Guidance and conceptual tools to inform the design, selection and evaluation of research capacity strengthening interventions.

This practice note presents four conceptual tools intended to support the design, selection and evaluation of research capacity strengthening (RCS) programmes in low-income and middle-income country settings. The tools may be used by a wide range of RCS stakeholders, including funders, implementing parties and programme evaluators, to guide decision-making in lieu of largely as yet unavailable empirical evidence. The first conceptual tool guides decision-making regarding RCS intervention design, focusing specifically on the combination and integration of potential intervention activities. The second conceptual tool provides a framework for assessing the implementation challenges of potential RCS interventions in terms of: (1) the overall cost of implementing the proposed intervention in a given context; (2) the length of time required to complete full implementation of the proposed intervention in a given context and (3) the level of control the implementing partners would have over the proposed intervention in a given context. The third conceptual tool provides a means to consider the anticipated impact of potential RCS interventions in order to inform selection decisions (ie, which out of a number of potential RCS intervention options may be most impactful in a given setting given the intervention design and implementation challenges). The fourth and final tool is designed to support the evaluation of a collective RCS effort, whether that be multiple RCS interventions delivered within the context of a single or continuous programme or multiple RCS programmes delivered in a common setting. © Author(s) (or their employer(s)) 2021. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

There is limited evidence currently available to inform research capacity strengthening (RCS) initiatives in low- and middle-income countries.

To guide decision making in this ‘evidence-poor’ context, four conceptual tools are described that may inform the design, selection and evaluation of RCS interventions.

These tools may be used by a wide range of RCS stakeholders, including funders, implementing parties and programme evaluators, and may be applied irrespective of the scale and aims of a planned RCS initiative.

Introduction

International development donors and governments have strengthened their commitment and spend on research capacity strengthening (RCS) initiatives in low-income and middle-income countries (LMICs) in recent years.1 RCS support in a development context is provided on the basis that greater research capacity leads to socioeconomic growth through evidence-informed policy and practice, improvements in human capital and pro-poor products/technologies.2 Despite increasing investment, the evidence base pertaining to the design, selection and evaluation of RCS interventions in LMICs is limited at present confounding informed decision-making.3 There are few robust outcome or impact evaluations of RCS interventions or agreed evaluation frameworks with standardised metrics, and the body of RCS literature that is available is highly fragmented.4–7 The current state of the RCS evidence base reflects, in part, the inherent complexity of the RCS endeavour which may be characterised as a multifaceted, long-term, continuous process subject to a diverse range of influences and assumptions.5 Yet, it remains the case that there is not yet sufficient evidence to reliably inform which types of intervention, in which combinations, with which focus and in what proportion are required to build research capacity effectively and sustainably in LMIC settings.

In this practice note, we present a set of conceptual tools designed to support decision-making by people who fund, implement or evaluate RCS programmes in this evidence-poor context. The tools may be used to inform: (A) the design of RCS interventions in support of an overarching RCS goal, for example, by parties tasked with strengthening research capacity in a particular focal area or by teams preparing a proposal for an RCS funding call; (B) the selection of preferred RCS interventions from multiple intervention options, for example, by a peer-review committee assessing proposals submitted to an RCS funding call or by implementing parties comparing the pros and cons of one potential RCS intervention with an alternative and (C) the evaluation of a collective RCS effort over a defined period of time, for example, by funders reviewing their RCS investments within or across programmes or by senior management reviewing the span of RCS activities carried out within their institute.

The tools themselves have been informed by our collective research and applied experience supporting a wide variety of RCS programme monitoring, evaluation and learning undertaken as founding (IB) or current staff members of the Liverpool School of Tropical Medicine’s Centre for Capacity Research (CCR). The CCR has an explicit aim to ‘advance the science of capacity strengthening for sustainable development’, recognising: the historical and structural inequalities that have left some regions in need of capacity development; the need to ensure partners in such regions are sufficiently supported to equitable participate in, and lead, these capacity development efforts; and that, ultimately, capacity strengthening in these regions will be accelerated by the development and application of robust, research-derived evidence and learning.

RCS intervention design and selection

Multiple factors need to be considered when designing and/or selecting an RCS intervention. Intervention design/selection is first informed by the overarching goal of the RCS initiative, which whether modest or ambitious, should be agreed and understood among all relevant stakeholders including funders, implementing partners and intervention beneficiaries.8 The RCS goal must be clearly articulated from the outset, ideally within the frame of an initiative-specific theory of change (ToC) that may be standalone or nested within a broader set of ToCs collectively describing a pathway from intervention to higher level national impact.9 RCS intervention design/selection will also inevitably be influenced by the available resources (human, financial, physical) as well as contextual constraints. In almost any context and irrespective of the overarching goal and available resources, multiple and potentially diverse options for RCS intervention will be viable. For example, RCS interventions may target individual researchers, the research institutions to which they belong or other aspects of the broader research ‘system’ essential to research production and uptake.10 Capacity strengthening needs may also be vast and highly variable, ranging from training provision to infrastructure development to legislation/policy formulation.11 Thus, different types of intervention may be required to support and achieve the same overarching RCS goal and efficiencies may be gained or lost depending on the degree to which any new intervention complements prior, concurrent or future RCS efforts.

Navigating this ‘choice overload’, exacerbated by the lack of robust guiding evidence, can be a challenge. The following conceptual tools provide means to inform RCS intervention design and/or selection, focusing on:

Configuration—what activity, or combination of activities, would the RCS intervention(s) consist of, and to what degree would multiple activities be integrated?

Implementation complexity—where various RCS intervention options exist then how complex would each intervention option be to implement relative to another?

Anticipated impact—what might the anticipated impact of alternative RCS interventions be?

Configuration

RCS interventions consist may potentially consist of any number of activities. Figure 1 depicts multiple configurations of an RCS intervention drawing on the example of four common RCS activities. As shown, each activity may be delivered in isolation or in varying degrees of combination. For example, an intervention could consist of the purchase of laboratory equipment (one activity), or the purchase of laboratory equipment plus the provision of biosecurity training to laboratory staff (two activities, partial integration), or the purchase of laboratory equipment, plus the provision of biosecurity training to laboratory staff, plus the upskilling of a laboratory manager to ensure regular maintenance and proper use of the equipment (many activities, substantial integration). The optimal number and level of integration between activities within a RCS intervention will always be highly context dependent; however, it may reasonably be assumed that the potential for impact will be greater if additional activities are delivered (eg, providing activities ‘X+Y+Z’ in conjunction will have greater impact potential as compared with delivering activity ‘X’ in isolation) and that the impact potential will be even greater if the full suite of RCS activities are closely integrated.

Figure 1

Research capacity strengthening intervention activities and their potential configuration.

Implementation complexity

Implementation of an intervention, irrespective of which activity or combination of activities it consists of, may be considered in terms of: (1) the overall cost of implementing the proposed intervention in a given context; (2) the length of time required to complete full implementation of the proposed intervention in a given context and (3) the level of control the implementing partners would have over the proposed intervention in a given context (ie, to what degree successful implementation depends on factors outside of the implementers’ control). Each of these factors may be placed on a scale ranging from low to high (for cost and time) or high to low (for control) providing a ‘measure’ of the likely complexity or degree of difficulty of implementing the underlying intervention, as depicted in figure 2. RCS interventions that can be implemented at low cost, quickly and with a high level of control may be considered inherently less complex than interventions that are costly to implement and require an extended time period and critical contributions from parties outside of the intervening partners’ control.

Figure 2

Assessing the relative complexity of implementing a research capacity strengthening intervention according to cost, time and control.

Anticipated impact

Complexity is not synonymous with effectiveness meaning less complex RCS interventions should not be favoured over more complex alternatives or vice versa, just because they are more or less complex. Thus, the anticipated impact of a proposed intervention, especially in relation to the overarching RCS goal, should be factored into the intervention design and/or selection process. For example, if two RCS interventions of differing complexity are being considered for implementation, yet the anticipated impact of the more complex intervention is similar to (or not significantly greater than) the less complex intervention, then logic may suggest the less complex intervention is the better alternative. Conversely, if the more complex RCS intervention has the potential to achieve substantially greater impact, then it may be the preferable alternative. Where possible, the anticipated impact of any one intervention should also be considered in relation to prior, concurrent or future RCS initiatives. For example, how might the proposed intervention complement previous or concurrent RCS interventions implemented within the same context, thereby amplifying the anticipated overall impact? Or how might the proposed intervention provide an essential platform to support future RCS initiatives that would otherwise not be possible or less impactful? This decision-making process would primarily rely on subjective measures in the short term to mid-term such as local understanding of the RCS context, anecdotal experiences from elsewhere and assumed causal pathways given the dearth of robust evaluation to inform evidence-based impact weightings at present. However, by ensuring appropriate evaluation processes are included to test the underlying impact assumptions, the evidence base to support more refined impact weightings of RCS interventions will develop exponentially.

Figure 3 illustrates a hypothetical RCS intervention comparison plot that factors in intervention configuration (shape of plot), implementation complexity (position of plot) and anticipated impact (size of plot).

Figure 3

Research capacity strengthening interventions plotted by configuration, implementation complexity and anticipated impact.

RCS evaluation

In addition to informing RCS intervention design and selection, the proposed conceptual tools provide a means to assess a broader RCS effort in a focal area over a defined period. For example, using the measures represented in figure 3, a funder or group of funders could plot various RCS interventions supported through a single or multiple funding initiative(s) over a given period or a senior management team could review a series of RCS initiatives carried out in their respective institution. The tools could be applied using only the three ‘measures’ of configuration, implementation and anticipated impact described above or as illustrated in figure 4 could further incorporate measures of RCS goal and level. ‘Goal’ in this case recognises that across a portfolio of RCS interventions/programmes, multiple RCS goals may be represented, while ‘level’ refers to whether the interventions/programmes focus on individual-level, institutional-level or systems-level RCS or some combination thereof. Figure 4 illustrates three hypothetical outcomes of this type of exercise. As shown in figure 4A, the collective RCS intervention effort clusters towards the simpler end of the scale both in terms of the complexity of the intervention structure and their respective implementation requirements, and the interventions themselves are highly uniform (in terms of goal, level and impact). In figure 4B, the range of interventions are more diverse and in figure 4C, they are more diverse again.

Figure 4

Plotting the collective research capacity strengthening (RCS) effort.

Applying the conceptual tools in this way as depicted in figure 4 would expose both gaps and duplication in the broader RCS effort, identify opportunities for cross-programme learning where interventions are sufficiently similar to support comparative analysis (which would be especially potent if effective communication channels can be established across the multiple RCS funding agencies for the purpose of shared learning) and provide a means to assess the overall level of intervention diversity. Achieving intervention diversity is important not only at a ‘systems’ level to support the wide range of capacities needed across a research system but also within specific components of the broader research system. For example, strengthening research support and management capacity within a single university may require interventions ranging from the building of physical infrastructure to the provision of leadership training to managerial staff to the automation of complex procurement processes.11 It may also be argued that when there is a lack of evidence on which RCS interventions are most effective, adopting a ‘diversification’ strategy would both reduce the risk of concentrated investment in what subsequently may prove to be less-effective interventions and provide the necessary platform to support robust evaluation of different intervention types. Based on this diversification principle, of particular concern may be those where the profile of the collective RCS effort resembles the scenario depicted in figure 4A, implying a glut of relatively simplistic, low cost and easy to implement RCS interventions with a similar focus. A concentration of RCS effort as depicted in figure 4A is unlikely to be sufficiently transformative to realise research-driven impacts on socioeconomic development.

Conclusion

This practice note has presented a set of conceptual tools intended to support the design, selection and evaluation of RCS interventions and programmes in LMIC settings. The tools may be used by a wide range of RCS stakeholders, including funders, implementing parties and programme evaluators, to guide decision-making in lieu of preferable, but largely unavailable, empirical evidence. Use of the tools may be refined as the RCS evidence base develops, especially in regard to informing the impact weighting of alternative RCS interventions, and the tools themselves may contribute to the development of the evidence base by facilitating both cross-programme learning and greater diversity in RCS intervention, and therefore, greater opportunity to ‘test’ different intervention types. The conceptual tools are largely neutral with respect to underlying RCS approaches such as whether research excellence or research equity should be prioritised in the capacity strengthening effort12 or the appropriate balance between individual-level, institution-level or broader systems-level RCS investments.13 The conceptual tools may also be applied in those cases where the RCS intervention is ‘embedded’ within a larger research programme as opposed to being the primary programme objective in its own right. Accordingly, the proposed conceptual tools may be used by multiple RCS stakeholders for multiple purposes and may be applied across RCS interventions irrespective of both scale and the underlying aim.

The tools, while designed to support considered decision-making in an evidence poor environment, are not in themselves sufficient for a full consideration of RCS intervention design, selection and evaluation. For example, key factors in the RCS process such as intervention sustainability or intervention ownership/equitable partnership could potentially be overlooked if the tools are applied without an adequate appreciation of their respective importance. Similarly, RCS design, selection or evaluation decisions that have resulted from the application of the presented tools should not be considered absolute. Rather, reflection processes should be built into any subsequent RCS intervention or evaluation exercise allowing for the possibility of real-time readjustment based on lessons learnt, emerging evidence or change in circumstance. There may even be value in reapplying the tools across the lifespan of an RCS intervention/evaluation as a part of the reflection process.