Evaluating the impact of integrated development: are we asking the right questions? A systematic review.

Background: Emerging global transformations - including a new Sustainable Development Agenda - are revealing increasingly interrelated goals and challenges, poised to be addressed by similarly integrated, multi-faceted solutions. Research to date has focused on determining the effectiveness of these approaches, yet a key question remains: are synergistic effects produced by integrating two or more sectors?  We systematically reviewed impact evaluations on integrated development interventions to assess whether synergistic, amplified impacts are being measured and evaluated.
Methods: The International Initiative for Impact Evaluation's (3ie) Impact Evaluation Repository comprised our sampling frame (n = 4,339). Following PRISMA guidelines, we employed a three-stage screening and review process.
Results: We identified 601 journal articles that evaluated integrated interventions. Seventy percent used a randomized design to assess impact with regard to whether the intervention achieved its desired outcomes. Only 26 of these evaluations, however, used a full factorial design to statistically detect any synergistic effects produced by integrating sectors. Of those, seven showed synergistic effects. Conclusions: To date, evaluations of integrated development approaches have demonstrated positive impacts in numerous contexts, but gaps remain with regard to documenting whether integrated programming produces synergistic, amplified outcomes. Research on these program models needs to extend beyond impact only, and more explicitly examine and measure the synergies and efficiencies associated with linking two or more sectors. Doing so will be critical for identifying effective integrated development strategies that will help achieve the multi-sector SDG agenda.

Introduction

Rationale

Twenty-first century global trends such as rapid urbanization and dramatic climate change are forcing the international community to rethink solutions to challenges that are increasingly multi-faceted and interrelated. Indeed, the Sustainable Development Goals (SDGs) – an ambitious framework of 17 goals to end extreme poverty, fight inequality and injustice, and reverse climate change over the next 15 years – emphasize the integration of previously distinct development aims. The agenda states that “[t]he goals and targets we have decided on are integrated and indivisible and balance the three crucial dimensions of sustainable development: the economic, social and environmental” ( United Nations, 2015). This evolution in thinking indicates a firm shift away from narrowly isolated sectors of development toward what the authors refer to as “win-win cooperation.” An analysis of how each of the 169 SDG targets is related to others reveals a web of closely interrelated objectives, yet also points out that any policy and program integration founded on these underlying linkages would need to rest on evidence with regard to their means of implementation ( Le Blanc, 2015).

Thus, decisions about when and how to most effectively implement the integrated, multi-sector SDG agenda need to be driven by evidence, rather than by assumptions about the amplified results of ‘doing more together’. Importantly, a large volume of research carried out to assess various types of integrated programs – in this case meaning programs that intentionally link their design and delivery (i.e., implementation) across more than one core development sector – suggests that in many cases these cross-sector approaches are successful in achieving positive impacts. But are program evaluators addressing a critical question: is 1 + 1 > 2? For example, does adding a nutritional component to an agricultural program reach more people than doing the same work separately? Does integrating family planning into an environmental conversation program improve the targeted impacts of one or more components? Are these interventions generating amplified impacts that go beyond the sum of single sector interventions? To identify and apply effective integrated program approaches in low- and middle-income countries (LMIC), it behooves us to first understand which programs are being evaluated and how. In this paper we present the results of a systematic review designed to identify to what degree impact evaluations have measured synergies and interaction effects, and if evaluations have, then, which methodologies were used to do so.

Objectives

The objectives of this systematic review were threefold: to identify impact evaluations of integrated development programs; to assess whether these evaluations seek to statistically measure synergistic effects or efficiencies associated with integrated interventions; and, for those that do, to document if synergies are detected.

Methods

Our review consisted of a three-stage process ( Figure 1), based on the PRISMA guidelines ( Moher ), as well as recommendations from Waddington . We first established a sampling frame from which to screen and review published articles. Given our objectives, we required a large, relatively exhaustive development database, or combination of databases, that included a broad range of evaluations not limited to any particular development sector. The first stage in our process, therefore, entailed establishing a sampling frame from which to identify and review the highest number of relevant publications and is therefore a modified step to a conventional systematic review. Once the sampling frame was established, we screened each article’s abstract and identified evaluations of development programs that we defined as “integrated development.” We then reviewed the full text of each article in the integrated development subset and documented characteristics essential to the objectives of our review.

Figure 1.

Review process for systematic review on integrated, multi-sector programs.

Stage 1: Establish a sampling frame

The universe of development literature and evidence is extensive. Published articles from this broad field are scattered throughout numerous databases, span a multitude of sectors, and use a diverse range of keywords and terminology (or no indicative terminology at all). We explored many of these specialized databases and possible combinations of databases from which we could specifically identify the most evaluations of programs that integrated two or more traditional development sectors (described below). Fortunately, we found a high-quality database that was not specific to any particular sector that would serve well as a single sampling frame for our review. Moreover, while we were initially interested in a broad range of research designs we felt comfortable with using only impact evaluations as it helped to ensure a baseline level of rigor in study design.

The International Initiative for Impact Evaluation (3ie) Impact Evaluation Repository is an index of all published impact evaluations of development interventions. To be included in the repository, an impact evaluation must be published (as a journal article, book chapter, report, or working paper), take place in a developing country, examine the effectiveness of a specific development intervention, and use a specifically defined experimental or quasi-experimental estimation strategy.

During the creation of the repository, 3ie systematically searched more than 45 databases, search engines, journal collections, and websites with an aim to identify all published development impact evaluations ( Figure 2) ( Mishra & Cameron, 2014; Jorge Miranda, 2017, personal communication). At the time of our analysis (September 8, 2016), 3ie had reviewed more than 140,000 potential studies, rendering an index of 4,339 eligible studies ( Mishra & Cameron, 2014; Jorge Miranda, 2017, personal communication). The repository, including a full description of its inclusion criteria and review methodology, is available here. The search was limited to English language articles that evaluate a development program, project, or policy; to evaluations that occur in LMICs; and articles that use rigorous techniques to identify a counterfactual. Grey literature is included in the database, and the 45 sources include everything from EconLit to Google Scholar (full list can be found here). Key terms to identify impact evaluations included terms like ‘impact’, ‘effect’, or ‘random’, but differ slightly by database. The last 3ie update was completed in July 2016.

Figure 2.

PRISMA flow diagram for systematic review with 3ie impact evaluation repository review process.

As part of our due diligence, we sought to confirm that 3ie’s repository was a thorough and sufficient sole-sampling source. A library science specialist audited the methodology 3ie staff used to create the repository. Her objective was to assess whether the searches used were both sensitive (i.e., broad) and specific (i.e., focused) enough to ensure that the vast majority of relevant and eligible impact evaluations were included in the final repository. She reviewed the databases that were used and how they were searched with regard to subject scope, time frame limits, and geographic coverage. She concluded that although some lesser-known and regional databases were excluded from the 3ie repository the likelihood that any significant number of new references would be found was negligible. The databases that the 3ie repository was sampling from were searching many of these smaller databases, so she could find no evidence that the larger databases missed anything the smaller databases held. She concluded that the overall methodology design was strong and its implementation consistent. We therefore feel confident that using the 3ie repository as our sampling frame provided us with a sufficient index of development evaluations.

Stage 2: Screen articles in the repository for integrated development approaches

The purpose of this stage of the review was to identify all of the publications in the 3ie repository that evaluated integrated programs. No terms to denote integrated development have universal agreement. The concept of integrated or multi-sector development in published papers is described by many different terms (e.g., cross-sector, linked, combined, blended). Moreover, authors rarely self-identify their interventions in this way within an article, let alone an abstract. We could not, therefore, rely on key search terms to identify evaluations of programs that were integrated in nature. Instead, we manually reviewed the abstracts of every study in the 3ie repository (as of September 8, 2016) against FHI 360’s working definition of integrated development:

“Integrated development approaches intentionally link the design and delivery (i.e., implementation) of programs across more than one core sector.”

Note that our definition of integrated development encompasses studies that would be classified as “multi-sector” or “multi-disciplinary” by others. More precisely, our definition focuses on the integrated nature of the intervention itself and excludes programs that only integrate different subsectors within a core sector (e.g., health programs that link family planning and HIV/AIDS). Our definition also excludes programs that measure outcomes in multiple sectors stemming from a single-sector program (e.g., measuring both education and nutrition outcomes from an education-only project). Finally, explicit intentionality did not factor in the actual application of the definition, since very few evaluations were detailed enough to illustrate motivation or thought process.

Other definitions of integrated programs imply collaboration between the public and private sectors, integration of services between different established levels in a system (e.g., government ministries and local service providers), include different core development sectors, or require clear implementation processes to support integration (e.g., co-location of services, cross-training staff). While these approaches are often grouped together and utilize some of the same approaches and techniques, we broke out multi-sector integration here from public and private sector (or from multi-level systems integration) to create a group of evaluations that are similar enough to be evaluated together meaningfully.

There are no universal or definitive lists of development ‘sectors’. Global bodies and implementing organizations characterize thematic areas in fluid ways, at times bundling some fields (e.g., health and nutrition), and at other times ensuring they are distinctly separate. For this review, we used core sector categories and illustrative interventions and outcomes found in Table 1. This was shaped by what FHI 360 classifies as core sectors, as well as what 3ie and other development databases classify as core sectors. We used these sector categories to classify interventions as well as outcome measures.

Table 1.

Sector intervention and outcome examples.

Sector	Intervention examples	Outcome examples
Agriculture and food security	Farming, food supply chains, famine prevention	Food security, agricultural productivity, agricultural production, access to extension services
Economic development	Livelihoods, cash transfers, microfinance	Income, savings, household assets
Education	Early education, primary school, secondary school	Enrollment, grade completion, attendance
Environment	Environmental/land management, conservation, climate change	Deforestation rate, erosion, environmental knowledge and perceptions
Governance	Peace building, conflict management, election monitoring, democracy	Participation in government, inclusive government, effective management scores, corruption
Health	HIV, tuberculosis, maternal and child health, sexual and reproductive health, non-communicable disease, malaria, vaccination	HIV prevalence, vaccination rates, under- five mortality
Nutrition	Micronutrient provision, food fortification, feeding programs, diet diversification	Anemia, malnutrition, stunting, wasting, dietary diversity
Water, sanitation, and hygiene (WASH)	Water quality, management, supply	Access to water sources, access to sanitation, hand washing behaviors

All of the interventions in the studies reviewed fell within these sector categories. We added an “other” category to describe outcomes measured, to capture more amorphous, non-sector specific measures, such as “child labor.” For our review, cross-cutting topics such as gender, youth, civil society, and technology were considered aspects of, and relevant to, the interventions and outcomes in each sector, but not sectors in and of themselves. During the review we had also initially included ‘humanitarian’ as a sector. With further discussion and analysis, it was clear that this sampling frame was not inclusive of the humanitarian sector, nor is humanitarian work represented at the same level of the conventional development sectors included above. Therefore, the final analysis was completed without the humanitarian sector category in either intervention sector or outcome sector measured.

To enhance reliability, two individuals independently reviewed all of the abstracts and identified the sectors represented in the interventions being evaluated. If more than one sector was identified, the study was categorized as “yes” for integrated development; all other studies were marked as “no”. These two reviewers met at predetermined intervals to compare their results, and had an average of 89% agreement.

All discrepancies in coding were resolved at each comparison point after the team discussed the interpretation of the integrated development definition (resulting in 100% agreement). In the few cases in which the two reviewers could not agree on a study’s categorization, a third-party reviewed the abstract and made the final decision. Any study that members of the review team both categorized as integrated development was included in the second round of review. For cases in which the abstract alone did not contain enough information to make a determination, studies were advanced to the next round of review so that a final determination could be made during review of the full text.

Importantly, although the repository includes impact evaluations published in any form, our inclusion criteria for this review required a study to be published in a scientific journal. We excluded grey literature to manage the high volume of eligible papers but acknowledge that there is value in their examination for future reviews. Therefore, only those publications that were peer reviewed moved on to Stage 3.

Stage 3: Characterize evaluations of interventions identified as integrated

Full text articles of the subset of studies on integrated programs that were published in scientific journals were reviewed by two individuals. Each article was compared against a checklist, to ascertain the study’s scope and methodology (our checklist is presented with the corresponding results in Table 2).

Table 2.

Summary of integrated development impact evaluation characteristics.

Characteristic (N = 601)	Frequency
Self-identified term for the nature of intervention being evaluated (by study author, in the title or abstract)	Integrated Combined Multi-component Multi-faceted Other Did not self-identify	44 (7%) 25 (4%) 9 (1%) 2 (<1%) 19 (3%) 507 (84%)
General study design	Experimental Quasi-experimental Both	419 (70%) 174 (29%) 8 (1%)
Study arm combinations included in design	Integrated arm(s) + control arm Integrated arms(s) only Single-sector arm(s) + Integrated arm(s) Full factorial Other	366 (61%) 68 (11%) 132 (22%) 26 (4%) 9 (1%)
Partial factorial design (Randomized study that includes at least 1 single-sector arm — but not ALL single-sector arms — and at least 1 integrated arm and at least 1 control arm)	12 (2%)
Full factorial design (Randomized study that include ALL single-sector arms, at least 1 integrated arm, and at least 1 control arm)	26 (4%)
Qualitative component included in evaluation	60 (10%)
Sectors included in study intervention	Agriculture & food security Economic development Education Environment Governance Health Nutrition Water, sanitation, and hygiene	65 (11%) 238 (40%) 433 (72%) 25 (4%) 12 (2%) 456 (76%) 266 (44%) 44 (7%)
Number of intervention sectors included in design	2     3     4 5 or more	373 (62%) 132 (22%) 87 (14%) 9 (1%)
Sectors in which outcomes measured	Agriculture & food security Economic development Education Environment Governance Health Nutrition Water, sanitation, and hygiene Integrated outcome (bespoke) Other	40 (7%) 102 (17%) 149 (25%) 13 (2%) 8 (1%) 373 (62%) 174 (29%) 23 (4%) 1 (<1%) 54 (9%)
Number of sectors in which outcomes were measured	1     2     3 4 or more	347 (58%) 185 (31%) 60 (10%) 9 (1%)
Geographic area of study	Middle East/North Africa Sub-Saharan Africa Asia Latin American & the Caribbean Europe Oceania Number of countries represented	21 (3%) 213 (35%) 177 (29%) 188 (31%) 4 (<1%) 2 (<1%) 70
Cost analysis conducted	43 (7%)
Implementation/process evaluation conducted	41 (7%)

In particular, we noted the number of control, single-sector treatment, and integrated sector treatment arms in each evaluation. We further identified those evaluations which employed either a partial factorial or full factorial experimental design. For the purposes of our review, partial factorial designs included at least one single-sector arm (but not all single-sector arms), at least one integrated arm, and at least one control (no intervention) arm. Full factorial designs included all possible single-sector arms, at least one integrated arm, and at least one control arm. Factorial designs are exceptionally rigorous and permit evaluators to determine the effects of multiple interventions on an outcome. Since they include all possible combinations of intervention arms, full factorial designs are able to reveal differential effects of single-sector and multi-sector interventions and measure potential synergistic effects associated with integrated approaches.

Therefore, we specifically reviewed each full factorial evaluation to determine if the authors measured or detected synergy associated with the integrated study arm. For our review, we defined synergy as a statistically significant (p < 0.05) interaction effect in the integrated arm (between two or more intervention sectors), or as instances in which the effect size of the integrated arm of a program was greater than the sum of the effect sizes among the single-sector arms.

Given the extreme heterogeneity of the types of programs evaluated and outcomes assessed, we did not seek to collectively synthesize their substantive findings. Instead, the primary objective was to determine if and how impact evaluations of integrated programs are designed to measure or systematically document the synergy and efficiency assumed in multi-sector development.

Results

We reviewed 4,339 abstracts, comprising the entire 3ie repository as of September 8, 2016. After a two-step screening process, 601 articles were included in our final dataset for characterization ( Figure 2, Supplementary File 2). From the initial set of 4,339 articles from the 3ie repository, 3,543 were excluded (2,380 did not meet the definition of integrated and 1,163 were not published in a scientific journal). The full text of the remaining 796 were assessed for eligibility. One hundred and ninety-five were excluded with full text review (193 did not meet the definition of integrated and two were not available to reviewers). This left 601 studies included in the analysis. The list of articles is included here as a supplement ( Supplementary File 3), and each article may also be found in a searchable online database.

The majority of evaluations (70%) employed a randomized controlled design to assess the effectiveness of their interventions. However, only 26 (4%) of the 601 studies reviewed used a full factorial design and only 12 (2%) employed a partial factorial design. The majority of evaluations (61%) assessed the effectiveness of an integrated intervention by comparing one or more integrated arms to a no-treatment control only. A minority of evaluations included a comparison of integrated arms only (11%), or contained single-sector arms and integrated arms but no control arm (22%). Few evaluations included qualitative (10%) or cost analyses (7%) components. Most articles (84%) did not identify the interventions being evaluated as “integrated”, or any other related term ( Table 2).

With regard to what types of interventions and desired outcomes were being assessed, the three sectors most often represented in the intervention design — in order of highest to lowest frequency — were health, education, and nutrition. The same three sectors were also most common in terms of outcomes measured, with nutrition slightly outpacing education for second most common. The top three most common sectors that each sector was integrated with is also illustrative (see Table 3). For example, agriculture and food security is most commonly integrated with economic development, followed by nutrition and health. The proportion of studies that measured outcomes within each sector are displayed as well ( Table 4). For example, 29.5% of studies that include WASH as an intervention sector include an education outcome measure. Seventy-six percent of studies that include an environmental intervention include an economic development outcome measure.

Table 3.

Most commonly integrated sectors and sector combinations.

Sector	Most Commonly Integrated with…	Second most commonly integrated with…	Third most commonly integrated with…
Agriculture & food security (n=65)	Economic development (n=57)	Nutrition (n=20)	Health (n=19)
Economic development (n=238)	Health (n=171)	Education (n=137)	Nutrition (n=103)
Education (n=433)	Health (n=344)	Nutrition (n=193)	Economic development (n=137)
Environment (n=25)	Economic development (n=20)	Agriculture & food security (n=12)	Governance (n=5) Health (n=5)
Governance (n=12)	Economic development (n=9)	Agriculture & food security (n=5) Environment (n=5)	Education (n=4)
Health (n=456)	Education (n=344)	Nutrition (n=192)	Economic Development (n=171)
Nutrition (n=266)	Education (n=193)	Health (n=192)	Economic development (n=103)
Water, sanitation, and hygiene (n=44)	Education (n=29)	Health (n=27)	Nutrition (n=11)

Table 4.

Proportion of outcomes by sector intervention.

		Outcomes
		Agriculture & food security (n=40)	Economic development (n=102)	Education (n=149)	Environment (n=13)	Governance (n=8)	Health (n=373)	Nutrition (n=174)	Water, sanitation, and hygiene (n=23)
Interventions	Agriculture & food security (n=65)	32/65 49.2%	40/65 61.5%	6/65 9.2%	7/65 10.8%	0/65 0%	10/65 15.4%	15/65 23.1%	1/65 1.5%
	Economic development (n=238)	38/238 16.0%	100/238 42.0%	65/238 27.3%	13/238 5.5%	8/238 3.4%	87/238 36.6%	32/238 13.4%	2/238 0.8%
	Education (n=433)	11/433 2.5%	40/433 9.2%	142/433 32.8%	2/433 0.5%	4/433 0.9%	279/433 64.4%	116/433 26.8%	16/433 3.7%
	Environment (n=25)	8/25 32.0%	19/25 76.0%	4/25 16.0%	10/25 40.0%	1/25 4.0%	6/25 24.0%	1/25 4.0%	1/25 4.0%
	Governance (n=12)	4/12 33.3%	9/12 75.0%	4/12 33.3%	1/12 8.3%	2/12 16.7%	3/12 25.0%	1/12 8.3%	1/12 8.3%
	Health (n=456)	15/456 3.3%	57/456 12.5%	100/456 21.9%	2/456 0.4%	4/456 0.9%	346/456 75.9%	122/456 26.8%	12/456 2.6%
	Nutrition (n=266)	12/266 4.5%	27/266 10.2%	87/266 32.7%	3/266 1.1%	3/266 1.1%	142/266 53.4%	156/266 5.9%	5/266 1.9%
	Water, sanitation, and hygiene (n=44)	4/44 9.1%	4/44 9.1%	13/44 29.5%	1/44 2.3%	0/44 0%	29/44 65.9%	9/44 20.5%	19/44 43.2%

For the 38 studies that represented either a partial or a full factorial design, we assessed whether the effectiveness of an integrated intervention — in terms of study outcomes — was evaluated. Of the 26 that were full factorial, seven reported findings that showed that the integrated arm was most effective ( De Brauw ; Haque ; Leventhal ; Nga ; Nga ; Olsen ; Widen ). Eight demonstrated mixed findings, or did not report the effectiveness of the integrated arm as compared to the other arms ( Awasthi ; Duflo ; Gilgen & Mascie-Taylor, 2001; Halliday ; Jinabhai ; Kim ; Leventhal ; Tahlil ). In some cases, the added value of integration was reported in another study, or it was stated that the combination was not intended to have effects on the outcomes of the separate sectors, so the data was not fully analyzed or reported. In terms of mixed findings, some studies demonstrated tradeoffs, where integration added value to certain outcomes, but was deleterious for others. Finally, 11 evaluations found no added value of integration ( Attanasio ; Dangour ; Desai & Tarozzi, 2011; Fenn ; Friis ; Gilgen ; Gowani ; Mwaniki ; Nahar ; Rohner ; Walker ).

The seven studies that found amplified impacts included four studies that examined the combination of nutritional supplementation and deworming ( Haque ; Nga ; Nga ; Olsen ), one study that examined the effect of nutritional supplementation and antiretroviral prophylaxis use in women and infants ( Widen ), one study examined a school-based psychosocial adolescent health intervention for girls ( Leventhal ), and one study examined the promotion of orange-flesh sweet potato through integrated agricultural and nutritional activities ( De Brauw ). Four interventions took place in Asia and three in sub-Saharan Africa. All seven found improved outcomes in the integrated arm as compared to single sector and control arms.

Only three of the full factorial studies incorporated cost analysis, and two of the three found that the integrated arm was cost-effective. The third did not perform a cost analysis on the integrated arm.

We also reviewed the findings reported in the 60 studies that included a qualitative component. We found only 1 study that intentionally documented synergy via the qualitative inquiry – the others used the method to investigate other aspects of the intervention.

Discussion

Our screening of 4,339 records in the 3ie Impact Evaluation Repository identified 601 journal articles that describe studies of programs we defined as integrated. Our full text review of these 601 articles revealed several interesting trends. First, researchers do not use standardized terms for describing integrated development programs. In fact, the majority of authors did not use any term at all to indicate the integrated or multi-sector nature of the interventions they were evaluating. This finding validates our manual screening methodology. Had we used a key term search strategy, we likely would have missed many relevant studies. Interestingly, 46% of the full factorial evaluations addressed integration or synergy in their abstracts, as compared to only 16% of all studies identified as integrated.

Next, only 26 evaluations employed a full factorial design. Though randomized controlled designs are sufficient to confidently detect the impact of these types of programs, a full factorial design is the only design that enables researchers to statistically measure whether the impact is related to the synergy presumed to result from integrated, multi-sector programming. With so few full or partial factorial studies completed, and such diversity in the outcomes found, we are not able to evaluate effectiveness. We can, however, see trends in where research is being done, and the need for research that does consider integration as a component of the program to be evaluated.

We recognize that full factorial designs are often costly and time-consuming, and may not be feasible or even advisable in many scenarios. Having such strong counterfactuals does offer credible information for internal validity. Like many RCTs, however, they have a limited ability to suggest external validity. Factorial studies may not be necessary for the types of integrated approaches that have been confirmed in the past to deliver synergy. Given their expense, they may best be used in cases where proof of concept is needed, but not necessary for understanding how to apply a promising or proven model in other contexts. Methods used in implementation science and adaptive programming can similarly shed light into the effectiveness of integrated approaches in new settings. Factorial or not, using a mixed method approach to include examination of cost efficiencies and qualitatively assessing synergies offer value for determining how integration factors in to the program findings. While this review was not focused on these types of evaluations, we found that few impact evaluations on integrated approaches included these components.

Limitations

Creating and applying a definition of integrated development was a subjective process. To address this, we utilized two independent coders and employed inter-coder agreement procedures to enhance reliability in our screening process. Assigning core sectors to studies was also a subjective process, and in some cases assigning sectors to an intervention was difficult (e.g., depending on its particular aim, aid to small-scale farmers could conceivably be an economic development, agriculture, or nutrition intervention). We attempted to mitigate this by providing definitions and examples of core sectors to both reviewers, and once a type of intervention was categorized in one way it remained consistent across studies.

Another potential limitation is that the 3ie repository may not be exhaustive. The 3ie database limits by language (English) and the last update at the time of our searches was completed in July 2016. Limiting to English could systematically exclude publications from certain regions, or exclude certain groups of evaluations. Eligible publications in regional or small databases not included in the search strategy could have been overlooked. Given the size of the repository, however — more than 4,300 publications — including a small number of studies that may be absent in the 3ie repository would not have changed the substance of our findings.

We also recognize that in the past 4–5 years an increasing proportion of impact evaluations are being written up as working papers ( Cameron ) and may not be published in peer-reviewed journals. Although our judgment is that it would not be significant, we do acknowledge that reviewing to include grey literature could conceivably shift some of the trends we highlight here. Our entire database of impact evaluations for integrated development found within the 3ie repository – peer-reviewed and grey literature – can be found here.

Finally, a large portion of the studies in the impact evaluation repository (and therefore this review) focus on the health sector. This is more a representation of the volume of research conducted in the health sector in general rather than an indication that health-focused approaches have a higher importance or more effectiveness than other sectors. Due to different evaluation cultures within different sectoral communities, and the ease to which some interventions lend themselves to certain types of evaluations, health is almost certainly overrepresented here than if this review had a broader methodological sampling frame.

Further research

There are many avenues which we could not pursue during this research that would further support our understanding of how integrated projects are being studied, and the results of those studies. A review of observational studies, or a systematic review of qualitative or cost analyses studies could also illustrate meaningful trends. There have been targeted reviews on various combinations of sectors that have included deeper examination of the results (since the subject is narrower) (e.g., Lukas, 2008; Ruel ). This review can be used to highlight areas that could be further explored with a finer comb. Further reviews of different types of integration – including private and public sector, and multi-sector outcomes for example, mentioned above – could also produce interesting results.

Conclusions

Our systematic review is not intended to determine whether or not integrated development approaches work. We know from the high number of randomized evaluations included here that report positive findings that in many contexts integrated, multi-sector interventions have produced impact. What our systematic review does indicate, however, is that very few impact evaluations to date were designed to specifically examine the synergistic and interaction effects that are potentially associated with integrated programming. In other words, to what extent is the integration itself producing the results versus other factors? And if so, why and how? Impact evaluations of new or yet-to-be-proven integrated programs need to be better designed to intentionally assess not only their overall impact but the explicit value added of linking two or more development sectors, in terms of service delivery outcomes, participant perspectives, and cost.

Addressing these gaps is essential as the international community pivots toward a more cross-cutting global development agenda. Implementing this agenda will likely deploy more promising and innovative silo-breaking programs. We must ensure that our research designs and measurement strategies keep pace accordingly.

This systematic review aims to identify and provide an overview of studies that measure the synergistic effects of integrated development approaches. To achieve this goal, the study screened and reviewed articles included in The International Initiative for Impact Evaluation's (3ie) Impact Evaluation Repository. The review addresses an important research question. However, I still see a number of limitations that are not fully addressed in the current version of the paper.

As discussed in my previous review, the authors exclude studies that were not published in peer-reviewed journals in the screening and review of the articles. I do not think this was an appropriate methodological choice. In addition, the authors do not take into consideration trade-offs between internal validity and construct validity in their conclusions.

In their new version the authors acknowledge that it would be valuable to include grey literature in future reviews. They mention that grey literature was not reviewed to manage the high volume of eligible papers. The authors also acknowledge that their exclusion of grey literature could have led to systematic biases in their review. However, they do not discuss specific reasons on why grey literature could have led to different results. The authors only report that “Although our judgement is that it would not be significant, we do acknowledge that reviewing to include grey literature could conceivable shift some of the trends we highlight here.” I would like to ask the authors to explain that judgement in more detail. Studies that measure synergistic effects are often underpowered to detect small but meaningful effects. As a result, publication bias may be larger among studies that focus on the synergistic effects of integrated development approaches. This important limitation is not discussed in detail in the current version of the paper, and I am not convinced that publication bias has not resulted in systematic biases in the review. Many systematic reviews have shown that excluding grey literature can lead to systematic biases.

The authors also acknowledge that the external validity of RCTs that study synergistic effects is likely limited. They, however, do not acknowledge that the construct validity of RCTs that aim to measure synergistic effects is likely smaller than for RCTs with only one treatment arm. The reason is that researchers often drive the implementation of programs that are evaluated to measure synergistic effects; as a result most studies that measure synergistic effects estimate the efficacy and not the effectiveness of development programs. I would have liked to see a discussion of how many studies that measure synergistic effects focused on implementation of government programs. My expectation is that this percentage is very small, which further reduces the external and construct validity of RCTs that aim to measure synergistic effects. I would like to ask the authors to discuss these limitation in somewhat more detail.

This paper already provides an important contribution to the literature, but I think the paper can be further strengthened by discussing these two limitations in more detail.

I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

- This is an easily readable and transparent/clear article that makes new and useful contributions to the area of integrated development interventions.

- For the most part the authors addressed reviewer comments from the first peer review report.

Rationale and Objectives

- The concerns raised about the rationale and objectives in the first peer review form were adequately addressed by the authors. The objectives of the study now more directly match the rationale and the results. In particular, the authors have clarified that they are interested in the study designs of evaluations on integrated development interventions.

Methods

- The authors have done well to clarify the search strategy/sampling of the International Initiative for Impact Evaluation (3ie) Impact Evaluation Repository, which was less clear in the original manuscript. They justify their use of the Impact Evaluation Repository as the only source of articles.

- The authors do well to clarify their definition of integrated development intervention and how they used the definition to inform the assessment of potential studies to include in their review.

Results and Discussion

- Appreciate the addition of two tables expanding on commonly integrated sectors and proportion of outcomes by sector intervention. Very clearly presented.

- Authors expanded well on limitations and future directions of the research.

- Can the authors add a qualifier about why they limited it to English-only articles?

- Can the authors clarify what they mean by context. The authors simply discuss integrated development interventions as having positive impacts in different contexts, but do not define what they mean by context. Do the authors mean geographical location?

We have read this submission. We believe that we have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

I am glad that my review was helpful to the authors and they have addressed many of my suggestions. I am happy to recommend that this paper be published in this esteemed journal.

I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

This systematic review aims to identify and provide an overview of studies that measure the synergistic effects of integrated development approaches. To achieve this goal, the study screened and reviewed articles included in The International Initiative for Impact Evaluation's (3ie) Impact Evaluation Repository. The review addresses an important research question. The approach is rigorous and the objectives of the study are clear. However, the authors also exclude studies that were not published in peer-reviewed journals in the screening and review of the articles. I do not think this was an appropriate methodological choice. In addition, the authors do not take into consideration trade-offs between internal validity and construct validity in their conclusions. I highlight these issues in more detail below in addition to some other smaller comments. The authors make an important contribution to the literature but I would appreciate it if they would take these comments into consideration when they resubmit the paper.

Excluding studies that are not published in peer-reviewed journals:

The authors explain that they exclude studies that were not included in peer-reviewed journals. It is unclear why they make this choice. It is well known that the findings of peer-reviewed papers can systematically differ from the findings of articles that are not published in peer-reviewed journals. This may be either because of quality differences or publication bias. Risk of bias assessments can mitigate concerns regarding the former. Therefore, it is unclear why the authors exclude the studies that did not appear in peer-reviewed journals. The authors highlight that of the 26 full factorial papers, seven demonstrated that the full factorial arm was most effective. It is very well possible that this percentage may be systematically different in articles that were not published in peer-reviewed journals because of publication bias. So excluding non-peer reviewed studies may well have led to a bias in the results. It may have been more appropriate to conduct a risk of bias assessment.

Internal validity versus construct validity

The authors argue in favor of designing and implementing impact evaluations with full factorial designs. I do not disagree with this view. It is important to conduct studies with full factorial designs. However, the reality is that such impact evaluations are very hard to implement in practice in settings without full control of researchers. It is much more challenging to design and implement impact evaluations with a full factorial design when working within government systems or when the intervention is implemented at scale. In contrast, it may be relatively easy (with an emphasis on relatively) to implement a full factorial design when the intervention is designed and implemented by researchers.

The impact evaluation with full control of the intervention may have a high interval validity but the construct validity of the latter design may be much more limited. Estimated effect sizes from full factorial designs can probably not be credibly extrapolated to settings in which the government has full control over the program. So we need to be careful when we recommend impact evaluations with full factorial designs. Their construct validity may be limited if the government or another large implementing partner is unlikely to implement the program with a full factorial design in practice. Or the take-up of the program is much more limited. For example, Bold et al. (2013) demonstrate that the effects of remedial education programs can differ significantly depending on whether they are implemented by an NGO or by the government.

Similarly, full factorial designs may sometimes not be relevant. For example, we conducted a study on the effects of a teacher training program that aimed to promote positive gender socialization in Karamoja, Uganda . This cluster-randomized controlled trial included two treatment arms and a control group. The first treatment arm only included the teacher training. The second treatment arm included the teacher training + an intervention that included text messages related to the contents of the training . In practice, it would not have been useful to design and implement the latter intervention without the teacher training. But as a result the evaluation does not qualify as a full factorial design according to the definition of this study. I do not think this is a limitation of the study by Chinen et al. (2017). Instead, it shows that full factorial designs are not always the best solution to measure synergistic effects of development programs.

Other Comments

The researchers also highlight the option of analyzing synergistic effects using qualitative research. It would be helpful to learn a bit more about the specific qualitative research the authors would have in mind to measure synergistic effects.

The 3ie Impact Evaluation repository gives a comprehensive overview of impact evaluations in international development. However, it is very well possible that including studies from other databases would have changed the results. It would be good to acknowledge this.

The authors do not assess the conditions under which integrated interventions are more effective. However, the authors highlight this as one of the goals of the study. I would remove this objective.

I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

The paper and its subject is extremely timely and relevant given the integrated development focus of the SDGs. I found the methodology, rationale for the sampling frame, phases of the literature review to be well reasoned and laid out. The subject matter of this paper, especially the 38 studies, could prove to be extremely helpful for implementation practitioners, researchers, and policy makers as sector-specific communities grapple with the complexity of designing and delivering integrated development programs in support of the SDGs.

I submit, below, a few constructive suggestions for the authors’ consideration to tighten and focus (in some places) or expand (in others) the paper to make it even more useful for the broader development community. I trust these are helpful. Thank you for the opportunity to review this paper.

The authors may want to consider adding the word “impact” into the title itself. “Evaluating the Impact of Integrated Development: are we asking the right questions? A Systematic Review”, perhaps?

In the Introduction section, the authors correctly note, “To effectively advocate for integrated, multi-disciplinary approaches to development, it behooves us to understand under which circumstances integrating two or more development sectors enhances impacts in amplified or synergistic ways.” (emphasis added). I did not find this addressed in the Findings section. I strongly recommend that the Findings section be bolstered to focus on the findings from the 38 studies to discuss this critical element. Or, consider deleting this as a rationale for the paper to focus it on the objectives mentioned.

The authors also state, “In this paper we present the results of a systematic review designed to identify if and how synergies and interaction effects between sectors in integrated interventions are being measured.” It will help to clarify that “how” refers to the methods used (experimental vs quasi experimental; # of sectors for which outcomes are measured, etc.) rather than the specific outcomes being measured and evaluated in the integrated programs. [That said, ‘diversity of outcome measures used in integrated programs’ is a paper that I would urge the authors to consider writing next.]

Definitional clarity:

I recommend including a little more discussion around the definition that the authors choose for” integrated development” – what other definitions did you consider? Why did you find them lacking? How did you arrive at this definition? How did you define “delivery” of programs? How did you assess, in your review of the papers, “ intentional linkage”? How did you assess, in your review, if the interventions were linked by “design” (which I would consider being important because it demonstrates a planned approach)?

I struggled with semi-equating integrated development with “multi-disciplinary”. Multi-sectoral is probably the term the authors may want to stick closer to. Multi-disciplinary is more of a research term with definitions of its own.

I would imagine that there is wide diversity in the outcomes that each sector uses. It may help to give an example for what counted as an outcome for each sector. It will help ground the paper a bit more for the reader.

In the Methods and Limitations section, there are a few steps and limitations missing – language, focus on LMICs, etc. This has been well covered by another reviewer, so I will not repeat these here.

In the Discussions section, I recommend two things for the authors’ consideration:

An emphasis on the 38 studies. I would have liked to see a separate table that digs into these studies a little more. – what did these show? Where and on what type of integrated development do these focus vis-à-vis the 601 studies? What are they measuring? What synergistic outcomes did you see? Why do you think this is the case?

A discussion on gaps - what gaps did you see?

In the Conclusions section, consider adding a paragraph on the utility value of this important endeavor for various audiences. What are the key takeaways for this audience? Or, perhaps, consider adding an impact evaluation agenda? What are the implications for reaching the SDGs from your viewpoint?

I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

The authors present an interesting review of integrated development approaches and are right to highlight this as an important area of research. The structure of the review is clear and easy to follow the argument and ideas of the authors.

Introduction - Rationale Introduction - Objectives  Methods Results Discussion and Conclusion

It would be helpful to have “integrated development intervention” defined earlier in the manuscript, especially because authors mention that there has yet to be consensus on a definition in the literature. Providing an example of an integrated development intervention (including a brief description of which features make it an integrated development intervention) would be helpful in explaining why they are so important in the context of the SDGs.

The statement below was part of the rationale for conducting this study, but the objectives do not reflect this rationale and neither do the results or discussion sections of the study. Specifically, the authors do not analyze the circumstances when an integrated intervention works. I recommend that this sentence be removed or rephrased to reflect the objectives and results/discussion of the study.

"To effectively advocate for integrated, multi-disciplinary approaches to development, it behooves us to understand under which circumstances integrating two or more development sectors enhances impacts in amplified or synergistic ways."

The authors might consider circumstances of evaluation in terms of implementation, commitment by participating sectors, funding available, or context (sociocultural, political, institutional, geographical, economic, etc – and this context needs to be defined). For example, how did the institutional structure of sectors involved in an integrated intervention influence the interventions success or failure? How does the geographic context of the integrated intervention influence its success or failure? Etc.

The secondary objective of the review seems unnecessary because the authors should state the characteristics of the included studies in the results section as part of the systematic review, therefore this does not need to be an objective.

Regarding the first objective, it would be helpful to have clarification of what the authors mean by “how.” By the end of the article it was clear that the authors mean “how” in terms of evaluation study design (e.g. experimental, factorial) rather than “how” in terms of type of evaluation (e.g. process, outcomes, impact). However, it would be helpful to have this clarified in the objective statement. Specifically, the authors should clarify that they mean to summarize how impact evaluations have been conducted on integrated development programs.

Selecting the International Initiative for Impact Evaluation (3ie) Impact Evaluation Repository, which only has impact evaluations, as the sole source of studies for this review is another reason why the authors should consider clarifying that the review aims to summarize how impact evaluations have been conducted on integrated development programs.

In response to criticism that the MDGs only focused on LMICs, the SDGs were supposed to apply to all countries, irrespective of “development’” status. Based on the definition of integrated development interventions/approaches provided by the authors, these approaches are not dependent on location, i.e. an integrated development approach can take place in any country as long as it “intentionally links the design and delivery of programs across more than one core sector.” 3ie search strategy limited the scope to LMICs and LMIC regions. If the SDGs – which are meant to apply to all countries – are invoked as rationale for this systematic review, then it might be helpful to have a comment from the authors for excluding integrated approaches in LMIC contexts.

The 3ie repository search strategy limited inclusion criteria to English language articles. The authors make no comment on how this could have excluded potentially relevant evaluations for their review, especially if evaluation reports were published by governments and especially considering that these evaluations were conducted in LMIC contexts.

This is not a typical systematic review because the authors did not design the search protocol or conduct the literature search stages of the systematic review. The authors provide sound justification for their decision to use the 3ie repository to cover these aspects of the systematic review process. However, the authors might consider repositioning this article as a modified approach to a systematic review. The authors provide a link to the inclusion criteria and review methodology. However, it may be helpful if the authors provided brief details of the search protocol and search strategy, for example: language criteria, key terms, databases chosen, inclusion of grey literature. These are aspects of the systematic review search that helps the reader assess the appropriateness of the search and the articles included for analysis.

It would be helpful for the following sentence on page 4 to be rephrased: “The 3ie review process has no restriction on publication date; however, the systematic review upon which it is based was completed in July, 2016.” I needed to review the 3ie repository to understand what this sentence means – that a search of 45 databases was conducted in August 2016, which is where the authors sourced their articles for this review.

The authors should also comment on the limitations of this strategy in their limitations section of the manuscript.

The authors consulted a library information science specialist to audit the methodology. The specialist affirmed that the 3ie repository search included the most relevant databases, however, there were lesser known databases excluded from the search. The authors do not search these databases separately. It would be beneficial if the authors commented on/justified their decision to exclude these databases, despite feedback from a specialist that they may contain relevant articles for the review. The search in these databases could have used the same terms as the 3ie repository search strategy.

The authors state that there is no universal agreement on an integrated development approach definition and then they present their definition (presumably FHI 360 because of author affiliations, although this should be stated more clearly), which is used for the review. If space is available, it would be helpful to see other definitions of integrated development approaches and justification by the authors as to why their definition was chosen for the review. It would be beneficial to comment on why the authors support a definition that excludes interventions that integrate subsectors of a core sector or interventions that measure multiple sector outcomes for interventions that do not have a multi-sector component. Arguably, these two intervention approaches could also have amplified impacts worth understanding. This discussion around definition may be most appropriate in the introduction section of the manuscript.

The authors should be more explicit on how the definition of integrated development approaches influenced the review process and inclusion/exclusion criteria.

The authors should clarify how they decided on the list of sectors for their review such that another set of researchers would come up with the same list of sectors if they replicated the study.

The authors excluded grey literature articles, and based on the results, setting this criterion excluded 1,163 potentially relevant articles. The authors should justify their decision to exclude this grey literature, especially considering that many impact evaluations may not be published in peer reviewed literature.

This is another point that the authors should comment on in the limitations section.

It would be interesting to know who conducted the evaluations included in the review (e.g. NGOs, government, UN agencies) and for whom the evaluations were conducted (e.g. consultants hired by government). This information would provide additional insight on the state of integrated development approaches and provide an understanding of who is interested in these evaluations and who is applying the findings of these evaluations.

It would be interesting to know if there were any sectors that were commonly integrated (e.g. health and nutrition sectors). The commonality of pairings may provide interesting insight on the current state of integrated approaches to development in terms of which sectors are most often paired.

In the table, the authors present “sectors included in a study intervention” (e.g. 238/601 (40%) of studies included the economic development sector in the intervention) and “sector with outcomes measured” (e.g. 102/601 (17%) of studies had economic development outcomes measured). It may be interesting to have the authors present the numerator as “sector with outcomes measured” over the denominator as “sectors included in a study intervention.” This would illustrate, of the articles that had X development sector involved, what proportion measured X sector outcomes. In the case of the economic development sector, 43% of studies (102/238) that included the economic development sector also had economic development outcomes measured.

It seems unnecessary for the authors to comment on whether the included articles had a qualitative component considering that the eligibility criteria for the review excluded qualitative evaluation design methods.

The results show that of the 26 full factorial evaluations, seven reported the integrated arm as more effective, eight had mixed findings, and 11 reported no added value of the integrated arm. This finding relates to the author’s objective to “document if synergies are detected.” The authors should comment on this finding in the discussion section, rather than only commenting on the number of full factorial evaluations conducted and the feasibility considerations associated with this evaluation study design.

Specifically, what synergies were detected/how are those synergies defined? How does this finding relate back to the authors definition of integrated development approaches?

The authors comment that cost efficiency evaluations and qualitative assessments of synergies are also valuable methods to understand the impact of integrated interventions, yet these methods to evaluation were excluded from the review. The authors might consider expanding on why they excluded studies that used these methods and/or comment on whether future research should include evaluation of integrated approaches using these methods.

It might be interesting for the authors to comment on future directions for research on integrated development interventions.

The authors state that “Our systematic review is not intended to determine whether or not integrated development approaches work. We know from the high number of randomized evaluations included here that in many contexts integrated, multi-sector interventions have produced positive impacts.” However, the authors did not provide readers with any results on the effectiveness of integrated approaches using RCT evaluations in the results section of the manuscript.

Also, what do the authors mean by context? Location?

We have read this submission. We believe that we have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however we have significant reservations, as outlined above.