Evaluating capacity strengthening for social and behavior change communication: a systematic review.

International social and behavior change communication (SBCC) programs often include capacity strengthening (CS). Quality evaluations of CS can help justify investing in these activities and guide the design of future CS activities. To inform and improve future CS efforts, a comprehensive examination of ways in which activities aimed at strengthening capacity for improved SBCC are assessed is needed. Unfortunately, systematic literature reviews about the assessment of CS activities in SBCC programs are rare. This systematic review helped fill this gap and explored ways in which CS interventions for improved SBCC in low- and middle-income countries (LMICs) evaluated their success. A search of electronic research databases yielded a total of 1033 potentially eligible publications. Reviewers identified 19 eligible publications that assessed the effects of activities for improved SBCC capacity. Reviewers identified seven findings, including the fact that evaluating CS for improved SBCC is rare, with only three publications having focused exclusively on evaluating SBCC capacity. This current review also identified several shortcomings around the quality of writing as well as sufficient detail to support certain claims and conclusions, especially around issues of sustainability. Until quality evaluations of CS activities are better documented, future CS activities for SBCC will find it difficult to identify effective CS approaches and demonstrate their contribution to improved SBCC in LMICs. The review discusses several implications and offers practical recommendations regarding ways to improve the evaluation of CS activities in SBCC.

INTRODUCTION

Social and behavior change communication (SBCC) in the international health context plays an important role in introducing and maintaining desired health behaviors and norms. SBCC uses a range of communication approaches, such as mass media, social media, digital communication, community-level activities, interpersonal communication and advocacy to influence social norms and behaviors (USAID, n.d.). Capacity for the planning, implementation, monitoring and evaluation of SBCC activities is necessary for impactful and sustainable SBCC programming. Capacity strengthening (CS) for SBCC may involve a diverse range of SBCC practitioners including SBCC organizations, government partners, health service delivery organizations that integrate SBCC within their activities, journalists, community-based organizations and peer educators. Understanding the various existing measurement and evaluation approaches to strengthening capacity for SBCC is necessary to guide CS activities for SBCC. Guidance for assessing CS activities, in general, is warranted given the dynamic nature of capacity (Ebbesen ; LaFond and Brown, 2003) and the numerous challenges in evaluating capacity (Ebbesen ).

When it comes to SBCC and the larger field of health promotion, evaluators have argued that communities and their capacities are dynamic and therefore merit appropriately tailored evaluation approaches (Labonte and Laverack, 2001). The complexity of measuring capacity has been noted in the international development sector, as well. Woodhill argued that social systems and institutions are (Woodhill, 2010) inherently complex because people and organizations are unpredictable and because social systems and institutions are dynamic networks of separate entities. Ebbesen et al. proposed using a mix of qualitative and quantitative (Ebbesen ) methods which may increase the strength of measurement when evaluating CS activities in health promotion programs. Articulating the ways in which CS activities are hypothesized to affect change (e.g. logic model, theory of change [ToC]) can also enhance their evaluation (Wigboldus, 2010). There is no single standard for assessing evaluations of CS for SBCC interventions (Hawe ; Ebbesen ; LaFond and Brown, 2003). However, evaluators have developed several frameworks to help guide evaluation of capacity in the international health sector (LaFond ), in health promotion (Hawe ; Catford, 2005) and in SBCC (Health Communication Capacity Collaborative, 2016).

A single review has examined evaluations of CS efforts for HIV SBCC programs (Lettenmaier ). This current systematic review examines a broader question to consider how CS for SBCC is evaluated across all health areas in low- and middle-income countries (LMICs).

METHODS

The review involved three steps: (i) literature search online, (ii) selection of eligible publications via abstract and full-text review and (iii) data extraction and quality assessment via full-text review. A team of six reviewers participated in screening and reviewing publications, four of which helped write this manuscript.

Inclusion and exclusion criteria

A publication abstract/summary needed to meet the following criteria in order to merit initial consideration for inclusion in the review:

The publication was published between January 1995 and July 2016.

The publication described a SBCC/health communication intervention that took place in an LMIC(s).

The publication was available in full-text form.

The publication described an activity that aimed to build the capacity for improved SBCC or health communication.

A full-text publication with eligible abstract/summary needed to meet the following five criteria to merit inclusion in the final review:

The publication was written in English.

The publication was gray literature (e.g. internal evaluation reports produced by an organization) or peer-reviewed literature (e.g. journal articles)—entire journal issues, conference descriptions or proceedings, literature reviews, editorials and opinion pieces were not eligible.

The publication described effects of an activity on the capacity of participants (could include additional evaluation content).

The publication described the evaluation methodology.

The publication included at least three of the following characteristics.

clearly defined whose capacity the activity aimed to strengthen

described CS activity in detail

described barriers and facilitators that affected the success of the CS activity

specified what type of capacity was being evaluated

discussed recommendations or lessons learned

The team of reviewers defined capacity broadly, as the ability of a person or entity to achieve objectives over time. A CS activity for SBCC, by extension, is a set of actions that facilitate the processes by which SBCC capacity is intentionally improved (LaFond ). To be as inclusive and comprehensive as possible, reviewers included publications that described effects of an intervention on the capacity of participants. In other words, the final sample of publications included not solely CS evaluations but also publications which contained some assessment of CS activities or a description of the effects of CS activities on capacity. This approach meant that publications with minimal focus on evaluating CS activities for improved SBCC were eligible in this review, which was necessary in part due to the small number of formal evaluations of CS for SBCC activities. This approach, however, also introduces the limitation of potentially including less rigorous assessments. However, for simplicity’s sake, the publications are heretofore described as ‘evaluations’, due to the focus of this article’s analysis.

Literature search

Starting in October 2016, a reviewer searched for publications in EBSCOhost, POPLINE, Scopus, PubMed and Cochrane Library using terms related to evaluation, CS and SBCC. In addition, the reviewer identified gray literature associated with a wide variety of international donor agencies, multilateral agencies and foundations (e.g. the World Health Organization, the Danish International Development Agency). DistillerSR software automatically removed duplicates identified from the multiple searches.

Selection of eligible publications

In the first phase of assessing eligibility, four reviewers screened publication abstracts/summaries—two reviewers per publication. They included publications that described the evaluation of CS activities for improved SBCC in LMICs. In cases where two reviewers disagreed on the assessment, a third reviewer conducted a separate independent screening of the abstract/summary to break the tie and determine whether or not to include the publication. Publications that fulfilled the abstract/summary eligibility criteria underwent full-text review by one of four reviewers. The full-text review assessed eligibility according to the above-mentioned criteria.

Capacity strengthening activities and their evaluations

Examination of the CS activities and their corresponding evaluations, although not the primary focus of this review, provides context for understanding our assessment of CS for SBCC evaluations. First, reviewers extracted information about the scale and scope of CS activities, where CS activities took place, who participated in them, and what kinds of SBCC capacity the activities sought to strengthen. Second, in order to describe evaluations, reviewers extracted information related to the goals of the evaluation activities, types of data collection, evaluation design and the findings presented. Reviewers classified the capacity-related findings by the SBCC Capacity Ecosystem framework (Health Communication Capacity Collaborative, 2016) to reflect whether findings described capacity at an individual, organization or system level. This framework posits that CS activities for SBCC can influence change at these three levels. Given that a goal of international development is to create and sustain capacity over the long-term, reviewers also examined whether and how evaluations assessed the sustainability of CS efforts.

Quality of writing

One reviewer further analyzed the publications and extracted data used to generate the final manuscript. During this process, this reviewer assessed the quality of each written publication in two steps. First, the reviewer assessed general readability of the publication. The easier the publication is to understand and follow, the more likely that a reader will find the evaluation useful and apply the findings and recommendations to future programs. A publication’s readability was diminished by grammatical errors, overuse of jargon (e.g. project-specific acronyms) and poorly organized content. Second, the reviewer gauged whether publications sufficiently documented their activities and evaluation to support their conclusions. e Linking the publication’s CS-related recommendations/conclusions back to specific evaluation findings help ensure validity and credibility.

RESULTS

Initial searches yielded 1646 publications. After automatic removal of duplicates in Distiller, 1033 publications remained. The abstract/summary review yielded 243 potential publications, and the full-text review resulted in a final sample of 19 eligible publications (see Figure 1).

Fig. 1:

Study flow diagram for the publication selection process.

Description of capacity strengthening activities

Before considering the quality of the written publications, reviewers first examined the CS activities for SBCC that were evaluated. The CS activities described in the study sample took place in 12 different countries; 7 publications described activities in Asia and 12 publications described activities in sub-Saharan Africa. The participants of the CS activities were varied. Most commonly they included community groups, community-based organizations or their members (n = 8, Adams, 2007; ADAPT-CAETIC Development Consortium, 2014; Chakravarthy ; Field-Nguer ; Hien ; Save the Children, 2005; Stauffer ; Martin and Freimuth, 2011) and community health workers/volunteers (n = 6, Ahluwalia, ; Field-Nguer ; Hien ; Pyle ; Boothby and Veatch, 2007; Stauffer ). Publications did not necessarily evaluate the capacity among all CS activity participants. (For additional details about these publications, see Table 1.)

Table 1:

SBCC CS evaluation characteristics (N = 19)

	CS activity and evaluation characteristics
Publication (year of publication)	Country(ies) in which evaluations took place	Purpose of CS assessment activitiesa	Capacity strengthening activity participants	Data collection activitiesb
Qualitative
Pyle et al. (2007)	Bangladesh	− Evaluate program objectives including extent to which sustained improvements in municipal health systems were achieved	Community committees, municipal government	- Individual interviews with donor staff and unspecified persons   - Group interviews with unspecified persons   - Meetings with past evaluators of the program
Stauffer et al. (2016)	Bangladesh	− Not stated	Clinicians, community health workers (CHWs), community sales-agents	- Individual and group interviews with ‘target group representatives’ and stakeholders   - Phone interview with implementing program staff
Andina et al. (2013)	India	− Evaluate effectiveness of program’s technical assistance in improving local capacity, identify lessons learned and program legacy	Government, clinicians, private media organizations	- Document review   - Individual interviews with unspecified individuals   - Meetings with stakeholders
International Business and Technical Consultants Inc. (2015)	Kenya	- Determine extent to which program achieved goals of strengthening the capacity of public sector institutions to promote and oversee social marketing (SM) and SBCC initiatives, of one or more Kenyan entities to implement SM and SBCC initiatives, of increasing the synchronization of national and United States Government -funded SM and SBCC   - Identify lessons learned regarding how to establish sustainable SM programs   - Identify recommendations for making future SM programs more effective in delivery sustainable services	Ministry of Health (MoH) staff, non-governmental organizations (NGOs)	- Document review   - Group interviews with various participants, partners and stakeholders   - Secondary data analysis
Martin and Freimuth (2011)	Kenya, Namibia, South Africa	- Evaluate C-change’s progress toward improving capacity in strategic planning, program design, implementation and monitoring and evaluation of communication programs.   - Review the interest and commitment of USAID/Global Health Missions for SBCC CS in the follow-on SBCC project	Public health students, public health professionals, NGOs, media organizations	- Document review   - Observation of intervention activities   - Individual interviews with various participants and stakeholders   - Individual and organizational capacity assessment
ADAPT-CAETIC Development Consortium (2014)	Madagascar	- Determine extent to which the community committees fulfill their roles and responsibilities in managing community health systems, including support to CHWs   - Assess the effectiveness of the program’s capacity building on local community health committees capability in the rural areas of Madagascar	Community committees	- Individual interviews with community committees, and other stakeholder   - Group interviews with stakeholders
Rispel et al. (2010) c	South Africa	- Determine the main activities of the peer educators   - Identify barriers and facilitators of peer educator actions   - Identify the impact of the training program on the community trainers	Master trainers, ‘community trainers’ also called peer educators	- Individual interviews with master trainers, community trainers and other stakeholders   - Group discussions with stakeholders   - Survey of community members
Wills and Rudolph (2010) c	South Africa	− Explore the experiences of the first cohort of trained health promoters and to use the findings to determine whether and how the training program contributes to capacity building	Health promoters	− Individual interviews with health promoters
Al-Iryani et al. (2011) c	Yemen	- Highlight factors that enabled community peer education   - Understand the role of the community focal points, local council members   - Understand the effect of life skills on the life of the peer educators and their peers   - Understand the impact of the program on the sexual behavior of the targeted young people, including sex workers	Peer educators, community focal points	− Individual and group interviews with peer educators, community focal points and other stakeholders
Family Health International (2006)	Zambia	− See how five program elements contributed to program’s strategic aims. Elements included capacity building within the Ministry of Education at all levels and four others geared towards adolescents	Health promoters, student peer educators	- Individual interviews with peer educators, teachers and other stakeholders   - Observation of peer educators lessons
Quantitative
Ahluwalia et al. (2010) c	Tanzania	− Assess the sustainability of the impact of community-capacity and training provided to CHWs maternal and infant health outcomes, service uptake, emergency obstetric care, emergency transport systems and other formal and informal components in the community	CHWs	− Survey of CHWs and other stakeholders
Namagembe et al. (2012) c	Uganda	− Examine the effect of team-based training courses on clinical and laboratory skills related to malaria management relative to baseline	Clinicians and other health facility staff (e.g. lab workers, information system assistants), same training given to all groups, results not disaggregated	− Observation of clinicians post-training at various time points using structured checklist
Mixed methods
Adams (2007)	Cameroon	Not stated, but the purpose of one data collection activity (survey of health promoters) was to assess the impact of the program on their behavior, sexual and reproductive health, socio-economic impact, as well as the impact on other young people, families and communities	Health promoters, community-based organizations (CBOs)	- Document review   - Survey of health promoters
African Youth Alliance (2005)	Tanzania	- Assess extent to which the interventions met their objectives (increased use of services)   - Capture successes, challenges and lessons learned of both the facility and outreach efforts	Clinicians, peer educators, other health facility staff	- Individual semi-structured interviews with clinicians, peer educators and other stakeholders   - Group interviews with clinicians, peer educators, client youth   - Pre- and post-training for clinicians   - Exit interviews with facility clients   - Mystery clients
Field-Nguer et al.   (2015)	Tanzania	- Learn which of the program’s CS interventions have been most effective in improving the ability of individuals and institutions to design, implement and evaluate SBCC programming   - Identify what areas of behavior change research, design or implementation has program successfully strengthened the capacity of Tanzanian individuals, institutions and communities   - Identify internal or external factors have influenced the project’s ability to achieve its objectives	MoH staff, training institutes, media institutions, health promoters, young educated adults/SBCC professionals, CHWs	- Document review   - Individual interviews with various types of   program participants   - Group interviews with program   participants   - Individual pre- and post-training questionnaire   - Institutional capacity assessments
Save the Children (2005)	Tajikistan	Not stated	Community committees, children, health facility staff	- Document review   - Exit interviews with pregnant mothers   - Individual interviews with health facility staff and stakeholders   - Group interviews with health facility staff, children, community committees and other stakeholders   - Secondary data analysis   - Survey of community members
Chakravarthy et al. (2012) c	India	- Describe the steps in the development of the community mobilization process; show how a model was adapted to the different social and economic environments of the sex workers   - Assess the progress of the program in developing organizations and empowering sex workers to play an active role in their local and district-wide organizations	Peer educators (sex workers), CBOs	− Individual semi-structured interviews with CBO members, peer educators and other stakeholders
Hien et al. (2008) c	Vietnam	- Evaluate the effectiveness of an educational program   - Assess the usefulness of a participatory style of education and the applicability of an intersectoral approach in the educational process   - Evaluate the appropriateness of the facilitated training from the point of view of participants	CHWs, CBOs, traditional authorities, same training given to all groups, some results disaggregated	- Pre- and post-test of participants and controls   - Training feedback survey of training participants
Boothby and Veatch (2007)	Indonesia	- Examine whether the project completed what it set out to do in terms of activities, trainings, material development and services that were planned to achieve specific objectives or result   - Identify what changes—if any—resulted from people’s participation in the supported program	CHWs, clinicians	- Individual semi-structured interviews with clinicians and other stakeholders   - Group interviews with CHWs and other stakeholders   - Individual interviews with unspecified number of nurses   - Pre- and post-training test for health providers

For brevity’s sake, this column includes only assessment objectives related to CS activities.

Sample sizes are not provided as this information was partial or missing for many publications.

Peer-reviewed publication.

CS activity participants often received training on the relevant technical health areas (e.g. HIV/AIDS) as well as SBCC-specific skills such as interpersonal communication, counseling and/or health education. Several publications provided little detail regarding the skills, tasks or roles their CS activities aimed to reinforce (Adams, 2007; Andina ; Boothby and Veatch, 2007; FHI, 2006; Hien ; Pyle ; Save the Children, 2005). For example, participants might be trained on ‘approaches to public education’ or how to perform ‘awareness campaigns’ (Boothby and Veatch, 2007). It was often difficult to assess the strength of CS findings in the reviewed publications for two reasons. First, publications often had only limited information on the content of CS activities or the specific aims of the CS activities. Second, CS activities were often integrated into larger SBCC, health and development programs.

Two publications that described CS interventions mentioned a ToC which outlined how capacity for SBCC would be strengthened (Al-Iryani , Martin and Freimuth, 2011) but neither mention was accompanied by a detailed description of how the ToC was integrated into CS activities.

Description of evaluations

All publications specified at least one evaluation goal related to CS for SBCC except for two publications which did not (Stauffer ; Save the Children, 2005). Among the reviewed publications, only three described interventions and evaluations focused exclusively on CS for SBCC (Martin and Freimuth, 2011; Field-Nguer ; Wills and Rudolph, 2010). The first of these three described the evaluation of the global C-Change project; this evaluation used mixed methods to describe shifts in capacity at the individual, organization and system level (Martin and Freimuth, 2011). The second was an evaluation of the Tanzania Capacity and Communication Project (TCCP) that used qualitative methods to assess capacity at all three levels of the SBCC Capacity Ecosystem. The third publication that focused exclusively on CS for SBCC used qualitative methods to describe the learning of South African health promoters that participated in a university course (Wills and Rudolph, 2010). Two of these three evaluations referenced use of SBCC capacity assessment tools. The TCCP evaluation provided a web link to a tool which assessed organizational SBCC capacity across six domains and described how the project used scores on communication strategy and design to plan technical assistance for a local communication organization (Field-Nguer ). The C-Change project’s capacity assessment tool was different than that used in the TCCP evaluation and was described in less detail (Martin and Freimuth, 2011). The C-Change project adapted different versions of the tool for assessing SBCC capacity of individuals, organizations and donors/networks but did not clearly define what categories were assessed and did not discuss assessment findings (Martin and Freimuth, 2011).

In terms of the levels of the SBCC Capacity Ecosystem framework, all (n = 18) but one publication assessed CS activities that aimed to strengthen individual capacity (ADAPT-CAETIC Development Consortium, 2014). Six others described evaluation findings of activities that aimed to influence change at the organization level (ADAPT-CAETIC Development Consortium, 2014; Field-Nguer ; IBTC, 2015; Pyle ; Andina ; Martin and Freimuth, 2011), and five included a system-level focus (Chakravarthy ; Field-Nguer ; Andina ; Wills and Rudolph, 2010; Martin and Freimuth, 2011). Only three publications evaluated capacity at all three levels of the SBCC Capacity Ecosystem (Field-Nguer ; Andina ; Martin and Freimuth, 2011). None of the publications described assessments that were explicitly linked to TOCs.

The reviewed publications differed in terms of evaluation design and methods. In terms of evaluation design, 18 publications described non-experimental designs, and only one study (Hien ) used an experimental design which featured random assignment of community leaders from various communities to a training intervention. Most publications used either qualitative methods alone (n = 10) or both qualitative and quantitative methods (n = 7). The two that used only quantitative methods focused on assessing individual performance (Namagembe ; Ahluwalia ).

Several publications used quantitative measures to describe changes in capacity for SBCC. A few publications quantified the success of CS efforts by describing the percentage of participants that demonstrated specific counseling practices before and after an intervention (Namagembe ; Save the Children, 2005) or reported the percentage of trained participants that were practicing activities such as patient education (Ahluwalia ) and implementing awareness campaigns (Boothby and Veatch, 2007). Four publications described the use of pre- and post-intervention assessments (e.g. participant written tests, survey or observation) to assess the capacity of individuals (Save the Children, 2005; AYA, 2005; Field-Nguer ; Hien ).

In terms of addressing sustainability, some publications acknowledged the important link between CS and sustainability of SBCC interventions (FHI, 2006; Martin and Freimuth, 2011). However, this review found that sustainability was not generally a focus of CS for SBCC evaluation findings. For example, publications often described the hypothesized sustainability of some, but not all, of the implemented CS activities for SBCC (AYA, 2005; Pyle ; Stauffer ; IBTC, 2015). More troubling was the fact that two publications mentioned sustainability of capacity outcomes as part of the stated scope of evaluation but did not include evaluation findings related to sustainability in the actual publication (IBTC, 2015; Andina ).

Only three publications explicitly collected data about the sustainability of capacity outcomes (Ahluwalia ; Namagembe ; Martin and Freimuth, 2011). In the first publication, Ahluwalia et al. examined whether village health workers (Ahluwalia ) were still active 7 years after the end of the Community-Based Reproductive Health Project and found many were. This project trained workers to counsel pregnant women, communicate maternal health care messages and refer women experiencing pregnancy-related complications to district hospitals. Namagembe et al. observed clinician’s malaria counseling skills in Uganda (Namagembe ) repeatedly over a period of 12 months after clinicians received a 6-day training that touched upon health education for adherence, prevention and follow-up. They defined adequate patient education as clinicians advising their patients on at least six out of eight topics (e.g. malaria treatment and prevention, completion of treatment). That study found that participants’ interpersonal counseling skills were improved up until 12 weeks after the training, but fewer participants did so at one year after the training. Lastly, Martin and Freimuth asked respondents questions about (Martin and Freimuth, 2011) whether supported SBCC communities of practice were ‘self-sustaining’, to assess the sustainability of these networks. It is not clear what results link to this question as the publication describes both a web-based and more informal face-to-face community of practice. The publication described the web-based community of practice as ‘struggling’. An interviewee referred to the latter community of practice as ‘structural CS, because it creates a new institutional framework for SBCC’ (Martin and Freimuth, 2011). All three publications presented some evidence to speak to the sustainability of CS for SBCC efforts.

Other publications often lacked concrete findings relating to the long-term sustainability of CS efforts. Five publications posited the sustainability of CS efforts without any explicit mention in their methods or process of conducting their evaluation, to suggest that they collected data to assess sustainability (ADAPT-CAETIC Development Consortium, 2014; Field-Nguer ; Save the Children, 2005; Pyle ; Adams, 2007).

Publications presented factors such as funding and stakeholder buy-in when discussing the likelihood that their CS efforts were sustainable. For example, Pyle et al. discussed that ‘dependency’ of partners on the evaluated project (Pyle ) was a concern and a possible threat to the ongoing activity of municipal committees that serve an SBCC coordination role. Three publications mentioned a lack of funding as a threat to the gains made by or the continuance of CS activities for SBCC in particular (FHI, 2006; Field-Nguer , IBTC, 2015). For example, one publication noted that only 8% of the program budget was dedicated to CS although CS was the sole focus of one of the program’s primary objectives (Field-Nguer ). Another publication discussed factors such as the ‘voiced commitment’ among teachers in a student health education program (Save the Children, 2005) to argue that capacity for SBCC would endure. On the flip side, another publication noted little involvement and activity among community health committees over the course of the project as explanation for why capacity gains were not likely to endure (ADAPT-CAETIC Development Consortium, 2014).

Quality of the writing

Reviewers noted several concerns regarding the quality of publication writing. This section details the findings related to the general readability of publications before describing the strength of arguments related to recommendations and conclusions made about CS for SBCC. In terms of readability, reviewers found clarity of writing and organization of the publications to be highly variable. The three evaluations focused exclusively on CS for SBCC were clearly written and organized (Field-Nguer ; Martin and Freimuth, 2011; Wills and Rudolph, 2010). Similarly, the seven peer-reviewed publications were generally easy to read and well organized (Rispel ; Wills and Rudolph, 2010; Al-Iryani ; Ahluwalia ; Namagembe ; Chakravarthy ; Hien ). With few exceptions (Save the Children, 2005; Stauffer ; Field-Nguer ; Martin and Freimuth, 2011), however, gray literature publications were generally harder to read and follow. In two instances, gray literature publications contained many instances of incorrect grammar or spelling (ADAPT-CAETIC Development Consortium, 2014, Pyle ). One of these two publications was difficult to follow because the publication’s paragraphs were serially numbered in a different way than the table of contents (ADAPT-CAETIC Development Consortium, 2014). Furthermore, some publications used so many project-specific acronyms that they were difficult to follow (Pyle ; Andina ; IBTC, 2015).

The second part of examining publication quality involved assessing whether publications provided sufficient detail to support the claims made.

Lack of detail regarding program activities, evaluation methodology and findings challenged the validity and credibility of certain publications. In terms of lack of detail concerning the assessment methodology used, six publications lacked clarity about the methods used to assess the effects of CS activities versus other activities (Boothby and Veatch, 2007; Chakravarthy ; FHI, 2006; IBTC, 2015; Pyle ; Stauffer ). In addition, several publications provided minimal information regarding their sample sizes or did not clearly explain which stakeholders participated in which data collection activities (AYA, 2005; Andina ; Field-Nguer ; Martin and Freimuth, 2011; Pyle ; Save the Children, 2005; Stauffer ).

Regarding CS activities and findings, several publications contained minimal text or content describing the relevant SBCC capacity skills or competencies that activities aimed to strengthen (Adams, 2007; Ahluwalia, ; Andina ; Boothby and Veatch, 2007, Stauffer ; Namagembe ). In three instances, publications did not clearly define the extent of the technical assistance provided to CS activity participants and, by extension, did not distinguish between the CS activities and the findings of their evaluation (Adams, 2007; IBTC, 2015; Andina ). For example, Adams discussed e CS activities (Adams, 2007) and evaluation findings concomitantly, making it difficult to distinguish one from the other. The fact that it was difficult to distinguish CS activities from findings complicated subsequent arguments about the activities’ contribution to changes in SBCC capacity. As highlighted earlier, only two interventions explicitly mentioned the use of a programmatic ToC (Al-Iryani , Martin and Freimuth, 2011) and none detailed whether or how a ToC informed their CS evaluation.

Regarding whether or not publications presented evidence to support their CS-related recommendations/conclusions, findings were mixed. The fifth screening criteria used to screen full-text publications only needed to satisfy three out of five sub-criteria, one of which was having recommendations/conclusions/lessons learned. As a consequence, a couple publications did not provide any recommendations, conclusions or lessons learned regarding capacity for improved SBCC (AYA, 2005; Namagembe ). Among the 17 publications included in the final sample that provided this content, 11 publications did not clearly support all their recommendations and conclusions related to CS activities for improved SBCC. Six publications did link all recommendations and conclusions to their findings (ADAPT-CAETIC Development Consortium, 2014; Rispel ; Wills and Rudolph, 2010; Al-Iryani , AYA, 2005; Chakravarthy ).

Arguments linking CS to SBCC capacity were often weak and were found in both gray (IBTC, 2015; FHI, 2006) and peer-reviewed literature (Stauffer ). The following two examples help demonstrate this trend. In Bangladesh, a mid-term evaluation of a social marketing project repeatedly touted the need for more training of providers, private sector partners and government staff. However, exit interview results showed client satisfaction was high. The publication’s authors were not clear on how provider behavior, specifically, was lacking except for one observation of poor referral form documentation as an issue and that trained providers did express a need for training (Stauffer ). The evaluation’s repeated emphasis on training seemed out of proportion given the evidence provided.

A second example was a publication that concluded that an NGO-gained capacity by taking over management of their health communications and marketing (HCM) project. The publication argued that the project had ‘empowered board and management and the capacity to implement HCM [project] activities’ (IBTC, 2015). The publication described the NGO’s and the project staff’s joint implementation of SBCC campaigns but did not explain how the NGO specifically gained capacity relative to what it had previously accomplished before the CS activities. Authors also supported the empowerment claim by saying the management staff were all Kenyan and the NGO’s board was more than 50% Kenyan. While the supporting findings are positive, the author’s claim of empowerment is weak without a larger conversation of what empowerment is, how nationality correlates with empowerment, and what the NGO’s capacity was before the implementation of CS activities (IBTC, 2015). These two examples represent a pattern found across the reviewed publications.

Some publications generally supported most, but not all, of their conclusions and recommendations. Two such examples were evaluations that focused exclusively on assessing CS for SBCC (Martin and Freimuth, 2011; Field-Nguer ). Authors of the TCCP evaluation report concluded that the project successfully built the capacity of the Ministry of Health and Social Welfare and Tanzania Commission for AIDS in designing, developing, implementing and monitoring national-level mass media campaigns (Field-Nguer ). However, none of the findings presented in the report demonstrated improvement in the capacity of the Commission. The publication only describes the technical assistance and training of Commission officials. Similarly, Martin and Freimuth argue that media professionals (Adams, 2007) need different capacities strengthened than those required by SBCC professionals and that C-Change should identify these capacities and tailor CS approaches accordingly. While their recommendation may be true, it was not clear which findings supported that recommendation (Martin and Freimuth, 2011). Compared to other publications, evaluators of the TCCP and C-Change projects assessed an array of diverse CS activities (Martin and Freimuth, 2011; Field-Nguer ).

In the process of assessing the publication quality, reviewers noticed a separate trend in the way arguments related to CS activity effectiveness were supported by indirect proxy measures. Publications sometimes implied that capacity had been built based on behavioral outcomes of the intended audience of the SBCC intervention (e.g. peer education) as opposed to the outcomes of the CS activity itself (e.g. peer educator training). In at least one case, a publication noted that while the intervention had addressed family planning providers’ perception and behavior, evaluators could not know if the mentoring of providers translated into different counseling practices without looking at contraceptive commodity sales data (IBTC, 2015). In other words, the evaluators seemed to argue that they could not assess changes in family planning counseling capacity directly and would need supplementary data about client behavior (IBTC, 2015). Elsewhere, evaluations described the health behaviors of CS activity participants, themselves, though the goal and design of the health promotion program suggested these participants were to counsel or educate others (Adams, 2007; FHI, 2006). All 19 publications provided at least a minimal description of how CS activities affected their participants. However, several publications dedicated markedly more emphasis on describing shifts in social norms or behaviors of the SBCC intervention’s audience and less emphasis on describing the effects of CS activities on its participants (Al-Iryani ; Rispel, ; Andina ; Boothby and Veatch, 2007). The priority was often placed on measuring behavioral outcomes and their determinants rather isolating the direct effects on capacity.

DISCUSSION

The current systematic review identified seven overall findings with distinct programmatic and research implications for CS in SBCC. First, there is a scarcity of literature about specific evaluation of CS for SBCC. Although CS has become a ‘buzz word’ in international development (Hawe ), published evaluations of CS activities that aim to build SBCC capacity in LMICs are rare. Only 19 publications met the study’s eligibility criteria, and only three focused solely on CS findings. This review’s findings complement previous literature which has noted that evaluation of CS efforts for SBCC is rare (Lettenmaier ). Future evaluations should publish their findings more widely in order to share lessons learned with others interested in implementing CS activities in SBCC and generate further knowledge about what works and what does not work.

Second, the review found that among publications that described evaluation findings of CS for SBCC, most did not emphasize the assessment of CS activities. The details of how CS activities were evaluated was often lost in description of large multi-arm interventions that were not focused on SBCC or CS. Lack of detail about the CS activities and evaluation, including about a program’s ToC, made it difficult to assess the process of change and the impact of CS activities. This finding is consistent with the understanding that CS is rarely a stand-alone activity and is more often paired with other programmatic activities (Hawe ; Labonte and Laverack, 2001) such as service delivery or health interventions. In addition, this finding is a concern given the need to identify the most appropriate and effective CS activities for SBCC. Future evaluations of CS for improved SBCC should clearly and thoroughly describe both the CS activities as well as how they are evaluated, including the hypothesized ToC. This is important so that readers can fully understand what is being presented and so that those implementing SBCC programs can apply their learning when designing future CS activities.

Third, the current systematic review found that evaluation assessments of CS for SBCC generally employ non-experimental designs. Of the reviewed publications, only one described an experimental design. This finding is not necessarily a weakness, even though experimental designs are typically the gold standard of program evaluation, given experimental designs, may not be the most appropriate for CS activities. In practice, CS activity participants often volunteer or are specifically chosen, introducing self-selection bias and making random assignment difficult (James, 2001; LaFond and Brown, 2003). In addition, experimental designs of CS activities may sacrifice rich contextual knowledge and a better understanding of why activities were effective, in favor of objectivity and learning whether or not an intervention worked (Berwick ). CS evaluators may purposefully opt for non-experimental designs in part to understand the role of context and what components of CS activities are effective. Ultimately, when designing an evaluation of CS for improved SBCC, a critical consideration for practitioners is that the evaluation study design should appropriately factor in and account for the complexity and situational context of the CS intervention.

Fourth, in terms of the SBCC Capacity Ecosystem, while evaluations commonly addressed organization-level capacity and individual-level competencies, they rarely addressed system-level capacity. This finding comes as no surprise to those who evaluate capacity. The documented predominance of capacity assessment at the ‘micro level’ (e.g. individual level) as opposed to the ‘macro level’ has persisted in capacity assessment for some time (LaFond and Brown, 2003; Brown et al, 2001). This bias may be related to the fact that system-level capacity takes longer to develop and that evaluators have traditionally struggled to measure capacity at that level (Ebbesen ; USAID, 2015). In contrast, individual competencies are often less challenging to delineate, observe, document and therefore measure than system-level changes in a community of practice or professional network of partner organizations. System-level change involves a larger number of stakeholders and more intangibles compared to individual-level change. This finding in respect to the SBCC Ecosystem may also reflect the scarcity of programmatic investment in system-level CS activities. It was often unclear from the reviewed publications why system-level activities were omitted from the scope of evaluation activities. This finding is a concern, in part, because system-level competencies such as prioritizing SBCC funding, sustaining SBCC training and harmonizing SBCC implementers are key to allowing individual-level capacity to flourish within an SBCC Capacity Ecosystem (Health Communication Capacity Collaborative, 2016). Future CS interventions—as well as their evaluations—should take a more ecological approach and aim to affect change beyond the individual level. A more ecological approach to CS efforts and their corresponding evaluation also calls for greater financial support from funders, given the resource-intensive nature of intervening at the organization- and system-level.

Fifth, publications highlighted several challenges regarding assessing sustainability of CS activities. Although CS is key to ensuring sustainable gains in development (LaFond ; Gurman ), most publications did not explicitly assess sustainability. Publications did, however, cite continued funding as a barrier to sustainable change in SBCC capacity. Conclusions about the sustainability of CS activities were not typically linked to evaluation findings. It was not clear whether these programs did not evaluate sustainability of their efforts or whether they did evaluate sustainability but did not present that data in the reviewed publications. Either way, these publications suggest that even when programs place importance on sustainability of its efforts, a gap exists in terms of documenting and sharing that evidence.

The lasting impact of CS for SBCC activities hinges on programmatic, financial and contextual factors including stakeholder buy-in and continued funding. While evaluations must work within the constraints of available resources and program timelines, CS activities often continue until the end of a funding cycle, making it practically difficult to assess whether capacity outcomes were sustained over time and after the end of a program. The terms and conditions of donor funding likely hinder the frequency with which programs can evaluate the success or sustainability of CS efforts. Funders should rise to this challenge by increasing funding for the evaluation of CS activities. Moreover, funders must identify mechanisms for earmarking evaluation funds that extend beyond the end of program activities in order to be able to explore issues of sustainability. Such commitment could enable better documentation and knowledge sharing about which types of CS activities can generate sustainable change.

Sixth, authors identified the shortcomings in terms of quality, both in terms of the writing as well as providing sufficient detail and documentation to support claims. Problems of clarity in the writing, particularly in gray literature, made it difficult to understand what kind of CS activities were most effective at strengthening capacity for SBCC. The substance, content and specific aims of CS activities available in the reviewed literature were often vague or unspecified. Core program components, such as ToC, were also not routinely integrated into the evaluation publications. These shortcomings make it more difficult to obtain a complete picture of the CS activities as well as of their evaluation. Evaluation practitioners must prioritize clear and detailed documentation so that others can easily identify and apply valuable lessons learned into their own work.

Finally, publications did not consistently support all of their recommendations and conclusions with evaluation findings. This appeared to be partially a consequence of the fact that CS was not a focus of most evaluations or publications. Evaluations sometimes inferred the success of CS activities by indirect proxy measures, relying on behavioral outcomes of the SBCC interventions that the CS activities may have supported rather than outcomes of the CS activities themselves. Such indirect arguments for or against capacity may be common for various reasons. In the broader field of health promotion, a direct trade-off typically exists between strengthening the capacity of health promotion workers and focusing on the implementation of health promotion activities that produces an actual health gain (Hawe ). This would explain not just the frequency of indirect measures of capacity but also the rarity of evaluations focused on CS for SBCC. In addition, assessing capacity is difficult. Evaluators face challenges such as the dynamic nature of capacity, multi-level organizational contexts, the long-time frame needed to develop capacity and the difficulty of attributing cause or credit for changes in capacity (Ebbesen ; Gurman ). This finding is a concern for those in SBCC programs seeking evidence-based publications to justify the time and funding needed for quality CS activities.

CONCLUSIONS

Great opportunity exists to improve both the quantity and quality of documented capacity assessment for SBCC. Evaluation practitioners should consider a broader view of CS, incorporating a more ecological perspective that measures change beyond the individual-level and selecting study designs that embrace complexity and context. Moreover, they should also ensure evaluations are well documented (e.g. CS activity objectives, a program’s ToC) and support their conclusions related to capacity with findings in order to enhance understanding and to support their claims. Finally, funders should invest in rigorous CS activities that allow for the assessment of sustained capacity outcomes after the end of CS activities. By combining greater attention to complexity and detail, with financial support that allows for more rigorous and comprehensive evaluations, future evaluations will better demonstrate the potential and lasting impact of CS activities on improved SBCC and health promotion.