An illustration of how responsive feedback in a social marketing tobacco control intervention in Ghana enabled managers to make decisions that increased intervention effectiveness.

This report illustrates how a feedback loop, set up to provide data and insights to a donor and designers/implementers of a social marketing tobacco prevention intervention in Ghana, helped adapt the original design of the intervention to one that was more suited to the social and media contexts of Ghana. The designers/implementers had previously, successfully implemented a tobacco control intervention with adolescents in Botswana. This experience had informed the initial intervention design in Ghana. As the feedback generated by evaluators started demonstrating just how different the Ghanaian social and media contexts were from the Botswanan one, implementers started making changes to their selection of channels, resulting in a design which was quite different from the original one. The close involvement of the donor in this process enabled implementers to make rapid changes to the design of the intervention. This illustration adds to a small but growing literature establishing the importance of feedback loops to improve the design and implementation of development interventions. Copyright:

Disclaimer

The views expressed in this article are those of the authors. Publication in Gates Open Research does not imply endorsement by the Bill & Melinda Gates Foundation.

Introduction

“Why do some behavioral interventions succeed and others fail?” is a question at the forefront of every implementer’s mind. Are there systematic processes that, if followed during the design and implementation of a behavioral intervention, increase the likelihood of the intervention’s success? Specifically, can timely learning or responsive feedback during the implementation of an intervention help increase the likelihood of a project’s success?

The answer to this question occupies the attention of “the Curve” community of practice (CoP) supported by the Bill & Melinda Gates Foundation ( Viswanath ). This CoP provides a platform to 20 leading implementation and research organizations and 5 donors that, as a group, have focused on this question since 2018. The Curve provides participating organizations the opportunity to think deeply about this question, to learn from each other’s experience and to develop illustrative examples of responsive feedback, with the objective of providing examples which may be of broader interest within the development field.

Here, we illustrate how feedback can help by providing an example of the process of learning during the implementation of a behavioral intervention in Ghana. In 2016, the tobacco control team within the Global Policy and Advocacy (GPA) team at the Bill & Melinda Gates Foundation decided to fund an impact evaluation of a social marketing campaign to prevent tobacco use in Ghana. The tobacco team felt that an objective impact assessment would not only help guide the design and measurement of future social marketing campaigns, but could also contribute to an investment case for prevention-related communication. A preventive approach for tobacco control is extremely important in Africa, where the prevalence of tobacco use is still relatively low and the tobacco industry sees a prime opportunity for market growth. The impact evaluation was designed to respond to several critical questions: which messages are most effective; which media are most effective; how cost-effective is the campaign; and, perhaps most importantly, what is the magnitude of the campaign’s impact.

The implementing organization, Good Business (GB), had previously designed social marketing interventions in Botswana and Uganda. These interventions had been implemented with the aim of changing knowledge, attitudes, and perceptions around tobacco use. GB had hired researchers to measure the effects of its intervention among adolescents who were exposed to project activities through indicators such as the number of girls who spontaneously listed smoking as something they were opposed to and the number of girls who thought smoking made them more attractive. This ability to track behavior among teens exposed to the intervention had enabled GB to feel confident about continuing with the implementation of planned activities. While results from the campaigns were generally positive, the foundation’s tobacco control team wanted independent verification that the approach worked, and that positive changes in attitudes and norms related to smoking could be attributed to the campaign. In other words, they wanted a comparison of the effects of the campaign with what would have happened in its absence. The tobacco control team felt that a population-based impact evaluation would add to the evidence base and help them build an investment case for using social marketing to prevent teen tobacco use in sub-Saharan Africa.

The Tulane University School of Public Health and Tropical Medicine (Tulane) was selected to conduct the evaluation using a quasi-experimental, panel design. The evaluation relied on population-based household surveys conducted with randomly selected 13–17 year-old girls in project and non-project areas in Ghana. The full methodology and results of the evaluation are detailed in Hutchinson . These surveys were conducted before activities were initiated and at the end of the project. A unique, additional feature of the evaluation was a feedback loop comprised of multiple rounds of mobile phone surveys designed to measure exposure and message comprehension among the project’s target audience during the project’s 20-month implementation period. The level of exposure to a social marketing intervention is considered a key determinant of its anticipated success ( Hornik, 2002). Indeed, some experts will not consider assessing the population level impact of an intervention that reaches less than 20% of its target audience.

Intervention description

The SKY campaign built upon GB’s experience of implementing a social marketing teen tobacco prevention intervention in Botswana. The Botswana campaign had shown how important a determinant peer influences were on teen tobacco use ( Persons, 2017). Figure 1 shows the Theory of Change (ToC) of the SKY Girls tobacco control intervention. Based on the Botswana experience, the Ghanaian ToC emphasized the importance of a multi-channel brand and community platform to motivate teenage girls to join a movement that fostered a sense of belonging. By connecting to other girls and by expressing themselves through positive choices, the ToC expected adolescent girls to gain self-confidence and a sense of identity that would enable them to reject tobacco use.

Figure 1.

Theory of change for SKY girls tobacco control.

In Ghana, formative research conducted by the implementer to design the campaign showed that teen girls were usually aware of the health harms of tobacco and were reluctant to smoke, but the paramount need for social inclusion among fellow teenagers often dictated their decision-making. Teen girls in Ghana lacked the confidence to resist the social pressure of smoking. The choice to smoke had become part of a teenage girl’s struggle with identity formation.

The SKY social marketing campaign was designed to unite girls behind the idea of ‘being true to themselves’, to create an environment that enabled girls to express their true feelings, including their desire not to smoke, and to feel supported by a community of peers. In partnership with the Ghanaian advertising agency Now Available Africa, GB launched a platform focused on girls’ empowerment and resilience. Using content around passion points such as music and fashion to draw girls in, the campaign weaved in messaging about choosing not to smoke as the smart, ambitious, aspirational choice.

To reinforce the message, the campaign was executed across multiple media channels, including free magazines, radio programming, free original movies, social media, and live events in Accra. The local agency worked closely with teens to develop content owned by the audience and in direct response to their interests. As part of this campaign, Ghanaian teens were featured throughout the media channels – in the magazine, in live discussions on radio, and on social media.

Project management

We focus our discussion on the interaction between the implementers and the evaluators, on the type of feedback provided and how the feedback provided enabled implementers to adapt the design of the intervention to the media and cultural context of Ghana.

A primary program officer (JP) from the tobacco control team of the Bill & Melinda Gates Foundation managed both the implementation (primed by GB) and the evaluation (primed by Tulane) grants with support on the evaluation component from a colleague in the foundation’s Global Delivery team. The primary program officer believed that evaluators should maintain their independence but should provide insights and learnings to the implementer on a regular basis to optimize the use of data in management decision-making. The evaluation grant was designed to work in tandem with the implementation of the project: research timelines were aligned with implementation timelines, indicators were linked to the project’s theory of change (ToC).

With this management model in mind, in December 2016, the primary program officer organized an in-person, project kickoff meeting attended by the UK-based implementer, the US-based evaluator and the Seattle-based donor. This provided an opportunity to discuss project objectives and evaluation approaches. Following the kickoff, a cadence of monthly Skype calls was put in place which included the primary program officer, her colleague from Global Delivery (SA), the implementer and evaluator. These meetings occurred consistently over the 20-month duration of project implementation.

From the outset, evaluators focused on learning about the project design, its underlying assumptions and the activities planned. Understanding the project’s ToC and learning about the outcomes expected from the planned activities had immediate implications for the design of survey instruments used for assessing exposure to project activities and measuring the impact of the campaign on outcomes.

As part of their contribution to this interaction, GB reviewed and commented on survey instruments and provided input on measurement indicators they considered most meaningful from a programmatic perspective. For example, given their previous experience with the challenges of retaining very young adolescents’ attention during an interview, implementers insisted on shortening the length of the household survey questionnaire. Once the baseline survey had been completed, Tulane shared the findings with GB for their interpretation and programmatic decision-making.

Impact evaluation findings

The findings showed that exposure to the SKY campaign was high in Accra. Of girls who lived and attended school in Accra, 84% were exposed to at least one of the SKY channels and 38% were exposed to four or more channels. Of the girls exposed to SKY, 58% said SKY was important to them, with 39% citing the free magazine as the best way to connect to SKY.

The data suggests that the campaign was able to reduce the pressure girls felt to smoke cigarettes and increase teen girls’ confidence in making decisions for themselves. About half of girls exposed to the campaign felt that SKY helped them to reject shisha and cigarettes, think about their future and personal goals and feel confident to make decisions for themselves ( Hutchinson ). Four out-of-ten girls exposed to SKY felt that it helped them connect with other girls in their peer group – a part of GB’s goal of building a community of teens that supports girls in rejecting the things they didn’t want.

In the multivariate analysis, the intervention was associated with an 11 percentage point decrease in the perceived pressure to smoke cigarettes, a 10 percentage point increase in the likelihood of conversations with friends about smoking, and similar magnitude of increases in girls’ perceived ability to make choices and in their disagreement with the justification for shisha use.

Feedback loops

In Botswana, GB had used a multi-channel strategy which used a radio show hosted by teen girls, television, the SKY Girls magazine and outdoor events. Social media was a critical component of the Botswana social marketing teen tobacco control campaign. The SKY Girls Facebook page was the hub of the community, allowing girls the freedom to express what they liked (“ sure ka yone”/SKY) and what they did not (“shapo ka yone”), to share their views and to be supported by other girls. The SKY Girls Facebook page was the third most popular page for teen girls in Botswana and was central to the social marketing campaign’s strategy of amplifying the message.

In contrast with Botswana, the Ghana baseline household survey, conducted by the evaluators prior to the start of the campaign, showed that radio listenership was extremely low among Ghanaian teenage girls. A large proportion of Ghanaian girls were not available online. The use of Facebook, which had been central to the campaign in Botswana, was relatively low in Ghana. Adding to the complexity, television could not be used in Ghana because of the risk of campaign messages contaminating the comparison group. Taken together, this meant that a substantially different channel strategy would have to be adopted for the social marketing campaign in Ghana with no reliance on mass media and reduced reliance on social media.

Several other factors made the Ghanaian context different. Adolescent girls’ mobility was much more restricted in Ghana than in Botswana. To the implementers, this highlighted the importance of schools as a place to reach adolescent girls. School-based activities, which were limited in Botswana because they were considered “uncool”, and were not part of the original plan in Ghana, became vital for the social marketing campaign in Ghana. School-based activities helped build awareness of the brand quickly without the need to advertise on television. The local advertising agency used, Now Available Africa, recommended schools as a great means of reaching girls and ensuring a depth of engagement and being more cost-effective. They also emphasized that school activations were not considered “uncool” in Ghana.

It took several months after the baseline household survey for the mobile phone surveys to be implemented. These phone surveys began providing data on campaign exposure. GB were surprised by the initial low level of exposure. This also meant that the number of teens captured in the phone survey who were exposed to SKY was small – limiting an analysis of the profile of Ghanaian girls who were exposed to the campaign - an analysis that is extremely important for marketers to ascertain whether a campaign is reaching those it plans to reach.

To overcome this limitation and provide insights on girls who were exposed to the campaign and to study message comprehension and message effectiveness, the evaluators decided to analyze the SKY Girls database. This was a database on teens exposed to project activities, maintained by GB and Now Available Africa. Analysis of this database was not originally part of Tulane’s scope of work. The analysis showed that Facebook was less effective in delivering messages than the SKY Girls magazine. Earlier analysis of the baseline household survey had shown that a small proportion of adolescent Ghanaian girls used Facebook. Based on both insights, the implementers chose to reduce use of Facebook as a platform for messaging about the harms of tobacco, and instead used it as a medium to engage adolescents on how they made decisions on important issues. Thus, both the channel and messaging strategy changed based on insights from multiple sources of data. GB felt that they were able to adapt their content strategy to match the way in which adolescent girls were using different communication channels.

As the approach taken in Ghana was considerably different from the one in Botswana, it was important for implementers to feel confident that the programmatic strategy they were choosing – a strategy that was very different from what they had expected to use – led to increased exposure to SKY among teenagers.

Subsequent rounds of mobile phone surveys showed steadily increasing levels of exposure over time. Surveys showed that the channels being used in Ghana were working and that there was a steady increase in exposure to the campaign. Analysis showing that adolescents really liked the project brand helped increase the implementer’s confidence in the new approach.

The intervention evolved considerably, as results from multiple rounds of mobile phone surveys came in. Rather than relying on Facebook and interactive radio shows, schools became the catchment area for the intervention. The implementer had expected to reach people through live, public, events, but this was not feasible in Ghana since Ghanaian girls’ mobility was low.

Evaluators used data from the baseline household survey to conduct segmentation analysis - which groups individuals into a small number of homogenous segments – to aid campaign message development. This helped the implementer develop an understanding of sub-groups within the population who could be targeted with messages that may be particularly appealing to them. The implementer also conducted focus group discussions (FGDs) to understand how exposure to certain project activities, such as the movie or brand activations, were received. Continuous listening to adolescents’ response to project activities through quantitative (and some qualitative) tools helped ensure that implementers understood the social and cultural context of adolescent girls’ behavior. In Botswana, adolescent girls spent more time out of the house with their friends than in Ghana where girls were more influenced by parents who exercised greater control over their movement. This made reaching adolescent girls through social media challenging in Ghana and shifted the project’s focus towards school-based activities. There was also a more direct attempt to keep parents in Ghana informed and to secure their support for SKY.

The Ghanaian experience helped intervention designers to modify their initial ToC. Figure 2 shows the modified ToC of the SKY Girls tobacco control intervention. The modified ToC included a greater emphasis on the risk associated with certain types of tobacco, notably shisha. The outcome measures were tightened and rationalized in the new version of the ToC. Finally, the modified ToC took the desired behavior, non-use of tobacco by teenage girls, explicitly into account.

Figure 2.

Evolved theory of change for SKY girls tobacco control intervention.

Discussion

The feedback loop established in Ghana – a combination of the use of phone survey data for programmatic purposes and feedback gathered through the implementers’ regular interaction with the audience through in-person and online interactions - enabled GB to implement a different intervention than what they would have implemented in the absence of insights about the consumer.

A key feature of the feedback loop was the close involvement of the donor at all stages of project design, implementation and evaluation. Decisions could be taken quickly since the donor understood the types of issues that came up on the implementation and the evaluation side. This allowed implementers to shift to new channels and strategies that were more suitable to the Ghanaian context. The creation of free full-length movies is one such adaptation; the movies spoke directly to the Ghanaian teen audience, access and screening were designed around community needs, and the channel worked well with the evaluation design – distribution could be closely managed to minimize contamination of the control, non-project areas. Evaluators appreciated that the donor team did not micromanage the project, enabling them to have flexibility to make changes to the surveys when needed and possible. They felt that the inclusion of an evaluation expert on the donor team made it easier for them to provide the rationale for their analyses and survey design choices to implementers and for the intent-to-treat approach used for analyses of impact using the baseline and endline household surveys.

What didn’t work as well

Feedback loops worked only partially at the outset of the project: while the evaluators did discuss the quasi-experimental design of the evaluation and its reliance on population-based representative surveys during the December 2016 face-to-face meeting, in-depth input on the evaluation design could not be obtained from implementers because of their lack of familiarity with a quasi-experimental design and the type of data it generates.

Low exposure to project activities in the first telephone survey (conducted about 2–3 months into the project) did not ring true to the implementers who were used to relying on project-generated outputs (such as numbers of people exposed to project activities or discussions on the Facebook page) that are not population-based. Implementers also had concerns about the reliability of the phone survey data, since a higher proportion of girls reported exposure to SKY on television than they expected. Evaluators felt that had they been able to work more closely with the implementers in discussing the evaluation design and the types of results it could generate at the project’s outset, the study design could have been adjusted to provide data that would have been even more useful to implementers. Additional face-to-face meetings between evaluators and implementers at the project’s outset may also have resulted in more insights with practical implications for the intervention.

Implementers felt that had population-based survey findings been available to them earlier - there was a 2-month gap between data collection for the baseline household survey and availability of the first set of results showing that radio and Facebook exposure was low among teens - they would have been able to make changes to intervention design sooner. Implementers did start making changes to the intervention design within 3 months of the launch of the campaign.

Implementers appreciated the evaluators’ explaining the study design to them at the start of the project. Implementers also felt that the initial face-to-face meeting with the evaluators made a big difference to their working relationship. It deepened the understanding of each other’s work and increased the willingness to be flexible in the design of both the survey and the intervention in support of the end goal and made subsequent communication easier. For example, implementers felt that they could request the evaluators to add specific questions that were important from an implementation perspective to phone surveys.

Implementers found that quantitative surveys were not ideal in helping them gain an understanding of adolescent girls’ behavior. Their experience of focus group discussions (FGDs) with very young adolescents in Botswana had showed them that girls who initially reported being non-smokers would then reveal that they smoked once they became more comfortable with the moderator. Implementers felt that greater use of qualitative methods in the evaluation would have been helpful. They were concerned about the length of the quantitative household survey questionnaire and the duration of time spent in the interview by such a young population. The survey questionnaire took an average of 40 minutes in Ghana. Their experience with a similar age group in Botswana had shown that very young adolescents started losing interest in responding to a questionnaire after about 10 minutes. Indeed, the low reported rates of actual smoking at baseline and endline surveys show the limitations of a quantitative survey implemented in a home setting in being able to capture a hidden behavior such as teen smoking.

In spite of the regular communication between implementers and researchers, evaluators did not learn about one of the project’s new activities that was piloted during the last 4 months of the campaign –- community activation trucks to reach teens in Ghana – until near the end of the implementation period. Evaluators were, however, still able to include a question measuring exposure to this activity in the endline household survey. Had evaluators known of this activity earlier, they would have been able to measure exposure to it through the mobile phone surveys as well.

By asking a few open-ended questions in the quantitative surveys, evaluators were able to determine that the indirect approach taken by implementers towards convincing girls not to smoke by focusing on the negative physical aspects of smoking was working: teens gave avoidance of getting yellow teeth and bad breath as their main reasons for not smoking. The emotional appeal to teen girls of maintaining their attractiveness appears to have worked.

Evaluators felt that implementers found the evaluation design to be very challenging. The evaluation focused on measuring population-level coverage. However, this did not provide insightful information for implementers during the initial phase of implementation - since exposure to project activities was very low initially. In other words, although the surveys employed relatively large sample sizes (over 2,000 for telephone surveys), low exposure to the intervention at project beginning meant that little analysis could be done around message comprehension (since there were not enough girls exposed to the campaign in the sample survey).

Evaluators felt that they could have been more helpful if they had been more successful in explaining what could and could not be obtained from the evaluation design. They felt that a more flexible evaluation design would have been more suitable than the fixed, quasi-experimental design that they had chosen. They did, however, make the best use of all available data by conducting additional analyses of data from the project database, additional analyses of baseline household survey data and data from the phone surveys. For example, evaluators substituted one of the mobile phone surveys which was based on a sample of respondents from the household survey for a mobile phone survey which was based on a sample of respondents from the project activities database. The latter database collected data on teens exposed to project activities.

Data availability

No data are associated with this article.

This paper describes the process through which a social marketing approach developed in Botswana was transferred to Ghana, with refinements in strategy informed from an iterative relationship and information sharing between program implementers and evaluators. The paper is well written and informative, and is an interesting case study in that it captures a unique case where a relationship that is often kept at arm’s length (implementers, evaluators and donors) is instead united to lead to a more evidence-based adaption.

It would be helpful to introduce how the data that informed this case study were collected and analyzed. Does it represent only the perspective of the authors, or was there a more systematic approach to data collection and interpretation?

There is a bit of a contradiction in the description of the tenor of the relationship between the implementing team and evaluating teams, which merits clarification and nuance. On page seven, there were concerns expressed from the implementing team that community survey results were late to be shared, and therefore not used to alter the program strategy. But the next paragraph notes that the implementers had a constructive relationship with the evaluative team. These two statements are at odds with each other.

I think the discussion could be bolstered with a deeper exploration of the applicability of the case study lessons to other situations or programmatic/geographic areas. This case study really focused on how channels of communication or messages were altered or changed in emphasis, which makes sense given that it was a social marketing intervention. Would a similarly iterative approach work for non-behavioral intervention refinement? How could it be useful for supply side/health system improvement strategies?

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

The article describes a process through which the authors, who were part of a team funding an evaluation of a behavior-change communications campaign targeting adolescent girls' (potential) tobacco use in Ghana, which was in turn based on a previously tested, successful such intervention in Botswana. The key contribution of this article is to describe how data/insights from a feedback loop set up for various parties to the intervention/evaluation was used to adapt the content and mode of delivery of the intervention being evaluated in a way that is likely to have improved its suitability to the context and its effectiveness.

The article is well written and clearly walks the reader through the process by which the team utilized the data and insights emerging as part of the implementation of the intervention to identify ways in which needed to be adapted, and to facilitate these changes. It also identifies ways in which the feedback mechanisms set up and utilized did not work as well as intended, and offers suggestions for learning from these insights.

Overall, I found the article to be a useful and thought-provoking read. I will summarize below some thoughts on specific and broader ways in which I think it could be further improved.

Specific suggestion: Beyond these specifics, the article also raised a broader question for me, which it would be interesting to have the authors reflect upon in the article should it be revised further. This is about the order in which decisions about what to implement to tackle a problem in a given context ought to be taken. While the article does a great job of showing us how data and feedback can make an intervention more robust, it also raised questions for me that an intervention developed in Botswana was sought to be evaluated in a new context, i.e. Ghana, without initial investigative work around critical pieces of the new context, such as teenagers' preferred modes of social or conventional media engagement (e.g., TV, radio, etc). While the iterative feedback loops established for the project clearly helped the team adapt to critical differences between the previous and proposed context, the article left me with a niggling set of queries about why an evaluation was proposed/greenlighted without this sort of investigation of the context having been carried out as a preliminary step. It would be interesting for the authors to address this question head-on, because it would appear to speak to a key set of questions around the ways in which the questions the field asks 'Does this intervention, which worked in Country X, work in Country Y"? may be in need of revision (perhaps, for example, we should instead ask ''Do the conditions that made this Intervention, which worked in Country X, exist in the same way in Country Y? And if not, how should we revisit the motivating question and intervention design").

The authors describe the previous intervention in Botswana as "changing knowledge, attitudes, and perceptions around tobacco use", and refer to measurements of effects of the campaign on "indicators such as the number of girls who spontaneously listed smoking as something they were opposed to and the number of girls who thought smoking made them more attractive". However, they then refer to the ability to track behavior of teens exposed to the intervention. It seemed at least to this reader that the article would benefit from making a clearer distinction between knowledge or attitudes or perceptions (which the intervention targets) and tobacco-related behaviors, e.g., the (non)uptake, (dis)continuation etc of tobacco use, (which it does not measure, as far as I could tell). Recognizing the inherent difficulty of capturing not only measures of such behaviors but also the extreme difficulty of establishing a counterfactual in a case such as this (where the appropriate measure of behavior change would seek to capture behavior that did not take place but, absent the intervention, would have), I do not think the focus on attitudes, beliefs, etc is in any way a problem, but would merely suggest that the distinction between these and behavior be carefully acknowledged and not elided.

The authors might want to refer to, and possibly cite, recent research which also takes seriously the need for field-based inputs into the development and refinement of behavior change interventions. An example is Buttenheim et al. (2019) [1], which takes a similarly thorough approach to intervention design and refinement. It would be interesting to see the authors situate this research within the broader body of literature that engages carefully with the need to ensure that interventions are adapted to the context they are being implemented in.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.