Informed health choices intervention to teach primary school children in low-income countries to assess claims about treatment effects: process evaluation.

BACKGROUND: We developed the informed health choices (IHC) primary school resources to teach children how to assess the trustworthiness of claims about the effects of treatments. We evaluated these resources in a randomised trial in Uganda. This paper describes the process evaluation that we conducted alongside this trial.
OBJECTIVES: To identify factors affecting the implementation, impact and scaling up of the intervention; and potential adverse and beneficial effects of the intervention.
METHODS: All 85 teachers in the 60 schools in the intervention arm of the trial completed a questionnaire after each lesson and at the end of the term. We conducted structured classroom observations at all 60 schools. For interviews and focus groups, we purposively selected six schools. We interviewed district education officers, teachers, head teachers, children and their parents. We used a framework analysis approach to analyse the data.
RESULTS: Most of the participants liked the IHC resources and felt that the content was important. This motivated the teachers and contributed to positive attitudes. Although some teachers started out lacking confidence, many found that the children's enthusiasm for the lessons made them more confident. Nearly everyone interviewed thought that the children learnt something important and many thought that it improved their decision-making. The main barrier to scaling up use of the IHC resources that participants identified was the need to incorporate the lessons into the national curriculum.
CONCLUSION: The mostly positive findings reflect the trial results, which showed large effects on the children's and the teachers' critical appraisal skills. The main limitations of this evaluation are that the investigators were responsible for both developing and evaluating the intervention. © Author(s) (or their employer(s)) 2019. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

RCT Entities:   Population Interventions Outcomes

Use of data collection triangulation.

Having used a modified CERQual approach, we have high confidence in most of our findings.

The study investigators were responsible for both developing and evaluating the intervention.

Background

Adults and children are confronted with claims about the effects of treatments (any action intended to maintain or improve health) in their everyday interaction. Many of these claims are unsubstantiated, unreliable, inaccurate or biased.1 2 Failure to use treatments supported by reliable evidence may result in unnecessary suffering and can waste scarce resources, especially in low-income countries. This could be avoided if people were able to assess the trustworthiness of treatment claims and make informed healthcare choices.3 4 However, several studies have shown that people commonly lack the ability to understand the risks and benefits of treatments, and fail to apply key concepts that are essential for appraising claims about the effects of treatments.5–8

The aim of the informed health choices (IHC) project is to help address this problem by developing and evaluating learning resources to enable people to assess claims about treatment effects and make informed healthcare choices. We first developed a list of IHC Key Concepts that people need to understand in order to assess claims about the benefits and harms of treatments.4 We determined which of these concepts could and should be taught to primary school children.9 We then spent 3 years developing the IHC primary school resources10 using a human centred design approach,11 that included several cycles of idea generation, prototyping solutions, piloting in schools and making improvements grounded in teacher and student feedback and classroom observation. These resources facilitate the teaching of 12 of the IHC Key Concepts to grade 5 children (10–12 years) (figure 1).9 10 We also developed a podcast to teach some of the same concepts to the children’s parents.12

Figure 1

The informed health choices primary school resources.

Teaching primary school children to assess claims about treatments can capitalise on children’s curiosity and enthusiasm to learn.13 Through children sharing what they are learning at school, it might also have an indirect effect on their family members’ abilities to determine the reliability of claims. Teaching children to assess claims about treatment effects can provide a foundation for a more scientifically literate and healthier society.

We evaluated the effects of IHC primary school intervention on children’s ability to assess treatment claims in a cluster randomised trial in Uganda.14 The trial showed that the intervention had a large effect on the children’s ability to assess treatment claims. It also had a large effect on the teachers’ ability to assess treatment claims. We measured these outcomes again after 1 year, and both children and teachers had retained what they learnt.15 In a linked randomised trial, we evaluated the effects of the IHC podcast on the ability of parents of the primary school children to assess treatment claims.16

The objectives of this process evaluation were to:

Identify factors affecting the implementation, impact and scaling up of the intervention.

Identify potential adverse and beneficial effects of the intervention.

These objectives differ from the four objectives in the protocol for this study,17 as described in online supplementary file 1. These changes were made in order to present the findings of both the qualitative and quantitative analyses more coherently and succinctly.

Methods

This was a multimethod study using qualitative and quantitative data. Our main focus in this paper is on the qualitative analyses. Some of the quantitative results are reported elsewhere.15

Qualitative data collection and analysis

Frameworks

We used a framework thematic analysis approach to guide data collection and analysis.17 We started out by developing two frameworks. The first addressed the factors that could affect the implementation, impact and scaling up of the school resources (table 1), and the second addressed potential adverse and beneficial effects of the resources (table 2).

Table 1

Framework for factors that could affect the implementation, impact, and scaling up of the school resources

Domain	Factors and sources	Explanation
Teachers	Skills and competencies	Teacher’s education and experience in relation to the lessons being taught.
	Understanding of the content being taught	Teacher’s understanding of the content.
	Sufficient training	The extent to which the teachers received sufficient training in teaching the lessons
	Self-efficacy	Teacher’s confidence in teaching the lessons.
	Fit to the teacher’s teaching style and context (eg, class size)	Teachers’ comfort with the instructions or ability to adapt the instructions to their style and context.
	Attitudes	Teachers’ attitude towards new material (change), science, critical thinking and independent thinking by pupils (or their role as authorities in the classroom).
	Beliefs	Teachers’ beliefs about the teaching methods or content (eg, what treatments work or the concepts).
	Emotions	Teachers’ emotions, such as stress or anxiety.
	Motivation	Teachers’ motivation to teach the material.
	Positive learning environment	Teachers’ ability to create a positive learning environment; for example, encourage discussion, respond positively to questions, engage pupils.
Pupils	Literacy	Pupils’ ablity to read and understand the material.
	Attendance	Pupils’ attendance or reasons for poor attendance (eg, long distance to school or inability to pay school fees).
	Motivation to learn	Pupils’ motivation to learn the new material.
	Attitudes	Pupils’ attitudes towards learning, towards authorities, towards science, towards critical thinking.
	Beliefs	Pupils’ beliefs about the content (eg, what treatments work or the concepts).
	Home environment	The extent to which the pupil’s home environment encourages or discourages learning from the lessons.
	Differentiated instruction	The extent to which pupils different learning needs are met.
	Peer influence*	Positive or negative attitudes of other pupils towards the material.
Teaching materials	Value of the material	The extent to which the materials are valued by the teachers and pupils.
	Compatability with the curriculum	The extent to which the material fits with the rest of the curriculum and how it is taught.
	Appropriateness of the material*	The extent to which the materials are relevant, challenging and engaging.
	Credibility of the material*	The textent to which the teachers and pupils perceive the material as credible.
School system and environment	Time constraints	The extent to which there is sufficient time to accommodate introducing the new material.
	Competing priorities*	The extent to which other priorities for the school, teachers or pupils limit introducing the material (eg, preparing for exams).
	School organisation and management*	The extent to which the school provides an environment that supports adoption of new subjects, material and teaching methods.
	School resources, particularly human resources*	The extent to which the school has adequate resources to introduce the new materials (eg, human resources, student/teacher ratio, teacher workload, classroom space and classroom resources, such as blackboards and acoustics).
	Attitudes and beliefs of head teacher and other teachers*	Attitudes or beliefs of colleagues that influence the teacher’s interest in and ability to teach the material.
	Parent and community involvement*	Parents’ attitudes towards the new material or how things are done at the school.
	Regulations*	Regulations (eg, Ministry of Education policies and regulations) that affect introducing the new material.
	Political environment *	Elements of the political environment that affect introducing the new material; for example, authoritarianism or teacher strikes.
	Bureaucracy*	Bureaucratic arrangements that delay or limit introduction of the new materials, or facilitate introducing them.
	Incentives and disincentives	Incentives or disincentives to introduce the new materials for teachers or head teachers.

*There were no key findings in relation to these factors or they were subsumed under another factor.

Table 2

Framework for potential adverse and beneficial effects

Potential adverse effects	Corresponding beneficial effects
Conflict between children and teachers due to children challenging their teachers	More open and engaging discussion of the basis of diverse claims or beliefs.
Conflict between children and parents due to children challenging their parents	Better understanding between children and parents due to children conversing with their parents about what they are learning and parents feeling more engaged with what their children are learning and engagement of parents in discussions of health issues.
Distrust of health professionals or conflict between children and health professionals*	Appropriate questioning of health professionals, better understanding and better healthcare.*
Conflict due to undermining of religious beliefs*	Engagement of children and others in discussion about religious beliefs and science.*
Shortened enjoyment of the innocence of childhood*	Increased enjoyment of school and childhood.
Nihilism or cynicism*	Healthy scepticism and appreciation science.

*There were no key findings in relation to these factors.

The first framework (table 1) was developed iteratively by reviewing existing frameworks and studies of barriers and facilitators to implementing changes in schools,18–22 and frameworks of barriers and facilitators to implement changes in health professional practice.23 The framework shown in table 1 has been modified from the framework in the protocol, based on the results of the framework analysis.

Use of the IHC school resources might have had adverse or beneficial effects that were not measured in the trial, including effects on relationships between children and others, and on beliefs, attitudes or behaviours. We developed a framework of potential effects and impacts based on pilot and user testing of the resources; discussions with education researchers, policy-makers and teachers; potential beneficial effects identified by the National Curriculum Development Centre in Uganda, and reviewing the literature. The framework shown in table 2 has also been modified based on the results of the framework analysis.

Sampling

The intervention was implemented in 60 schools in Uganda.14 For qualitative data, we sampled six of these schools. Within each school, we included all the head teachers, all the grade 5 teachers who participated in the IHC lessons, two children and two parents. In order to capture the opinions, views and experiences of a wide range of participants,24 we purposively sampled the six schools based on geographical location (rural, semiurban or urban) and ownership (public or private). We also sampled schools with variation in the extent to which teachers used the resources as intended. Within the schools, we sampled children with variation in performance on end-of-term examinations and based on how well they understood the IHC lessons. We sampled parents who had also participated in either the intervention or control group of the IHC podcast trial and who had varying levels of education. Finally, we aimed to include all of the five district education officers in the central region of Uganda, where the trial took place.

Data collection

We collected qualitative data using lesson evaluation forms, observation, individual interviews and focus group interviews (box 1).

Lesson evaluation forms: We included qualitative data from the lesson evaluation forms completed by teachers from the six schools selected for the process evaluation.17 Data included teachers’ suggestions for improving the informed health choices (IHC) materials specific to each lesson, what they liked and did not like and what facilitated their achievement or non-achievement of the intended lesson objectives. These were also entered into a spreadsheet.

Classroom observations: Each class in the 60 schools in the intervention group of the trial was observed at least once by a trained research assistant or one of the investigators (AN or DS) early on in the trial. In addition, we observed six classes a second time, after the teachers had become more familiar with using the resources. For observations, we used data collection forms to note how well the teachers adhered to the lesson plan, any problems that the teachers or children had with the lesson, and any aspects of the lesson that went particularly well.17 The teaching was done in English and notes were taken in English, although teachers occasionally used a local language (predominantly Luganda). We noted the extent to which the children followed the lesson and participated actively. After the lesson, we collected the exercise books and recorded how well the children did on the exercises for the lesson that had been observed.

Interviews: We used individual interviews in order to gain an in-depth understanding of issues and further obtain detailed information about personal feelings, perceptions and opinions about the IHC intervention. Individual interviews took place in the participants’ own environment, thus included children’s schools’ compounds, empty classrooms, empty teachers’ staffrooms and policy-makers’ personal offices at the district headquarters.

Interviews with district education officers, head teachers, and teachers: We conducted face-to-face interviews with 4 of the 5 district education officers, head teachers from 5 of the 6 schools and 10 grade 5 teachers who used the IHC primary school resources in the trial. We collected these data following completion of the intervention (which included nine lessons). We chose to only use face-to-face interviews to obtain in-depth data from district education officers and head teachers because it was not feasible to organise group discussions with them. We developed and used semistructured interview guides.17 The guides focused primarily on questions related to barriers and facilitators (table 1), and strategies for scaling up use of the resources. We also included questions about potential adverse and beneficial effects (table 2). The interviewer included prompts for each of the domains and factors in the frameworks (tables 1 and 2), asking interviewees to reflect on their experiences and perceptions from their different perspectives. The adult interviews were scheduled to last an hour. A summary of each interview was provided to the interviewee for further comment.

Interviews with children: We used a semistructured interview guide for face-to-face interviews to elicit the views of children who used the IHC primary school resources in the trial.17 We interviewed a total of 12 children, two from each of the six schools in the process evaluation. The children’s interviews were scheduled to last no more than 30 min. The children were individually interviewed after we obtained parental consent and their assent. Although all the interviews took place on the school premises (classrooms, assembly halls and school compounds), in order to ensure confidentiality, the interviews were conducted in full view of a responsible adult (teacher) but not in close proximity to allow the children to freely share their experiences using the materials.

Focus group discussions: We used focus group discussions in order to gain detailed information about the different groups’ feelings, perceptions and opinions.

Focus group discussions with children and teachers: We used focus group discussions with children and teachers to explore barriers and facilitators to using the resources, as well as potential adverse and beneficial effects. We carried out three focus group discussions with children and two with teachers. The focus group discussions included six to eight participants, with clear ground rules (including confidentiality) agreed in advance. The children’s focus group discussions were scheduled to last 45 min while the adults (teachers) focus group discussions lasted 1.5 hours. Each group was moderated by AN or DS using a guide17 and assisted by an observer who took notes. We used an iterative process to develop focus group guides drawing on issues emerging from the initial individual interviews to revise the questions and to create prompts for the discussions. We conducted further interviews after we had conducted some focus group discussions and had some preliminary findings from these. This allowed us to explore issues for which more detailed data were needed, or to capture the views of particular subgroups (eg, poorer children or children who were not doing well in class).

Interviews with parents: We used a semistructured interview guide for the individual face-to-face interviews with parents whose children used the IHC primary school resources in the trial.17 This interview guide included questions about the parents’ perspectives on how the resources were used, barriers and facilitators to their use, potential adverse and beneficial effects, and potential effects of the school resources on parents. We interviewed a total of 13 parents of children who used the IHC primary school resources in the trial. The parents were selected from participants in the podcast trial. We included both parents who listened to the IHC podcast series and parents in the comparison group of the podcast trial, who did not listen to the podcast series, in order to gain a balanced view of opinions.

Interviews with the lead investigators: AN and DS were responsible for implementing the intervention in the field. Given the importance of their role in the trial and the process evaluation, two other investigators (SL and CG) in turn interviewed AN and DS to capture their thoughts and experiences and their reflections on the findings from the other data sources.

The questions and prompts used in the interviews and focus group discussions were guided by the two frameworks described above. For all of the interviews and focus group discussions, one of the study investigators (AN or DS) carried out the interview or facilitated the discussion. A research assistant was responsible for observation and note taking during the interviews. All of the interviews and focus group discussions were conducted in English except for interviews with two head teachers, one teacher and all but one parent; which were done in Luganda. AN, DS and NKS are fluent in Luganda. We recorded and transcribed all of the interviews and focus group discussions. Observations during class lessons, interviews and focus groups were recorded using structured forms and entered into a spreadsheet.

We conducted a total of 44 individual interviews: 12 with children, 6 with head teachers, 10 with grade 5 teachers, 13 with parents and 3 with district education officers (2 individual interviews, 1 joint interview); and five focus group discussions (three with children and two with teachers). In addition, we observed at least two of the lessons taught in the six schools selected for the process evaluation. The amount of data we collected was guided by considerations of the variation in issues emerging from the data collection and the extent to which we are able to explain these variations; our time and resource constraints; and the need to avoid large volumes of data that cannot be easily managed or analysed as emphasised in literature.24

Data analysis

We used a framework thematic analysis approach, guided by the two frameworks described above (tables 1 and 2), and following the stages of familiarisation, coding, charting and interpretation of the data. We applied both of these frameworks to the same data set.

Two of the investigators (AN and DS) independently read and reread the transcripts. They then coded the data, using the factors included in the two frameworks, but also searching for additional factors. AN, DS, ADO, CG and SL then reviewed summaries of the coded data and considered additional factors suggested by the data. Disagreements were resolved by discussion. They also coded the qualitative data collected through the classroom observations and the teachers’ evaluation forms. For each framework, the definitions and boundaries of each of the frameworks’ factors were discussed among the investigators, and both frameworks were revised in line with the codes and categories that emerged from the data. We then charted the data by writing a summary of the findings for each framework factor. We then considered the extent to which the quantitative data from the teachers’ evaluation forms17 supported those findings and whether they suggested additional findings relevant for each framework that were not captured in our analysis of the qualitative data. Finally, using the summarised data, we explored the range and nature of phenomena, and possible explanations for the findings.

Quantitative data collection and analysis

All 85 teachers in the 60 schools in the intervention group of the trial completed a questionnaire at the end of the term during which the IHC lessons were taught.17 They also completed a lesson evaluation form after each lesson.17 The quantitative data we included from the lesson evaluation forms required teachers to rate suitability of the IHC materials on a Likert scale of 1–6 (1=lowest, 6=highest). Statistical analyses were performed with R (R Core Team, Vienna, Austria; V.3.4.3; using packages tidyverse and knitr).

Appraisal of the certainty of the findings of the process evaluation

We assessed the certainty of the findings using a modified version of the GRADE-CERQual ('Confidence in the Evidence from Reviews of Qualitative research') approach25 26 (see online supplementary file 2). Although CERQual has been designed for findings emerging from qualitative evidence syntheses, several components of the approach are suitable for findings based on multiple primary sources of qualitative data. So far as we are aware, this is the first time that a modified version of GRADE-CERQual has been used to assess findings from a single study rather than from an evidence synthesis.

Integration of the findings of the process evaluation with the findings of the trial

We used a logic model approach to organise the findings of this process evaluation with the findings of the trial. First AN and DS organised the findings into chains of events that may have led to the outcomes of the trial and additional outcomes that we explored (potential adverse and beneficial effects). Findings and outcome measures were categorised as follows in the logic model:

Attributes of the intervention.

Effect modifiers.

Intermediate outcomes.

Observed and potential effects.

After discussion, the investigators revised the logic model iteratively until there was agreement on a final model.

Patient and public involvement statement

Primary school children, parents, teachers, headteachers and policy-makers (district educational officers) participated in providing structured feedback. They were not otherwise involved in the design of the process evaluation or the analysis. This study was preceded by the prioritisation of key concepts9 and the development of the IHC resources.10 Those processes engaged key stakeholders (teachers, children and policy-makers) in the prioritisation of key concepts for inclusion in the resources, brainstorming workshops to generate ideas for the resources, and consultation workshops to discuss the feedback gathered during user-testing and piloting of earlier versions of the resources.

Results

The main findings are summarised in online supplementary file 2, and we have integrated the findings into a logic model in figure 2. Quantitative data are summarised descriptively in online supplementary file 3. We have used the logic model to organise the results.

Figure 2

Logic model. IHC, informed health choices.

The intervention

Value of the IHC intervention

Most of the children, teachers and parents liked the IHC materials because they found them beneficial, interesting and fun for the children. Teachers valued that the IHC content addressed both social and academic issues.

Compatibility with the curriculum

All the teachers that taught the IHC lessons emphasised that the IHC content was important that it should be added to the curriculum, and that it was compatible with the primary school curriculum for science. We found strong support for this finding in the quantitative data in the teachers’ end-of-term assessment: 99% thought that what the children learnt was very important or important; and 95% agreed or strongly agreed that they liked the content of the lessons.14

Compatibility with teaching styles

Most of the teachers found that the design of the IHC lessons was compatible with their teaching styles, especially the use of multiple examples in the teachers’ guide. Several teachers, however, felt that specific aspects of the IHC lessons were in conflict with their teaching styles, such as summarising the content of each lesson. These were not perceived as major challenges.

Differentiated instruction

The teachers felt that the Teachers’ Guide27 allowed sufficient flexibility for teachers to employ different ways of doing things to accommodate different teaching styles. Observations and video recordings from the trial document different teaching styles used to teach the IHC lessons. Examples of the videos can be found here. In the end-of-term teachers’ assessment, 90% agreed or strongly agreed that the instructions for how to teach the lessons fit with their teaching style; 92% agreed or strongly agreed that they liked the way the teaching materials were organised; and 88% agreed or strongly agreed that they adapted the instructions to fit their teaching style.

Teachers pointed out that the way the materials were structured enabled children of different capabilities to participate in the IHC lessons more than they would participate in other lessons. However, several teachers found the material challenging for children who were not fluent in English or who had poor reading skills. This is consistent with the finding in the trial that the effect of using the IHC primary school resources was larger for children with better reading skills.14

Training and understanding of the content being taught

The introductory workshop that was part of the IHC intervention was seen as appropriately conducted by the teachers with many crediting it for the smooth implementation of the project. The majority of the teachers acknowledged that the IHC content was new to them, and some expressed concern about their understanding of the IHC content. All participants thought that the introductory workshop was too short. Despite these concerns, 94% of the teachers agreed or strongly agreed in their end-of-term assessment that they understood the content of the lessons and 97% responded that they learnt very much or much. This is further supported by the proportion of teachers who had mastered the material by the end of the term (72%) compared with the teachers in the control schools (15%).14

Adding on to the curriculum

The IHC lessons were viewed as an add on to what was already in the curriculum. Several teachers emphasised the importance of incorporating the IHC lessons into an already a packed primary school curriculum. This was also a cause of concern for the district education officers, and one of the main reasons why some schools declined to participate in the study.14

Effect modifiers

Incentives

Several teachers identified incentives that motivated them to teach the IHC lessons. These included having head teachers and school owners that were supportive, ongoing support from the research team and simply having enough textbooks.

Teachers’ competencies

Teachers said they had diverse competencies that were important for delivering the IHC content to the children, such as communication and teneral teaching skills. Most felt that it was important that they were science teachers.

Positive learning environment

The majority of the teachers shared ways in which they created a positive learning environment for the children during the IHC lessons, including use of role playing, using relevant examples and allowing children to express their opinions.

Teachers’ beliefs

Many teachers had beliefs that were in conflict with some of the examples and sometimes directly in conflict with a key concept, particularly the concept that widely used treatments or treatments that have been used for a long time are not necessarily effective or safe. Although many teachers expressed concerns about it being difficult to reconcile their beliefs with the IHC content, the quantitative data suggest that this may not have had an important impact on their ability to apply the concept or teach it.

Children’s beliefs

Children were less likely to identify conflicts between their beliefs and the IHC lessons than the teachers were. However, the children we interviewed seemed to struggle with the key concept that personal experiences or anecdotes (stories) are an unreliable basis for assessing the effects of most treatments. Nonetheless, 30% more children in the intervention schools answered both questions on this concept correctly compared with children in the control schools.15

Intermediate effects

Teachers’ motivation

The majority of the teachers felt motivated while teaching the IHC lessons because they felt that the content was important for the children and for themselves. Others felt that the way the IHC programme was introduced to them at the introductory workshop motivated them. Others felt that the support provided by the IHC team and their head teachers motivated them. This included provision of adequate materials, carrying out observational visits, progress calls and encouragement.

Teachers’ self-efficacy

While the content was new for most teachers, this affected their confidence differently. Some felt that their professional training equipped them to teach new material. Many teachers also noted that the children’s positive response and enthusiasm about the lessons made them more confident. However, others initially lacked confidence. A few teachers felt uncertain as a result of teaching the lessons, particularly when they felt put on the spot by children asking them questions. Nevertheless, by the end of the term, 95% of the teachers strongly agreed or agreed that they were confident about their ability to teach the lessons, and 94% strongly agreed or agreed that they liked teaching something new. In the lesson assessments, most teachers indicated that it was easy for them to teach the lessons and that they were comfortable teaching the lessons.

Teacher’s attitudes

Teachers mostly had positive attitudes towards the IHC lessons, particularly in relation to the content being new and valuable to them as well as to the children. Several of them vividly expressed how they felt during the lessons. One noted: ‘It was something I can’t even describe, being a new idea brought to me, I loved it so much. I was teaching learners, as I was also teaching myself.’

There was strong support for this from the quantitative data: 98% liked very much or liked teaching the lessons; 99% thought that what the children learnt was very important or important; 94% thought that the children learnt very much or much from the lessons and 98% strongly agreed or agreed that it is important to teach children to think critically.

Children’s motivation to learn

The children that attended the IHC lessons were motivated. They enjoyed the lessons and looked forward to them. They liked both the design and the content of the book, including the pictures, the characters, the games and being able to colour in the pictures in their exercise books.

Most of the children indicated that they experienced the IHC lessons positively. They found the lessons enjoyable and the books interesting. Teachers noted that children were more active during the IHC lessons compared with their normal lessons, that they were enthusiastic, and that they were eager to attend. The majority of the parents also mentioned that their children enjoyed the IHC lessons, noting that they were reading their books and sharing what they had learnt with people at home. This is consistent with the trial results. When asked ‘How much did you like what you learned as part of the lessons with The Health Choices Book?’, 78% of children in the intervention group responded that they ‘liked the lessons very much’ and another 16% responded that they ‘liked the lessons’.

Time constraints

Although the majority of the teachers were able to complete all the nine IHC lessons, this was not always to their satisfaction due to other competing priorities. The quantitative data from the lesson evaluation forms showed that the amount of time different teachers used preparing and teaching lessons was similar across lessons, but varied among teachers. Most teachers reported using between 5 and 30 min preparing for each lesson and between 45 and 90 min teaching each lesson. Most teachers felt that they spent close to the right amount of time preparing and teaching each lesson, but some felt that they spent too much or too little time preparing and teaching each of the lessons. There was little correlation between the amount of time a teacher used preparing and teaching each lesson and whether they thought that was too much or too little time (see online supplementary file 3).

Attendance

Teachers pointed out that absenteeism was a common problem. The children whose parents had paid the school fees on time said that they attended all the lessons. In the trial, 10% of the children in the intervention schools and 29% in the control schools did not complete the test. To a large extent, absenteeism was attributed to parents’ failure to pay their children’s school fees in time.

Scaling up

Parent and community involvement

Teachers, parents and district education authorities emphasised the importance of involving other stakeholders, including parents and the communities at large, for the lessons to be effective.

Collaboration with policy-makers

Several teachers emphasised the importance of working in collaboration with the Ministry of Education and the National Curriculum Development Centre to ensure the IHC lessons are incorporated into the primary school curriculum. The majority of the teachers, parents and children interviewed supported spreading the IHC programme to other schools and other age groups, including infant, middle and upper primary sections.

Potential beneficial and adverse effects

Beneficial effects

Nearly everyone interviewed thought that both the children and teachers learnt important lessons from the IHC resources and many felt that the lessons improved their decision-making. Several parents observed that children gained confidence and started asking important questions about the benefits and harms of treatments before decisions were made. Teachers also noted impacts on their own learning and decision-making. A teacher gave this example: ‘Somebody came to school and asked us to buy some food supplements. Then I asked myself: ‘Are these foods really well researched?’ In fact, I did not buy (them).’

Some teachers and children also noted beneficial impacts on English and numeracy skills.

Adverse effects

Although teachers found the IHC lessons enjoyable, some reported having experienced stress because of teaching something new and it being additional to their usual subjects.

The majority of teachers and parents expressed concern about the potential conflicts between themselves and the children resulting from children sometimes challenging their authority such as asking questions or refusing to take instructions from those in authority. However, there were no reports of actual conflicts.

Discussion

Facilitating factors

The findings of this process evaluation are consistent with the findings of the trial, in that most of the factors that were identified were facilitators rather than barriers to the implementation of the intervention (figure 2). The findings suggest that children, teachers and parents appreciated the IHC school intervention. Teachers found the IHC lessons compatible with the curriculum and their teaching styles; and the materials enabled teachers to apply differentiated instruction. Effect modifiers included teachers’ skills and competencies and positive learning environments. These contributed to intermediate effects, including teachers’ motivation, self-efficacy, positive attitudes and a positive overall experience, which in turn contributed to the IHC resources having a large effect on the ability of the children to assess claims about treatment effects.

Children, teachers, parents, head teachers and district education officers valued the IHC primary school resources and their content. We believe this is due, in large part, to the fact that we spent 3 years developing them using a human-centred design approach.11

Through this approach we tried out many ideas, developed some into prototypes and gradually improved these in cycles, based on extensive feedback from users and through observation of use in classrooms. This enabled us to identify and resolve problems early on, leading to solutions that people valued, for example, use of a comic book format, creating characters that appealed to the children, adding local language vocabulary, building in activities teachers could carry out in large classes without extra materials and adding examples that were familiar in the East African context.

The IHC primary school resources were designed to support teachers who were not familiar with the content, and the workshop was designed to introduce the project in general (goals and expectations), to answer questions and to offer clarifications. It was not designed to teach the content to the teachers. Although some teachers were concerned about the duration of the training workshop, most of the teachers in the intervention group mastered the IHC key concepts after teaching them. This is likely attributable, at least in part, to their having learnt the IHC key concepts as they taught the children. This suggests that, although some teachers perceived the need for a longer workshop, this might not be necessary either for them to master the IHC key concepts or for them to teach them effectively. It might be helpful to inform teachers about this and to reassure them.

Support from the school authorities and from colleagues played an important role in ensuring that the teachers were allowed adequate time to teach the IHC lessons. This likely contributed to the effectiveness of the intervention in the trial.14 Effectively scaling up use of the IHC lessons will undoubtedly require the support of school authorities to ensure that teachers have sufficient time.

The children enjoyed the lessons and looked forward to them. Their positive attitudes towards the materials and the lessons likely played an important role in the effect that the intervention had on their ability to assess treatment claims. Many teachers noted that the children’s positive response and enthusiasm about the lessons made them more confident.

The IHC materials facilitated switching from English to local languages. This made it possible for children to ask questions in the language with which they were most comfortable. This is not common for subjects in the normal upper primary school curriculum in Uganda. Children clearly stated that they generally appreciated being able to use more than one language during the IHC lessons. Several studies have noted the benefits accrued over time on functional literacy (reading and writing) when children are able to study in their mother tongue.28–30

Impeding factors

Many teachers identified conflicts between their beliefs and personal practices and the IHC content. These were particularly in relation to herbal remedies and the concept that widely used treatments or those that have been in use for a long time are not necessarily beneficial or safe. Paying particular attention to examples and IHC key concepts about which the teachers may have conflicting beliefs might help to address this. For example, it might help to acknowledge in the Teachers’ Guide27 and the training workshop that teachers and others commonly believe that treatments are effective when, in fact, their effects are uncertain or there is evidence to the contrary. Open discussion about disagreements, the logic and the evidence underlying relevant IHC key concepts; and use of compelling examples that do not directly challenge the teachers’ beliefs might be helpful. For example, ineffective interventions that were widely used for decades or centuries, which are no longer used, such as bloodletting, could be used as examples.

Children’s beliefs may be less resistant to change than adults’ beliefs,31 which may be reflected in the fact that children were less likely to identify conflicts between their beliefs and the IHC lessons than the teachers were. On the other hand, some children struggled with the key concept that personal experiences or anecdotes are an unreliable basis for assessing the effects of most treatments. It also might be helpful for this key concept to provide teachers and children with a variety of examples, including some that are less likely to challenge their prior beliefs about the effects of specific treatments.

Absenteeism, likely attributable to a large extent to the parent’s failure to pay tuition fees, was another barrier identified by the teachers and the lead investigators. This is a systemic problem, not specific to the IHC lessons.32 Another important barrier, which was not identified in the process evaluation, but is undoubtedly the biggest challenge, is the cost of the intervention.14

In the context of our trial, we were not able to remove anything from teachers’ already heavy workloads. If the IHC lessons were incorporated into the curriculum, rather than taught as an add-on, as in the trial, that might reduce the burden on teachers and increase the effectiveness of the intervention. A second reason why the intervention might be even more effective when scaled up, would be that teachers would not be teaching it for the first time after the first year and the material would no longer be new to them. On the other hand, they would not have support from the research team. Some teachers suggested that this might have facilitated implementation of the intervention, even though the research team only observed lessons when they visited schools and did not provide feedback or help. In addition, teachers indicated that the training workshop, which was taught by the two principal investigators was important. It is uncertain whether it would be feasible to offer a comparable workshop to all of the teachers in the country who would be responsible for teaching the IHC lessons.

Potential effects

The children shared what they learnt with their families. Although having a child in a school that used the primary school resources had little, if any, short-term effect on the parents’ test scores for parents who participated in the podcast trial,16 after 1 year the mean score of parents with a child in an intervention school was 4% higher than that of parents with a child in a control school and 12% more parents had a passing score.33 This finding is promising in terms of the potential for the intervention to benefit families of the children and not just the children themselves.

Other potential benefits that were identified, besides the value of what was learnt, include improved decision-making by teachers, as well as by children, improvements in English and numeracy, and improved relationships between children and adults.

The most common concern was the risk of conflict between children and authorities, including teachers, parents and healthcare professionals. The concern was that children questioning claims made by authorities might be viewed as disrespectful and undermining of their authority. Undermining of religious and cultural beliefs was also identified as a potential adverse effect. There were incidents of children challenging authorities, including teachers, parents, head teachers and members of the research team. However, none of the participants in the trial or the process evaluation reported actual conflicts and none were observed.

Strength and limitations

An important limitation of this study is that the investigators were responsible for both developing and evaluating the intervention. This could have led us to emphasise participants’ positive experiences of the intervention when designing the process evaluation and when collecting and analysing the data. In addition, as most respondents were aware that the lead investigators were responsible for the intervention itself, there may have been a desire to please the investigators by giving positive reports.34 Although we tried to address this by making it clear to the respondents that the IHC materials were being tested, not them, there may have been some desirability bias.35

An important strength of the study is our use of data collection triangulation. We gathered data using a variety of methods, and explored whether the findings from different sources challenged or supported each other. In addition, based on a modified CERQual approach, we have high confidence in most of our findings. Exceptions are our findings regarding children’s beliefs, time constraints, incentives and disincentives (see online supplementary file 2).

Conclusions

The extent to which children, teachers, parents, head teachers and district education officers valued the IHC primary school resources played a key role in facilitating the success of our intervention. The key barrier that we identified was that the IHC lessons were an add-on, rather than being incorporated in the national curriculum.