Evaluating an oral health programme for the prevention of dental decay among school-aged children in China: protocol for a mixed-methods study based on the RE-AIM framework.

INTRODUCTION: The objective of the Comprehensive Intervention of Oral Disease for Children (CIODC) in China is to prevent dental decay for school-aged children and provide free prevention services in pilot areas beginning in 2008. It is a potentially affordable, acceptable and effective prevention strategy to use for more school-aged children in the future. There is a shortage of robust evidence regarding the cost-effectiveness, feasibility and scalability of prevention strategies for dental decay for school-aged children in China. This study aims to provide a comprehensive evaluation, including an economic evaluation and process evaluation, to better understand how and why the public health programme may be effective and economical. METHODS AND ANALYSIS: Mixed methods will be used in this study. Cost-effectiveness analysis (CEA) will be conducted from a societal perspective, based on a modelling study over 6 years (from age 7 to 12) in terms of the incremental cost-effectiveness ratios per dental decay averted. The Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework informed the process evaluation. An estimated 48-80 semistructured interviews with service providers, patient parents/caregivers and decision-makers under the logic model will be used in the progress evaluation to describe the feasibility and sustainability of CIODC. ETHICS AND DISSEMINATION: The study has all necessary ethical approvals from the Ethics Committee of Anhui Medical University (number 2021H030). All participants will provide informed consent prior to participation. Findings will be disseminated through conference presentations and scientific publications in peer-reviewed journals. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

A strength of our study includes the use of implementation theories and quantitative and qualitative research methods for analysis.

The design of the process evaluation will provide insights into how the implementation of such a programme can be expanded in practice and brought to scale.

Limitations to our current approach include the process evaluation being based on Anhui province data, while the economic analysis is based on national data for China.

Our sample for the interviews, although large for qualitative research, is a small subset of the participants in the prevention programme.

Introduction

Burden and prevention of dental decay in children

Dental decay is one of the most challenging diseases faced by children globally, with a high prevalence and inadequate information regarding the awareness and prevention among children.1The Global Burden of Disease Study 2016 estimated that 486 million children worldwide suffer from caries of primary teeth.2 China is no exception, and the annual incidence of dental decay in children aged 5 years and 12 years was 70.9% and 34.5% in 2017, respectively.3 However, the treatment proportions of their dental decay were only 4.1% and 16.5%, respectively.3 The short-term consequences of untreated caries include pain, hospitalisation and emergency visits due to abscess and systemic infection and even death.4 In the long term, children who have experienced dental decay are much more likely to have diminished oral health and higher risk of caries lesions in permanent teeth.5 As previously reported, the ratio of indirect economic burden (such as the loss of productivity) to direct economic burden (such as medical treatment cost) was 1:1.6 This implies a significant burden on society due to limited awareness of prevention and inadequate medical treatment.

The pilot programme of the Comprehensive Intervention of Oral Disease for Children (CIODC) in central and western China aims to prevent dental decay and provide free prevention services for children aged 7–9 years old. It was conducted in terms of counties (or districts in urban) in 2008 among 22 of 34 provinces (autonomous regions and municipalities) and then gradually broadened yearly among pilot counties (districts) in provinces where possible.7 Thus far, in 2021, the programme is practised in certain pilot districts/counties in all 34 provinces (autonomous regions and municipalities). CIODC programme is funded by the central and local government and is organised and managed by the local Center for Disease Control and Prevention (CDCP). The provincial government chooses the intervention counties (districts) in the pilot area, and then the intervention elementary schools are chosen from the pilot counties (districts) according to certain service workloads (eg, numbers of teeth with pits and fissures and numbers of children with dental decay). Subjects aged 7–9 years in these schools were assigned to the intervention groups. In general, the intervention groups received oral health education, oral check-ups and professional preventative fissure sealants. In contrast, the other children not covered by CIODC only received usual care without any intervention or oral health check services for dental decay unless they actively sought healthcare. Before CIODC intervention, the community staff and elementary teachers will receive yearly professional training in oral health education.7 CIODC flowchart is shown in figure 1. In 2020, a total of 3000 children received pit and fissure service; the coverage proportion was approximately 22%.8 According to the government’s development planning, CIODC programme should keep expanding coverage, with the coverage proportion intended to be 28% in 2025.8

Figure 1

CIODC flowchart. CIODC, Comprehensive Intervention of Oral Disease for Children.

Evaluation of the public programme to improve oral health for children

Dental sealants have been confirmed as efficient treatments for preventing dental caries in children,4 5 and CIODC programme in China has been shown to reduce the incidence of dental caries by approximately one-third.9–11 However, there have been few health economic analyses of dental care, which makes it difficult to assess the cost-effectiveness of interventions.12 13 Furthermore, there is a shortage of robust evidence on CIODC regarding the feasibility and scalability of the prevention strategy for dental decay for school-aged children in China.14 To make evidence-based decisions about the effective and efficient allocation of scarce health funding and resources, decision-makers require information about the potential cost-effectiveness and the feasible implementation approach to determine if and how the intervention is effective and efficient in the longer term.15 In addition, for evidence-based complex interventions such as CIODC, implementation research to understand how it can be effectively scaled-up to more school-aged children is important. This study aims to evaluate the economic effectiveness of CIODC and identify a feasible and acceptable approach to expand implementation, for preventing dental decay in school-aged children in China.

Aim and objectives

Based on the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework and logic model, the current study aims to conduct a process evaluation, including an economic evaluation of the implementation of CIODC programme in China. The economic evaluation aims to measure the cost-effectiveness of CIODC. The process evaluation also seeks to explore reasons for any variations in clinical effectiveness and address questions about the generalisability of CIODC across all children aged 7–9 years old if it is found to be economical and effective.

The findings will inform decision-makers on how to optimise the delivery of prevention services for CIODC and the programme’s value. The fundamental role of the process evaluation is to demonstrate how the programme can be implemented into practice and policy if proven effective. A deep appreciation of the needs of children and their families will be valuable for the health system and policy reform in China. The core aims of this evaluation are to demonstrate:

Identify the costs and health outcomes of CIODC, including numbers of tooth dental decay prevented and disability-adjusted life years (DALYs) averted during the 6 years (age 7–12 years).

Estimate the cost-effectiveness and the incremental cost-effectiveness ratios per dental decay averted and DALYs per child averted of the intervention group, as compared with the usual care group.

Calculate the budget impact on public finance expenditure when the coverage of CIODC expands gradually to cover all children aged 7–9 years old.

To explore if CIODC was implemented as designed.

To understand whether, how and why CIODC had an impact through exploring the providers’ and parents’/caregivers’ perspectives of dental decay prevention.

To explore if CIODC is generalisable, scalable and sustainable through exploring health providers’, parents’/caregivers’ and decision-makers’ experiences of CIODC and its perceived impact.

Methods and analysis

Mixed methods evaluation will be used in this study. The economic evaluation will be conducted using the cost-effectiveness analysis. The RE-AIM framework informed the process evaluation. Semistructured interviews under the logic model will be used in the progress evaluation to describe the feasibility and sustainability of CIODC. The methods and data sources are shown in table 1.

Table 1

Methods and data sources

Domains	Data sources	Index	Output
Reach	Published report and database of Anhui CIODC Programme	Percentage of children covered by CIODC Number of teeth covered by CIODC, person times of oral health education and check-ups in Anhui per year	Coverage of CIODC of Anhui case Measuring short-term output for patients
Effectiveness and cost- effectiveness	The published literature review on a national level Published reports of health outcome follow-up surveys of CIODC on a national level Published report of the Global Burden of Disease study	The implementation costs of CIODC and the economic burden and loss of working time due to dental decay on family and societal level The morbidity and treatment percentage of child aged 7–12 Health outcomes of intervention children compared with usual care children, including the numbers of decayed teeth prevented and DALYs pre-child averted The disability weights of dental decay	Identify the costs and health outcomes of CIODC Evaluation the cost-effectiveness and ICERs of CIODC Evaluating the budget impact of scaling CIODC in Anhui
Adoption	Qualitative interviews of Anhui case	Reports of fidelity and satisfaction and suggestions of key stakeholders (actual providers of intervention services, parents/caregivers)	Measuring short-term output for actual service providers Explore if CIODC was implemented as designed
Implementation	Qualitative interviews of Anhui case Programme documentation	Reports of implementation barriers for providers Inputs of CIODC to operate for key stakeholders, including funding, work times, knowledge, skills and expertise, and resources	Inform how and why CIODC had an impact Identify the implementation barriers
Maintenance	Documentation of Anhui case Qualitative interviews in Anhui	Retrospectively from 2008, to assess the growth of the programme Willingness to provide expand services of providers Willingness-to-pay of parent/caregiver to prevent dental decay	Determining the feasibility and strategies of scaling in Anhui (including the economic feasibility)

CIODC, Comprehensive Intervention of Oral Disease for Children; DALYs, disability-adjusted life years; ICERs, incremental cost-effectiveness ratios.

Modelling the cost-effectiveness of CIODC based on quantitative data

A CEA will estimate the total and incremental costs of CIODC from a societal perspective, compared with the status quo, that is, children with usual care services. The costs in this study include the intervention costs and the economic burdens due to dental decay. The costs of the implementation, regular oral health checks and health education will also be calculated as part of the total intervention costs and will be assigned only to the intervention subjects. Additionally, we will calculate the human costs of providing intervention services for community and school workers and dentists, given the consideration that providing these services would increase their workload. The dentist provides pit and fissure sealing services and regular oral health checks, while community and school workers contribute to delivering oral health education and help to the oral health check services. The medical costs of intervention services will be calculated based on the market price, such as the cost of sealer. The average implementation cost of CIODC will be collected from the published literature.

The economic burden of dental decay in this study was defined as the direct costs and indirect costs due to dental decay on social and patient perspective. Household costs include costs of treatment services related to dental decay and expenditures on food and traffic for visiting dentists, which are estimated based on the hospital fee records and published literature. Given that parents/caregivers take time off work to visit the dentist with their children due to dental decay, we estimated the product loss of households associated with dental decay using the loss worktime×annual gross domestic product per capita.

Considering that the preventive effect of CIODC lasts for years, and the eruption time of permanent molars is 12 years of age, the follow-up period of outcomes in this study is defined as 6 years (ages 7–12). The primary outcome will be the numbers of decayed teeth prevented; the secondary outcome will be DALYs per child averted. The primary outcome data of CIODC will be estimated based on the literature on a national scale, and disability weights of caries will be based on the Global Burden of Disease study in 2013 estimates of dental decay disability, leading to less than perfect health.16

The CEA will be presented in terms of incremental cost-effectiveness ratios (ICERs), calculated as the total cost between CIODC individuals versus the individuals with usual care divided by the cumulative effect. ICERs will be calculated for the numbers of dental decay prevented and DALYs averted.

Costs will be presented in 2022 prices in China RMB. All costs will be adjusted for inflation using the China Consumer Price Index. Moreover, costs and outcomes will be converted to present values using an annual discount rate of 5% in the base case17 and annual rates of 0% and 8% in the sensitivity analysis. Several sensitivity analyses for the other vital parameters that may affect the result of the economic evaluation will be carried out to assess the impact on the cost-effectiveness results, including the clinical effectiveness, probability of dental decay for school-aged children and the medical costs of dental decay.

Furthermore, budget impact analysis will be conducted to simulate the total financial budget when the public programmes cover all children aged 7–9 years in Anhui province and China, using the total number of children multiplied by the average cost of CIODC per child. Consequently, these analyses will enable a framework for resource allocation to CIODC to reduce the oral disease burden for the child and their family.

The affordability of CIODC will be explored using a range of criteria, including the WHO-CHOICE standards, together with an analysis of the fiscal space for programme delivery using a health economic evaluation method and probability analyses to determine a set of cost-effectiveness thresholds.18 CEA was performed with TreeAge Pro Healthcare 2021 (TreeAge Software, Williamstown, Massachusetts).

The RE-AIM framework for the process evaluation

The RE-AIM framework is adopted to evaluate the programme process,19 and table 1 describes the data sources and the analysis indexes. The RE-AIM framework is a method of examining the relationships between the context, mechanisms and outcome of a programme. Researchers have used it to develop, evaluate and disseminate research findings using qualitative and quantitative methods.20–22 The framework’s five domains reveal the proportion and representativeness of the participants involved in programmes, the impact and acceptance of programmes and how the programmes work. As a result, the RE-AIM framework helps understand the efficacy of programmes implemented in real-world community settings.20 The context of each child with dental decay can impact the oral health behaviour of their patients and caregivers, which will affect improvement in the child in terms of quality of life. Thus, the process evaluation could help explore reasons for any variations in programme effectiveness and address questions about the transferability of this CIODC for expanding gradually to all school-aged children.

Logic model

A logic model will be added to the process evaluation.23 24 The purpose of a logic model is to articulate the underlying assumptions about how the expected outcomes of a programme will be reached in the short, medium and long term.23 25 Therefore, a logic model is a valuable tool to guide the evaluation, including its inputs, activities and outcomes.26Our logic model will be based on initial semistructured interviews with key stakeholders. Clarification with key stakeholders provides the basis for developing an evaluation framework (including process, outcomes and impact) that aligns the key evaluation questions, indicators and available data.

The logic model (see figure 2) delivers visual representation of the underlying rationale or logic of CIODC. It helps to understand how CIODC worked and informed the design of CIODC framework, systematic data collection, data analysis and reporting. The process of developing the logic model provides a structured way to explore the complexity of what CIODC offered to those involved and enabled the research team to work collaboratively with key stakeholders to document a growing understanding of how the model was being implemented.

Figure 2

Logic model of CIODC. CIODC, Comprehensive Intervention of Oral Disease for Children.

The inputs in the model will identify all the significant resources that are necessary for CIODC to operate. There are five types of information included in the model: funding, work time of community or school workers and clinical dentists, CIODC knowledge and resources, clinic prevention skills and expertise and relationships with local government. The model identifies nine key activities and involves five types of participants and uses colour to highlight the logical outcomes of CIODC activities corresponding to each participant group.

The development of a logic model does not assume that a programme is static or unchanging. A logic model may be reviewed after an evaluation is completed to ensure that the programme learns from the evaluation findings and adapts appropriately. The logic model in the overall RE-AIM framework is used to highlight the research impact, serve as a template to consolidate the findings and be a dynamic structure with change to be made if required, with the RE-AIM framework. This implies that our understanding of the causal mechanisms of CIODC may change with the iterative analysis of the process evaluation data. There is also a post hoc examination of the process evaluation findings in light of the main results of the evaluation to help inform the final logic model for how CIODC truly impacted the efficacy outcomes and inform the generalisability of CIODC.

Process evaluation of CIODC based on the qualitative and quantitative data

The process evaluation was conducted using an Anhui case study for the following reasons. First, CIODC is a public programme carried out with a unified framework among pilot areas; thus, a case study is a feasible method to obtain the necessary information about the implementation process. Second, Anhui was one of the first pilot provinces conducting CIODC (carried out since 2008), with a middle-income level in China (the annual per capita gross domestic product for Anhui in 2020, ranking 13 among 34 provinces, municipalities and autonomous regions). As a case study, Anhui province may serve as a reference point.

Reach data from CIODC documents of Anhui province can be achieved through an analysis of the programme process (eg, numbers of teeth treated with pit sealant and numbers of children receiving fluoride treatment) and the total human resource input (eg, staff numbers of community or pilot schools involved in oral health education and oral health check, working hours of regular oral health checks per child, frequencies of the oral health checks per year and hours per oral health education and training session and frequencies per year).

Semistructured in-depth interviews based on a logic model will be conducted among key stakeholders, including parents/caregivers, community workers, primary school workers, clinical dentists and decision-makers to understand their CIODC prevention and oral healthcare perspectives. This information will be used to seek adoption, implementation and suggestions on how CIODC strategy could be expanded on a large scale. Methods for our interviews are described as per the consolidated criteria for qualitative research. Semistructured interview guidelines based on our study objectives will be developed and tested. The critical areas covered include overall views of the health and socioeconomic impact of the decayed tooth, children’s parents/caregivers, healthcare service providers and decision-makers. Early findings will be discussed with the study team, and minor changes made to the interview guidelines if needed (see figure 2).

Participants for the study will be recruited using maximum variation purposive sampling. A multistage sampling strategy will be used to obtain samples. First, four cities (counties) from different geographic locations in Anhui Province will be selected purposively. Second, 1–2 communities from urban and rural areas will be selected in each city (county). Third, a list of two to four health providers, two decision-makers and about two to four parents/caregivers whose child is aged 7–9 years old will be chosen in each community. The local site staff will contact them, and where relevant, reasons for not participating will be elicited. The interviewer face-to-face will formally consent them. Interviews with patients or caregivers will be conducted either individually or together, either at a community or school site.

The benefits of the conversations with parents/caregivers separately from the other interviewees would be to more comprehensively explore their reasons for participating in the project. Health service providers will be invited to participate in interviews in the local dental clinics or hospitals. The interviews with community and school workers will be conducted in their workplace, and the decision-maker’s interview should be carried out separately in their office. Written informed consent will be obtained from all interviewed participants. As per qualitative research methods, the interviewer will carry out preliminary thematic data analysis at the end of each interview and discuss any highlight results at each interview group. According to the sampling matrix, we will interview equal numbers from both the usual care and CIODC groups. That is, an estimated 48–80 interviews will be conducted, though the final number of interviewees will be determined by the saturation of themes and resources permitting. Interviews will be conducted face-to-face, audio-recorded, professionally translated and transcribed verbatim. These interviews will be conducted in Mandarin and then translated into English from September 2021 to October 2022.

The researchers are multidisciplinary and composed of programme investigators, dentists and research fellows from China and Australia. The team has a background in stomatology, health economics and public health, with varied experience in qualitative research. Team members who have a good understanding of CIODC and are known to the principal investigators, dental decay but will not have established relationships with the patients or the child’s caregivers suffering from dental decay. The interviews will be uploaded into a software programme NVivo V.9 (QSR International, Melbourne, Victoria, Australia) to assist with data management.

Preparation of future implementation based on interview data

In the interview guides, questions based on the RE-AIM framework were included (table 1). Stakeholder interviews will be used to collect the responses of the parents/caregivers, dental service providers, community and school workers and decision-makers as the stakeholders. The questions are designed to revolve around five domains of the RE-AIM framework. Taking the parents as an example of a stakeholder, the critical factors for the unwillingness to attend the preventive services of dental decay will be identified, followed by the expected and acceptable service approach. That is, when and where the preventive services will be provided, and how much the parents/caregivers should pay out-of-pocket for these services.

A crucial role of the process evaluation is to inform how CIODC can be implemented to practice and policy if proven effective. It is well recognised that the generation of good quality evidence does not always translate into improved child oral health outcomes.10 Financial carries, such as high out-of-pocket costs for caries preventive services for households, or an increased budget for the local government, are possible reasons for CIODC not being delivered once shown to be efficacious. Understanding these cost barriers can inform how remuneration and payment systems may be shaped to facilitate implementation beyond existing sites. Incorporating an assessment of stakeholders’ perception of how CIODC can be practically funded, delivered and broadened is, therefore crucial.

Thematic analysis will be used for the qualitative study and to establish themes within the subheadings of the process evaluation framework (figure 2). Constant iterative comparisons between sources, for example, parents/caregivers and preventive service providers, will be carried out to identify common and distinctive themes. Contextual information from other process data and costs to parents will be used to triangulate the emerging themes. The qualitative process data will provide descriptive data on the approach to provide CIODC. The examination of such contextual findings from the process evaluation is an example of how we could gain a deeper understanding of the assumed causal mechanisms of CIODC (as depicted in the logic model in the overall process evaluation framework). Such insights will help inform the final logic model of how the preventive intervention truly impacts the clinical effectiveness and oral health outcomes and inform the generalisability of the preventive project of child oral health.

Patient and public involvement

Children as patient, the parents/caregivers of children, together with other stakeholders, will be involved in the process evaluation stage, including the interview on the implementation and expansion of CIODC and dissemination of the findings. The dental doctors, community workers and CDCP are implementers of CIODC in pilot areas, and these are the stakeholders in this study. Patients and the public will not be involved in the development of the research question or in the design of the study. The study is designed to collect the patients and public’ opinions of CIODC, and surveys are addressed to them. Patients and stakeholders will receive oral and written information about CIODC, and then the burden of the interview will be assessed by patients and stakeholders themselves. After signing an informed consent form, participants will be assessed for eligibility, and data collection will begin.

Ethics and dissemination

The protocol has all necessary ethical approvals from the Ethics Committee of Anhui Medical University (number 2021H030). All participants will provide informed consent prior to participation. The study will comply with the data protection act names and other identifying details will NOT be included in any study data electronic file. The findings from this study will hold value not only academically but also to key stakeholder groups including parents/caregivers, community workers, dental service providers and decision-makers as well as regulators of social care and the broader public health landscape. The findings will be disseminated within academia and the wider policy sphere, through conference presentations and scientific publications in peer-reviewed journals.

Study status

This study was affected by COVID-19. The quantitative data, including the cost and effectiveness, were collected in 2020, and the health economic evaluation has been preliminarily analysed. The process evaluation started in September 2021. Pilot interviews were conducted with parents/caregivers in September 2021; completion of the community or school worker and health provider interviews was planned for April 2022, and completion of the decision-maker interviews is expected by October 2022, with ongoing preliminary and iterative analyses.