Protocol for process evaluation of SMART Mental Health cluster randomised control trial: an intervention for management of common mental disorders in India.

INTRODUCTION: In India about 95% of individuals who need treatment for common mental disorders like depression, stress and anxiety and substance use are unable to access care. Stigma associated with help seeking and lack of trained mental health professionals are important barriers in accessing mental healthcare. Systematic Medical Appraisal, Referral and Treatment (SMART) Mental Health integrates a community-level stigma reduction campaign and task sharing with the help of a mobile-enabled electronic decision support system (EDSS)-to reduce psychiatric morbidity due to stress, depression and self-harm in high-risk individuals. This paper presents and discusses the protocol for process evaluation of SMART Mental Health. METHODS AND ANALYSIS: The process evaluation will use mixed quantitative and qualitative methods to evaluate implementation fidelity and identify facilitators of and barriers to implementation of the intervention. Case studies of six intervention and two control clusters will be used. Quantitative data sources will include usage analytics extracted from the mHealth platform for the trial. Qualitative data sources will include focus group discussions and interviews with recruited participants, primary health centre doctors, community health workers (Accredited Social Health Activits) who participated in the project and local community leaders. The design and analysis will be guided by Medical Research Council framework for process evaluations, the Reach, Effectiveness, Adoption, Implementation and Maintenance (RE-AIM) framework, and the normalisation process theory. ETHICS AND DISSEMINATION: The study has been approved by the ethics committee of the George Institute for Global Health, India and the Institutional Ethics Committee, All India Institute of Medical Sciences (AIIMS), New Delhi. Findings of the study will be disseminated through peer-reviewed publications, stakeholder meetings, digital and social media platforms. TRIAL REGISTRATION NUMBER: CTRI/2018/08/015355. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

A strength of our study is its use of implementation science theories and guidelines for process evaluation to frame the study design.

This study combines data from an open-source medical record system with qualitative methods to understand trends, patterns and differences in outcome.

One limitation could be the overlap between the implementation team and the evaluation team.

Introduction

India has a significant burden of mental disorders with an estimated 115 million people in need of mental healthcare.1 The National Mental Health Survey of India (2015–2016) found substance use, depressive disorders and anxiety disorders to be prevalent in about 10% of the population.1 Despite the significant burden, access to mental health services is severely limited and it is estimated that nearly 95% of individuals with common mental disorders (CMDs) are unable to access care in India2 leading to large treatment gaps. Studies report that in low-income and middle-income countries, the treatment gap for any mental disorder is between 75% and 85%.3 One study found that in low-resource settings such as India, only 1 in every 27 individuals with depression who recognised need for treatment, could access minimally adequate treatment from a trained mental health professional.4

This large treatment gap is due to several factors, on both demand and supply sides. Low awareness about mental health in the community and high level of stigma related to mental illness are key demand side factors for poor help-seeking for CMDs.5 On the supply side, several systemic barriers limit access to mental health services. Among these are the lack of a trained mental health workforce and absent/minimal mental health services at the primary care level, inadequate supply of psychotropic drugs at primary healthcare facilities and limited budget for mental healthcare.6

Our formative research has demonstrated that addressing both supply and demand side factors by conducting a community-based anti-stigma campaign and implementing a technology-enabled mental health services delivery model by primary health workers, has the potential to increase access to mental healthcare for those at risk of CMDs and reduction in depression and anxiety scores.7–10 In this research, task sharing by primary health workers helped facilitate the process, and technology was seen as an enabling factor in streamlining delivery of mental healthcare.10

Based on these findings, we developed Systematic Medical Appraisal, Referral and Treatment (SMART) Mental Health—a hybrid effectiveness-implementation cluster randomised controlled trial (cRCT) that is being implemented in two Indian states. The cRCT protocol is available elsewhere.11 The goal of SMART Mental Health is to reduce psychiatric morbidity due to psychological stress, depression and risk of self-harm (collectively referred to here as CMDs for the project) in individuals identified at high risk of these conditions. The coprimary outcomes are:

(1) The mean difference in Patient Health Questionnaire-9 (PHQ-9) scores at 12 months in people identified at high-risk of CMDs and (2) the difference in mean behaviour scores at 12 months in the total population.

In this paper, we outline the protocol for a process evaluation of the SMART Mental Health. Process evaluations provide important insights into how an intervention is implemented, leading to understanding what strategies either worked or did not work, explaining differences in outcome, and to gain insights into the experience of the target population for whom the intervention was designed. The aims of the process evaluation are to:

Assess implementation fidelity and understand how the intervention was implemented.

Understand perceptions about effectiveness and acceptability of intervention components by different stakeholders.

Identify and explain facilitators of and barriers to implementation of the intervention.

Explain variations in outcomes and unexpected consequences across sites.

Explain any adaptations to the intervention during the study and their possible impact on the outcomes.

Methods and analysis

Theoretical framework

The process evaluation has been integrated into the cRCT design with an early formative study conducted to understand the feasibility of implementing the project components. It draws on multiple theories and frameworks (table 1). The Medical Research Council guidelines for process evaluation will provide an overall conceptual framework.12 According to this framework, the three broad areas of enquiry in a process evaluation are ‘ implementation’ (what is implemented and how); ‘mechanism of impact’—(how intervention produces change) and ‘context’—(how context affects implementation and outcomes). The framework also emphasises the need to spell out the key causal assumptions made in the programme theory.

Table 1

Theories to be used in the study

Theory	About theory	Purpose of using the theory
Theory guiding overall design and conceptual framework of the process evaluation
MRC Framework23	A framework for designing and carrying out process evaluation of complex interventions. Process evaluation should answer questions related to three components: Implementation (what is delivered and how?) Mechanisms of impact (how does the delivered intervention produce change?) and Context (how does context affect implementation and outcomes?) Along with the context and the mechanism of impact, it emphasises the need to spell out the key causal assumptions or the programme theory.	The framework is used to provide the overall conceptual design of the process evaluation. The three components (implementation, mechanism of impact and context) will be the broad areas of inquiry in the process evaluation.
Theories that will inform specific domains of inquiry in the study
RE-AIM13	A framework which provides five key dimensions on which a behaviour change intervention can be evaluated. These include Reach, Effectiveness, Adoption, Implementation and Maintenance of an intervention.	The framework will be used to evaluate the ‘Implementation’ component of the programme.
Normalisation Process Theory14	A theory which focuses on how complex interventions become ‘normalised’ or embedded in routine practice. It helps to understand facilitators and barriers in adoption and routinisation of an intervention. Includes four main components: coherence (sense making), cognitive participation (engagement), collective action (work done for intervention to happen), and reflexive monitoring (taking measure of costs and benefits of the intervention).	The model will be used to explain differences in routinisation of mHealth component in the post-trial maintenance phase.

MRC, Medical Research Council.

We will also use the Reach, Effectiveness, Adoption, Implementation and Maintenance (RE-AIM) framework13 to understand and describe the reach, effectiveness, adoption, implementation and maintenance of the intervention. The normalisation process theory (NPT)14 will help to understand the factors that influence integration and routinisation (becoming part of routine practice) of novel interventions in specific settings. NPT is grouped into four broad sub-constructs which influence normalisation or routinisation of novel interventions (coherence, cognitive participation, collective action and reflexive monitoring). RE-AIM and NPT will be used to evaluate how the programme was implemented to understand barriers to and facilitators of its routine use by primary health centre (PHC) doctors, and community health workers commonly known as ASHAs (abbreviation for Accredited Social Health Activists) and community participants.

Broad thematic areas of inquiry will include the context, implementation and mechanism of impact (table 2). Under the theme ‘context’, social, political, cultural and health system level factors impacting on implementation of the intervention will be explored. Differences between the sites, programme adaptations that were a result of change in context (eg, the COVID-19 pandemic), and site-specific barriers and facilitators that impacted the programme implementation and outcome will be enquired into. Under ‘implementation’ the process evaluation will assess the implementation of the two intervention components—anti-stigma campaign and mHealth based service delivery—using the RE-AIM parameters. It will also investigate the experiences of end users of the intervention. Finally, the process evaluation will explore the ‘mechanism of impact’ by critically examining any variations in outcomes or unexpected outcomes.

Table 2

Conceptual framework for process evaluation

Broad area of enquiry	Domains of inquiry	Key questions/process measures	Data source
Context	Differences in context	What are the differences in social, economic, cultural and health system level, between the sites and among the clusters? Do contextual differences influence how programme is delivered in different settings?	Secondary data;Formative research dataInterview with project staff
	Significant changes in context and programme adaptions	What are some of the key contextual factors which influenced the overall implementation of the intervention (eg, COVID-19 pandemic)? What were some of the context specific adaptations that were made to address emerging challenges?	Interview with project staffProject documentation on operational challenges
	Barriers and Facilitators	What are some major barriers faced in implementing the intervention components? What are some of the factors which acted as facilitators in implementation of the intervention components (anti-stigma campaign, mHealth, training and capacity building?	Interview with project staff
Implementation	Implementation fidelity	Was the intervention delivered as it was planned?	Programme records and documents;Observation and ratingInterview with project staff
	Intervention Reach	What was the coverage of the different anti-stigma campaign methods, in terms of: Total persons reached (including gender-wise break-up) Villages and clusters covered Number and proportion of high-risk cohort reached Number and proportion of non-high-risk cohort reached Key stakeholders reached What was the reach of the mHealth services in terms of : Number and proportion of high-risk cohort in the intervention arm provided counselling or follow-up services by Accredited Social Health Activists (ASHAs)? Number and proportion of persons from high-risk cohort provided services in village level health camps Number and proportion of high-risk cohort from the intervention arm who sought care at the PHC Number and proportion high risk-cohort from the control arm who sought care for CMDs What was the reach of IVRS messages to ASHAs and high-risk individuals in terms of Total calls made Calls completed as proportion of total calls Calls not picked up as proportion of total calls Average time of a call made Did the ASHAs face any challenge in reaching out to any category of high-risk individual in their village?	Project records and documents Backend data Interview with project staff Interview with ASHAs
	Intervention effectiveness	What was the perception of the community and key stakeholders about the utility effectiveness content of the Information Education Communication materials the antistigma? What are some of the key take home messages that people absorbed from the campaign? What was the perception of ASHAs about impact of anti-stigma campaign in their village? What is the association between exposure to anti stigma content with changes in KAB scores and care seeking? What is the perception of ASHAs about effectiveness of technology health mental health service delivery in managing CMD in the community? What is the perception of PHC doctors about effectiveness of technology health mental health service delivery in managing CMD in the community? What was the perception of ASHAs about the utility of messages received through IVRS?	Community satisfaction survey done at the end of drama performanceOutcome survey data;Backend data;FGD with community membersInterview with community leaders (like elected village heads, influential village elders and religious leaders),
	Intervention acceptability and adoption	What was the perception of ASHAs about using Electronic Decision Support System (EDSS) for providing care (challenges, perceived benefits, potential for routine use of mHealth)? What was the perception of PHC doctors about using EDSS for providing care (challenges, perceived benefits, potential for routine use of mHealth)? What were the patterns of use of EDSS by ASHAs in terms of : Average time take by ASHAs to administer GAD7 and PHQ 9 over time (during screening, during monitoring) Association between gender of high-risk patient and average time taken by ASHAs to complete screening Association between GAD7 and PHQ 9 scores and average time taken to complete test by ASHAs Cluster-wise difference in average time taken by ASHAs to administer GAD7 and PHQ 9 Association between ASHA’s age and education with average time taken to administer GAD 7 and PHQ9 What were some key features of use of EDSS by PHC doctors in terms of: Average time taken for diagnosis and identification of treatment plan using Mental health Gap Action Programme (mhGAP) over time Association between type of CMD and time taken for diagnosis and identification of treatment plan using mhGAP What was the perception of high-risk patients about ease of getting treatment through mHealth?	Backend dataInterview with ASHAsFGD with ASHAs;Interview with doctorsInterview with PHC support staffInterview with health officials (ASHA co-ordinator, Chief Medical Officer)Patient interview
	Post-trial maintenance	What was the proportion of ASHAs who continued to provide routine care compared with those who discontinued? What are the factors which explain differences in the uptake of the intervention among ASHAs? To what extend is patient adherence associate with routine care and follow-up provided by the ASHAs What are the cluster level differences in no of CMD patients provided treatment during the post-trial phase? What are the factors which explain these differences? To what extent has use of EDSS become routine practice among PHC doctors? What are factors explain differences in adoption/ routinisation of EDSS in different PHC clusters?	Backend dataInterview with ASHAsInterview with PHC doctorsInterview with PHC support staffInterview with project staff
	Health service use	What are the barriers or facilitators that patient from intervention cluster face while accessing care in the PHC?How many high-risk individuals identified in the intervention arm did not seek care? What are factors which can explain this?What are the factors which explain treatment adherence among high-risk patients who sought care?What are the cluster-wise differences in service utilisation, treatment adherence and number of referrals to specialist centres? What are the factors which can explain this?	Backend dataInterview with high-risk individualsInterview with ASHAsInterview with doctorInterview with project staff
Mechanism of impact	Variation in outcomes	What kind of cluster level variation is overserved in in the outcomes? What works, for whom and in what context?	Outcome dataBackend data;Interview with ASHAsInterview with doctorInterview with project staff
	Unexpected outcomes	What are some unexpected outcomes and what factors can be attributed to them?	Outcome dataBackend data;Interview with ASHAsInterview with doctorInterview with project staff

CMDs, common mental disorders; FGD, focus group discussion; GAD7, Generalised Anxiety Disorder-7; IVRS, interactive voice recording system; KAB, Knowledge Attitude Behaviour; PHC, primary health centre; PHQ9, Patient Health Questionnaire-9.

Study setting

SMART Mental health is being implemented in 133 villages serviced by 44 randomly selected PHC in West Godavari district of Andhra Pradesh (South India) and Palwal and Faridabad districts of Haryana (North India).

Study design

The process evaluation will use a mixed-method multiple case study design with PHC clusters constituting a ‘case’. Up to eight case studies will be included. Each case will be selected purposively based on the principle of maximum variation in terms of health service delivery context, implementation challenges and outcomes.

Intervention description

The intervention comprises two key components; an antistigma campaign, and a technology-enabled mental health service intervention delivered through task sharing. The capacities of community health workers known as ASHAs and PHC doctors will be enhanced, by providing training in identifying and managing stress, depression, or suicide risk using a technology enabled electronic decision support system (EDSS).

In the preintervention phase, ASHAs will be trained to use the EDSS to screen individuals at high risk of stress, depression, self-harm or suicide using digital hand-held tablets. The tablets have two preinstalled, standardised screening and assessment tools—the PHQ915 16 and the Generalised Anxiety Disorder-7 (GAD-7)16 17 questionnaire. The screening process classifies whether participants are at high risk of CMDs based on the PHQ-9 and GAD-7 scores. Because a substantial proportion of people at risk of CMDs undergo natural remission over a period of time18 a second screening of all people initially identified at high risk is undertaken by the ASHAs within 6 months of the first screening to identify those who remain at ‘high risk’.

Additionally, a Knowledge Attitude Behaviour19 scale is administered to assess levels of stigma associated with mental disorders in the community, a Barrier to Access to Care Evaluation-Treatment Stigma20 questionnaire to assess stigma perceptions related to help-seeking for mental disorders and the EuroQol 5-Dimension-3 Level scale21 to assess quality of life. Questions related to history of psychiatric morbidity, availability of social network/support, treatment history and costs incurred in treatment (which will be used for economic evaluation) are also asked.

In the intervention phase, the two major intervention components will be implemented to those PHCs randomised to receive SMART Mental Health (figure 1). The logic model for how the intervention strategy is hypothesised to meet its aims has been provided (figure 2).

Figure 1

Study schema for smart mental health.11 ASHA, Accredited Social Health Activists; CMD, common mental disorder; GAD7, Generalised Anxiety Disorder−7; PHQ9, Patient Health Questionnaire-9.

Figure 2

Logic model of smart mental health. ASHAs, Accredited Social Health Activists; EDSS, electronic decision support system; IEC, Institutional Ethics Committee; PHCs, primary health centres.

The anti-stigma campaign uses audio-visual and print material tailored to the local community and delivered to both high-risk and non-high-risk individuals, with the aim of reducing negative knowledge, attitudes and behaviours related to mental disorders. The second component of the intervention is a technology enabled mental health service delivery model. An mHealth platform will be used for screening, diagnosis, referral and management of CMDs by community level health workers (ASHAs) and PHC doctors. Health workforce capacity building is a crucial input which will be embedded throughout the intervention. The ASHAs will follow-up individuals at high-risk of CMDs to support access to care from the PHC doctors. When the patient reaches the PHC, the doctors will use an EDSS based on WHO’s Mental Health Gap Action Programme Intervention Guide.22 Clinical data will be shared between the ASHAs and doctors using a secure cloud-based server. For follow-up care, the ASHAs will have an algorithm enabled priority listing that will provide them with a traffic-light system to prioritise and track the progress of individuals in her village. They will use this to follow-up patients, paying particular attention to the highest priority individuals and enquiring about their treatment adherence and mental well-being.

Following the intervention, a post-intervention phase up to 9 months will assess the sustainability of the intervention without external influence of the trial team. In this phase, the components of the intervention will be rolled out in the control arm too. Support for ASHAs and doctors by project staff will be minimal. Staff will assist ASHAs and doctors to resolve any technical problems with the tabs and provide initial support and troubleshoot any issues.

Control arm

In the control arm, ASHAs will be provided with the names of individuals at high risk of CMDs and they will support those individuals to seek care and provide them with relevant information of mental healthcare providers. PHC doctors in the control arm will be informed that there may be patients who may seek care for CMDs. The ASHAs and the doctors in the control arm will not be provided with access to the EDSS. The anti-stigma campaign will be delivered in a less extensive manner. Besides pamphlets and brochures, all the other anti-stigma components will be shared with the study participants. The live drama shows however, will not be conducted. Only videos of the drama will be shown. The ASHAs will draw on their existing training and experience on mental health to support individuals as needed.

Data collection

Quantitative data source includes analysis of the usage analytics extracted from the mHealth platform. This includes (1) user metrics from each tablet used by ASHAs and PHC doctors; (2) screening and treatment data about each high-risk individual in the intervention cohort; (3) data from the priority listing application (used by ASHAs) which provide information on treatment status and high-risk individuals who need to be followed up and (4) data from the interactive voice recorded system used to send messages to ASHAs and high-risk individuals (to facilitate treatment adherence and follow-up). These data will be used to assess reach, effectiveness, adoption, maintenance and service utilisation of the intervention.

Qualitative data will include key informant interviews and focus group discussions (FGDs) with PHC doctors, ASHAs, hospital administrators, service users and any other relevant stakeholders such as family members of service users and community leaders. The qualitative study data will explore perceptions of key stakeholders about the effectiveness and acceptability of intervention components and challenges in implementation. A detailed data collection plan has been discussed in table 3.

Table 3

Qualitative data collection plan

FGDs
Type of group/Individual	Some areas of inquiry	Number planned per PHC	Total planned(6 intervention and 2 control PHC clusters will be selected for the case study)
ASHAs	Facilitators and barriers experienced in delivering the intervention in the community Perception about effectiveness of different intervention components like anti-stigma campaign, technology-based decision support system and use of IVRS Perceptions on training appropriateness, effectiveness and methods Factors that influenced treatment seeking by high-risk cohort Overall experience of participating in the trial	1	8
Project field staff	Barriers or facilitators experienced in implementation of the intervention Perceived factors which explain high/low treatment seeking in different PHCs Key challenges and lessons learnt in implementation of intervention components like antistigma campaign, technology-based decision support system and use of IVRS Views on impact of the intervention in the community Perceptions on training appropriateness, effectiveness and methods		3
Study participants from high-risk cohort in intervention arm who sought treatment(To purposively select individuals who (1) went to PHC (2) Went to camp (3) got treated by psychiatrist (4) started treatment but discontinued)	Perceptions about different intervention components like antistigma campaign, technology-based decision support system and use of IVRS Facilitators and barriers in treatment seeking Experience of care and perception about quality of care Perceived benefit if any as a result of treatment received Positive/negative experiences as a study participant Perception about benefits/effectiveness of the intervention	2(1 with men one with women)	12
Study participants from non-high-risk cohort in the intervention arm	Perception about the different components of the antistigma campaign (eg, Live drama, pamphlets etc) Key takeaway messages from the antistigma campaign Perceived changes if any related to mental health stigma Positive/negative experiences as study participant	2(One with men and one with women)	12
Study participants from high-risk cohort in the control arm (including both who sought treatment and who did not seek treatment)	Reasons for seeking or not seeking care Facilitators and barriers in the community to seeking care for CMDs Experience as a study participant	2(One with men and one with women)	4
Total FGDs			39
In-depth Interviews
PHC doctors	Experience of using technology-based decision support system to diagnose and manage CMDs Challenges faced in trial participating Perceived effectiveness of intervention components (antistigma campaign, mHealth) in improving management of CMDs in the community. Possible facilitators and barriers to scaling up Overall experience of participating in the trial	1	8
Village heads/community leaders of the village	Their role in this programme if any Views about the programme and its impact Feedback and suggestion if any	1	8
Study participants from high-risk cohort in intervention arm who who did not seek treatment	Perceptions about different intervention components like antistigma campaign, technology-based decision support system and use of IVRS Reasons for not seeking care Facilitators and barriers in treatment seeking Positive/negative experiences as a study participant Perception about benefits/effectiveness of the intervention	1	12
Government health officials	Perception about effectiveness of the intervention in reducing treatment gap for CMDs Perceived facilitators and challenges in scaling up the intervention Their role if any in the programme	2 (per district)	6
Total interviews			24

ASHAs, Accredited Social Health Activists; CMD, common mental disorder; EDSS, Electronic decision support system; FGDs, focus group discussions; IVRS, interactive voice recording system; PHC, primary health centre.

At the end of the post-intervention phase, a detailed comparative case study of two PHCs with be undertaken. It will include one PHC with high utilisation of EDSS and one with low utilisation. The case study will provide insights into barriers and facilitators in adoption and routinisation of EDSS and explain differences in levels of utilisation of mHealth in different PHC clusters. Interviews with all key stakeholders (including PHC doctors, ASHAs, supervisors associated with the PHC) will be used to develop the case study.

Data analysis

For quantitative data, basic descriptive analyses will be conducted. For qualitative data analysis interview transcripts will be read independently by two persons. A priori codes based on the conceptual framework (table 2) will be used to code the data. Additional thematic findings emerging through the data will be added to the coding framework. Data will be coded using NVivo V.12.0. Both qualitative and quantitative data across case studies will be triangulated to arrive at the findings.

Patient and public involvement

In the formative phase, community feedback was sought through FGDs, to make antistigma content culturally and contextually relevant. The study findings will be shared with the public. Findings will be disseminated through publication in peer reviewed journals, meetings, digital and social media platforms.

Ethics and dissemination

SMART Mental Health cRCT was approved by the George Institute for Global Health, India and the Institutional Ethics Committee (IEC), All India Institute of Medical Sciences (AIIMS), New Delhi. The trial has been registered (CTRI/2018/08/015355) with the Clinical Trial Registry-India, National Institute of Medical Statistics, Indian Council of Medical Research (ICMR). The project has received requisite approval the Health Ministry’s Screening Committee (HMSC), ICMR.

Trial status

At the time of writing this paper, the intervention phase of the trial had begun in both sites. Clinical Trials Registration was completed on 16 August 2020. Randomisation of clusters in Haryana was done on 21 September 2020 and in Andhra Pradesh on 4 December 2020. Key intervention components were being delivered in Andhra Pradesh and postintervention activities and follow-up surveys were being planned in Haryana.