Evaluation of capacity-building strategies for mental health system strengthening in low- and middle-income countries for service users and caregivers, policymakers and planners, and researchers.

BACKGROUND: Strengthening of mental health systems in low- and middle-income countries (LMICs) requires the involvement of appropriately skilled and committed individuals from a range of stakeholder groups. Currently, few evidence-based capacity-building activities and materials are available to enable and sustain comprehensive improvements. AIMS: Within the Emerald project, the goal of this study was to evaluate capacity-building activities for three target groups: (a) service users with mental health conditions and their caregivers; (b) policymakers and planners; and (c) mental health researchers.
METHOD: We developed and tailored three short courses (between 1 and 5 days long). We then implemented and evaluated these short courses on 24 different occasions. We assessed satisfaction among 527 course participants as well as pre-post changes in knowledge in six LMICs (Ethiopia, India, Nepal, Nigeria, South Africa, Uganda). Changes in research capacity of partner Emerald institutions was also assessed through monitoring of academic outputs of participating researchers and students and via anonymous surveys.
RESULTS: Short courses were associated with high levels of satisfaction and led to improvements in knowledge across target groups. In relation to institutional capacity building, all partner institutions reported improvements in research capacity for most aspects of mental health system strengthening and global mental health, and many of these positive changes were attributed to the Emerald programme. In terms of outputs, eight PhD students submitted a total of 10 papers relating to their PhD work (range 0-4) and were involved in 14 grant applications, of which 43% (n = 6) were successful.
CONCLUSIONS: The Emerald project has shown that building capacity of key stakeholders in mental health system strengthening is possible. However, the starting point and appropriate strategies for this may vary across different countries, depending on the local context, needs and resources. DECLARATION OF INTEREST: S.E.L. received consulting fees from Lundbeck.

There is a growing awareness that strengthening mental health systems to effectively prevent mental ill health and care for people with mental health problems requires a broad perspective, taking into account the interconnectedness of human and financial resources beyond diagnosis and provision of treatment. Most mental health-related capacity building in low- and middle-income countries (LMICs) focuses on training clinicians and/or lay people to identify and treat people who need care to reduce the treatment gap. However, health system change also relies on support from other key stakeholders to achieve comprehensive improvements. Three stakeholder groups are particularly crucial, but are rarely considered, as target groups for strengthening mental health systems in LMICs: (a) service users and caregivers; (b) policymakers and planners; and (c) mental health researchers. In general, health policy and health systems research in relation to mental health in LMICs is a neglected field.[1] Improvements of mental health systems and hence mental health outcomes require commitment and understanding from policymakers and planners to allocate and coordinate budgets appropriately, and to plan for appropriate and inclusive local and national policies. A critical consideration is having insights from service users and caregivers communicated effectively, to ensure that any system or policy reform is appropriate and relevant to their needs and preferences.[2-4] To facilitate this cycle, we need advocates and practitioners who are knowledgeable and equipped with real-world evidence about how to design a system that effectively addresses the mental health needs of the consumers in an equitable manner and operates efficiently within the available resources.

Mental health researchers also play a key role in developing and communicating needed evidence to these stakeholders. Although there are good models for researcher development,[5] capacity and evidence are lacking in many LMICs. Two systematic reviews[6,7] have clearly highlighted the paucity of evidence, first in relation to building the capacity of policymakers and planners to strengthen mental health systems in LMICs and second involving service users with mental health needs and their caregivers in health policy planning, service monitoring and research. The goal of this study was therefore to evaluate capacity-building activities carried out as part of the Emerald (Emerging mental health systems in low- and middle-income countries) project. These activities targeted three groups: (a) service users and caregivers; (b) policymakers and planners; and (c) mental health researchers. Emerald is a multicountry initiative to develop evidence and capacity for mental health system strengthening in Ethiopia, India, Nepal, Nigeria, South Africa and Uganda.[8] In this paper we present the engagement and participation of each stakeholder group; changes in relevant mental health system knowledge; and overarching structural and institutional changes in research capacity.

Method

Emerald capacity-building activities

Details regarding Emerald capacity-building activities are reported in detail elsewhere.[8-10] Briefly, a range of targeted activities were delivered in each of the six Emerald participating countries. Activities were tailored to local needs, context and resources and according to the target group.[10]

Service users and caregivers

For service users and caregivers (Table 1), the primary activity was a 1- to 3-day workshop to raise awareness about treatment and the rights of people with mental illness and to increase advocacy and involvement among service users and caregivers. (Training manuals developed for Emerald are included in supplementary Appendices 1A and 1B available at https://doi.org/10.1192/bjo.2019.14.) As part of the Emerald programme, efforts were also made to train primary care workers and managers to support service user involvement and to encourage PhD students to develop research in the area.

Table 1

Tailoring of the service users and caregivers short-course delivery and target participants to country context

	Ethiopia	India	Nepal	Nigeria	South Africa	Uganda
Target audience	Service users and caregivers in the existing PRogramme for Improving MEntal health care project, district and primary care health centre heads and district health office planners	Service users and caregiver organisation at national and state level; service users and caregivers in Sehore district, Madhya Pradesh	Service users and caregivers from several primary health centres in the Chitwan district and staff from service users organisation in Kathmandu	Service users and caregivers	Service users from several primary health care facilities in the Dr Kenneth Kaunda district	Service user organisations/groups; leaders in service user organisations
Goals	Increased awareness of the meaning and potential benefits of involvement of service users and caregivers and receptiveness to the concept; equipping service users, caregivers and primary health centre heads/district health office staff with a framework for engagement acceptable to all	Awareness on system issues involving community advisory board group and other service users/caregiver organisations in advocacy	From the workshop we hoped: (a) to develop an appropriate, common and contextually suitable term for the word ‘service users’; (b) to discuss the findings of the capacity-building studies and develop common consensus on various aspects of service users involvement in Nepal	Short term: improve awareness and equip with advocacy skills and evidence; long term: empower to engage service providers, facility managers, government agencies, mass media and the general public	To improve awareness of the importance of service user advocacy to improve mental health services and to empower service users to engage in such activities	Short courses; advocacy sessions at both national and district levels
Plan for Emerald resources	Multifaceted intervention to increase service users involvement in mental health services and systems at the grass-roots level (feasible and relevant to be integrated into plans for mental healthcare scale up)	Workshops with national and state-level organisations	Workshops/group sharing	Capacity-building workshops	Capacity-building workshops including workshop materials	Modules, video, summary notes
Duration, days	2 for service users/caregivers; 1 for primary healthcare leads	1	3	2	1	1

Policymakers and planners

For policymakers and planners (Table 2), workshops in mental health system strengthening were run, with country teams selecting modules from the following domains: mental health awareness-raising, the chronic care model and mental health system planning. Each site also developed and maintained an ongoing dialogue with policymakers, providing technical support and facilitating collaboration between researchers and policymakers. As it was not possible to run workshops in Nepal, only engagement activities described in Table 2 were used for policymakers and planners. A course overview and materials are provided in supplementary Appendices 2A–C.

Table 2

Tailoring of the policymakers and planners short-course delivery and target participants to country context

	Ethiopia	India	Nepal	Nigeria	South Africa	Uganda
Target audience	Federal Ministry of Health and regional focal persons for mental health	Members of mental health policy group and programme officers of the state health societies	Staff from Ministry of Health; Department of Health Services, Mental Hospital, Tribhuwan; University Psychology Department, TU Teaching Hospital; Nepal Human Rights Commission; Nepal Health Research Council; National Women's Commission; Ministry of Law and Justice; Ministry of Women, Children and Social Welfare; Central Child Welfare Board and Ministry of Home (Nepal Police)	Federal Ministry of Health; Ministry of Defence; national-level policy-makers for mental health; hospital directors; national associations/agencies of psychiatrists, psychologists and primary health care; World Health Organization country office staff; national police headquarter staff; director of prison medical services; NGO staff; Human Rights Commission	National, provincial and district mental health directorate staff – aligned with the new national mental health policy framework and strategic plan, adopted by the Department of Health in July 2013 (to 2020)	Ministry of Health policymakers, district policymakers (including sector managers); school management committees; leaders in higher institutions of learning; civil society organisations working in the mental health field (including user organisations)
Goals	That healthcare planners and managers have improved awareness about mental health and are better equipped to coordinate the mental healthcare scale up within their regions	Share learnings of mental health system with the group and work towards sustaining linkages with policymakers and planners and build technical capacity of planners at state and national level in appraising Program Implementation Plans for mental health	To orient policymakers about the need for mental health system strengthening; to highlight that by treating mental illness we also contribute to other physical health outcomes; to make them aware that cost-effective mental health interventions exist and with little efforts of policymakers much can be achieved in the field of mental health	Short term: increased awareness and sensitisation of the salience of mental health in overall health system planning and delivery; long term: commitment towards supporting mental health integration into general medical services with increased prioritisation and funding for mental health programmes	Improved capacity for mental health service planning for provincial and district health planners	Improved response and delivery of effective programmes on mental health; improve cost-effectiveness of interventions; improve programme sustainability
Planned approach	One-off workshop, convened by the Ministry of Health and run by Emerald	Workshop with policymakers and planners at national level in Delhi and in various states including Madhya Pradesh	Two-phase engagement model: large group meeting with discussion of key concepts; followed by small group meetings (e.g. lunch) on specific topics	Initially as sensitisation and capacity-building workshops for policymakers; subsequently, sustained process of continued engagement using the platform of the National Action Committee on mental health	Build on existing engagement process with the Department of Health. Country principal investigators already involved in technical support to Department of Health with respect to mental health reform and implementation	Coffee breaks; lunchtime meetings; policy briefs; short trainings lasting a few hours; sharing of modules
Duration, days	0.5	1–2	1 for large group meeting; also lunch/dinner meetings spread over several months	1–2	1–2	1

Mental health researchers

To increase capacity among mental health researchers (Table 3), short courses were provided in mental health systems research, implementation science research and service user involvement in research in addition to further training about leadership and writing skills. A course overview and materials are provided in supplementary Appendices 3A–C. Ten PhD students were linked to Emerald and two MSc fellowships were offered on a competitive basis to individuals based in Emerald LMIC partner countries. PhD students also received support via a peer-led forum that was designed to bring PhD students together via a network to share information and experiences, and to identify needs and organise targeted e-learning opportunities delivered by members of the Emerald group.

Table 3

Tailoring of the mental health researchers short-course delivery and target participants to country context

	Ethiopia	India	Nepal	Nigeria	South Africa	Uganda
Target audience	PhD students and faculty at Addis Ababa University and Jimma University	Students and researchers (state and national level); consultants in health sector (state and national level)	Researchers from Nepal Health Research Council, research staff of New Era, Transcultural Psychosocial Organization Nepal, Crehpa and HERD (research-based organisation), Masters-level students from psychology (Tribhuwan University) and public health (Institute of Medicine)	Early-career researchers from a multidisciplinary background (psychiatry, psychology, health economics, public health, non-governmental organisations)	Students; clinicians; health professionals; researchers working in these areas (e.g. Human Sciences Research Council (HSRC), Medical Research Council)	Students at medical school in various universities and others undertaking courses related to mental health; clinicians (e.g. continuing medical education); health professionals especially those undertaking mental health related research projects
Goals	For PhD students, a broadening of their training with a view to equipping them in postdoc work; for faculty, to increase the number of health systems projects and publications	Improved delivery of mental health services/programmes and bridging the gap between researchers and implementers and to facilitate more effective services that are cost-effective	To orient participants to system-thinking perspectives and explain key concepts of health system strengthening; to impart knowledge on methods for measuring and monitoring health system performance and improvement	Short term: stimulate interest in health systems research and implementation science; long term: develop capacity to design, conduct and implement health system research that will contribute to knowledge and improve functioning	Improved delivery of mental health services/programmes and more effective services that are cost-effective	Improved capacity to undertake mental health research for both students and clinicians; improved response and delivery of effective programmes on mental health; improved cost-effectiveness of interventions; improved programme sustainability
Duration, days	5	3	5	2	2	2
Delivery of capacity-building	Face-to-face classroom teaching	Lecture sessions for researchers and students; workshops for consultants	Face-to-face classroom teaching, group work and case sharing	Workshops with face-to-face interaction	Course delivered face to face to researchers already working in this field	Lecture sessions; face to face; continuing medical education workshops

Participants

As countries differed in their recruitment methods these are described separately for each country and each target group in the Appendix.

Assessment of capacity-building results

Evaluation of capacity-building activities covered a range of domains. Although the Emerald project collected qualitative and quantitative evaluation data,[9] we focus on the quantitative findings here. Overall, the quantitative evaluation focused on process information and outcomes for each of the three target groups, in addition to agreed overarching indicators of structural or institutional change. Process information covered the absolute numbers of people who registered and completed each training module. To better understand the reach of the training, we collected information about participant characteristics such as gender, whether they were based in a public institution and whether they were from outside of the capital city.

To assess outcomes, participants were asked to complete a questionnaire before and after each training course/workshop. Questionnaires were tailored for each target group and covered participant satisfaction with the training and changes in knowledge (questionnaires are available on the Emerald website, see https://www.emerald-project.eu/home/). In terms of knowledge outcomes, we first examined the improvement in responses to knowledge items (averaged across items). We also assessed the total number of questions that demonstrated a positive pre–post improvement.

Change in institutional capacity was assessed in two ways. First, information was collected via questionnaire from MSc and PhD students, and early-career, mid-career and senior researchers about the impact of the Emerald programme on grant and paper involvement and international collaboration. Outputs (papers and grant participation) were collected in an identifiable email survey. All other feedback about, for example, satisfaction with the Emerald project were considered to be more sensitive and thus were collected anonymously via a GoogleForm document.

Second, senior researchers from each of the Emerald partner sites were also interviewed in relation to: (a) organisational self-sufficiency in delivering short courses, (b) how well equipped the department or institution was for the supervision of PhD students in the area of mental health systems research/implementation research (such as expertise, numbers of supervisors), (c) institutional capacity for delivering Masters-level training in health systems research, implementation science or non-communicable/long-term disorders and (d) the extent to which the Emerald programme contributed to any of these organisational changes and/or had become embedded in institutional training.

Research ethics committee approvals

Ethical approval for this study was obtained from King's College London, the World Health Organization and the institutional review boards of each of the participating sites.

Results

Process information

Almost all individuals who registered (94–100%) also attended the courses. The majority of participants in all stakeholder groups were men, although this was almost evenly split in the service user and caregiver workshop (55% men), whereas women were most clearly underrepresented in the policymakers and planners’ short course (87% men). There was a balanced representation from individuals living outside the capital city, in particular for the service user and caregiver workshops and the researcher course on service user and caregiver involvement, where approximately two-thirds of participants came from outside of the capital city. Most short-course participants were working in the public sector (65–87%), except for the service users and caregiver workshop where only 7% who attended were working in the public sector (Table 4).

Table 4

Process information and participant details for the researcher, policymaker/planner and service user/caregiver short courses

	Researcher short course on implementation science	Researcher short course on mental health system strengthening	Researcher short course on service user involvement in research	Policymakers and planners’ short course	Service user and caregivers’ workshop
Courses, n	6	6	4	1	4
People registered for course, n	167	126	79	23	132
People completing course, n (%)	167 (100)	126 (100)	78 (99)	23 (100)	124 (94)
Women, n (%)	66 (40)	46 (37)	Data not available	3 (13)	60 (45)
From outside capital city, n (%)	68 (41)	70 (56)	52 (66)	Data not available	85 (64)
Working in public sector, n (%)	113 (68)	82 (65)	56 (71)	20 (87)	9 (7)

Capacity-building satisfaction outcomes

Table 5 shows that high levels of satisfaction were reported for the short courses across all three target groups. Policymakers and planners reported the highest level of satisfaction with 78% strongly agreeing (22% agreeing) that the teaching standard was high and 89% strongly agreeing (11% agreeing) that their expectations had been fulfilled. For all satisfaction outcomes, at least 95% of respondents reported agreement or strong agreement that they were satisfied with the standard of teaching and that their expectations had been fulfilled.

Table 5

Satisfaction and knowledge outcomes for researchers, policymakers/planners and service users/caregivers across all countries

	Researcher short course on implementation science(n = 114)	Researcher short course on mental health system strengthening (n = 121)	Researcher short course on service-user involvement in research(n = 66)	Policymakers and planners’ short course(n = 13)	Service user and caregivers’ workshop(n = 124)	Total
Satisfaction
Standard of teaching was high, %
Strongly agree	47.0	45.3	49.3	77.7	68.3	57.5
Agree	49.6	53.0	44.0	22.3	25.6	38.8
Neither agree nor disagree	3.3	1.7	6.7	0	3.8	3.1
Disagree	0	0	0	0	2.4	0.5
Expectations have been fulfilled, %
Strongly agree	50.5	47.5	56.3	88.8	65.1	61.6
Agree	47.4	50.4	42.0	11.2	32.1	36.6
Neither agree nor disagree	2.1	2.1	1.7	0	1.6	1.5
Disagree	0	0	0	0	1.3	0.3
Knowledge
Change pre–post, mean %	+1.8	+9.7	+52.3	+17.9	+21.7
Positive/negative direction of individual items, questions	5 positive; 5 negative	12 positive; 0 negative	10 positive; 0 negative	8 positive; 0 negative	6 positive; 0 negative

Knowledge outcomes

On average, there was an improvement in knowledge across all short courses, with the greatest improvement in the researcher course on service user involvement in research (with a mean improvement in knowledge of 52.3% across items) and the lowest level of improvement in the researcher short course on implementation science (improvement of 1.8%). At the individual-item level, all short courses except for the researcher short course on implementation science showed an improvement in each item. See Table 5.

Overarching indicators of structural or institutional change

Emerald researchers and MSc/PhD students surveys

Almost all Emerald MSc/PhD students completed the email and online surveys (91%, n = 10), and 67% (n = 20) and 53% (n = 16) of Emerald researchers completed the anonymous online and email surveys, respectively. Among those who responded, all Emerald researchers and MSc/ PhD students attended at least one of the seven Emerald annual meetings in person, with the vast majority finding the meetings at least somewhat useful. In terms of MSc/PhD student supervision, all students reported being at least somewhat satisfied with the quality of supervision.

Over half (60%, n = 6) of participants of the MSc/PhD online survey reported having been involved in the early career research support group, with half (50%, n = 5) saying that they had found these meetings somewhat useful and half (50%, n = 5) saying that they had not found it useful. There seemed to be good cross-partner interaction between Emerald researchers with the majority of early-, mid- and senior career researchers reporting a lot or quite a lot of input from Emerald researchers outside their country. In terms of future career plans, 70% (n = 7) of MSc/PhD respondents said they felt somewhat equipped for their future career plans, and 30% (n = 3) said they felt very equipped. In total, 80% (n = 8) reported that their PhD or MSc had contributed ‘quite a lot’, and 20% (n = 2) ‘a lot’, to them feeling equipped for their future career plans. All ten MSc/PhD respondents reported that they planned to continue working in research, with all of them saying that Emerald had prepared them well to continue working within research either ‘a lot’ (40%, n = 4) or ‘quite a lot’ (60%, n = 6).

In relation to outputs, participants of the PhD email survey had submitted 1.25 papers on average relating to their PhD work (a total of 10, range 0–4) and a further 27 papers were planned (per person mean of 3.38; range 2–5). PhD respondents were also involved in 1.75 grant applications, on average, during their PhD (range: 0–4). Of 14 applications, 43% (n = 6) were successful. Participants of the researcher email survey reported an average of 5.8 paper submissions related to Emerald (range 0–16). In terms of grant applications, the researchers reported involvement in an average of 4.1 applications during Emerald (range: 0–10). Of these, 45% were successful (Table 6).

Table 6

Anonymous online capacity-building survey results of Emerald researchers and PhD students

	Early-career researchers (n = 5)	Mid-career researchers (n = 7)	Senior researchers (n = 8)	PhD/MSc students(n = 10)
Attended at least one annual Emerald meeting, n (%)	5 (100)	7 (100)	8 (100)	10 (100)
Found meetings useful or somewhat useful, n (%)	5 (100)	6 (86)	8 (100)	10 (100)
Meeting supervisor at least once per month, n (%)	–	–	–	6 (60)
Somewhat/very satisfied with supervisor meeting frequency, n (%)	–	–	–	7 (70)
Somewhat/very satisfied with supervisor meeting quality, n (%)	–	–	–	10 (100)
Very/somewhat supported with teaching by supervisors/Emerald researchers, n (%)	–	–	–	4 (67)
A lot or quite a lot of input from Emerald researchers outside your country, n (%)	4 (80)	6 (86)	7 (88)	–
Reporting that Emerald contributed a lot or quite a lot to a positive career change, n (%)	1 (20)	3 (43)	2 (25)	–
Reporting that Emerald contributed a lot or quite a lot to feeling equipped for future career plans, n (%)	4 (80)	7 (100)	6 (75)	10 (100)
Reporting that Emerald contributed a lot or quite a lot to being prepared to continue working in research, n (%)	3 (60)	5 (71)	6 (75)	10 (100)
Future career plans, n (%)
Academia	2 (40)	3 (43)	6 (75)	10 (100)
Public sector	4 (80)	1 (14)	3 (38)	5 (50)
Private sector	1 (20)	0 (0)	1 (13)	0 (0)
Non-governmental organisation	0 (0)	0 (0)	0	4 (40)
Further education/postdoc	2 (40)	2 (29)	0	6 (60)
Career break	0 (0)	0 (0)	1 (13)	0 (0)
Remain in position	0 (0)	0 (0)	1 (13)	NA
Number of Emerald-related papers submitted, mean (range)	5.8 (0–16)			1.25 (0–4)a
Involvement in grant applications during Emerald, mean (range)	4.1 (0–10)			1.75 (0–4)a

NA, not applicable.

PhD students only.

Changes in institutional research capacity

All Emerald LMIC partners experienced improvement in their capacity to conduct health systems research, with the change ‘very much’ attributed to Emerald by four institutional partners and ‘to some extent’ by the other three institutional partners. The average values across participating institutions of change in capacity and associated attribution to the Emerald programme are presented in Fig. 1.

Fig. 1

Change in capacity and attribution of change during Emerald project by research area, averaged across institutions.

Discussion

A total of 24 short courses involving 527 participants were implemented and evaluated for the target groups of service users and caregivers, policymakers and planners, and mental health researchers across the six Emerald countries. This was complemented by concerted training, including mentoring of junior researchers and development of resources to improve the research capacity of institutions associated with the Emerald project. Our evaluation suggested that short courses and workshops for each of the target groups were associated with high levels of satisfaction and led to improvements across target groups, although the implementation science module of the short course for researchers showed only a slight improvement. In relation to institutional capacity building, all of the Emerald LMIC partner institutions reported an increase in their research capacity for most aspects of mental health system strengthening and global mental health, and a large part of these positive changes were attributed to the Emerald programme.

The level of improvement varied across institutions and was lower where baseline capacity in the area was already strong. Developments in capacity were also reported by PhD students, MSc students and other Emerald researchers. Students and researchers reported being involved in publishing research papers, submitting grant applications and supervising students. These findings suggest that the Emerald model of delivering and evaluating tailored capacity-building activities could provide an important step towards strengthening the human resources for researchers needed to support improved mental health systems in six LMICs.

The Emerald project demonstrated several areas of improvement across the six participating countries; however, countries also differed widely in their baseline capacity, human, financial and political resources and needs; and thus, capacity-building strategies varied in each country. For example, Ethiopia had no service user organisations and only one caregiver organisation based in the capital city, whereas Uganda already had three service user organisations with 16 900 members spread throughout the country.[11] Country-level adaptations were made to all of the short courses, to fit in with the individual countries' local contexts and needs. This highlights the challenges in developing training materials that could be applicable across a diverse group of countries and the importance of training local facilitators to be sensitive to the group needs when delivering and facilitating the workshops. As a result, the level of appropriateness of training materials was diverse and required careful situation analysis[8] to ensure that the facilitator delivering the workshop had a good grasp of this context.

There were some areas of the capacity-building activities that need further attention. In particular, the implementation research course for researchers did not demonstrate improvements at the level shown in the other short courses. It may be that for this course the materials were being continuously developed while the evaluation was not modified alongside the development of the course materials. Sites noted that it was particularly useful to tailor the course to the country-specific context; however, some details such as those related to economic evaluation were limited given the lack of data and specific expertise in this area available in the participating countries.

Strengths and limitations

The findings from these capacity-building activities and their evaluation add to the sparse literature on capacity building and mental health system strengthening in LMICs, in particular for policymakers and planners, and service users and caregivers.[6],[7] There are, however, several limitations that should be considered when interpreting the findings. In relation to the short courses, it was difficult to assess practice and/or the behavioural impacts as our evaluation used proxy indicators based on self-report. Self-report indicators may exaggerate the behavioural impact or change. In some contexts, it may be possible to supplement survey responses with analysis of publicly available documents of health system responses to community mental health needs to examine the impact on mental health system strengthening. However, the quality and comprehensiveness of public reports were not of high quality in the sites where Emerald activities were delivered. Additionally, it is difficult to know exactly how much of an impact could be attributed to the Emerald programme using these more general types of outcomes that are not precisely tied to Emerald.

Moreover, these broader system impacts may take time to become apparent and our evaluation timeline did not allow for a long-term follow-up to assess the impact of the short courses. Our evaluation of institutional research capacity did permit a longer-term follow-up by collecting information about subjective experiences and academic outputs and resources attributable to the 5-year Emerald project. We were not able to compare the impacts to a control group that did not receive the capacity-building activities and so it is difficult to know what kind of changes in institutional capacity would have resulted without Emerald. Nevertheless, our evaluation demonstrated a high level of productivity among associated researchers and institutions.

Implications and future directions

Evidence-based capacity-building is an important aspect of mental health system strengthening in LMICs. The Emerald project activities and evaluation have shown that building capacity in mental health system strengthening in LMICs is feasible and generally welcome by participants and beneficiaries. Focusing on three distinct and interrelated target groups of service users and caregivers, policymakers and planners, and mental health researchers also showed the potential for interaction between these groups. For example, equipping service users and caregivers with greater knowledge, awareness and receptiveness to mental health research and service planning could facilitate greater involvement in a synergistic way if policymakers, planners and researchers are also aware of the benefits of involving service users and caregivers. Similarly, building the capacity of mental health researchers could increase the evidence needed by policymakers and planners to improve the quality and efficiency of mental health service planning. In order to better understand the effects of capacity-building activities, potential synergies and areas needing improvement, evaluation needs to be an integral part of the delivery of these activities.

The evaluation framework used by the Emerald project might serve as a model for the assessment of capacity-building across the three selected target groups of stakeholders in LMICs. Although the starting point and appropriate strategies for this may vary across different countries, making training and evaluation materials freely and publicly available should further increase capacity and involvement in mental health system strengthening in the future.

Moreover, future evaluations of capacity-building activities can build and improve on the Emerald framework by, for example, considering applying triangulation techniques to assess the impact on a broader group of stakeholders and considering additional outcomes. We are currently piloting other evaluation methods at the local level that may strengthen our understanding of this process. For example, there is currently one Emerald linked PhD student in Ethiopia who is conducting in-depth action research to assess the impact of the capacity-building activities. In terms of specific measures, the Emerald programme also planned to incorporate an assessment of attitudinal changes among policymakers but the attitude questionnaires we developed were not acceptable to policymakers and planners.

Future evaluation frameworks should consider other ways of assessing attitudinal change and reduction in stigma, possibly using less direct proxies of this outcome. The Emerald project has made an important step to develop our understanding of the capacity-building process and further strengthening of mental health systems and increasing engagement of a range of stakeholders in this process will require us to continue to advance and improve on the delivery, implementation and evaluation of these activities. Short course and MSc module materials are openly available to facilitate capacity building.

Short course recruitment methods for each country and stakeholder group

	Mental health researchers	Policymakers and planners	Service users and caregivers
Ethiopia	Advertised within Addis Ababa University and targeted invitations to researchers from regional universities	Mental health focal staff from the MoH and regional efforts to scale up mental healthcare	Identified from recipients of integrated primary mental healthcare in Sodo district, in collaboration with the district health office
India	Advertised within Public Health Foundation of India and Sangath	Ongoing engagement with policymakers from the MoH, Government of Madhya Pradesh and members of the National Mental Health Policy Group	Representatives from national-level service user organisations, members of the PRIME Community Advisory Board Group and recipients of mental healthcare in Sehore district
Nepal	Short courses provided for researchers from local research organisations by invitation, as well as the National Health Research Council	Ongoing engagement with policymakers from the key MoH departments tasked with mental and primary healthcare, with selection of participants done by MoH	Identified from recipients of integrated primary mental healthcare in Chitwan district, taking part in the PRIME programme
Nigeria	Short courses were delivered for the three modules (mental health systems; implementation science and service user involvement). Advertised nationally during annual postgraduate research seminars with participants attending from all over the country. This was supplemented by targeted invitations to researchers from the different regions of the country	Capacity building for mental health policymakers and planners at national and regional levels. Recruitment was by targeted invitations to regional and national officials	Capacity building for service user and caregiver organisations. Recruitment was by targeted invitations to known groups from different regions of the country
South Africa	Short courses provided for researchers and clinicians focused on mental health systems and implementation science, recruited through local advertising and networks	Improved capacity for mental health service planning for provincial and district health planners, identified through existing policy and planning partnerships	Recipients of psychosocial rehabilitation in the Dr Kenneth Kaunda district in collaboration with the South African Federation for Mental Health
Uganda	This category targeted Masters-level students and psychiatrists involved in research programmes by invitation	From our engagement with key policymakers at the MoH headquarters including the national mental health focal person	By invitation, the participants were identified by the respective caregiver/service user organisations

MoH, Ministry of Health; PRIME, PRogramme for Improving MEntal health care.