Leveraging massive open online courses to expand quality of healthcare education to health practitioners in Rwanda.

Improving the quality of healthcare delivery is increasingly a global health priority. However, quality improvement training opportunities that provide theoretical foundations and basic skills for patient safety and other quality initiatives have been limited or historically out of reach, especially in low-and-middle income countries (LMICs). To address this gap, the Harvard Initiative on Global Health Quality (HIGHQ) created and launched a massive open online course (MOOC) in 2014 focused on patient safety and quality of care using the edX platform. More than 30 000 students from across 195 countries registered for the online course. This paper summarises an innovative educational partnership between the course team and one of these countries, Rwanda, to develop a blended-learning model to bolster participation in this new course among Rwandan healthcare professionals. Although a small country, Rwanda was among the top performing countries for attracting course registrants and was the leading country for the proportion of enrollees who ultimately completed the course. Further, half (21 of 42) of Rwanda's district hospitals opted to appoint a PH555x course facilitator at their site to help lead regular meetings and discussions about the course content at their facility. The majority of Rwandan enrollees were health professionals (63%) and 81% reported that PH555x was their first experience taking an online course. Among those participating in the 'flipped' component at hospital sites, 94% reported that the course helped them to think of specific ways to improve healthcare quality at their facility. In this paper, we describe this innovative public-private educational model, challenges to implementation and lessons learned that may be helpful for future MOOC developers who wish to augment learning opportunities among healthcare professionals in LMICs. © Author(s) (or their employer(s)) 2019. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Introduction

Improving healthcare quality is central to strengthening health systems and improving the health and well-being of patients worldwide.1 2 However, educational opportunities for health practitioners have been limited on this topic, especially in low-and-middle-income countries (LMICs). One increasingly used approach to address broader education access gaps in LMICs, including those for healthcare professionals, is massive open online courses (MOOCs).3–7 There is some controversy regarding how effective they have been in ensuring educational access to people who may benefit most.5 7 Yet in resource-limited settings, MOOCs have the potential to allow learners to engage with previously inaccessible content and can facilitate innovation in ‘technology-enabled pedagogy’ and other ‘blended learning’ approaches.3 5 7

To expand educational opportunities for healthcare providers, policymakers and patients on the fundamental principles of healthcare quality, Harvard University launched an edX course entitled ‘Improving Global Health: Focusing on Quality & Safety’ (PH555x) in 2014.8 This MOOC, hosted through HarvardX and created by faculty from the Harvard Initiative on Global Health Quality (HIGHQ), was the first of its kind to provide a freely accessible overview of quality of care for a global audience. In addition to the online course, the Rwandan Ministry of Health and HIGHQ partnered to augment the PH555x course modules with a blended learning educational programme for clinical practitioners based at district hospitals throughout Rwanda.

In this summary, we provide a detailed overview of the PH555x course along with overall enrolment and completion rates globally. We then describe this novel public–private academic partnership between PH555x and the country of Rwanda to both broaden awareness of the course nationally, while also developing a mechanism to host in-person course discussion groups among healthcare professionals from district hospitals. We then summarise lessons learned from implementing the PH555x Rwanda Programme that may be beneficial to other initiatives that wish to explore this type of model.

Course description

PH555x was an 8-week course organised into eight modules covering a variety of domains related to healthcare quality (table 1). Each module was released on a weekly basis and consisted of a set of 3–7 min core videos that feature a guided conversation between course faculty and guest experts, which addressed each module’s learning objectives. The videos were augmented by required and recommended readings, discussion boards, multiple choice questions and live Twitter sessions with course faculty. There were no prerequisites to enrol in the course and it was freely available through the edX platform. In terms of assessment, a student’s completion rate was based on homework (eg, multiple choice questions assessing knowledge from readings and video content integrated throughout each module) (60%), self-rated participation (eg, verifying contributions to discussion boards on a weekly basis) (20%), and a final writing assignment (20%). Students who achieved a score of at least 60% or above were eligible for a course completion certificate. Of note, there was no separate assessment scheme for the Rwanda PH555x facilitated sessions as described further below; course completion was based on meeting the aforementioned course requirements as captured on an individual edX account.

Table 1

Weekly course structure, featured experts and module learning objectives

Module	Featured course expert(s)	Learning objectives
All lessons	Ashish K. Jha, K.T. Li Professor of Global Health, Harvard University, Senior Associate Dean for Research Translation and Global Strategy at Harvard T. H. Chan School of Public Health, and Director of the Harvard Global Health Institute	Understand the following course learning objectives designed to expose participants to: The importance of focusing on quality for improving population health. A framework for understanding healthcare quality. Approaches to quality measurement. The role of information and communication technology in quality improvement. Tools and contextual knowledge for quality improvement.
Week 1: Burden	David Bates, Chief Innovation Officer, Brigham and Women's Hospital	Understand the terms burden, harm, preventability and what they mean for healthcare quality. Identify the similarities and differences in measuring quality in high-and-low-income countries. Identify quality measurements specific to ambulatory care. Discuss the frequency of medication errors as one measure of quality.
Week 2: Measurement	Jishnu Das, Lead Economist, World Bank Group Niek Klazinga, Head, Healthcare Quality Indicators Project, OECD Health Division	Understand quality variation across healthcare systems and geographies. Discuss categories of healthcare resources and how they are allocated based on geography, disease frequency, political factors and more. Understand global healthcare quality indicators. Compare how countries with and without large-scale healthcare data infrastructure can measure and understand quality. Understanding healthcare quality.
Week 3: Standards	Allen Kachalia, Chief Quality Officer, Brigham and Women's Hospital	Understand the role of the legal regulations in creating and measuring quality. Identify the ways in which accreditation, staffing ratios, compensation, deterrence and negligence impact healthcare quality. Discuss practitioner licensing across geographies. Compare patient compensation for healthcare mistakes across geographies.
Week 4: Quality improvement	Maureen Bisognano, President Emerita and Senior Fellow, Institute for Healthcare Improvement Agnes Binagwaho, Vice Chancellor, University of Global Health Equity and Former Minister of Health of Rwanda	Understand the Plan Do Study Act framework for implementing and evaluating healthcare quality standards. Identify key characteristics in healthcare leadership. Understand the role of accountability in healthcare quality.
Week 5: Health information technology and data	Julia Adler-Milstein, Associate Professor, University of Michigan	Understand a Health Information Exchange. Discuss the role information technology plays in gathering, storing and analysing healthcare data. Identify the role and utility of electronic health records. Understand how health data is gathered, stored and shared across geographies.
Week 6: Management	Raffaella Sadun, Associate Professor, Harvard Business School T.S. Ravikumar, Former Director and CEO, Jawaharlal Institute of Postgraduate Medical Education and Research	Understand the role that managers can plan in implementing and improving quality and patient outcomes. Identify the ways in which the quality of management itself can be measured. Discuss the Virginia Mason Hospital system as an example of management and improving healthcare quality. Discuss the example of the Jawaharlal Institute of Postgraduate Medical Education (JIPMER) and its role in improving quality.
Week 7: Role of patients	Ronen Rozenblum, Director of Business Development, Centre for Patient Safety, Research, and Practice, Brigham and Women's Hospital Felix Greaves, Deputy Director, Science and Strategic Information, Public Health of England	Understand the meaning of patient-centred care. Identify ways in which the patient experience can be measured and therefore improve health outcomes. Discuss process improvements that can lead to better patient-centred care. Understand hospital and doctor ratings online. Identify the impacts that social media can have in shaping attitudes about a healthcare provider.
Week 8: Public systems	Julio Frenk, Former Dean, Harvard T.H. Chan School of Public Health Ed Kelley, Director, Department of Service, Delivery, and Safety, WHO	Discuss healthcare reform in Mexico as an example of government influencing health outcomes. Understand the role the WHO plays in measuring and contributing to health outcomes.

While basic principles of quality hold across contexts, distinct barriers to assessing and improving quality have been identified in LMICs.9 It was therefore imperative for the course to involve experts in the measurement and improvement of quality in such settings. In particular, Rwanda—a country of 12 million people in East Africa—was identified as a national case study for the PH555x course given the health sector’s well-documented health gains over the past two decades.10 11 Despite being devastated in 1994, Rwanda’s health sector emerged as an outlier in their achievement of health-related Millennium Development Goals, including those focused on maternal health, infant mortality and the burden of HIV/AIDS.12–14

Launching the Rwanda-Harvard PH555x collaboration

When making plans to feature Rwanda’s Minister of Health as a guest expert in the overall PH555x course, this spurred conversation regarding how to optimise the availability of the PH555x content to a key audience of interest: health professionals at Rwanda’s district hospitals. One of Rwanda’s rural district hospitals had prior experience in leveraging a ‘flipped classroom’ model15 through a separate edX course, which focused on the social determinants of health.16 The Ministry of Health (MOH) wished to build on this experience and engage health practitioners more broadly with the content proposed by the PH555x course as this was well aligned with the country’s health sector priority to build capacity in quality improvement among its healthcare professionals.

Similar to other blended learning approaches designed to promote capacity building, the Rwanda-Harvard PH555x collaboration sought to complement free online content with high value discussion groups at the hospital level, promoting context-specific learning.3 4 7 The aim was to foster opportunities for engagement with the online supplementary course material in a more learner-centred model that allowed for multidirectional discussion between different types of clinical practitioners working to improve the quality of care in Rwanda.

To achieve this desired engagement, prior to the PH555x online course launch, the MOH sent an official invitation to all individuals working in Rwanda’s health sector to encourage them to enrol. Further, the MOH specifically asked the leaders of its 42 district hospitals to consider naming an active member of the hospital’s Quality of Care Committee as the site facilitator. The facilitator was asked to promote the course to hospital staff, coordinate optional, in-person discussion groups that would allow for members at that hospital to discuss the course content together, and arrange relevant local speakers or experts to expand on the content and discuss ongoing quality improvement programme or challenges at their site. To mitigate anticipated barriers to participation for clinical leaders, approval was obtained from Rwanda’s Medical and Dental Council so that all who received a course completion certificate were eligible to obtain Continuing Professional Development (CPD) credit.

To support the initiative, a PH555x Teaching Fellow with graduate level training in public health and health policy served as a liaison with the MOH and all district hospital PH555x facilitators. Funding was available through edX to support the Teaching Fellow’s time in supporting the overall course discussion boards (as is generally done across other edX courses). For the Rwanda programme in particular, the Teaching Fellow also supported the hospital course facilitators by distributing weekly sample session guides (online supplementary file 1), sending reminders regarding course milestones and requirements, sample discussion prompts and helping connect learners with any edX platform-related questions to the edX technical staff. All of these electronically available materials, which were provided both in email as well as through a shared drive folder to all named facilitators, were provided with the expectation that facilitators would customise the session to add value and context specific to their clinical setting and participants.

Overall, enrollees from Rwanda had two potential ways of participating in the PH555x course: (1) they could opt to participate in the freely available edX course on their own time, as was true for any other enrollee, and thus had no exposure to the discussion groups facilitated at the district hospitals, or (2) participate in hospital-based discussions as organised by the site facilitator throughout the 8-week course and complete whatever aspects of the programme that interested them.

Programme outcomes

For this article focused on an innovative public–private educational partnership, the key outcomes of interest include overall PH555x course enrolment and completion rates globally, which serve as a proxy for engagement with course content. In addition, for the Rwanda initiative specifically, we summarise district hospital participation levels within Rwanda as well as characteristics of participants who were in communication with the site-based facilitators.

Overall course enrolment and completion rates

A total of 30 150 students from 195 countries enrolled in PH555x for the course’s inaugural launch in 2014. The majority of these enrollees came from the United States (USA) (n=7635). Rwanda ranked 13th of all countries with the number of registrants (n=396) (table 2) and first in terms of ratio of registrants to overall population.

Table 2

Top 20 list for course registration, completion and population size, by country

edX Rank*	Country	Number registered	Number viewed	Number certified	% Certified(of registered)	% Certified (of viewed)	Population size†	Size rank†
1	USA	7635	2343	330	4.3	14.1	318 892 103	4
2	India	1899	655	116	6.1	17.7	1 236 344 631	2
3	UK	1140	379	53	4.6	14.0	63 742 977	23
4	Brazil	1084	460	30	2.8	6.5	202 656 788	6
5	Canada	979	320	44	4.5	13.8	34 834 841	38
6	Mexico	911	384	33	3.6	8.6	120 286 655	12
7	Nigeria	685	225	53	7.7	23.6	177 155 754	8
8	Saudi Arabia	602	222	37	6.1	16.7	27 345 986	47
9	Australia	581	187	35	6.0	18.7	22 507 617	56
10	Colombia	545	191	16	2.9	8.4	46 245 297	30
11	Egypt	423	148	15	3.5	10.1	86 895 099	16
12	Japan	421	158	8	1.9	5.1	127 103 388	11
13	Rwanda	396	290	179	45.2	61.7	12 337 138	74
14	Peru	389	129	8	2.1	6.2	30 147 935	43
15	Pakistan	382	118	19	5.0	16.1	196 174 380	7
16	Germany	378	154	20	5.3	13.0	80 996 685	18
17	Turkey	368	151	11	3.0	7.3	81 619 392	17
18	Spain	345	133	25	7.2	18.8	47 737 941	29
19	Philippines	330	114	16	4.8	14.0	107 668 231	13
20	China	304	110	10	3.3	9.1	1 355 692 576	1

*edX rank by course enrolment (only top 20 countries provided but additional data available on request).

†Population size and size rank were obtained from CIA World Factbook; 2014 estimates presented given course launch of that same year. Available at: https://www.cia.gov/library/publications/download/download-2014/index.html (rank#2119).

Among countries with the greatest number of course enrollees, Rwanda ranked first worldwide in terms of the proportion of enrollees who received course completion certificates based on the assessment guidelines noted above. A total of 45.2% (179 of 396) of Rwandan students completed the course (table 2). Since some enrollees may have never started the course, however, we examined completion rates among those who ‘viewed’ the course at least once. Rwanda’s completion rate on this metric (percent of students certified who viewed the course) was 61.7% (179 of 290). As a comparison, only 14.1% of those who registered and viewed the course at least once from the USA actually completed it. Rwanda’s completion rate on either metric is markedly higher than the typical completion rate for edX courses and for the majority of MOOCs overall (generally less than 10%).17 Of note, some countries not featured in table 2 had higher completion rates among students that viewed the course (ie, Armenia (n=15 students; 86% completion rate)); however, these countries had only a small number of enrollees in the course overall (<20 students).

District hospital participation and Rwanda programme enrollees

The Rwandan MOH invited all of its 42 district hospital leaders to promote widespread enrolment in the course throughout Rwanda, as well as consider naming a course facilitator at their site (figure 1). In total, half of Rwanda’s district hospitals (21 of 42) opted to name a PH555x course facilitator. In terms of activities, some facilitators hosted weekly gatherings with local speakers, food and other discussion-based learning experiences. Other sites met together only a few times over the duration of the course, but facilitators would regularly engage their colleagues by sending weekly course reminders through email and other text-based communications (eg, What’sApp, SMS).

Figure 1

Location of Rwandan registrants in the PH555x course.

A total of 140 individuals completed an anonymous pre-course survey that was distributed by Rwanda’s hospital-based facilitators to their contacts who were considering participating in the discussion groups at the hospital sites. The majority of respondents (63%) were healthcare providers, yet participants spanned a variety of disciplines, including hospital administrators (8%), researchers (6%) and data analysts (3%). The majority (81%) of respondents indicated that PH555x was their first exposure to an online course.

At the conclusion of the course, a total of 47 individuals completed the anonymous post-course survey, which hospital facilitators circulated to their contact lists. It was not possible to delineate if these individuals actually participated in the hospital-based discussions, but all had at least interacted with the course to some degree. Among these post-course survey respondents, 94% agreed (with 81% strongly agreeing) with the statement that the course helped them to think of ‘specific ways to improve the quality of care at my workplace’. In terms of next steps, qualitative comments from both individual participants and the facilitators suggested that the majority of participants hoped to continue their discussion groups at their hospital, engage with more trainings related to specific quality improvement projects at their healthcare facility and exchange in dialogue regarding quality improvement with other hospitals throughout the country.

Lessons learned

The successful enrolment and completion rates among Rwandan enrollees in PH555x—regardless of whether or not they participated in a discussion-based group at a district hospital—suggests the value of identifying innovative public–private educational partnerships to broaden access to the basic tenants of quality improvement. In table 3, we summarise key facilitators and barriers to optimising engagement among Rwandan healthcare professionals in PH555x.

Table 3

Barriers and facilitators to optimising PH555x course participation in Rwanda

Barriers to participation

Limited time window to prepare for national roll-out Internet connectivity Competing time demands on facilitator given normal work duties to complete Language barriers (the course was not provided in French or Kinyarwanda) Voluntary nature of programme

Facilitators to participation

Promotion of course to health sector by Ministry of Health Appointment of facilitators at district hospital level Provision of alternatives to enhance access to content when internet connectivity barriers were experienced (eg, provide transcripts of video text, audio files or option for smaller/low-quality videos) Appointment of course liaison (teaching fellow) to serve as point of contact for facilitators and to help create weekly session guides Incentive to complete course for local continuing professional education licensing Integration of course content with ongoing health sector initiatives to improve quality and safety

Regarding the educational partnership between HIGHQ and Rwanda’s MOH, the following underlying principles were helpful for ensuring the launch of this collaboration.

Alignment with national priorities and standards

First, the content of PH555x was relevant to Rwanda’s national policy priorities to improve quality and safety in the health sector. Concurrent to the course launch, district hospitals were actively engaged in Rwanda’s hospital accreditation process, which was a key tool for quality improvement discussed in the course.18 Although this was a motivating experience that had strong alignment with the course content, it would have been helpful to provide enrollees the option of participating in the course beyond the designated 8-week period as they faced this competing although complementary focus on accreditation. As such, future MOOC providers that wish to partner with the public sector to promote broadened access should identify ways that the course content is aligned with national objectives and can actively support existing policies and strategies, while also being sensitive to time demands on relevant stakeholders.

Leveraging online course content to promote local capacity building

Second, the objective of broadening the accessibility of knowledge—a foundational mission within MOOC learning and other blended learning models in LMICs7 15—has strong alignment with Rwanda’s commitment to building local capacity and promoting professional development within the health sector. Since each district hospital already had a quality committee at the time of the course roll-out, members from this committee were natural candidates to serve as facilitators and motivators for recruiting colleagues at their hospital to participate in the course. Further, physicians had the incentive to participate as PH555x was deemed eligible for CPD credit. The MOH wanted all participants to see the unique role that they—whether they be an administrator to physician—had in improving quality of care at their site. Fortunately, the post-course survey suggested that the course had this desired effect in helping Rwandan health practitioners complete the course feeling as though they had specific ideas on how to improve healthcare quality at their workplace. Although not within the scope of this article, future impact evaluation work would be helpful that demonstrates whether changes in hospital-based process and outcome quality measures occurred in those hospital settings that engaged with the course content versus those that did not.

Flexibility and pragmatism to reduce logistical barriers and to promote access to content

Third, as has been reported in other studies related to launching MOOCs in a resource-limited setting, it was important to be flexible to modify course content in real time in order to enhance access.7 16 A key logistical challenge was that of limited, regular internet connectivity at the district hospitals. Although Rwanda has a robust information and communications technology (ICT) infrastructure, a reliable, high-capacity internet connection was not universally available at all times and across all participating sites when the course launched in 2014. Among those hospitals that did participate, a number reported that internet connectivity was a barrier to accessing the course content as originally presented on the edX platform as the default system required more bandwidth than was commonly available. To mitigate this barrier, the HarvardX course team provided hospital facilitators with lower resolution videos, audio files, video transcripts, and—in the very few cases it was requested—flash drives containing all course content—so their colleagues could proceed through the course content over that 8-week timeline. Future MOOCs that wish to broaden access across settings that struggle with limited internet connectivity (not only in terms of availability but speed to download large files) should consider these sorts of creative alternatives to accessing content with their partners at the outset.

Encouragement of local experts to discuss their perspectives on a particular course topic

Fourth, this partnership encouraged greater buy in among enrollees by allowing for the local context to influence how the content was delivered. By encouraging district hospitals to host these discussion groups and customise the content related to that week’s course module, they were encouraged to coordinate meetings with key leaders at their hospital site to discuss current, locally relevant updates that directly related to the module of interest. For instance, for the session on electronic medical records, the facilitator at Rwinkwavu District Hospital invited their site’s information and technology experts to comment on this dimension of healthcare quality.

Limitations and next steps

Although we believe this PH555x Rwanda programme represents a novel public–private educational partnership to broaden desired healthcare quality content to Rwandan health practitioners, there are a number of limitations to consider. First, this programme was implemented in a single country that had prior experience (although less extensive) with an edX course,16 thus it is not possible to assume that this approach is generalisable across other settings. However, it is our hope that future MOOC developers with an interest in partnering with LMICs to expand health practitioner engagement in relevant course topics may find this model useful for their own experience and encourage them of the value of liaising with relevant governmental leaders when attempting a national roll-out. Second, we acknowledge that the outcomes of interest in this innovation education report, namely course enrolment and completion, cannot be assumed as markers of skill, knowledge or have an impact on improving the focus area of PH555x: healthcare quality. Although it does not capture the totality of all Rwandan participants and cannot fully be attributed to the hospital-based component of the course, it was nonetheless encouraging to see that the large majority of respondents felt that the course provided them with specific ideas on how to improve quality at their workplace. Yet, as previously mentioned, future research in this space should aim to measure actually differences in quality outcomes as a result of course participation. Third, in terms of evaluating the impact of the ‘flipped’ component of PH555x at the district hospitals versus non-exposed Rwandan participants, we were unable to analyse the differences between these potentially interesting comparison groups for a variety of reasons. This included the fact that the overall goal when creating this national programme was not to test learning models but rather to increase exposure to the course via any route possible. Further, the flexible and variable nature of participation at the hospital-based sites and lack of granular data among course completers precluded us from doing a post-hoc analysis of these subgroups. As such, future MOOCs that strive to engage LMICs in this type of blended learning model and wish to have a more precise understanding of the impact of their flipped model should aim to design the course in a way that captured the desired measures for this level of analysis.

In summary, a unique educational partnership between HarvardX and the Rwanda MOH helped to bolster health practitioner participation in an MOOC focused on patient safety and quality. This was accomplished by strategic promotion of the course throughout the health sector, as endorsed by the MOH, and efforts to coordinate a complementary hospital-based course discussion group programme that half of Rwanda’s district hospital sites opted to participate in over the 8-week course. Lessons learned from this partnership can be used to improve future initiatives that strive to bolster LMIC access to relevant course content. Further, this partnership demonstrates the potential for future MOOC courses to collaborate with the public sector to expand exposure to relevant knowledge and skills, including those focused on the goal of improving healthcare quality, to health professionals and beyond.