The impact of an mHealth monitoring system on health care utilization by mothers and children: an evaluation using routine health information in Rwanda.

Maternal and child mortality rates remain unacceptably high globally, particularly in sub-Saharan Africa. A popular approach to counter these high rates is interventions delivered using mobile phones (mHealth). However, few mHealth interventions have been implemented nationwide and there has been little evaluation of their effectiveness, particularly at scale. Therefore, we evaluated the Rwanda RapidSMS programme-one of the few mHealth programmes in Africa that is currently operating nationwide. Using interrupted time series analysis and monthly data routinely reported by public health centres (n = 461) between 2012 and 2016, we studied the impact of RapidSMS on four indicators: completion of four antenatal care visits, deliveries in a health facility, postnatal care visits and malnutrition screening. We stratified all analyses based on whether the district received concurrent additional supports, including staff and equipment (10 out of 30 Districts). We found that community health workers in Rwanda sent more than 9.3 million messages using RapidSMS, suggesting the programme was successfully implemented. We found that the implementation of the RapidSMS system combined with additional support including training, supervision and equipment provision increased the use of maternal and child health services. In contrast, implementing the RapidSMS system alone was ineffective. This suggests that mHealth programmes alone may be insufficient to improve the use of health services. Instead, they should be considered as a part of more comprehensive interventions that provide the necessary equipment and health system capacity to support them.

Key Messages

RapidSMS Rwanda was widely implemented and used to track pregnant mothers and their infants.

The system increased the use of some health services. This only occurred, however, where the programme was augmented with continuous training, supervision and the provision of medical equipment.

MHealth interventions may require infrastructure, equipment and health system capacity to have their desired impact.

Introduction

Poor maternal and child health outcomes remain a major concern in many regions (World Health Organization, 2015). In fact, while the adoption of the millennium development goals in 2000 was followed by a reduction in child and maternal mortality, rates of maternal and infant mortality remain unacceptably high, particularly in sub-Saharan Africa (Requejo and Bhutta, 2015; World Health Organization ). For example, the World Health Organization has estimated that in 2015 almost 70% of maternal deaths and 50% of child deaths occurred in the region (United Nations Children’s Fund ; World Health Organization ). This has increased interest in policies and programmes that can increase utilization of key maternal and child health services.

One popular approach has been maternal and child health interventions delivered using mobile phones; so-called ‘mHealth’ interventions (Agarwal and Labrique, 2014; GSM association, 2016; Kruk ). This has been driven by the increasing penetration of mobile phones in low and middle-income countries. Given that mobile technology is accessible by >95% of the global population, mHealth interventions can potentially reach many individuals, including those living in settings with limited infrastructure (Kruk ; Rebecca ). These interventions have aimed to collect data, change behaviour, manage logistics and improve quality of care (African Strategies for Health and Management Sciences for Health, 2016; GSM association, 2016; Labrique ). Globally, a number of mHealth interventions have been piloted, and to date the majority of projects in Africa have focussed on maternal and child health (African Strategies for Health and Management Sciences for Health, 2016).

Globally, there have been limited rigorous evaluations and mixed evidence on the performance and effectiveness of mHealth interventions. A randomized trial conducted in Zanzibar found that an mHealth intervention combined with a voucher increased the percentage of pregnant women who completed four antenatal care (ANC) visits (Lund , 2014). Similarly, a quasi-experimental study in Bangladesh found that sending SMS reminders to mothers living in hard to reach areas significantly improved vaccination coverage (Uddin ). Beyond these examples, however, a systematic review has questioned the methodological quality of the existing body of research on mHealth programmes (Lee ; Sherri, 2016). In addition, we know of no evaluations of an mHealth programme that has been scaled nationwide. Thus, there is a paucity of real-world evidence on the impact of mHealth programmes.

Rwanda has used mHealth interventions to try and improve maternal and child health outcomes. The country has seen a considerable improvement in maternal and child health services use since 2000 (Farmer ). For example, the most recent demographic and health survey found that 91% of pregnant women delivered at a health facility [National Institute of Statistics of Rwanda (NISR) ]. There remain, however, significant opportunities to improve care-seeking behaviour in Rwanda by pregnant women and new mothers. For example, in 2014–15, only 44% of pregnant women complete the recommended four ANC visits prior to giving birth [National Institute of Statistics of Rwanda (NISR) ]. With mobile network coverage reaching 90% of the population, Rwanda scaled up an mHealth system known as RapidSMS nation-wide. At the same time, additional support for maternal and child health was rolled out in several districts. Although the system and this support have been in pace for several years, there has been no rigorous evaluation to date. Therefore, we conducted a longitudinal study to retrospectively evaluate the impact of RapidSMS both with and without this additional support on the use of important maternal and child health services.

Methods

Study intervention

The development and piloting of the RapidSMS system has been discussed elsewhere (Ngabo ). The RapidSMS system is a two-way communication system between community health workers (CHWs) and the Ministry of Health. CHWs are the first point of contact with the health system and create a link between communities and health facilities. Each village in Rwanda had three elected CHWs, one of whom oversees maternal and child health, and all of whom could access the system. In brief, CHWs in Rwanda were given mobile phones to report data on maternal and child health indicators using text messages. RapidSMS data were collected during pregnancy, and from birth until 2 years of age, and included a number of indicators: ANC, delivery, maternal mortality, postnatal care, anthropometric measurements and child mortality. The system generated automatic reminders for clinical appointments that were sent to the CHW, including ANC, the probable delivery date and postnatal care, with the aim of increasing routine care attendance and the proportion of health facility deliveries. Mothers were not messaged directly. RapidSMS was also designed to quickly link mothers to emergency obstetric care by notifying ambulance services (Ngabo ).

Simultaneously with this national scale up, UNICEF, using a Korea International Cooperation (KOICA) Grant, provided more comprehensive support to complement RapidSMS in 10 of 30 districts. These districts were selected based on poor maternal and child health indicators that were available at the district level (such as the facility delivery rate and infant mortality) and their distance to the capital, Kigali. This programme included the recruitment of two NGOs to support districts with the implementation and use of RapidSMS, ongoing training sessions for CHWs on providing home-based care, and providing supervision meetings with health centres every calendar quarter. It also included the provision of equipment to District Hospitals and Health Centres for treating newborns, including CPAP machines, radiant warmers, bed nets, suction devices and other supplies.

Data sources

We used data from two sources: the database of messages sent by CHWs, and the Rwanda Health Management Information System (HMIS), which contains routinely collected facility-level data on the provision of maternal and child health services. We used data on all messages sent by CHWs through June 2016 from the RapidSMS database. Data in the HMIS system are collected from each health facility in Rwanda by a designated individual, leading to a very high rate of data completeness (Nisingizwe ). As the monthly reporting forms for the HMIS system were substantially changed in January 2012, we used data from that date through June 2016.

Outcome measures

We studied the impact of RapidSMS on the following process of care indicators using outcomes data from the HMIS database:

Receipt of four recommended ANC visits: the number of mothers who completed the standard schedule of ANC visits.

Deliveries in a health facility: the number of births taking place in a public health facility.

Total postnatal care visits: the number of mothers attending any of the three standard postnatal care visits.

Child malnutrition screenings: the number of children screened for malnutrition using biometric measurements of height, weight and middle upper arm circumference.

Each of these indicators was reported monthly by each health facility. To allow comparisons between districts and account for population growth over time, we calculated per-capita monthly rates using estimated catchment populations for each health centre derived from the 2012 Census data.

Statistical analysis

We used interrupted time series analysis, one of the strongest quasi-experimental research designs (O’Keeffe ; Moscoe ). This method has the distinct advantage of accounting for pre-existing trends in the outcomes, which are very common in the above indicators (Wagner ). As RapidSMS was scaled-up at different times across Rwanda, we determined the month in which CHWs from each health facility first sent 50 or more messages. For each facility, we then obtained data from HMIS for 14 months prior. We chose this time length to include as many facilities as possible, as scale-up started in March 2013. Of the 481 health centres in the HMIS system, 461 health centres (96%) had data available for at least 24 months after they initiated use of the system. We excluded the 20 health centres that did not cross the 50 messages threshold before July 2014 as we did not have adequate post-intervention data. All analyses were thus conducted using ‘study time’, which is the number of months relative to each health centre’s index month.

Our interrupted time series models took the following form:

Where ‘time’ represented the month in study time (i.e. 1, 2, 3…), ‘RapidSMS’ indicated whether the health facility had implemented RapidSMS, and ‘post’ represents the number of months since implementation in month t. Any immediate change in the outcome would be captured by β and any change in the ‘trend’ of the outcome after RapidSMS initiation would be captured by β.

We stratified our analysis based on the 10 UNICEF-supported districts and the other 20 districts within Rwanda. For each outcome, we included only health facilities that reported complete data at every time point. This included nearly all health centres for ANC, deliveries and malnutrition screenings (430, 426 and 426, respectively) and fewer for postnatal care visits (88). As our observations may have been correlated over time, we used a generalized least squares model with a first-order autoregressive structure (Wagner ). In order to aid interpretation of rate changes, we used established methods to calculate percentage changes at one year following RapidSMS implementation (Zhang ).

Results

Use of the RapidSMS system

Between 2012 and June 2016, >45 thousand CHWs sent nearly 9.4 million unique SMS messages, for an average of 205 per CHW (median: 166, inter-quartile range 110–253). RapidSMS messages provided information on >2.5 million mothers. As shown in Figure 1, the number of messages per capita was fairly similar between supported and non-supported districts, with a rate of around 25 messages per 1000 inhabitants after scale-up.

Figure 1.

Average number of RapidSMS messages per 1000 catchment population in both UNICEF-supported and non-supported health centres

We compared the number of deliveries recorded in RapidSMS to both the HMIS data and the most recent Census projections to assess the completeness of reporting by CHWs. In 2015, RapidSMS captured 192 264 births. In that same year, there were 311 376 births in health facilities reported in HMIS. As about 91% of births occurred in a health facility, this would suggest a total of 342 171 births in 2015. This is very comparable to the Census projections of 343 077 total births in 2015 [National Institute of Statistics (NISR) Rwanda, 2014]. Overall, both figures suggest that RapidSMS captured about 56% of all births in 2015.

ANC visits

The rate of pregnant women who completed the recommended regimen of four ANC visits is shown in Figure 2. Although rates of attendance increased across the study period, we found no change in either the level or trend in both supported and non-supported districts (P = 0.51 and 0.70 for supported and 0.38 and 0.50 for non-supported districts).

Figure 2.

Interrupted Times Series analysis of receipt of four standard ANC visits per 1000 catchment population in supported and non-supported Districts

Health facility deliveries

The rates of delivery in a health facility are shown in Figure 3. The declining trend in facility deliveries is consistent with other data suggesting a declining fertility rate in Rwanda over our study period [National Institute of Statistics of Rwanda (NISR) ]. As shown in Figure 3, the start of RapidSMS with additional support did not change the level of health facility deliveries, but we did find an increase in the trend (estimate = 0.015, 95%CI: 0.007–0.023, P < 0.001). Together, this represents an 18% increase at 1 year. In comparison, we found no change in either the level or trend in non-supported districts (P = 0.69 and 0.70, respectively).

Figure 3.

Interrupted Time Series analysis of facility deliveries per 1000 catchment population in supported and non-supported Districts

Postnatal care visits

We found a differential impact of RapidSMS on postnatal care visit rates in supported vs non-supported districts. In supported districts, we found that the start of the RapidSMS programme and the additional support was associated with a 0.11 visits/1000 catchment population increase in the level of PNC visits (95% CI: 0.033–0.179, P = 0.007). The change in the trend was also positive, but was not statistically significant (P = 0.11). Taken together, these estimates represent a 100% increase at 1 year over existing trends. In contrast, we found no change in the rate of PNC visits in non-supported districts (P = 0.13), coupled with a small drop in the trend of −0.005 PNC visits/1000 catchment population/month (95% CI: −0·009 to −0·002, P = 0·007) (Figure 4).

Figure 4.

Interrupted Time Series analysis of total PNC visits per 1000 catchment population in supported and non-supported Districts

Malnutrition screening

Similar to the other outcomes, we also found an impact of RapidSMS coupled with additional support on malnutrition screening. Figure 5 shows that rates of malnutrition screening demonstrated no level change (P = 0.47), but we found a decrease in the trend of 0.180 after the RapidSMS programme with additional support started (95% CI: 0.121–0.240, P < 0.001). In contrast, we found no change in either the level or trend of malnutrition screening rates in non-supported districts (0.29 and 0.54).

Figure 5.

Interrupted time series (ITS) analysis of total PNC malnutrition screenings per 1000 catchment population in supported and non-supported District

Discussion

In recent years, mobile health programmes aiming to improve maternal and child health have been implemented in many countries. Our findings indicate that the RapidSMS system in Rwanda was extensively used, as millions of SMS messages were sent over study period. Despite this widespread use, our analysis only found evidence of an impact on the use of key maternal and child health services where an additional support package consisting of training of CHWs, equipment and supervision were provided. As both RapidSMS and the additional supports were simultaneously implemented in supported districts, we cannot attribute the improvements to the RapidSMS programme alone, but rather to the intervention package as a whole.

We feel this difference likely resulted from the non-supported Districts lacking the health care capacity for RapidSMS alone to impact outcomes. For example, prior qualitative research with CHWs in Rwanda has suggested that a lack of training and a lack of supervision were both barriers to providing effective care in the community (Condo ). Similarly, other research examining Rwandan District hospitals has found the some equipment necessary for providing paediatric emergency care (Hategeka ). This leads to the following conceivable interpretations: One, it might be that the RapidSMS system required additional support to be effective. Two, it might be that RapidSMS was ineffective and the increase in facility deliveries was solely the result of the additional support. Put another way, mHealth initiatives may not be successful in contexts lacking the necessary health system capacity to ensure their effective use. However, an mHealth programme designed to improve vaccination coverage in Bangladesh was successful despite the fact that they did not provide additional support to health facilities (Uddin ). The main difference between these interventions, however, was that SMS reminders were sent to CHWs in Rwanda and directly to mothers in Bangladesh.

Notably, although RapidSMS was originally designed to monitor pregnancies, this is where the impact of the programme was most limited. We believe that this is likely the result of late pregnancy detection, which limits the possibility of completing all four recommended ANC visits. This would not change because of RapidSMS reminders, as it would be too late to complete missed appointments. Further, our results showed an increase in the trend of facility deliveries in supported districts, but not in non-supported districts. It is worth noting here that the intervention increased health facility delivery rates despite the fact that rates in Rwanda are already quite high at 91%.

Our finding of an increased rate of PNC visits is also particularly important, as recent estimates indicate that 55% of pregnant women in Rwanda do not have any [National Institute of Statistics of Rwanda (NISR) ]. We also found increases in nutritional screening in supported districts. This finding is possibly linked to the provision of equipment, as the CHWs in supported districts were provided with the equipment necessary to take the required measurements. It is unclear, however, whether this increase in malnutrition screening will lead to better health outcomes, as effects would likely take numerous years to accrue and there are several other ongoing nutrition-focussed initiatives in Rwanda (Ministry of Local Government ).

Limitations

There are a number of limitations to our analysis that merit mention. First, the supported districts were not randomly selected, so differed in their pre-existing outcomes. It is worth noting, however, that this non-random selection would not have biased our interrupted time series analysis, as it accounts pre-existing level and trend differences in outcomes between these areas. Given our use of routinely collected data, our selection of indicators was limited to the data available in HMIS. Thus, we were unable to analyse some important indicators such maternal and child mortality. In addition, while the HMIS system produced fairly consistent data estimates over time, there might be unknown data issues. However, as those indicators has been measured the same way over time, any upward or downward bias would likely not impact our conclusions. Also, given our study period, we can only detect changes that would have taken place in the 24-month period following the implementation of RapidSMS in each health centre. Finally, it is possible that our interrupted time series analysis could have been biased by other interventions being implemented at the same time as RapidSMS. However, as RapidSMS was scaled-up over the course of many months, such an intervention would have to have been closely timed in every health centre. We are unaware of any such interventions, nor were any of the participants in a stakeholder engagements workshop we organized as part of this evaluation.

Conclusion

We found that the implementation of the RapidSMS system was effective at increasing the use of maternal and child health services when combined with additional support including training, supervision and equipment provision. In contrast, implementing the RapidSMS system alone was ineffective. This suggests that while mHealth interventions such as RapidSMS might function as a component of a successful package intervention, it also suggests that there are limitations in their ability to promote change independently if the necessary health system capacity is not present. In the future, the design of mHealth initiatives such as RapidSMS should be incorporated into more comprehensive interventions that also provide the necessary equipment and health system capacity to support them. What remains unclear, however, is whether RapidSMS was a necessary component of the impact we observed for the supported districts. As we were limited in our ability to assess this given the way in which the implementation was rolled out in Rwanda, future research should examine whether this is the case.

Funding

This study was funded by a research grant from United Nations Children's Fund (UNICEF) Rwanda. A steering committee composed of the University-based Researchers, UNICEF representatives, and the Rwanda Ministry of Health provided input from the inception through to publication of this analysis. However, final decisions on the study design, data collection and analysis, the decision to publish, and the content of the manuscript were made by the University-based researchers. Dr. Law received salary support through a Canada Research Chair and a Michael Smith Foundation for Health Research Scholar Award.

Conflict of interest statement. H.R., A.M., J.N. and E.N. had a portion of their salary paid from the grant. E.G. is employed by the Rwanda Ministry of Health. A.B. is employed by UNICEF.