Evaluation of a social franchising and telemedicine programme and the care provided for childhood diarrhoea and pneumonia, Bihar, India.

OBJECTIVE: To evaluate the impact on the quality of the care provided for childhood diarrhoea and pneumonia in Bihar, India, of a large-scale, social franchising and telemedicine programme - the World Health Partners' Sky Program.
METHODS: We investigated changes associated with the programme in the knowledge and performance of health-care providers by carrying out 810 assessments in a representative sample of providers in areas where the programme was and was not implemented. Providers were assessed using hypothetical patient vignettes and the standardized patient method both before and after programme implementation, in 2011 and 2014, respectively. Differences in providers' performance between implementation and nonimplementation areas were assessed using multivariate difference-in-difference linear regression models.
FINDINGS: The programme did not significantly improve health-care providers' knowledge or performance with regard to childhood diarrhoea or pneumonia in Bihar. There was a persistent large gap between knowledge of appropriate care and the care actually delivered.
CONCLUSION: Social franchising has received attention globally as a model for delivering high-quality care in rural areas in the developing world but supporting data are scarce. Our findings emphasize the need for sound empirical evidence before social franchising programmes are scaled up.

Introduction

Diarrhoea and pneumonia are leading causes of childhood morbidity and mortality despite being targeted by global investment in health for decades. In the last 5 years, these diseases were responsible for almost 25% of deaths among children aged 1 to 4 years worldwide. In India, nearly 500 000 children younger than 5 years died from diarrhoea or respiratory infections in 2013. These poor health outcomes were partially due to the low quality of care in both public and private health sectors, including absenteeism, poor knowledge and the know–do gap, i.e. the gap between knowledge of appropriate care and the care actually delivered.–

Efforts to improve the quality of primary care in developing countries have focused on a variety of strategies, ranging from training and performance incentives to organizational innovations in the private sector and the use of new technologies, such as telemedicine. One prominent approach is social franchising, which is similar to commercial franchising. The aim is to improve a socially desirable outcome, such as health, while generating sufficient revenue to be self-sustaining. In social franchising, a franchisor offers a standardized, branded set of products or services through franchisees who pay a subscription fee to join the franchisor’s network. Franchisees, who are typically existing local providers, in turn, receive training and follow service delivery protocols established by the franchisor. They also benefit from marketing, branding, supply chain management and diagnostic services organized by the franchisor. Many franchisors use new information technologies to improve the efficiency and coordination of their network of franchisees. However, despite the rapid growth of social franchising in developing countries, there is little evidence that it affects the quality of care provided on a large scale,– except for a recent programme in Myanmar where the introduction of social franchising was accompanied by a substantial increase in the number of health-care workers.,

In this study, we investigated the effectiveness of the World Health Partners’ Sky Program in improving the knowledge and performance of health-care providers in Bihar, India. In addition to its effect on participating health-care providers, it was hoped that the programme would also improve the performance of other local providers through spillover effects and by encouraging competition. Understanding of the programme’s effect on the quality of care could help to explain why a programme in rural Bihar costing over 23 million United States dollars (US$) failed to increase the appropriate treatment of childhood diarrhoea or pneumonia.

The programme was originally developed as a hub-and-spoke model: SkyHealth providers with telemedicine facilities at the hub would link to peripheral rural health providers with smaller, more basic facilities (called SkyCare providers). SkyCare providers typically offered basic primary care and symptom-based treatment and could refer patients to SkyHealth providers. SkyHealth providers had access to telemedicine technology that was able to connect physicians at the World Health Partners’ central medical facility in Delhi to patients in rural areas via an audiovisual interface. Using this technology, physicians were able to examine patients by auscultation, assess their blood pressure and pulse and obtain electrocardiogram results if needed. In addition, SkyCare providers could offer mobile phone consultations with these physicians or refer patients to SkyHealth providers.

Although planned as a hub-and-spoke model in which SkyHealth providers would be responsible for the empanelment of SkyCare providers, the programme was instead implemented as a two-tier model without referral networks in which the two types of provider were recruited directly by World Health Partners. Existing informal health-care providers or pharmacists in rural areas were approached by World Health Partners’ field representatives, given information about the programme and offered the opportunity to join the network by paying a franchisee fee. In early 2014, the franchisee fee for a SkyHealth centre was US$ 500 in addition to an investment of approximately US$ 1000 to set up the telemedicine centre. SkyCare providers paid a franchisee fee of US$ 17 plus a small fee of US$ 0.17 for each mobile phone consultation. The training offered reflected the heterogeneity of providers in the network: SkyHealth providers received 6 days of training on the diagnosis and treatment of infectious diseases, whereas training lasted 3 days for SkyCare providers. Both types of provider received some training on protocols for basic service delivery, marketing services, diagnostic services and ensuring a predictable supply of adequate-quality drugs. SkyHealth providers were also trained in operating computers for telemedicine services.

Methods

We assessed the knowledge and performance of health-care providers in 80 study clusters across 11 districts in Bihar at baseline between June and September 2011 and again at follow-up between June and September 2014. These 80 clusters were randomly selected, using Stata version 13.1 (StataCorp. LP, College Station, United States of America) from the 360 study clusters included in the Bihar Evaluation of Social Franchising and Telemedicine project before the Sky Program was implemented. Clusters comprised catchment areas surrounding a central village that met eligibility criteria for the possible establishment of a SkyHealth telemedicine centre – criteria primarily included the availability of a broadband connection, health-care infrastructure and potential investors in the franchisee network. To evaluate the effect of the Sky Program on the quality of care provided to households, we interviewed a sample of 64 households in each study cluster and selected the five most frequently visited providers for our baseline sample (i.e. we implicitly randomly selected providers on the basis of their market share). This process was repeated at follow-up but we also added providers from the Skycare programme network if they were not among the five most frequently visited providers at follow-up. In total, we carried out 810 assessments: 395 at baseline and 415 at follow-up. We collected information on the health-care provider’s age, educational attainment, socioeconomic characteristics and health-care practice and background, including medical training and experience, types of illness treated and familiarity with technology. At recruitment, providers consented to taking part in vignette interviews and to being visited unannounced by standardized patients within the following 2 months. Our primary outcomes were: (i) providers’ knowledge of the appropriate treatment for childhood diarrhoea and pneumonia as assessed using vignettes; and (ii) providers’ performance in dealing with cases of these diseases as assessed using standardized patients. Our protocols and instruments are available from the corresponding author on request. This study, which formed part of the Bihar Evaluation of Social Franchising and Telemedicine project protocol, was approved by Duke University in the United States of America (approval no. 29755) and the Health Ministry Steering Committee in India (No.12/2008/30-HMSC/4).

Vignettes

Two interviewers presented hypothetical patient vignettes to health-care providers. One interviewer recorded the diagnostic questions asked by providers in response to the vignettes and the other one read scripted responses to these questions.– Vignettes were devised to represent cases of childhood diarrhoea and pneumonia commonly encountered locally. The two vignettes used featured a father seeking treatment for his 2-year-old son: in the one for diarrhoea, the child had had loose stools for 2 days; in the one for severe pneumonia, the child had had a fever and cough for 5 days and difficulty breathing (full details of the vignettes are available from the corresponding author). Appropriate treatment for diarrhoea was defined according to 2005 World Health Organization guidelines as the provision of oral rehydration solution, with or without zinc supplements and without the prescription of unnecessary or potentially harmful drugs. The appropriate treatment for severe pneumonia was defined as the provision of antibiotics, without the prescription of potentially harmful drugs, and referral to a hospital for further evaluation.

Standardized patients

We assessed health-care providers’ performance in practice using a standardized patient method in which enumerators trained as standardized patients visited providers unannounced.,,– We used a proxy, standardized patient, case method, in which a father seeks treatment for his 2-year-old son who has the same clinical symptoms described in the vignettes. The cases and methods used for standardized patients were developed by members of our study team and have now been used in numerous settings.,, Knowledge and performance scores were calculated for the vignettes and standardized patients, respectively, according to item response theory and based on whether the correct diagnostic questions had been asked and the appropriate examinations had been proposed.,

Statistical analysis

We used a difference-in-difference method to compare changes in providers’ performance between areas in which the programme was implemented and areas in which it was not.– Implementation areas were those in which at least one SkyHealth or SkyCare provider was active, as recorded by our field staff at follow-up data collection. Although the study was originally designed as a large-scale, cluster randomized study, substantial deviations from the planned randomization occurred. Hence, we employed a quasiexperimental difference-in-difference analysis. Specifically, we used multiple linear regression to explore the association between changes in providers’ knowledge and performance (i.e. outcomes) and an indicator variable denoting the period before or after implementation of the programme interacted with an indicator of whether or not the study cluster was in an implementation area.

The difference-in-difference method provided estimates of covariate-adjusted differences in the knowledge and performance of providers between implementation and nonimplementation areas at follow-up that took into account any differences that existed before implementation. Our estimates probably captured both the direct effects of the programme on participating providers and the indirect effects on other providers in study areas that resulted from market competition. The analysis also included an indicator for the year and district fixed effects to account for unobserved differences across districts at the two assessments. Since, at baseline, it was not possible to predict which providers would participate in the programme, we did not have baseline data for all providers to conduct a provider-level, difference-in-difference analysis. As a robustness check, we performed analysis of covariance (ANCOVA) regressions to compare providers in implementation and nonimplementation areas using follow-up data with district fixed effects and cluster-level average values for outcomes at baseline. In all regressions, standard errors were clustered at the cluster level to correct for the correlation of the error term across providers within the same cluster. We used Stata version 13.1 for all statistical analyses.

Assuming an intracluster correlation of 0.05, a sample of 400 providers (i.e. 5 providers in each of 80 clusters) would yield over 95% power at the 0.05 level of significance for detecting a standardized effect size of 0.4 on knowledge scores based on the vignettes. Similarly, a sample of 200 providers (i.e. 5 in each of 40 clusters) would yield over 99% power for detecting a standardized effect size of 0.66 on performance scores based on the standardized patients. These relatively large effect sizes reflect the projected impact of the programme at the time of planning.

Results

Overall, 50 of the 80 study clusters were in areas in which the programme was implemented. At baseline and follow-up, providers in implementation and nonimplementation areas were comparable in terms of demographic characteristics, infrastructure, clinical activity and the type of medicine practiced (Table 1; available at: http://www.who.int/bulletin/volumes/95/5/16-179556). However, given the criteria for enrolment in the programme, participating providers were younger than nonparticipating providers, had more experience using computers, had fewer years of clinical experience and reported working fewer hours. Relative to the baseline in 2011, providers in both implementation and nonimplementation areas reported lower patient volumes, longer working hours and higher consultation fees in 2014.

Table 1

Health-care providers’ characteristics, social franchising and telemedicine programme, Bihar, India, 2011–2014

Health-care provider’s characteristics	Baseline assessment in 2011 before the programmea was implemented (n = 395)			Follow-up assessment in 2014 after the programmea was implemented (n = 415)
	Nonimplementation areas (n = 149)	Implementation areas (n = 246)		Nonimplementation areas (n = 141)	Implementation areas
					Nonparticipating providers (n = 226)	Participating providers (n = 48)b
Age, mean	45.3	43.3		45.7	42.7	37.8
Proportion educated beyond high school, no. (%)	106 (71.1)	196 (79.7)		114 (80.9)	188 (83.2)	41 (85.4)
Proportion with a medical qualification,c no. (%)	34 (22.8)	47 (19.1)		28 (19.9)	37 (16.4)	7 (14.6)
Proportion who have ever used a computer, no. (%)	32 (21.5)	43 (17.5)		34 (24.1)	50 (22.1)	20 (41.7)
Experience in years, mean	18.7	18.1		18.6	17.0	12.6
Patient caseload per day, mean	19.6	17.5		15.7	14.1	11.3
Working hours per week, mean	49.5	50.2		57.4	57.8	56.0
Proportion who have run camps,d no. (%)	12 (8.1)	19 (7.7)		10 (7.1)	20 (8.8)	9 (18.8)
Proportion working in a public health facility, no. (%)	3 (2.0)	7 (2.8)		1 (0.7)	2 (0.9)	0 (0.0)
Infrastructure index,e mean	0.1	0.1		0.1	0.1	0.1
Consultation fee in Indian rupees,f mean	12.5	11.3		24.3	28.1	23.9
Proportion performing task, no. (%)
Holding consultations with patients	148 (99.3)	246 (100.0)		141 (100.0)	226 (100.0)	48 (100.0)
Administering treatment	127 (85.2)	213 (86.6)		117 (83.0)	197 (87.2)	43 (89.6)
Selling drugs	78 (52.3)	111 (45.1)		82 (58.2)	148 (65.5)	30 (62.5)
Performing laboratory-related tasks	7 (4.7)	14 (5.7)		3 (2.1)	0 (0.0)	1 (2.1)
Performing administrative tasks	97 (65.1)	147 (59.8)		110 (78.0)	173 (76.5)	35 (72.9)
Owning the health-care business	109 (73.2)	170 (69.1)		134 (95.0)	216 (95.6)	46 (95.8)
Proportion practising type of medicine, no. (%)
Allopathic medicine	141 (100.0)	226 (100.0)		129 (91.5)	216 (95.6)	47 (97.9)
Homeopathic or Ayurvedic medicine	117 (83.0)	197 (87.2)		35 (24.8)	50 (22.1)	13 (27.1)
Proportion treating disease type, no. (%)
Diarrhoea	141 (94.6)	238 (96.7)		137 (97.2)	222 (98.2)	44 (91.7)
Pneumonia	126 (84.6)	203 (82.5)		128 (90.8)	206 (91.2)	44 (91.7)

a The World Health Partners’ Sky Program employed social franchising and telemedicine technology to help improve the knowledge and performance of health-care providers.

b Participating providers included 20 SkyHealth providers and 28 SkyCare providers.

c Medical qualifications included MBBS (equivalent to a medical degree in the United States), BAMS, BUMS and BHMS (i.e. bachelor’s degrees in Ayurvedic, Unani and homeopathic medicine, respectively) degrees as well as diplomas in Ayurvedic medicine and some other medical degrees.

d Outreach and public service health-care camps conducted in communities by health-care providers.

e The infrastructure index was derived from the following variables: (i) the availability of electricity; (ii) the availability of back-up power; (iii) the number of consulting rooms; (iv) the number of beds for day observation; (v) the provision of tests; (vi) the ability to perform X-rays; and (vi) possession of a computer system. The index is a normalized sum of the scores for component variables.

f Consultation fees were adjusted to 2001 Indian rupees using the consumer price index.

Providers’ knowledge of childhood diarrhoea and pneumonia was greater at follow-up in both implementation and nonimplementation areas, as illustrated in Fig. 1, Fig. 2 (available at: http://www.who.int/bulletin/volumes/95/5/16-179556), Fig. 3 (available at: http://www.who.int/bulletin/volumes/95/5/16-179556) and Fig. 4: a greater proportion of all types of provider asked diagnostic questions about diarrhoea and pneumonia in response to the vignettes at follow-up than at baseline. However, there was no significant difference between the performance of providers who participated in the programme and those who did not. Fig. 3 and Fig. 4 show data on the diagnoses offered and treatments proposed in response to the diarrhoea and pneumonia vignettes, respectively, at baseline and follow-up. Providers in both implementation and nonimplementation areas gave correct diagnoses more often at follow-up. In implementation areas, the performance of participating providers was generally similar to that of nonparticipants, except that participating providers were more likely: (i) to wrongly report that prescriptions did not include antibiotics; (ii) to prescribe the correct treatment for diarrhoea; and (iii) to diagnose pneumonia in response to the vignette (details available from the corresponding author).

Fig. 1

Outcome of a social franchising and telemedicine programme on the questions asked and tests proposed by health-care providers for the childhood diarrhoea vignette, Bihar, India, 2011–2014

Fig. 2

Outcome of a social franchising and telemedicine programme on the questions asked and tests proposed by health-care providers for the childhood pneumonia vignette, Bihar, India, 2011–2014

Fig. 3

Outcome of a social franchising and telemedicine programme on the diagnosis made and treatment given by health-care providers for the childhood diarrhoea vignette, Bihar, India, 2011–2014

Fig. 4

Outcome of a social franchising and telemedicine programme on the diagnosis made and treatment given by health-care providers for the childhood pneumonia vignette, Bihar, India, 2011–2014

Notes: The World Health Partners’ Sky Program employed social franchising and telemedicine technology to help improve the knowledge and performance of health-care providers. Implementation areas included 50 study clusters and nonimplementation areas included 30 study clusters. Health-care providers’ knowledge was assessed at baseline in 2011 before the programme was implemented and at follow-up in 2014 after implementation using a hypothetical patient vignette of childhood diarrhoea as commonly encountered locally, to which they had to respond. Health-care providers in implementation areas were not all participants in the programme.

Notes: The World Health Partners’ Sky Program employed social franchising and telemedicine technology to help improve the knowledge and performance of health-care providers. Implementation areas included 50 study clusters and nonimplementation areas included 30 study clusters. Health-care providers’ knowledge was assessed at baseline in 2011 before the programme was implemented and at follow-up in 2014 after implementation using a hypothetical patient vignette of childhood pneumonia as commonly encountered locally, to which they had to respond. Health-care providers in implementation areas were not all participants in the programme.

Notes: The World Health Partners’ Sky Program employed social franchising and telemedicine technology to help improve the knowledge and performance of health-care providers. Implementation areas included 50 study clusters and nonimplementation areas included 30 study clusters. Health-care providers’ knowledge was assessed at baseline in 2011 before the programme was implemented and at follow-up in 2014 after implementation using a hypothetical patient vignette of childhood diarrhoea as commonly encountered locally, to which they had to respond. The knowledge score was the item response score based on questions health-care providers asked interviewers in response to the vignette. Health-care providers in implementation areas were not all participants in the programme.

Notes: The World Health Partners’ Sky Program employed social franchising and telemedicine technology to help improve the knowledge and performance of health-care providers. Implementation areas included 50 study clusters and nonimplementation areas included 30 study clusters. Health-care providers’ knowledge was assessed at baseline in 2011 before the programme was implemented and at follow-up in 2014 after implementation using a hypothetical patient vignette of childhood pneumonia as commonly encountered locally, to which they had to respond. The knowledge score was the item response score based on questions health-care providers asked interviewers in response to the vignette. Health-care providers in implementation areas were not all participants in the programme.

Table 2 shows the results of the difference-in-difference analysis of changes in providers’ knowledge as assessed using the vignettes. The change between baseline and follow-up in the proportion of providers in implementation areas who asked the correct diagnostic questions for pneumonia was significantly less than the change in the corresponding proportion of providers in nonimplementation areas (difference-in-difference estimate: –0.027; 95% confidence interval, CI: –0.054 to –0.001). The programme had no significant effect on any other indicator of providers’ knowledge of either childhood diarrhoea or pneumonia, including making the correct diagnosis, prescribing the correct treatment and prescribing harmful treatments less often. The robustness checks performed using ANCOVA regression analysis showed that the programme had no significant effect on eight of 10 measures of providers’ knowledge (details available from the corresponding author). However, at follow-up, providers in implementation areas were seven percentage points more likely to prescribe the correct treatment in response to the pneumonia vignette than those in nonimplementation areas (intergroup difference: 0.070; 95% CI: 0.015 to 0.129) and seven percentage points less likely to prescribe a harmful treatment (intergroup difference: –0.070; 95% CI: –0.127 to –0.011). These estimates were comparable to point estimates in the difference-in-difference analysis though the latter were not statistically significant.

Table 2

Effect of a social franchising and telemedicine programme on health-care providers’ knowledge measured in vignettes, difference-in-difference analysis, Bihar, India, 2011–2014

Knowledge measure	Assessments before and after programmea implementation		Difference-in-difference estimate (95% CI)b,c
	Respondents who acted as described
	Before (n = 395)d	After (n = 405)
Diarrhoea vignettee
Asked the correct diagnostic questions, mean %	21.8	32.1	−0.009 (−0.037 to 0.018)
Made the correct diagnosis, no. (%)	289 (73.2)	356 (87.9)	0.007 (−0.132 to 0.145)
Prescribed oral rehydration solution, no. (%)	287 (72.7)	355 (87.7)	0.063 (−0.050 to 0.176)
Prescribed the correct treatment, no. (%)	15 (3.8)	17 (4.2)	0.021 (−0.034 to 0.075)
Prescribed harmful treatment, no. (%)	355 (89.9)	371 (91.6)	−0.056 (−0.153 to 0.042)
Pneumonia vignettee
Asked the correct diagnostic questions, mean %	14.9	34.3	−0.027 (−0.054 to −0.001)
Made the correct diagnosis, no. (%)	225 (57.0)	341 (84.2)	0.012 (−0.126 to 0.151)
Prescribed antibiotics, no. (%)	333 (84.3)	356 (87.9)	−0.075 (−0.196 to 0.046)
Prescribed the correct treatment, no. (%)	33 (8.4)	39 (9.6)	0.055 (−0.017 to 0.128)
Prescribed harmful treatment, no. (%)	340 (86.1)	365 (90.1)	−0.058 (−0.151 to 0.036)

CI: confidence interval.

a The World Health Partners’ Sky Program employed social franchising and telemedicine technology to help improve the knowledge and performance of health-care providers.

b The difference-in-difference method estimated the difference in the change from baseline in 2011 to follow-up in 2014 in providers’ knowledge, as assessed using hypothetical vignette interviews, between implementation areas and nonimplementation areas.

c Each row shows the result of a regression model with the outcome variable specified in the first column. All regressions included the following control variables: (i) age; (ii) education; (iii) medical qualifications; (iv) experience; (v) knowledge score (i.e. the item response score based on questions health-care providers asked interviewers in response to the vignette); (vi) working hours; (vii) patient caseload; (viii) a dummy variable indicating whether the health-care facility organizes camps; (ix) a dummy variable indicating whether the facility is a public health facility; and (x) an index for the clinic’s infrastructure. We also included a binary variable indicating whether the cluster in which the health-care facility was located was in a programme implementation area and we included time and district fixed effects. All standard errors were clustered at the cluster level.

d Data on some analysis variables were missing for 10 assessments.

e Health-care providers’ knowledge was assessed using hypothetical patient vignettes of childhood diarrhoea and pneumonia as commonly encountered locally, to which they had to respond.

Providers spent an average of 1.64 min with the diarrhoea standardized patient at follow-up and prescribed an average of 1.76 drugs; at baseline, the corresponding figures were 1.59 min and 1.85 drugs. Fig. 5 shows that, in general, providers in both implementation and nonimplementation areas asked standardized patients diagnostic questions about diarrhoea less often at follow-up than at baseline, though they asked about the nature of the stools more frequently. For the pneumonia standardized patient, there was no clear pattern in the change between baseline and follow-up in the types of questions asked (Fig. 6; available at: http://www.who.int/bulletin/volumes/95/5/16-179556). Whereas all providers asked more often at follow-up about rapid breathing, the type of cough and a runny nose, providers in programme implementation areas asked less often about chest indrawing with flaring nostrils. The last feature is important because chest indrawing and flaring nostrils are key signs of respiratory distress that can help differentiate severe pneumonia, which is treated with antibiotics and urgent referral to hospital, from pneumonia, which is treated with antibiotics alone. Moreover, these questions are critical given that the child was not present and could not be examined. The performance of providers in giving the correct diagnosis and treatment was poorer with standardized patients than with vignettes, which is consistent with the gap between knowledge and practice previously reported for these providers. For the diarrhoea standardized patient, all providers prescribed treatment at follow-up but none was the correct treatment for simple diarrhoea (Fig. 7). Among providers in the programme, in contrast, 16.7% (8/48) prescribed the correct treatment at follow-up in response to the diarrhoea vignette. Similarly, 11.1% (1/9) of programme providers prescribed the correct treatment for pneumonia standardized patients at follow up, whereas 22.9% (11/48) proposed the correct treatment in response to the pneumonia vignette. Fig. 8 (available at: http://www.who.int/bulletin/volumes/95/5/16-179556) shows data on the diagnoses offered and treatments prescribed by health-care providers for the pneumonia standardized patient.

Fig. 5

Outcome of a social franchising and telemedicine programme on the questions asked by health-care providers of the childhood diarrhoea standardized patient, Bihar, India, 2011–2014

Fig. 6

Outcome of a social franchising and telemedicine programme on the questions asked by health-care providers of the childhood pneumonia standardized patient, Bihar, India, 2011–2014

Fig. 7

Outcome of a social franchising and telemedicine programme on the diagnosis made and treatment given by health-care providers for the childhood diarrhoea standardized patient, Bihar, India, 2011–2014

Fig. 8

Outcome of a social franchising and telemedicine programme on the diagnosis made and treatment given by health-care providers for the childhood pneumonia standardized patient, Bihar, India, 2011–2014

Notes: The World Health Partners’ Sky Program employed social franchising and telemedicine technology to help improve the knowledge and performance of health-care providers. Implementation areas included 50 study clusters and nonimplementation areas included 30 study clusters. Health-care providers’ performance was assessed at baseline in 2011 before the programme was implemented and at follow-up in 2014 after implementation using a standardized patient method in which enumerators trained as standardized patients visited providers unannounced to ask about their 2-year-old son who was described as having childhood diarrhoea as commonly encountered locally. Health-care providers in implementation areas were not all participants in the programme.

Notes: The World Health Partners’ Sky Program employed social franchising and telemedicine technology to help improve the knowledge and performance of health-care providers. Implementation areas included 50 study clusters and nonimplementation areas included 30 study clusters. Health-care providers’ performance was assessed at baseline in 2011 before the programme was implemented and at follow-up in 2014 after implementation using a standardized patient method in which enumerators trained as standardized patients visited providers unannounced to ask about their 2-year-old son who was described as having childhood pneumonia as commonly encountered locally. Health-care providers in implementation areas were not all participants in the programme.

Notes: The World Health Partners’ Sky Program employed social franchising and telemedicine technology to help improve the knowledge and performance of health-care providers. Implementation areas included 50 study clusters and nonimplementation areas included 30 study clusters. Health-care providers’ performance was assessed at baseline in 2011 before the programme was implemented and at follow-up in 2014 after implementation using a standardized patient method in which enumerators trained as standardized patients visited providers unannounced to ask about their 2-year-old son who was described as having childhood diarrhoea as commonly encountered locally. Health-care providers in implementation areas were not all participants in the programme.

Notes: The World Health Partners’ Sky Program employed social franchising and telemedicine technology to help improve the knowledge and performance of health-care providers. Implementation areas included 50 study clusters and nonimplementation areas included 30 study clusters. Health-care providers’ performance was assessed at baseline in 2011 before the programme was implemented and at follow-up in 2014 after implementation using a standardized patient method in which enumerators trained as standardized patients visited providers unannounced to ask about their 2-year-old son who was described as having childhood pneumonia as commonly encountered locally. Health-care providers in implementation areas were not all participants in the programme.

Table 3 shows the results of the difference-in-difference analysis of changes in providers’ performance as assessed using standardized patients. There was no evidence for either diarrhoea or pneumonia that the programme was associated with a significant improvement on any measure of provider performance. Robustness checks performed using ANCOVA regression analysis confirmed these findings.

Table 3

Effect of a social franchising and telemedicine programme on health-care providers’ performance on standardized patient assessments, difference-in-difference analysis, Bihar, India, 2011–2014

Performance measure	Assessments before and after programmea implementation		Difference-in-difference estimate (95% CI)b,c
	Respondents who acted as describedd
	Before	After
Diarrhoea standardized patient,en	178	170
Asked the correct diagnostic questions, mean %	24.1	22.3	0.052 (−0.014 to 0.118)
Prescribed oral rehydration solution, no. (%)	31 (17.4)	25 (14.5)	−0.104 (−0.243 to 0.035)
Prescribed harmful treatment, no. (%)	159 (89.3)	160 (93.0)	0.002 (−0.147 to 0.152)
Mean performance score	0.0	1.2	0.243 (−0.196 to 0.682)
Pneumonia standardized patient,e n	162	163
Asked the correct diagnostic questions, mean %	30.7	34.3	0.021 (−0.061 to 0.104)
Prescribed antibiotics, no. (%)	104 (64.2)	122 (73.9)	0.067 (−0.183 to 0.318)
Prescribed harmful treatment, no. (%)	106 (65.4)	129 (78.2)	0.137 (−0.087 to 0.362)
Mean performance score	0.0	0.9	0.091 (−0.436 to 0.617)

CI: confidence interval.

a The World Health Partners’ Sky Program employed social franchising and telemedicine technology to help improve the knowledge and performance of health-care providers.

b The difference-in-difference method estimated the difference in the change from baseline in 2011 to follow-up in 2014 in providers’ performance, as assessed in interviews with standardized patients, between implementation areas and nonimplementation areas.

c Each row shows the result of a regression model with the outcome variable specified in the first column. All regressions included the following control variables: (i) age; (ii) education; (iii) medical qualifications; (iv) experience; (v) performance score (i.e. the item response score based on questions health-care providers asked the standardized patient); (vi) working hours; (vii) patient caseload; (viii) a dummy variable indicating whether the health-care facility organizes camps; (ix) a dummy variable indicating whether the facility is a public health facility; and (x) an index for the clinic’s infrastructure. We also included a binary variable indicating whether the cluster in which the health-care facility was located was in a programme implementation area and we included time and district fixed effects. All standard errors were clustered at the cluster level.

d Not all health-care providers were assessed using standardized patients because standardized patients could be introduced without detection only in clinics with a high patient volume.

e Health-care providers’ performance was assessed using a standardized patient method in which enumerators trained as standardized patients visited providers unannounced to ask about their 2-year-old son who was described as having childhood diarrhoea or pneumonia as commonly encountered locally.

Discussion

We did not find evidence that the World Health Partners’ Sky Program improved the quality of care for childhood diarrhoea or pneumonia in Bihar. Although health-care providers who participated in the programme were almost seven percentage points more likely to propose appropriate treatments with vignettes, we found no significant difference in the quality of care provided in practice for diarrhoea or pneumonia. This failure could be attributed to the weak design and implementation of the programme. For example, only 3 to 6 days of training was offered to providers who signed up as franchisees, which was probably insufficient and was shorter than other training interventions that have successfully improved provider’s knowledge and practice., For example, a recent study showed that intensive, sustained training for similar informal providers in India significantly improved their performance. In addition, the programme did not significantly influence patient demand: in 2014, health-care providers in the programme accounted for only 3.5% of all providers in implementation areas and 6% of all private providers. Moreover, fewer than 3% of children with symptoms of diarrhoea or pneumonia were taken to programme providers.

Our study has several limitations. First, assessments of providers were carried out in only 80 randomly selected clusters out of the 360 study clusters. Second, our assessments involved standardized cases of childhood diarrhoea and pneumonia, which means that our findings may not be generalizable to other conditions. However, these two conditions have a high priority in India’s health system. Another limitation is that, although the standardized patient method is considered the standard for evaluating provider performance,,, standardized patients can be introduced without detection only in clinics with a high patient volume (e.g. SkyHealth providers), which excluded most small rural providers who form the majority of SkyCare providers.

Our findings do not necessarily imply that franchising or technology that provides support for clinical decision-making and patient management cannot improve the performance of providers in rural settings. Instead, their failure to improve performance points towards problems with implementation – the programme design may not have been sufficient to boost the quantity or quality of the health-care services delivered. Moreover, because the programme network covered only a small fraction of existing rural health-care providers – most of whom were informally trained, had degrees in alternative medicine or had undertaken apprenticeships with formal practitioners,, it is unlikely that the programme could have influenced health or health care at the population level. Our results also underscore the challenges faced by social franchising in identifying appropriate incentives for providers to adopt technology and improve quality and in understanding patients’ responses to quality improvements.

Social franchising has the potential to create sustainable networks of private health-care providers and improve health-care quality and health outcomes. However, the success of this concept depends on identifying models that can be empirically shown to improve health outcomes before they are scaled up with investments from governments and global donors.