Use of indicators of standards of care to improve tuberculosis program management in Ethiopia.

BACKGROUND: Systematic monitoring of health programs and on-site mentoring of health workers are essential for the success of health care. This operations research was designed to measure the effectiveness of a new mentorship and supervisory tool for supervisors.
METHODS: In 2011 the Help Ethiopia Address the Low TB Performance (HEAL TB) Project used WHO or national TB indicators as standards of care (SOC) for baseline assessment, progress monitoring, gap identification, assessment of health workers' capacity-building needs, and data quality assurance. Cut-off points were selected for poor, average, and best performers for each indicator. In this analysis we present results from 10 zones (of 28) in which 1,165 health facilities were supported from 2011 through 2015. Other zones were excluded from the analysis because they entered the project later. The data were collected by trained mentors/supervisors and entered into Microsoft Excel. We used rates and ratios to show the impact of the intervention.
RESULTS: The improvement in the median composite score of 13 selected major indicators (out of 22) over four years was significant (p = 0.000). The proportion of health facilities with 100% data accuracy for all forms of TB was 55.1% at baseline and reached 96.5%. In terms of program performance, the TB cure rate improved from 71% to 91.1%, while the treatment success rate increased from 88% to 95.3%. In the laboratory area, where there was previously no external quality assurance (EQA) for sputum microscopy, 1,165 health facilities now have quarterly EQA, and 96.1% of the facilities achieved a ≥ 95% concordance rate in blinded rechecking.
CONCLUSION: The SOC approach for supervision was effective for measuring progress, enhancing quality of services, identifying capacity needs, and serving as a mentorship and an operational research tool.

Background

Tuberculosis (TB) remains a major cause of morbidity and mortality in many countries and a significant public health problem worldwide. Ethiopia is one of the 30 high-TB, TB/HIV, and MDR TB-burden countries globally and TB remains one of the leading causes of death. According to the 2017 World Health Organization (WHO) report, the incidence of all forms of TB in Ethiopia was 177 per 100,000 [1]. Major progress in global TB control followed the widespread implementation of the DOTS and later Stop TB strategies in countries with a high burden of TB. Establishing a reliable monitoring and evaluation system in TB programs, with regular communication between the central and peripheral levels of the health system, is very important [2], [3], [4].

Since good-quality data are needed to monitor the performance of TB programs and identify gaps, systematic TB program supervision should be carried out to verify the quality of information and address performance problems. Data at both the health facility and district levels can be used to monitor performance and identify gaps [3], [5].

In analyzing these data, there is increasing recognition of the importance of using standard approaches to diagnose and treat TB patients, as well as to screen for and prevent TB, at all levels. A standard set of WHO-endorsed indicators captures the processes and outcomes of TB treatment, but these are often analyzed primarily at central levels rather than at the district or facility level. Even when workers at the health facility or district level do analyze their TB data, there is typically no standard description of what indicator values would be considered a “good” or “poor” outcome in that particular setting, nor any method for prioritizing which indicator values to target for improvement in the future. The standards of care (SOC) tool was developed in Ethiopia to address these needs and is described in this paper.

Various studies have linked supervision to standards of health care, performance improvement, and subsequent quality of care [4]. But the efficacy of supervision in changing providers’ practices is unclear: one study confirmed better provider performance with supervision than without supervision, while another study showed no significant difference [6], [7]. Other studies reported that community health workers allocated to a supportive supervision group performed significantly better than those in the group with standard supervision [8], [9]. Supervisors using an indicator-based checklist realized greater improvement in the performance of midwives as compared to standard supervision [10]. A systematic review showed that there is insufficient high-quality evidence to advocate for any particular approach of implementing supervision [7].

This intervention was designed to develop a tool for supervisors that serve as an objective tool of mentorship/supervision and at the same time helps to identify program gaps. The tool is also used to prioritize health facilities for resource allocation based on their objective performance.

Methods

Definition of terms

Although there is no agreed-on definition of standard of care in medical practice, for this purpose we defined SOC as the quality of care that a patient should get based on WHO or national performance indicators.

The setting

The Federal Ministry of Health has overall responsibility for the health of Ethiopians, a responsibility that it carries out by designing national policies, strategies, and regulations. The country is organized into nine federal states and two city administrations, each with a Regional Health Bureau, which is responsible for planning, implementing, monitoring, and evaluating health programs. Under the region, there are zones with Zonal Health Departments. The zones are divided into woredas (equivalent to districts), with Woreda Health Offices. The lowest-level administrative structure is a kebele (community), with a population of 3,000–5,000. Each kebele has two Health Extension Workers. Ethiopia has a three-tiered health care system: at the lowest level, primary hospitals, health centers, and health posts provide primary health care; general hospitals offer secondary care; and specialized hospitals provide tertiary care [11].

Development of the indicators of standards of care

The USAID-funded HEAL TB project supported the implementation of a comprehensive TB program in the Amhara and Oromia regions of Ethiopia, with a population of 55 million, between 2011 and 2016 and in collaboration with the Amhara and Oromia Regional Health Bureaus, developed indicators of standards of care (SOC). The indicators are WHO TB indicators, indicators from the national health management information system, and indicators customized to measure specific needs. The SOC tool was designed for serving as a baseline assessment, monitoring the progress of activities, identifying gaps, mentoring staff based on the gaps, and planning to address capacity-building needs. The SOC tool is also used to verify the quality of data, provide data for operational research, and modify implementation approaches based on results.

The indicators are categorized by case notification, including community TB; treatment outcomes; laboratory quality; drug management; TB infection control; and TB/HIV. The indicators include definitions, formulas to calculate achievements, data sources, and customized cut-off points classified as poor, moderate, or very good performance. These three levels of performance are labeled red, yellow, and green respectively (see the sample SOC indicators in Table 1). At the beginning, 28 indicators were applied, but after one year of implementation they were reduced to 22 to focus on the core indicators and decrease the load for the mentor.

Table 1

Sample reference sheet for indicators of standards of care.

Image, table 1

Implementation of the standards of care

The SOC was implemented in 28 zones and 471 woredas and 2,180 health facilities covering 55 million population in a phased manner. The project invested in managerial, monitoring, and evaluation capacity of the zonal- and woreda-level TB managers through tailored leadership and management trainings. Quarterly the woreda TB mangers mentor every health facility. To systematize supervision and make it objective the SOC tool was developed and managers trained to use the SOC indicators and interpret the findings for each indicator. Every quarter, the woreda TB focal person applies the SOC tool in the supervision of every health facility in their catchment area. With a health facility TB focal person, the woreda TB focal person measures progress, identifies gaps, designs an improvement plan, and at the same time checks the quality of reported data. This team discusses the improvement plan with the health facility manager and documents the actions taken or to be taken in the mentorship logbook kept in the health facility (Table 2). In the next quarter, the woreda TB focal person repeats the same approach and measures progress against the previous quarterly plan.

Table 2

Indicators of standards of care applied by woreda TB focal persons in health facilities.

Unit	Actions*
Outpatient departments (OPDs) The woreda TB focal person, together with the unit focal person, checks the OPD register and calculates the percentage of patients screened for TB and the linkage for diagnostic investigation.	If the result is red or yellow, they identify the cause of low performance, discuss it with the unit head, and design an improvement action plan. The agreed-on action points are documented in the logbook, and both sign beside the next steps.
Tuberculosis unit Together with the unit focal person, calculate the total case notifications for the health facility and compare it with the target for the catchment population. Calculate the interim treatment and final treatment outcomes.	In the same way as above, identify the cause of the gap and mentor staff based on the gap. Plan further training or supply management actions as needed. Design improvement plans with the unit head.
Laboratory Check if internal quality control for acid-fast bacilli reagents is done and documented per national guidelines. Verify that all sputum smear slides are properly labeled and documented in the laboratory register as well as properly stored. If all are perfect, randomize the slides per the national sample size guidelines and drop the slides off at the external quality assurance (EQA) center for reading.	The same measures as above are taken. In the microscopy aspect, if EQA finds discordant slides, an expert from the regional laboratory or higher-level expert travels to the health facility and identifies the cause of the discordance and mentors laboratory staff on-site or plans more training of the laboratory staff. If there is a microscope issue, the expert recommends a change.
TB infection control Check if triage of patients is taking place and if staff observe proper practices. Check if the rooms in the health facilities are appropriate for TB infection control and advise the health workers about how to improve infection control practices.	The woreda TB focal person assists the health facility to establish a triage system and provides advice about ventilation. The TB focal person mentors staff on proper disinfection and disposal of infectious materials.
TB drug management Check if the stock balance is updated regularly and measure the stock available.	If there was a stock-out in the quarter, identify the reason and agree on actions. The actions could be better stock monitoring and early request for refill.
Health management information systems unit The woreda TB focal person, together with the unit staff, recounts indicators and reconciles those data with those reported in the previous quarter.	If there is a discrepancy, identify the reason and design an improvement plan together.

*The woreda TB focal person then discusses all the findings in all departments with the facility manager, and they revisit the improvement plan.

After completing the quarterly supervision and mentorship of the health facility's staff, the woreda TB focal person presents information about progress, including the gaps identified and improvement plans developed, to the Zonal Health Department TB manager. The woreda TB focal person also presents the actions to be taken by the Zonal Health Department, Regional Health Bureau, or partners. Then the Zonal Health Department TB manager, together with the HEAL TB zonal teams based in the zonal health department office, plans supervision of health facilities with poor performance. During the supervision visit, the respective woreda TB focal persons are also represented and use the SOC to conduct rechecking jointly. The repeat supervision has two purposes: (1) checking the capacity of the woreda TB focal person and mentoring the TB focal person and (2) supporting the poor-performing heath facility to improve its performance. The zonal team also checks a few very good performers to substantiate the findings. If all the findings reported by the woreda TB focal person are correct, then supervision by the Zonal Health Department TB manager to that specific woreda is reduced (Fig. 1).

Fig. 1

The cycle of standards of care at the heath facility, woreda, zonal, and regional levels.

Semi-annually, all zonal TB focal persons, under the leadership of the Regional Health Bureaus, present the progress of their respective zonal TB implementation progress. This experience-sharing forum also serves as a mechanism to redesign implementation approaches. The project assisted in organizing meetings and mentoring the Regional Health Bureau and zonal TB managers in monitoring and use of data for program improvement and covered the logistics and workshop costs.

The zonal and woreda supervisors use a checklist that covers both qualitative parameters and SOC indicators. The supervisor records the SOC while visiting the different units in the health facilities. Once the SOC indicators are computed, they are compared with the cut-off points for the respective indicators.

The woreda TB focal persons collect data on SOC indicators from health facilities manually and report the data to the Zonal Health Department and HEAL TB zonal team. The data are entered into Excel software by Zonal Health Offices. IBM SPSS software for Windows (Version 20.0. Armonk, NY: IBM Corp.) was used in the analysis for this paper. We assigned the values 2, 1, and 0, respectively, to the indicators in the green, yellow, and red categories for calculating the composite score of each indicator improvements across time. The data presented are for 10 zones and 1,165 health facilities that were supported for four years, but the rest of the zones, which entered the project later, were not included in this analysis.

Results

Capacity building of Woreda Health Offices and Zonal Health Departments

The results presented below offer evidence of this improved capacity. Although the results presented in this paper relate to measurement of SOC, other capacity building efforts, such as training of health workers, drug supply management, and provision of equipment and commodities, were also part of the intervention.

Progress in performance based on standards of care

We compared the proportion of health facilities in the green category (better performers) for each indicator at baseline versus four years after intervention. We noticed significant improvement in the proportion of health facilities in the green category for all indicators. Fig. 2 shows improvements for selected major indicators.

Fig. 2

Progress in the performance of selected indicators of standards of care, 2011–2015.

The composite scores for SOC were computed by adding these values of each indicator in each quarter. Fig. 3 shows the trend in the improvement of the composite SOC indicators over four years for 13 major SOC indicators in the 10 zones. There was a significant quality improvement in the median composite score over the years (p = .000). Fig. 4 illustrates the trend in data quality for three selected indicators in health facilities in 10 zones of Amhara and Oromia from 2011 to 2015. The accuracy of reports (reported versus recounted) in the health facilities improved significantly over the four years (p < 0.05).

Fig. 3

Improvement in the composite score of 13 standard of care indicators in 10 zones, 2011–2015.

Fig. 4

Percentage improvement in the accuracy of reporting of selected TB program indicators in Ethiopia, 2011–2015.

Sputum microscopy EQA was not a practice at baseline, but by the end of the fourth year all 1,165 health facilities were implementing EQA, and 96.1% of the health facilities were achieving a ≥ 95% concordance rate in blinded rechecking [12]. As shown in Fig. 5, the TB cure rate and treatment success rate also improved in all health facilities in the 298 zones.

Fig. 5

Trend in treatment success and cure rates, 2011–2015.

Discussion

Regular quarterly mentorship significantly improved the performance of the TB program in health facilities. Building the capacity of government TB focal persons in mentorship contributed to strengthening their sense of ownership and ensured the sustainability of the TB program support. Evidence-based mentorship, feedback, and decision-making were the hallmarks of the SOC approach, as recognized by health facility managers and health workers. This approach allowed health facilities to track the progress of TB program indicators on a quarterly basis. The advantage of objective indicator-based mentorship over subjective checklist-based supervision in improving performance has also been demonstrated in a trial setting [12]. In a case study report from Malawi, quality supervision in an HIV program significantly improved the performance of health programs and data quality [13]. Linking good performance with an award system is reported to be an effective motivational tool [14], which should be used in the future in Ethiopia's TB program.

In this project, a mentorship visit was conducted every three months, soon after the submission of the quarterly reports by health facilities. This timing helped the health care system to monitor the quarterly performance of the health facilities, identify gaps, and discuss improvement plans for the subsequent implementation period. Furthermore, the supervising team counterchecks the accuracy of the quarterly report against the health facility TB register, laboratory register, and administrative reports, which are essential inputs to improve the national health information system.

The woreda TB focal persons were effectively discharging the mentorship and capacity-building responsibility for health facilities in their catchment areas. On average, the TB focal persons mentored more than 90% of the health facilities each quarter. Inaccessibility of health facilities and competing priorities were among the reasons for missing some health facilities. However, the TB focal persons prioritized high-TB-burden health facilities so that they were not missed in the quarterly supportive supervision. The study by Loevinsohn et al. has shown that regular supervision is needed to sustain gains [10].

The SOC tool assesses the comprehensive package of TB interventions, most of which are not covered by the routine health information system. Drug supply management, community TB care, TB infection control, data quality, screening for TB in outpatient departments, and TB screening among contacts of TB index cases are not monitored in the government's routine health information system. Information about these variables is critical to assess the status of case finding, drug supply management, infection prevention approaches, and the accuracy of the reports submitted by health facilities.

The SOC tool is not merely a data collection tool, however; it is a tool to monitor the progress of the comprehensive interventions, identify implementation gaps, assess the capacity of health workers, and decide on the improvement plan. The SOC tool helped the supervisors to analyze the gaps with the aim of pinpointing facility-specific challenges and health system needs that must be prioritized and addressed. Supervisors themselves were responsible for computing the TB-related indicators reported in the health information system and counterchecking their data against what was reported by the health facilities. If there was any discrepancy, the supervisor and health facility manager discussed whether there was a recording problem or an issue with health workers’ understanding of how to compute data on the indicators. The supervisor provided on-site feedback to improve the quality of recording and reporting.

The project has been covering the cost of quarterly supervision at the woreda level. On average, the woreda TB managers were in the field for one working day every quarter per health facility, at a cost of US$7.50 per facility. Although this project focused on TB, the TB focal persons supported other health areas during their visits, and integration of services is recommended to improve effectiveness and coordination with related programs. At the woreda and health facility levels, different health programs are coordinated by the same health worker as a focal person, so integrated mentorship could be easily implemented.

The results gained through use of the SOC tool have garnered acceptance locally, nationally, and even internationally. Adequate technical capacity has been created at the woreda and zonal levels. Moreover, the participatory approach followed since the inception of the tool fosters sustainability. Efforts have begun to make the SOC a nationwide tool and share it with other countries.

Conclusion

The use of SOC mentorship at the district level is a rational and effective approach to improve TB case finding, treatment outcomes, TB drug management, laboratory quality, infection control and comprehensive TB program monitoring. We also observed significant improvement in data quality after the SOC guided mentorship. The SOC is recommended for national application and could be replicated by other health programs for better monitoring and capacity building.

Ethical considerations

We received ethical approval from the ethics committees of the Amhara and Oromia Regional Health Bureaus to analyze the routine data and disseminate the findings. We used aggregate program-level reports for this analysis with the consent of the reporting institutions. No patient identifiers were included in the routine report.

Availability of data

There are no data other than those included in this article

Competing interests

The authors declare that they have no competing interests.

Funding

The United States Agency for International Development (USAID) supported this work through HEAL TB, under cooperative agreement number AID-663-A-11-00,011. The contents of the article are the responsibility of the authors alone and do not necessarily reflect the views of USAID or the US government.

Author contributions

MM conceived the idea and designed the tool together with BD and YK. MM, DH, YK, BG, SN, GN, YKH, DJ, NH, and ZG were involved in the implementation. PS and BKT critically reviewed the manuscript for intellectual content with all the research team and provided feedback on subsequent versions. All authors reviewed and approved the final version of the manuscript.