A pragmatic approach to measuring, monitoring and evaluating interventions for improved tuberculosis case detection.

The inability to detect all individuals with active tuberculosis has led to a growing interest in new approaches to improve case detection. Policy makers and program staff face important challenges measuring effectiveness of newly introduced interventions and reviewing feasibility of scaling-up successful approaches. While robust research will continue to be needed to document impact and influence policy, it may not always be feasible for all interventions and programmatic evidence is also critical to understand what can be expected in routine settings. The effects of interventions on early and improved tuberculosis detection can be documented through well-designed program evaluations. We present a pragmatic framework for evaluating and measuring the effect of improved case detection strategies using systematically collected intervention data in combination with routine tuberculosis notification data applying historical and contemporary controls. Standardized process evaluation and systematic documentation of program implementation design, cost and context will contribute to explaining observed levels of success and may help to identify conditions needed for success. Findings can then guide decisions on scale-up and replication in different target populations and settings.

Introduction

Disease registration systems are maintained in many countries for both communicable and non-communicable diseases. The tuberculosis (TB) registration and reporting system is amongst the first established and best developed, with approximately 200 countries and territories reporting data annually[1] using standardized forms developed through successive WHO guidance.[2,3] The purposes of the system include both operational management of TB control efforts and surveillance of local, national and international trends. Quarterly routine reports from the system are either published or can be made available, comprising counts of diagnostic activities, cases enrolled on treatment and outcome of treatment disaggregated by a number of different measures. It is this system that has permitted the tracking of progress in TB control, nationally and globally. Although millions of people are treated for TB and millions of lives have been saved since 1990, each year an estimated 3 million incident cases of TB are ‘missed’, either never diagnosed or diagnosed and not reported to national TB programs (NTPs).[1]

Reaching all individuals ill with TB is becoming an increasingly important goal for national and global policy makers.[4-6] The inability to detect all in need of care, has renewed interest in using active case finding (ACF) approaches as opposed to only waiting for people with undiagnosed TB to seek care, the so-called passive case finding (PCF). Recent multi-country initiatives[7,8] supported by the publication of guidelines[9-13] have focused efforts to improve TB case detection. The ultimate goal of improved TB case finding approaches should be reducing incidence and mortality through early detection and cutting transmission,[14] but evidence and guidance on different strategies to improve case detection is yet to fully mature.[10] The gold standard for evaluating the impact of an intervention is through randomized control trials (RCT) using measurable indicators in a well-defined population. To date, there have been two such trials measuring the impact of ACF on prevalence or incidence, which did not show a lasting impact on TB prevalence,[15,16] and one cluster randomized trial without controls showing that ACF could significantly reduce TB prevalence.[17] However, extrapolating their findings to other settings, is hampered by considerable variations in TB epidemiology, infrastructure and site-specific realities. RCTs are expensive, resource-intensive, require relatively stable conditions in the intervention and control zones, and require strict adherence to the study protocol for the entirety of the trial. Furthermore, the effect of improved case detection on transmission and incidence is likely to be seen only after several years of delay;[18] therefore, a trial must take place over a period long enough to measure such impact. Finally, programmatic evidence is important for feedback to shape and revise policy guidance.[19] NTPs and their partners aiming to reach, identify and successfully treat all people with TB as outlined in the recent post-2015 TB strategy[20] require efforts to measure the effectiveness of different approaches. Consequently, a need remains to identify pragmatic solutions for measuring program performance and for evaluating outcomes and feasibility of different intervention strategies.

In this paper we discuss the approaches to selecting a TB case finding intervention and describe a pragmatic framework for monitoring and evaluation based primarily on routinely collected surveillance data, optimizing the strengths and limitations of different research and evaluation methodologies. Finally, we suggest a number of important implications that must be considered before introducing or scaling-up case detection interventions as part of TB control programs.

Selecting approaches to improve case detection

The decision of which case detection strategies to introduce should consider the local context and be based on a careful assessment of the epidemiological situation, potential benefits and risks, costs and available resources.[14] Identifying key populations with high numbers of undiagnosed TB and understanding health-seeking behavior and the barriers to TB detection are necessary first steps when developing an intervention and an approach to measure its effectiveness. The pathways for identification of people with suspected TB, their diagnosis and treatment provide several entry points for potential improvement (Figure 1). The gaps between boxes are areas where people with TB may be ‘missed’. Five entry points for improving detection and notification are: improving awareness/knowledge, increasing access to care, better identification of people for testing, more sensitive and rapid diagnostic tests and stronger linkages for notification to NTPs.[4] The first four areas are entry points to increase case finding and treatment.The last approach focuses on improving notification rather than necessarily increasing detection. It can be used when a sizable proportion of ‘missed’ cases are due to under notification; for example, by linking the private sector to NTPs and improving recording and reporting within NTPs. Studies on improved case detection have focused on different entry points such as improving access,[15-17,21-26] accuracy of diagnostics[27-31] and notification,[32-34] and less frequently on measuring the effects of raising awareness[35] and identification of people for testing.[36]

Figure 1.

Pathway to care with barriers and possible interventions.

The diagnostic algorithm for screening and testing must be considered when undertaking these ACF interventions. Highly sensitive screening tests are preferred in combination with highly specific confirmatory tests. While culture testing (both highly sensitive and specific) is not feasible in many settings, caution must be taken when actively screening people with a low pre-test probability of TB, such as house-to-house approaches. Using smear microscopy alone may lead to false positive results due to lower specificity. A recent ACF intervention in Uganda found half of smear-positive cases to be culture-negative.[37] The use of chest x-ray as a screening rather than diagnostic tool can be highly sensitive for identifying people needing further testing.[38]

All interventions to improve TB case detection will attempt to identify and diagnose prevalent TB and cut the cycle of transmission. Based on the barriers addressed in the pathway to care, the intervention type, population targeted and testing procedures used, a monitoring and evaluation framework can be tailored.

A pragmatic framework for monitoring and evaluating TB case finding interventions

When introducing interventions aimed at improving case detection, program managers will want to know if cases are detected earlier, but also that they effectively target cases that would otherwise have been missed. This framework employs both intervention data (cases found by the project) and TB notification data from a larger population, and uses both historical and contemporary controls to evaluate the impact of a given intervention to improve TB case detection. The primary outcome is measured as additional TB cases above what would be expected without the intervention, and can include a number of other measures.

Measuring intervention specific effects

Interventions to improve case detection should collect data for calculating a number of project-specific indicators based on the type of interventions proposed. Many ACF studies are designed to measure the yield of screening efforts and use this to predict the prevalence of active and undiagnosed TB in a population. When people are asked about the presence of TB symptoms or approached about inclusion in diagnostic procedures, they are screened. A diagnostic test is then applied to a subset of those screened who have met criteria for further testing to identify TB. Direct yield of an intervention can be defined as the number of people identified with TB through efforts of the intervention. Contact investigation[39,40] and screening of prisoners,[41] people with HIV[42] or other easily identified risk groups lend themselves to measure direct yield. Interventions should attempt to collect a basic set of core data including: number of people approached, screened, screening positive, tested, diagnosed, initiated on treatment and successfully treated. Reviewing the numbers of people passing through each step of the screening process is essential to evaluate drop out or inefficiencies throughout the system, and to identify areas to be improved in order to optimize case detection. Intervention-specific indicators are detailed in Box 1. The number of people needed to screen (NNS) to identify a case of TB can provide a reasonable estimate of the effort required in an ACF intervention, however not all screening approaches are the same. Some screening efforts can be done at little or no cost, while others may require significant time, effort and financial resources.[43] Thus, interventions that only report yield or NNS without measures such as time, cost and effort are of limited value when trying to plan scale-up, while using resources efficiently.

Box 1.

Key data and indicators for evaluating TB case detection interventions

Intervention description and assumptions
Steps of the TB pathway to care targeted Screening, diagnostic and treatment algorithms used Types of test used and definitions of different types of confirmed TB Definition of individual suspected to have TB and criteria for testing	Use of incentives Use of patient enablers Acceptability of intervention for providers and for population Costs (system and patients)
Intervention-level case detection data and indicators
Number of people eligible for screening Number of people screened Proportion of people screened among those eligible (B/A) Number of suspected TB patients identified Proportion of people suspected TB patients identified among those screened (C/B) Number of people tested/evaluated for TB disease Proportion of people tested/evaluated for TB among suspected patients (D/C) Number of people diagnosed with TB Proportion of people diagnosed among those screened (E/B) and tested (E/D) Number of patients initiated on treatment Proportion of people initiated on treatment among those diagnosed (F/E) Number of patients successfully completing treatment Proportion among those initiated (G/F)
Time from onset of symptoms to TB diagnosis and change between baseline and intervention Time from TB diagnosis to treatment initiation and change between baseline and intervention Smear grading and positivity rate at diagnosis and change between baseline and intervention
Population-level case detection data and indicators
Target population Population description (size, location, etc.) for areas in which the intervention is being proposed List of BMUs where the intervention will be conducted Evaluation and control populations Population description (size, location, etc.) List of BMUs in these populations ○ Evaluation population usually contains more BMUs than those affected directly by the intervention Quarterly number of diagnostic tests conducted (historical and prospective) Quarterly historical and prospective disaggregated notification data (SS+, Bac+, SS-, EPTB) Quarterly cohort treatment outcomes
External factors affecting notifications in the evaluation and/or control populations
Contemporary trends and seasonal patterns Political, security or natural disaster events Start or discontinuation of other TB interventions Change in national TB guidance, changes in TB funding Changes in guidance on TB notification and health management information systems Major health systems related events and changes Disruption of essential supplies

Bac+: bacteriologically positive; BMUs: basic management units, EPTB: extrapulmonary TB; SS+: sputum smear positive; SS-: sputum smear negative.

Measuring detection/intention to treat, rather than actual treatment initiation and treatment completion as the endpoint for analysis will provide an overestimation of an intervention's impact. Even with PCF, pre-treatment loss to follow-up can be an important concern.[44-46] A recent study in South Africa found large differences in the cost per person for TB diagnosed, treated or cured through ACF.[47] Other studies have found small differences[48] but without measuring both diagnosis and treatment initiation, it will be harder to draw useful conclusions about the intervention.

Measuring change in notification

To be successful, interventions need to achieve direct yield, however intervention-specific indicators are insufficient to measure the larger impact of the activities on TB notifications. Studies have shown that a sizable proportion of TB cases identified during ACF (measured by direct yield and NNS) would presumably have found their way to treatment without the intervention.[7] The concept of ‘an additional TB case’ can be defined as a case that would not have been notified in the absence of the intervention. By tracking notifications in a larger intervention area and ideally a control population both historically and prospectively, the contribution of the intervention to increased case notification can be approximated.

Target, evaluation and control populations

The delineation of target, evaluation and control populations to measure impact of an intervention is an important step in setting up the evaluation framework underlining the difference between yield and additional cases. When choosing the ACF intervention the target population must be identified and described. Some populations, such as people with HIV or all diabetics are relatively easy to define. In other cases the target population will be more dispersed e.g., people living in slum areas with poor access to care. To show impact on TB notifications at a population level, the group targeted with the intervention needs to contain a substantial number of people with TB who are unlikely to present with symptoms to services (PCF) within a given future time period without the intervention.

The next step is to determine the population in which notification and treatment outcome data will be monitored and evaluated (evaluation population). To capture possible spill-over effects in neighboring non-intervention areas, and to account for potential redistribution of cases (e.g., shifting patients from one health facility to another without increasing the total number of patients receiving care) notification data will need to be monitored in an evaluation area geographically larger and more populous than the intervention area alone. The selection of this evaluation population is a delicate exercise. The evaluation population needs to be large enough to capture all potential effects of an intervention, but it should not be so large that any potential effect is diluted if the intervention targets only a small part of this population. In most cases it is possible and desirable to make use of the existing NTP notification system and select one or more basic management units where patients in the target population would normally present for care and be notified. While this approach works well in most interventions, small or dispersed target populations may require an adapted definition of evaluation population. Furthermore, it is important to verify where cases (e.g., migrant labourers and prisoners) diagnosed in a different catchment area than their registered domicile would normally be notified to ensure consistency in the intervention's notification approach.

The use of control populations is important to identify contemporary trends and other factors influencing case notification. Comparing year-over-year increases without taking into account both secular trends in notification and using control populations can give a false impression of the impact of an intervention,[8] or conversely, a lack thereof. Currently, there is a steady decline in national TB case notification in a number of high-burden countries such as Russia, Ethiopia and Kenya, while others such as Mozambique and Indonesia have increasing trends.[1] A few well-designed studies using TB case notification as an outcome measure have used control populations to help evaluate the effects of an intervention at a population level.[15,16,25,34] An intervention in Myanmar measuring the impact of involving private providers on TB case notification provides a clear example of a good use of a control population, as large increases in the intervention area were somewhat tempered by smaller gains in the control areas.[33]

Control populations should be comparable, but isolated from the evaluation population. They must share the same TB control system and ideally be similar in size and notification trend. A similar initial case notification rate supports plausibility of any measured difference between the two populations during the intervention. External factors influencing case notification in the evaluation and/or control population should be considered. Randomizing both control and intervention areas adds strength to the results.[21-22] If randomization is not feasible, external factors should be systematically monitored and documented, and their effect on case notification in both evaluation and control populations estimated. Examples of such factors are the introduction or closure of other interventions affecting case notification, or political and natural events that impact health seeking or shape the effectiveness of screening efforts (Box 1).

TB case notifications in both evaluation and control populations should be collected both historically and prospectively. Historical quarterly TB notifications, prior to the intervention period, need to be compiled in the absence of a local prevalence survey. At least 1 year of historical (baseline) data should be obtained to account for seasonal variation in notifications. Obtaining data for additional years would be ideal to better control for secular trends. In many situations a simple linear projection can be used to predict the standard passive system performance, while a logarithmic projection could be considered in other situations depending on the best fit. An illustration of target, evaluation and control populations for this framework are presented in Figure 2. In exceptional cases, where no TB control activities have been available and a new diagnostic center is established, there could be a baseline of zero.

Figure 2.

Conceptual model of monitoring and evaluating efforts to improve TB case notification. This figure is available in black and white in print and in color at International Health online.

Interpreting case notification results

Evaluating intervention results must be based on where the entry point lies in the patient pathway for improving case detection (Figure 1). For some interventions an improvement will be shown through better notification systems, others will rely on better sensitivity of diagnostic tests, while most of the interventions will focus on the direct yield of previously undiagnosed TB cases, the target and evaluation populations.

Interventions that can show direct yield of the ACF strategy with concurrent improvements in TB notifications, controlling for trend and control populations can demonstrate additional cases above the PCF system. However, a short period of increased TB case notification may be primarily due to identifying cases earlier and notifications will return to pre-intervention levels as soon as the back-log of prevalent cases is cleared. An intervention that is also successful in identifying cases that would otherwise not have reached any services is likely to show increased notifications over a longer period until an impact can be expected on transmission and incidence and notifications begin to decline. A study in Brazil documented increased TB notifications during the intervention but the effect was not sustained during a post-intervention monitoring period.The findings could mean that people were diagnosed earlier, rather than in increased numbers overall.[23]

An increase in bacteriological-confirmed cases in absence of an increase in all forms of TB may suggest a shift from unconfirmed to confirmed cases, rather than an increased number of people with TB treated. A recent RCT comparing Xpert MTB/RIF to smear microscopy found no difference in overall numbers of people treated,[27] highlighting the importance of reporting both bacteriologically-confirmed, as well as all forms of TB. A rapid, sensitive test could actually decrease overall notification in settings where many people are being put on treatment without bacteriological evidence, since such a test could convince a clinician to rule out TB. Analyzing the trends in bacteriologically-confirmed cases, all forms of TB (especially pulmonary TB), and the proportion of confirmed cases among all forms will provide a better understanding of the impact of different interventions. Increases in TB case detection are unlikely without substantial increases in laboratory testing and tracking the number of tests performed historically and prospectively can provide a good indicator of efforts for finding more people with TB.

The use of surveillance data (TB notification and treatment outcomes) has limitations; data completeness, consistency and reliability can vary over time and may be influenced by changes in surveillance practice (such as changing from paper-based to electronic registration, district boundaries, or changes in case definitions). Data outliers are not always easy to explain. These issues should be considered when analyzing the results of interventions. Therefore, systematic monitoring of external factors influencing notification in the evaluation and control population is crucial.

Other intervention benefits

A number of individual level benefits can be measured during ACF including early detection and improved treatment outcomes.[48]ACF interventions attempt to find people with TB earlier than they would have been detected (if at all) without the intervention.[24,49] Measuring changes in delay in diagnosis can be done by administering questionnaires about their health-seeking behavior to people identified with TB.[50] Documenting previous attempts to seek care for TB-related symptoms can be used to measure delay or missed opportunities.[51] There are a number of biases inherent in these types of measurements including recall and selection bias. The results of a systematic review showed that ACF may find cases earlier, and that smear grading is one of the better ways to measure this difficult indicator.[48] In addition to early diagnosis, other indicators of intervention success can include how the intervention affects the acceptability of screening[52] and the proportion of people identified with TB who immediately start treatment.[44]

Treatment outcome data for both individuals found though ACF as well as the larger evaluation population should be monitored. There is no clear evidence that ACF alone will improve treatment outcomes; but it does not worsen them.[48] A number of studies have shown that among actively found cases, better treatment outcomes can be obtained by providing treatment support to the patient.[21,25,53] Improved treatment outcomes through ACF may be mitigated by the possibility that people who delay or fail to seek care through PCF due to migration patterns, release from prison, substance abuse etc., may be harder to keep on treatment. Other mitigating factors include health systems unable to cope with the increased case load including drug shortages.

Implications for program decisions

The feasibility of scaling-up an intervention that has been proven to be effective and the likelihood of sustaining the same effect under regular program conditions are particular concerns for NTPs. For instance, ACF can easily lead to large increases in numbers of people to be tested, but laboratory capacity, supply lines and quality assurance systems may fail to keep up. A careful analysis of capacity and cost requirements is crucial. An assessment of the views of providers who need to be convinced of the usefulness of the proposed intervention and acceptance of the intervention by beneficiaries can predict future fidelity to a new approach when scaled up under routine program conditions.[54] The latter would include considerations of cost and time implications. To enable proper evaluation of interventions complete, quality notification data by BMU is critical. Routine data on people identified as symptomatic, people tested, diagnosed with TB and initiated on treatment would ideally be available.[55]

ACF will require more financial resources to identify people with TB than PCF. Program managers should take this into account when planning and evaluating, and the cost of the tests and the logistics around the activity must be properly planned. Similarly, the number needed to test is highly dependent on the screening criteria, as well as the diagnostic test used, and must be interpreted with caution. Other cost issues to consider are the ease of access to the population that is to be screened, the testing algorithm to be used, extra human resources needed, data monitoring and the periodicity of the interventions, whether they should be repeated or one-off events. Although the lack of proper cost-effectiveness analysis of ACF interventions has been raised as an important gap,[18,56] only recently has any study been published,[57] and more are needed. Calculating an adjusted ‘number needed to screen to find an additional case’ and related cost may provide a better measure of return on effort as it takes into account the fact that not all cases found through ACF are additional.

Replicating and scaling-up

A number of implementation-related and external factors are likely to influence the success of any given approach: the potential benefits and risks to the patients, acceptability of screening, level of effort and cost.[14] Readiness of health systems to deal with increased demands on diagnostic and care services, interpretation and acceptance of new guidelines and algorithms by the staff and adherence to quality assurance mechanisms may also impact effectiveness. Most published studies lack a critical description of program implementation and analysis of factors that may be considered essential for success or failure of an intervention, limiting the ability to predict replication of success in a different setting.[58]

Interventions to improve TB case finding are often introduced as a compound package of activities, building on assumptions of how best to address various identified barriers to access and early diagnosis. Elements such as staff support, incentives, patient enablers and system strengthening are often added to support active case finding activities. The level of synergy with the wider health system is also likely to influence performance.[54] Therefore, when evaluating interventions it is important to unpack them, identifying the components that are essential to their success and reviewing intervention delivery and implementation issues that may explain effectiveness.[59] Combining the analysis of case detection with a systematic description of the implementation processes will contribute to explaining differences in success levels of an ACF strategy when implemented in different settings and may assist in predicting success when replicated elsewhere.[58]

Conclusions

Increased interest in better ways to diagnose and successfully treat people with previously undetected TB has led to a myriad of new interventions and approaches. Introduction of these innovations requires a robust yet practical system to evaluate progress and their effect on case notification and treatment outcomes. When randomized controlled trials are not feasible, well designed, trend-adjusted before-after studies are a reasonable option and provide important evidence for what happens under routine conditions. Using these studies with a standardized process evaluation, and a systematic description of program implementation design, cost and context will help to identify the conditions needed for success. The robust TB surveillance system can support monitoring and evaluating of the impact of interventions on case detection. Continuous monitoring of both project and notification data during the introduction and maintenance of an intervention may be timely, uncomplicated and informative enough to indicate where fine-tuning of strategy or incremental changes in operational management lead to improvements in TB case detection. Moreover, periodic evaluation of the impact of an intervention on disease registration may provide evidence that is sufficiently persuasive to attempt to reproduce it in other settings, scale-up the intervention, or even draw tentative policy conclusions. Finally, the usage of disease registration systems for monitoring and evaluation increases skills in evidence-based reflection and decision-taking among TB professionals, and its continual application increases the relevance of data in the same operational settings that collect it. The resulting opportunity to improve data quality is beneficial to policy makers and researchers in addition to disease control programs themselves. Findings can then guide decisions on scale-up and replication in different target populations and settings.