An empirical approach to defining loss to follow-up among patients enrolled in antiretroviral treatment programs.

In many programs providing antiretroviral therapy (ART), clinicians report substantial patient attrition; however, there are no consensus criteria for defining patient loss to follow-up (LTFU). Data on a multisite human immunodeficiency virus (HIV) treatment cohort in Lusaka, Zambia, were used to determine an empirical "days-late" definition of LTFU among patients on ART. Cohort members were classified as either "in care" or LTFU as of December 31, 2007, according to a range of days-late intervals. The authors then looked forward in the database to determine which patients actually returned to care at any point over the following year. The interval that best minimized LTFU misclassification was described as "best-performing." Overall, 33,704 HIV-infected adults on ART were included. Nearly one-third (n = 10,196) were at least 1 day late for an appointment. The best-performing LTFU definition was 56 days after a missed visit, which had a sensitivity of 84.1% (95% confidence interval (CI): 83.2, 85.0), specificity of 97.5% (95% CI: 97.3, 97.7), and misclassification of 5.1% (95% CI: 4.8, 5.3). The 60-day threshold performed similarly well, with only a marginal difference (<0.1%) in misclassification. This analysis suggests that > or =60 days since the last appointment is a reasonable definition of LTFU. Standardization to empirically derived definitions of LTFU will permit more reliable comparisons within and across programs.

In sub-Saharan Africa, services for human immunodeficiency virus (HIV) care and treatment have expanded rapidly over the past decade and have provided life-saving antiretroviral therapy (ART) to over 2 million infected adults and children (1). Despite demonstrable health gains among enrollees, however, clinicians in many programs in the region are now reporting substantial patient attrition. In a study of 13 African cohorts, for example, Braitstein et al. (2) noted an average of 15% loss to follow-up at 12 months following ART initiation, with variability ranging from 0% to 44% across programs. In a review of 33 African cohorts, Rosen et al. (3) calculated a weighted mean attrition rate of 1.8%–3.3% per month, which they attributed largely to follow-up losses.

Although loss to follow-up is a commonly reported metric, it has no consensus definition. “Lateness” for scheduled appointments is often used to describe the phenomenon, but the actual time intervals employed vary greatly among programs. Our research group in Zambia, for example, has classified patients who are more than 30 days past their last scheduled appointment date as “late” in published reports (4–11). Médecins Sans Frontières (Doctors Without Borders) has defined loss to follow-up as being more than 2 months late for a scheduled appointment (12, 13); Yu et al. (14) used a 3-month interval from the time of a missed appointment in northern Malawi. Time since the last clinic visit has also been used to define loss to follow-up. Patients included in the ART-LINC collaboration were classified as lost to follow-up when 6 months had elapsed since their last visit (2, 15). A 3-month threshold from the last visit was used by Wools-Kaloustian et al. (16) to define follow-up loss in western Kenya. Before we can better understand the phenomenon of patient attrition, there is a need for standardized definitions of loss to follow-up based on empirical evidence, to permit consistent comparisons across and within programs.

MATERIALS AND METHODS

We analyzed data from a large programmatic cohort in Lusaka, Zambia, to determine the best-performing criterion for classifying patients who are late for scheduled appointments. The multisite Lusaka ART program is administered by the Zambian Ministry of Health and its local partners, with substantial support being provided by the President's Emergency Plan for AIDS Relief; the Global Fund for AIDS, Tuberculosis, and Malaria; and other donors. Clinical and programmatic characteristics have been described in detail elsewhere (4, 5). Briefly, HIV-infected adults and children are screened for ART eligibility on the basis of CD4+ cell count and clinical staging across 18 public sector sites. ART is initiated on the basis of national criteria (17), which closely follow those of the World Health Organization (18, 19). Adults initiating ART undergo an intensive clinical visit schedule over the first 3 months for assessment of potential side effects and encouragement of adherence; patients without complications proceed to once-monthly pharmacy visits and quarterly clinical visits. Medical history and appointment information is captured in an electronic medical record (20). From this database, data staff generate weekly lists of patients with missed appointments. Community health workers use collected locator information to contact patients with missed visits at their homes to encourage and facilitate clinic attendance (21).

We developed an empirical approach to determine the “most efficient” definition of loss to follow-up (Figure 1) based on sensitivity and specificity to predict a return to care in the subsequent year. The source population for our analysis was all ART-naive HIV-infected adults initiating HIV treatment at 18 Lusaka sites between April 1, 2004, and December 31, 2007. Patients who were active and those who were late for a clinical or pharmacy visit as of December 31, 2007, were included in the analysis. Patients who had formally withdrawn from the program or who had died prior to that date were excluded (22). We categorized loss to follow-up on the basis of number of days late, using thresholds from ≥1 day to ≥182 days (i.e., 26 weeks). For each cutpoint, we looked forward in our data set—from January 1, 2008, to December 31, 2008—to determine the proportion of persons who returned to care within the subsequent 1 year.

Figure 1.

Method used to evaluate the performance of many different intervals for defining loss to follow-up (LTFU) in a multisite human immunodeficiency virus treatment cohort, Lusaka, Zambia, 2004–2007. Once a specific interval is selected, patients are classified as “active” or LTFU as of December 31, 2007. We then document whether they return to care during the 2008 calendar year. Each bar represents a patient who has started antiretroviral therapy. The black portion represents the period for which a patient is active in care and not late for an appointment. If a patient becomes late for an appointment during the course of his or her follow-up, this is depicted by a thin black line. The white portion of each bar represents the interval in which lateness for a scheduled visit is “allowable” and not considered LTFU. The length of this white bar represents the threshold definition that we are primarily examining in this analysis. In the 2 panels, we demonstrate how differences in the LTFU threshold can affect the classification of patients. Patients categorized as a are LTFU as of December 31, 2007, and do not return to care during the next year (true positives). Those categorized as b were originally classified as LTFU but return to care (false positives). Those in c are classified as active on December 31, 2007, but fail to return for subsequent appointments (false negatives). Group d comprises patients who are classified as active at the cutoff date and remain active during the coming year (true negatives). Using this 2 × 2 table, we are able to calculate sensitivity (a/a + c) and specificity (d/b + d) for each LTFU threshold.

We evaluated the performance of the various definitions by calculating their sensitivity and specificity for determining loss to follow-up status and fitting receiver operating characteristic curves. “False-positive” cases were defined as persons who were classified as follow-up losses on December 31, 2007, but returned to care in the subsequent year. The false-positive rate for each cutoff definition of loss to follow-up (LTFU) was calculated as (non-LTFU prevalence) × (1 − specificity). “False-negative” cases for each cutoff definition of loss to follow-up were persons who were classified as active but never returned for later visits; this was defined as (LTFU prevalence) × (1 − sensitivity). The cutpoint that minimized the sum of false positives and false negatives was considered the most efficient loss to follow-up threshold—that is, a weighted sum of the sensitivity and specificity based on the prevalence of cases (23). If 2 or more time intervals had the same misclassification rate, we designated the shorter interval as the more efficient definition of loss to follow-up.

To determine whether length of enrollment affected the performance of our calculated definitions of loss to follow-up, we performed secondary analyses using different “enrollment cohorts.” We stratified patients according to the calendar year in which they started ART and, using the same method, calculated the optimal days-late threshold for defining loss to follow-up. These cohorts comprised persons starting ART between: 1) April 1, 2004, and December 31, 2004; 2) January 1, 2005, and December 31, 2005; 3) January 1, 2006, and December 31, 2006; and 4) January 1, 2007, and December 31, 2007. For each of these cohorts, December 31, 2007, was used as the cutpoint to classify patients as lost to follow-up, and we looked ahead 1 year to determine their subsequent status.

Two additional secondary analyses were performed. First, we restricted our study population to persons who were at least 1 day late for their clinical or pharmacy appointment as of December 31, 2007. We excluded the subset of patients who were active in care and had their next appointment following this cutoff date, because they could only contribute to “false-negative” misclassification. In contrast, those included in this subanalysis could potentially contribute to both “false-positive” and “false-negative” misclassification, depending on the definition of loss to follow-up used. Secondly, we tested the performance of definitions of loss to follow-up at other cutoff dates. For the primary analysis, December 31, 2007, had been chosen as a matter of convenience, but an underlying tenet of this method is that any date can be used, provided that a sufficient amount of follow-up time is available afterwards. We conducted this secondary analysis to confirm the robustness of our findings when other dates were used.

All analyses were performed using SAS, version 9.13 (SAS Institute Inc., Cary, North Carolina). Use of these programmatic data was approved by the University of Zambia (Lusaka, Zambia) Research Ethics Committee and the University of Alabama at Birmingham (Birmingham, Alabama) Institutional Review Board.

RESULTS

Between April 1, 2004, and December 31, 2007, 40,700 HIV-infected adults initiated ART at 18 program sites in Lusaka. As of December 31, 2007, 6,996 (17.2%) were known to have formally withdrawn from the program or to have died and were thus excluded. Of the remaining 33,704 patients, 23,508 (69.7%) had attended their last appointment and had a subsequent one scheduled and were thus considered “active.” Nearly one-third of patients (n = 10,196 or 30.3%) were at least 1 day late for a clinical or pharmacy visit. The median interval from the last missed appointment to the December 31, 2007, eligibility cutpoint was 123 days (interquartile range, 21–462). The distributions of the yearly enrollment cohorts are shown in Figure 2.

Figure 2.

Time from the last missed visit to December 31, 2007 (the date for determining loss to follow-up) in a multisite human immunodeficiency virus treatment cohort, Lusaka, Zambia, 2004–2007. Histograms are stratified by year of enrollment: A) 2004; B) 2005; C) 2006; D) 2007.

We evaluated the performance of various cutpoints for defining loss to follow-up, including sensitivity, specificity, and misclassification rate (Table 1), and then fitted receiver operating characteristic curves to these data (Figure 3). On the basis of our method, the best-performing definition was ≥56 days. This threshold had the lowest misclassification rate (5.1%; 95% confidence interval (CI): 4.8, 5.3), along with a sensitivity of 84.1% (95% CI: 83.2, 85.0), specificity of 97.5% (95% CI: 97.3, 97.7), a positive predictive value of 88.7% (95% CI: 87.9, 89.5), and a negative predictive value of 96.3% (95% CI: 96.1, 96.5). Interestingly, misclassification rates for ≥54 days to ≥69 days differed by less than 0.1% when compared with the misclassification rate of ≥56 days (Figure 4, part A).

Table 1.

Performance of Various Time Intervals After a Missed Appointment in Defining Loss to Follow-up Among Patients Enrolled in a Multisite Human Immunodeficiency Virus Treatment Cohort Between April 1, 2004, and December 31, 2007, Lusaka, Zambia

LTFU Thresholda, days	No. Classified as LTFU	No. Who Subsequently Returned to Care	Sensitivity, %	Specificity, %	Misclassification, %
≥7	9,269	3,308	92.6	87.9	11.2
≥14	8,397	2,539	91.0	90.7	9.3
≥21	7,686	1,907	89.8	93.0	7.6
≥28	7,063	1,377	88.3	95.0	6.3
≥35	6,730	1,110	87.3	95.9	5.7
≥42	6,499	938	86.4	96.6	5.4
≥49	6,295	798	85.4	97.1	5.2
≥56b	6,104	688	84.1	97.5	5.1
≥63	5,958	617	83.0	97.7	5.1
≥70	5,841	576	82.8	97.9	5.2
≥77	5,714	528	80.6	98.1	5.3
≥84	5,603	493	79.4	98.2	5.4
≥91	5,514	462	78.5	98.3	5.5
≥98	5,431	441	77.5	98.4	5.6
≥105	5,333	421	76.3	98.5	5.8
≥112	5,235	403	75.1	98.5	6.0
≥119	5,147	383	74.0	98.6	6.1
≥126	5,073	371	73.1	98.6	6.3
≥133	4,991	356	72.0	98.7	6.4
≥140	4,910	344	70.9	98.7	6.6
≥147	4,837	329	70.0	98.8	6.7
≥154	4,780	317	69.3	98.8	6.8
≥161	4,724	309	68.6	98.9	6.9
≥168	4,656	297	67.7	98.9	7.1
≥175	4,581	286	66.7	99.0	7.2
≥182	4,511	275	65.8	99.0	7.4

Abbreviation: LTFU, loss to follow-up.

For illustrative purposes, we show the performance of different LTFU time periods in 7-day increments.

The best-performing definition of LTFU coincided with the 56-day intervals.

Figure 3.

Receiver operator characteristic curves for adults initiating antiretroviral therapy in Lusaka, Zambia, between April 1, 2004, and December 31, 2007, by year of enrollment.

Figure 4.

Proportions of patients misclassified as active or lost to follow-up across different time intervals among adults initiating antiretroviral therapy in Lusaka, Zambia, between April 1, 2004, and December 31, 2007. In a series of sensitivity analyses, we stratified the overall population according to year of enrollment (part A); restricted our study population to only those patients who were at least 1 day late for a scheduled visit on December 31, 2007 (part B); and repeated our analysis using several cutoff dates other than December 31, 2007 (part C).

When we performed analyses of separate enrollment cohorts, the most efficient definition of loss to follow-up appeared to shorten as duration of time in the program decreased (Table 2). For persons enrolled for 3 years or longer (i.e., those who started ART prior to December 31, 2004), the most efficient definition of follow-up loss was 70 days. For persons enrolled for at least 2 years but less than 3 years (i.e., January 1, 2005– December 31, 2005), the most efficient definition of follow-up loss was 63 days. For persons enrolled for at least 1 year but less than 2 years (i.e., January 1, 2006–December 31, 2006), the most efficient definition of follow-up loss was 60 days. For persons enrolled for less than 1 year (i.e., January 1, 2007–December 31, 2007), the most efficient definition of follow-up loss was 47 days.

Table 2.

The Best-Performing Definition of Loss to Follow-up in a Multisite Human Immunodeficiency Virus Treatment Cohort, by Year of Enrollment, Lusaka, Zambia, 2004–2007a

Enrollment Dates for Cohort	Best-Performing Definition of Loss to Follow-up
	No. of Days	Misclassification, %	95% CI	Sensitivity, %	95% CI	Specificity, %	95% CI
April 1, 2004–December 31, 2004	≥70	3.0	2.5, 3.6	93.5	91.7, 95.0	98.1	97.5, 98.6
January 1, 2005–December 31, 2005	≥63	3.7	3.3, 4.1	92.6	91.3, 93.7	97.5	97.1, 97.8
January 1, 2006–December 31, 2006	≥60	4.3	3.8, 4.7	89.7	88.1, 91.1	97.2	96.8, 97.6
January 1, 2007–December 31, 2007	≥47	6.8	6.4, 7.3	68.3	66.1, 70.4	97.5	97.2, 97.8
Overall	≥56	5.1	4.8, 5.1	84.1	83.2, 85.0	97.5	97.3, 97.7

Abbreviation: CI, confidence interval.

The cutpoint for delinquency was December 31, 2007, for all cohorts.

We restricted the study population to persons who were at least 1 day late for their last clinical or pharmacy appointment as of December 31, 2007. When we evaluated the performance of loss to follow-up classifications at different time intervals, the best-performing definition was ≥56 days, identical to our primary analysis (Figure 4, part B). The proportion of patients misclassified at this threshold was 12.8% (95% CI: 12.1, 13.4); sensitivity was 89.8% (95% CI: 89.0, 90.6), specificity was 83.5% (95% CI: 82.3, 84.6), positive predictive value was 88.7% (95% CI: 87.9, 89.5), and negative predictive value was 85.0% (95% CI: 83.6, 86.1). We then evaluated the performance of definitions of loss to follow-up at cutoff dates other than December 31, 2007. This analysis yielded results nearly identical to those of our primary analysis (Figure 4, part C). The most efficient definitions using alternative cutoff days were: June 30, 2007 (62 days); September 30, 2007 (60 days); March 31, 2008 (62 days); and June 30, 2008 (59 days).

DISCUSSION

Minimizing the misclassification of patients lost to follow-up has great importance in cohort analyses and programmatic reporting. When patients are prematurely classified as lost to follow-up, they can be incorrectly censored and fail to contribute person-time to an analysis, thus contributing to underestimation of program coverage over time. For this reason, many cohort studies have utilized more liberal definitions of loss to follow-up to maximize the possibility of return. When this window is too wide, however, patients may be misclassified as active even when they will not return. A large proportion may in fact have died (14, 21, 24, 25). Here we have proposed a simple empirical approach to defining loss to follow-up in a way that can standardize program evaluation reports and research comparisons, both across and within programs. In our multisite program in Lusaka, use of ≥56 days’ lateness for a scheduled visit led to the fewest misclassifications of loss to follow-up. However, given the minute differences in misclassification seen in the intervals immediately preceding and following 56 days (Figure 4), we suggest the use of 2 months—or 60 days—as the lateness threshold for defining loss to follow-up. This interval is longer than the definition of 30 days that we have used in our previous program evaluations (4–11) but shorter than that used in many other programs (2, 14–16).

The strengths of this method are its relative simplicity and empiric approach, suggesting a standardized, evidence-based definition of loss to follow-up. We recognize several limitations to our analysis. This technique only considers the patient's last missed visit; we did not take into account previous missed visits and thus failed to utilize all available data. Our approach was designed to most accurately classify patients as either active or lost to follow-up at a single point in time, as would be required for program reporting or cohort analysis. This method is not conducive to more complicated analysis regarding factors associated with loss to follow-up, since persons who died or withdrew from the program prior to the date of classification were excluded. Although our definition of follow-up loss was robust across a number of sensitivity analyses, it is unknown whether it can be generalized to locations outside of our urban primary-care setting in Zambia. Several program characteristics may have affected our results, including an active patient tracking system for missed visits, the provision of free ART, and the use of sophisticated electronic medical records linking care data across multiple clinics. Similar analyses should be performed in other settings to determine how these and other program features influence optimal definitions of loss to follow-up.

Our method begins with the selection of an arbitrary date (i.e., December 31, 2007), from which patients are categorized as active or lost to follow-up according to a range of days-late thresholds. Once categorized as lost to follow-up, all patients are provided the same opportunity to return to care irrespective of time since the last visit; in this analysis, this was the calendar year of January 1, 2008–December 31, 2008. We recognize, however, that the likelihood that such a person will return to care varies inversely with the length of time since his or her missed visit. A patient whose last scheduled visit was 2 years ago, for example, is far less likely to return to care than someone who missed a visit just 2 months ago. Because the chances of such misclassification may be reduced among “older” enrollment cohorts, the performance of the “most efficient” definition of loss to follow-up was generally better in those groups. While specificity remained approximately the same, sensitivity increased among persons who had been enrolled in the program for a longer time (Table 2).

When we examined the proportion of ART patients misclassified as either active or lost to follow-up, the greatest change in misclassification was seen over the first 30 days. There appeared to be a broad nadir in the 30- to 90-day range, followed by a slight increase over the intervals that followed. While our method can determine the best-performing definition of loss to follow-up with great precision, as Figure 4 demonstrates, there probably exists a range of “acceptable” thresholds from the program perspective. This flexibility prompted us to recommend a 60-day threshold for defining loss to follow-up, which we found to be more intuitive and only marginally less efficient than the best-performing 56-day threshold in our setting.

We observed differences in the optimal definition of loss to follow-up when stratifying by enrollment date. Persons who had enrolled most recently had the shortest loss to follow-up windows; as time in the program increased, so did the length of this best-performing interval. The reasons behind this finding are unclear, but our exclusion of deaths and program withdrawals from the analysis may have played an important role. If 1 group were to have better status ascertainment—a likely scenario for earlier (vs. later) enrollment cohorts—then the differential exclusion of these patients could contribute to the variability from stratum to stratum.

When we investigated the interval between the missed visit and December 31, 2007, we found a bimodal distribution of these data for each enrollment cohort. A common feature of each was the prominent peak within 90 days, suggesting that the majority of missed visits had occurred only a few months earlier. However, we consistently observed a smaller peak for follow-up losses, generally coinciding with the year of enrollment for each cohort. This suggests a high rate of attrition early in a patient's ART course, with unreported early mortality being a likely and important contributor to these losses (26).

Arbitrary and variable definitions of loss to follow-up in ART programs are now extant, limiting the usefulness of monitoring and evaluation activities within and across programs. Here we have proposed a simple approach to determining the best-performing criterion for classifying patients as lost to follow-up. On the basis of data from the multisite Lusaka program, we suggest a 2-month (60-day) lateness threshold for defining loss to follow-up. Since the basic data needed to calculate such thresholds are found in many electronic medical records in sub-Saharan Africa (27), this method should be considered for program-, country-, and/or region-specific definitions of loss to follow-up.