The use of funnel plots with regression as a tool to visually compare HIV treatment outcomes between centres adjusting for patient characteristics and size: a UK Collaborative HIV Cohort study.

OBJECTIVES: A measure used for assessing the effectiveness of HIV care and comparing clinical centres is the proportion of people starting antiretroviral therapy (ART) with viral suppression (VS) after 1 year. We propose a method that adjusts for patients' demographic characteristics, and visually compares this measure between different sites accounting for centre size.
METHODS: We analysed viral load measurements for UK Collaborative HIV Cohort (UK CHIC) patients starting ART between 2006 and 2013. We used logistic regression to estimate the proportion with VS after 1 year of ART adjusted for patient mix (in terms of age and a combined gender/ethnicity/acquisition mode variable) and calendar year. We compared outcomes between centres using funnel plots which account for centre size.
RESULTS: The overall proportion of the cohort with VS 1 year after starting ART was 90% and increased from 83% to 93% between 2006 and 2013. VS was lower in younger individuals. White men who have sex with men (MSM) had the highest (94%), and black African (81%) and white (82%) heterosexual women the lowest proportions achieving VS. Comparing the unadjusted funnel plot with the adjusted, there were movements of some centres from outside to inside the 95% contour limits, which was largely explained by the patient mix of these centres.
CONCLUSIONS: VS 1 year after ART start was associated with demographic characteristics and centre size; therefore, to compare the performances of centres, adjustment for these factors is required. Adjusted funnel plot is an effective tool which accounts for both the demographic characteristics and the centre size. Social factors, rather than treatment decisions within the control of the centres, may drive differences in outcomes.

Introduction

In health care settings, there is a need to assess services. This is important in HIV services to obtain an overview of the health of the community, for commissioning services, and for monitoring of treatment outcomes. Measures should be carefully defined before being used to monitor aspects of care. Understanding how patient characteristics and centre size influence performance measures is important for their effective use in monitoring patient care in different settings.

In monitoring an HIV treatment programme, a number of measures have been chosen 1, 2, 3, 4, 5. A standard that is often used is the proportion of people starting treatment with antiretroviral therapy (ART) who have an undetectable viral load (VL) 1 year after starting ART. The proportion of people achieving an undetectable VL is an important indicator of programme performance as it is a marker of treatment adherence/success and has important implications for reducing onward HIV transmission 6 and for individual outcomes 7, 8. However, patient‐ as well as provider‐level factors are strongly associated with the achievement of an undetectable VL, including mode of HIV acquisition, age, gender and ethnicity 9, 10. As the characteristics of those attending different centres (the ‘case mix’) will vary, we need a tool that permits researchers to illustrate and compare performance measures, such as the proportion of people with an undetectable VL, after adjusting for any case‐mix differences.

The aim of this study was to propose a methodology for comparing HIV outcomes between centres, rather than understanding the root causes of those differences, which would require more detailed data on reasons for treatment failure. We used data from the UK Collaborative HIV Cohort (UK CHIC) study 11 to compare the proportions of people with an undetectable VL at 1 year after starting ART between UK HIV treatment centres. After investigating differences in case mix between the centres, we used funnel plots to describe the differences in this performance measure between centres and to visually illustrate the impact of case mix on the interpretation of results from these analyses.

Methods

The UK CHIC study was initiated in 2001 and collates routine data on HIV‐positive individuals who have attended a variety of clinical centres in the UK since 1 January 1996 (see Appendix 1). The project was approved by a multi‐centre research ethics committee. In accordance with data protection policy, data were provided in an anonymized format with any potential identifiers removed. The criteria for inclusion of an individual in the UK CHIC study are that they are HIV positive, have attended one of the collaborating centres at any time since 1996 and are aged ≥ 16 years. The present study was conducted within the National Institute for Health Research Health Protection Research Unit (NIHR HPRU) in Blood Borne and Sexually Transmitted Infections (University College London in collaboration with the London School of Hygiene and Tropical Medicine), and in Evaluation of Interventions (University of Bristol) in partnership with Public Health England (see Appendix 1).

In this study, eligible study participants were aged ≥ 20 years and started ART between 2006 and 2013. We excluded from analyses people who transferred care or died within 10 months of initiating ART and those who did not have at least one VL recorded during that period. Of the 19 eligible centres, data from two of them (labelled as 6 and 13) were excluded for the year 2012 and data from another one (labelled as 19) was included for the period 2009–2012 only, because of incomplete data in these centres for the periods of exclusion. We calculated the proportion of people with an undetectable VL at 1 year after starting ART (defined in our primary analyses as < 200 HIV‐1 RNA copies/ml, but in sensitivity analyses as < 50 copies/mL). We used the HIV‐1 RNA measurement that was recorded closest to 1 year after ART start, within a window of up to 13 months after start of ART, in order to allow for possible late recording of measurements. People whose only record was at ART start would be considered as not having suppressed VL at 1 year, if that measurement was > 200 copies/mL (or 50 copies/mL).

We tabulated the proportion of people with an undetectable VL by demographic characteristics and by calendar year. As gender, ethnicity and mode of HIV acquisition were highly correlated, we generated a combined variable (gender/ethnicity/acquisition mode) for the purposes of our analyses (with smaller categories combined as ‘other’). We graphed the proportion of people with undetectable VL by age, gender/ethnicity/acquisition mode, and centre, and used logistic regression to estimate the odds ratio (OR) for VL suppression by each factor.

Adjusting for case mix

We used logistic regression to model and predict the proportion of individuals with viral suppression in each centre, firstly unadjusted, and secondly adjusted for case mix (age and gender/ethnic/acquisition mode). We fitted separate models for each calendar year of ART initiation from 2006 to 2013 to account for the increasing trend in viral suppression over the period of our study. We first estimated the unadjusted and adjusted ORs of viral suppression at 1 year and used these to predict the unadjusted and adjusted probabilities of viral suppression for each person attending in that year. For each centre, the “observed” proportions of people with VL < 200 copies/mL were generated by computing the average over all years of the unadjusted and adjusted probabilities for people attending that centre.

Funnel plots

We chose to use funnel plots to compare graphically the unadjusted and adjusted measures as, by construction, they account for the centre size. A funnel plot shows the scatter of the observed measures for different centres compared to the expected value which is the average of all centres. The ratio of observed:expected measure is plotted against centre size. The contours representing the upper and lower confidence interval for the expected proportion are wider for small centres and narrower for large centres. The target value of the ratio (observed:expected proportion) is 1 and centres falling below the lower 95% contour limits are considered to have an observed proportion significantly lower than expected (Fig. S1). A detailed explanation of how to obtain adjusted funnel plots as used in this paper can be found in Spiegelhalter and in Forni and Gini 12, 13.

We constructed funnel plots for the ratio of the observed:expected proportion of people with undetectable VL plotted against centre size firstly using the unadjusted data and secondly adjusting for case mix. For the unadjusted funnel plot, we did not adjust for case mix, but did account for changes in viral suppression over time in each centre by taking the weighted average of the yearly proportions, with weights equal to the number of people attending that centre per year. For the adjusted funnel plots, the ratio of the expected proportion after adjustment to the observed proportion was calculated for each centre and plotted against centre size. We graphed a histogram of case mix by centre to inform the explanation of the movement of the position of centres between the unadjusted and adjusted funnel plots. In a sensitivity analysis, we constructed the funnel plots excluding the largest centre.

Results

The demographic characteristics of the 17 541 eligible participants are presented in Table 1. In the cohort, 90% of people overall achieved VL < 200 copies/mL 1 year after starting ART and this proportion increased from 83% in 2006 to 93% in 2013 (Fig. 1). The corresponding increase using the lower threshold for viral suppression of < 50 copies/mL was from 74% in 2006 to 87% in 2013, with an average of 83% viral suppression overall during this period. The proportion with undetectable VL was lower for younger people, for women, and for those with black African or black Caribbean ethnicity. Men who have sex with men (MSM) had the highest, and people who inject drugs (PWID) the lowest, rates of viral suppression (Table 1).

Table 1

Percentage of people with viral suppression at 1 year after antiretroviral therapy (ART) start by demographic characteristics

Demographic variable	Group	Number in cohort	Percentage in cohort	VL < 200 copies/ml at 1 year (%)
Age	20–29 years	2984	17.01	85.16
	30–39 years	6920	39.45	89.02
	40–49 years	5349	30.49	92.56
	> 50 years	2288	13.04	92.66
Gender	Male (M)	12,828	73.13	92.98
	Female (F)	4713	26.87	81.58
Ethnicity	White (W)	9499	54.15	92.55
	Black Caribbean	588	3.35	84.86
	Black African (BA)	4918	28.04	84.99
	Black unknown	426	2.43	84.04
	Indian/Pakistani/Bangladeshi	271	1.54	92.25
	Other Asian	401	2.29	93.27
	Other mixed (OM)	753	4.29	92.70
	Other (O)	331	1.89	92.15
	Unknown (U)	354	2.02	89.55
Acquisition mode	MSM	8955	51.05	94.21
	Injecting drug users	329	1.88	76.90
	Heterosexuals (HS)	6909	39.39	85.37
	Other (O)	116	0.66	86.21
	Unknown (U)	1232	7.02	88.07
Gender/ethnicity/acquisition modea	M/W/MSM	7310	41.67	94.20
	F/BA/HS	2984	17.01	81.43
	M/BA/HS	1463	8.34	91.52
	M/W/HS	655	3.73	91.76
	M/W/U	632	3.60	90.82
	F/W/HS	533	3.04	82.18
	M/OM/MSM	442	2.52	96.38
	Other	3522	20.08	87.39

Groups of the gender/ethnicity/acquisition mode variable are: white MSM (M/W/MSM), black African heterosexual women (F/BA/HS), black African heterosexual men (M/BA/HS), white heterosexual men (M/W/HS), white men of unknown acquisition mode (M/W/U), white heterosexual women (F/W/HS), and MSM of other mixed ethnicity (M/OM/MSM). All other combinations are included in the ‘Other’ group.

MSM, men who have sex with men; VL, viral load.

Figure 1

Trend in proportion of people with VL< 50 and < 200 copies/ml at 1 year after start of ART.

The proportion of people with viral suppression at 1 year increased with age (Fig. 2a). There were significant differences in viral suppression between HIV acquisition mode groups (Fig. 2b). In particular, white MSM had a higher proportion (94%) achieving VL < 200 copies/mL compared with other large groups, while the lowest proportions were observed for black African and white heterosexual women (81% and 82%, respectively). The proportion of people achieving viral suppression increased over calendar year for all age groups (Fig. 2c) and for all categories of patients (Fig. 2d).

Figure 2

Proportion (with 95% CI) of eligible participants with VL < 200 copies/mL at 1 year by age group (left) and by sex/ethnicity/acquisition mode group (right). Upper panels show overall proportion suppressed with 95% confidence intervals and lower panels show trends over time.

Centres with more patients tended to have a higher proportion of people who achieved viral suppression; furthermore, sampling variability was greater for smaller centres, as shown by the wider confidence intervals around this proportion (Fig. 3).

Figure 3

Proportion of people with VL < 200 copies/mL at 1 year by centre (left) and by number of eligible participants (centre size) (right).

Effects of adjustment for case mix

Table 2 compares the observed proportion of people with viral suppression for each centre with the expected proportions before and after adjustment for case mix. Compared to the unadjusted proportion, the adjusted expected proportion with viral suppression is closer to the observed proportion for most of the centres (highlighted with grey colour in the table).

Table 2

Difference between unadjusted and risk‐adjusted proportions of people with viral suppression

Centre label	Number of eligible participants	Observed proportion (%)	Unadjusted expected proportion (%)	Adjusted expected proportion (%)
1	1004	90.84	89.87	92.42
2	1432	92.67	89.91	91.23
3	4213	94.49	90.41	92.47
4	1920	91.77	89.68	91.13
5	1449	85.02	89.72	87.47
6a	1193	89.10	89.10	89.54
7	1299	87.61	89.90	90.05
8	338	88.17	89.60	88.28
9	578	82.70	89.61	85.97
10	543	87.66	90.20	86.83
11	625	90.72	89.83	89.22
12	721	90.43	89.44	86.58
13a	188	92.55	89.84	88.67
14	630	83.02	90.05	86.34
15	283	87.63	90.62	87.27
16	214	82.71	89.90	87.18
17	107	85.05	91.33	88.26
18	486	79.63	89.48	87.12
19a	318	89.94	89.87	86.65

Highlighted (light grey) centres are those whose case‐mix adjusted proportion was closer to the observed value compared to the unadjusted proportion.

Centres with missing data for some periods between 2006 and 2013.

The results for all UK CHIC centres included in our analysis are illustrated in the funnel plots (Fig. 4). In the unadjusted case, centres 5, 14 and 18 have lower than expected proportions of people with viral suppression (falling below or exactly on the 95% lower contour). However, after adjustment, these centres move inside the 95% contour limits. In contrast, while centre 3 seemed to do a lot better than expected prior to adjustment, it moved within the expected range after adjustment.

Figure 4

Unadjusted (left) and adjusted (right) funnel plots showing ratio of observed to expected proportion of people with undetectable viral load (VL) vs. size of centre.

The case mix of attendees varied by centre (Fig. 5). For example, centre 3 had a high proportion of white MSM, a group that is more likely to have a suppressed VL after 1 year compared with other groups, and low proportions of black African and white women, groups that tend to be less likely to have viral suppression. The opposite situation occurred with centres 5, 14 and 18, which had high proportions of groups that were less likely to have undetectable VL (i.e. black African women, white men with unknown mode of HIV acquisition and those with an ‘other’ mode of acquisition). These observations explain the movement of the points in the adjusted compared with the unadjusted funnel plots (circled points in Fig. 4). As centre 3 had many more attendees than the other centres and had the highest proportion of people with undetectable VL, it had a strong influence on the shape of the funnel plots, extending them to the right, and made a large contribution to the estimate of the average, that is, the position of the horizontal line representing a ratio of 1. When we excluded centre 3, centres 5 and 14 fell well inside the contour limits and only centre 18 remained outside the 95% contour lines in the unadjusted funnel plot (Fig. S2). On adjustment, centre 18 moved well inside the funnel, indicating that the proportion of people with viral suppression was as expected.

Figure 5

Stacked bar chart showing variation in case‐mix (gender/ethnicity/acquisition mode) between centres.

Discussion

Our study shows that funnel plots used in conjunction with regression modelling are a useful tool for monitoring centre performance for an important clinical outcome in a fair way that accounts for centre size and case mix. In this application, we have investigated differences between HIV treatment centres in the proportion of HIV‐positive people that have undetectable VL 1 year after starting ART. The funnel shape of the 95% confidence interval contours correctly allows for the greater sampling variation that occurs in smaller centres. The plots also provide a benchmark of the expected performance that is based on an average over all centres.

Our data clearly showed that the likelihood of having undetectable VL at 1 year after ART start was associated with individual characteristics. Furthermore, we showed that case mix was very variable between centres, with some treating mostly MSM and others treating mostly those infected via non‐MSM routes. While in the unadjusted funnel plots some of the centres were outside the expected 95% confidence limits, adjusting for case mix resulted in all centres being within the confidence limits on the funnel plots. The generation of funnel plots using the methodology described provides an objective way of comparing different centres with different case mixes.

The data on VL show that for the UK CHIC population there is an impact on viral suppression of age at start of treatment and sex/ethnic/acquisition mode group. Younger people are less likely to have undetectable VL than older people. Viral suppression is also more likely in MSM than in heterosexuals, in men than in women, and in white people compared with those of black African ethnicity. While these are not new findings 14, what we have done is use this information to standardize the population. The adjustments of the data to account for the variations in patient characteristics indicated that centres that were outside the 95% confidence limit all became acceptable performers. Similarly, centres near the boundary became closer to the expected value. In contrast, centre 3, which was the largest centre, treated mostly MSM and had a much higher than expected proportion of people with undetectable VL in unadjusted analysis, became closer to the expected value after adjustment for case mix. This shows how important the case mix is in determining a centre's assessed ‘ability’ to achieve viral suppression in its local population.

Limitations

While UK CHIC includes a good geographical representation of the UK epidemic, it is not fully national (with around 40% coverage). Previous work has shown that it is a representative sample of people receiving treatment for HIV infection in the UK 15. Funnel plots may be biased by the inclusion of data from very large centres. Because large centres contribute most to the calculation of the mean used for benchmarking, they may skew the data and bias the funnel plot. This will happen if large centres are unrepresentative, as is the case here where the largest centre was treating mostly MSM and had the highest proportion of people with an undetectable VL. The benchmarking line was biased upward and other centres were therefore more likely to fall below the lower 95% confidence interval. Therefore, large centres can exert undue influence on the funnel plot. If standards for commissioning are based on how the largest centres which cater mostly for MSM are performing, then other centres may appear to be falling behind. However, we investigated this by re‐running the funnel plots excluding centre 3.

There are a number of other limitations in our analysis of VL. We did not account for frequency of VL testing, VL assay, or presence of resistance in our model, which may have varied by centre and therefore affected our results. We hypothesized that VL measurements might be more frequent in those whose VL declined more slowly. However, we found that VL measurements were less frequent in those with unsuppressed VL at 1 year. We could only adjust for differences in a few key patient demographics, but this only partially captures the true case mix of individuals attending different centres. Other social factors not measured in our study, such as migration status, poverty, social support, and mental health, may affect individuals’ ability to adhere to ART. It is perhaps remarkable that the observed centre variation has been explained adequately, that is, within the statistical limits that might indicate it is attributable to chance, only by demographic case mix. Of course, we are only analysing the outcomes of people who start ART and who are monitored – clearly, some of the patients may never start ART, or may not regularly attend centres to have their VL measured. In our analysis, we did not include individuals who transferred their care in the first 10 months after starting ART or who died as they could not contribute to the assessment of VL at 1 year. However, we did include individuals who may have stopped ART before 1 year. Therefore we are analysing the effectiveness of HIV care rather than the efficacy of ART. The proportion of patients with suppressed virus would have been higher if we had restricted analyses to those who remained on and adherent to ART at 1 year. We do not know the reasons for lack of viral suppression, nor whether these varied by demographic group. For example, there may be some women who started ART in pregnancy to prevent mother‐to‐child transmission who did not continue treatment after delivery, although guidelines since 2010 have recommended lifelong ART for the mother's own health needs.

There is a considerable focus on the cascade of care in monitoring the HIV epidemic, with a goal to achieve 90:90:90, that is 90% of those with HIV diagnosed, 90% of those diagnosed on treatment, and 90% of treated patients achieving undetectable VL 16. When population targets are based on a set level then it does not matter what the case mix is when you are considering whether countries or centres have met the target. However, adjustment for case mix and consideration of centre size is important when direct comparisons are made between centres and any differences are attributed to inadequacies in the care that is provided. Our study demonstrates how illustrative it is to use the funnel plot as a method. Comparisons made in this way, are very helpful to those providing services, as a benchmark, and those commissioning, as an accurate performance measure.

Conclusions

Funnel plots accounting for expected variability attributable to centre size and adjusting for case mix are useful tools for monitoring performance in HIV treatment centres and for assessing comparative progress in a fair manner at the local level.

In this UK‐based study, there was a significant impact of age, sex, ethnicity and mode of HIV acquisition group on the proportion of people with undetectable VL 1 year after starting ART. Adjustment by these variables changed the relative positions of some of the centres in the funnel plot. Those centres that had higher or lower observed compared to expected proportion of patients with undetectable VL were no longer outside the 95% contour line in the adjusted funnel plot. Moreover, suppression of VL seemed to be lower at smaller centres, but the use of the adjusted funnel plot suggested that all of them fell within the expected region defined by the 95% contour limits. Our findings suggest that differences in outcomes of care between HIV centres are more likely to be attributable to patient case mix, which depends on geographical location and social factors, rather than treatment decisions within the control of clinicians.

Fig. S1 Showing the construction and interpretation of a Funnel plot.

Click here for additional data file.

Fig. S2 For comparison with Fig. 4 ‐ Unadjusted (left) and adjusted (right) funnel plots showing ratio of observed to expected proportion of people with undetectable viral load (VL) vs. size of centre omitting the largest centre (centre 3).

Click here for additional data file.

Table S1. Showing the demographic distribution of the excluded cases (transferred care and no VL measurement within 10 months).

Click here for additional data file.