Project ECHO Revisited: Propensity Score Analysis And HCV Treatment Outcomes.

Propensity score analysis is a statistical approach to reduce bias often present in non-randomized observational studies. In this paper we use this method to re-analyze data from a study that assessed whether patients receiving HCV treatment from providers in Project ECHO had different clinical outcomes than patients treated by specialists from an academic medical center (UNM HCV clinic) but in which treatment assignment was not randomized. We modeled the best estimated probability of treatment assignment, and then assess differences overall SVR and SVR in patients with genotype 1 infection by treatment arm using Stabilized Inverse Probability of Treatment Weights (SIPTW). Results show that after adjustment for SIPTW, HCV treatment outcomes were significantly better for the ECHO patients compared to the UNM HCV clinic patients. Higher proportions of patients treated by primary care providers achieved SVR and SVR with genotype 1 compared to those treated at UNM HCV clinic with 15.1% and 16.3% absolute differences, respectively. These results indicate that previously published results (showing no differences) were biased, and resulted in an underestimation of the treatment effect of ECHO on HCV treatment.

Introduction

A prospective cohort study from 2004 through 2008, conducted by Arora et al,1 comparing HCV treatment outcomes in patients treated for hepatitis C virus (HCV) infection at specialty clinics in an academic medical center (University of New Mexico (UNM)) and patients treated by primary care providers in Project ECHO showed no differences in sustained viral response (SVR) between the groups. A total of 57.5% (84 of 146) of patients treated at UNM HCV clinic achieved SVR and 58.2% (152 of 261) patients treated at ECHO sites achieved SVR. The difference in the proportion achieving SVR was 0.7 percentage points (95% confidence interval (CI); −9.2 to 10.7; P=0.89). Among patients with HCV genotype 1 infection, SVR was 45.8% (38 of 83 patients) at the UNM HCV clinic and 49.7% (73 of 147 patients) at ECHO sites (P=0.57). The impact of these results was significant as they showed that complex interferon-based HCV treatment could be effectively delivered outside of specialty care using a telehealth knowledge dissemination model. The ECHO (Extension for Community Healthcare Outcomes) model, developed as a platform to deliver complex specialty medical care and improve access of minorities and underserved populations to best practice care through an educational model of team-based interdisciplinary development - worked.1

This study’s limitations, however, were evident and discussed. As with many observational studies of clinical care, this was not a randomized trial; in a randomized trial patients and or providers would be randomly assigned to treatment or control arms. With no control over treatment assignment, treated – ECHO group, and non-treated- UNM HCV clinic group, may have had large differences on their observed covariates - imbalances, which could result in biased estimates of the treatment effects. While the study used a multivariable analysis approach to control for differences that were measured, such as race/ethnicity, age, and other factors, this type of covariance analysis may still not adequately eliminate bias. Thus, propensity score analysis is a statistical approach to balancing covariates and reducing bias in analyses of non-randomized observational studies.

We revisited the Project ECHO HCV study and utilized the propensity score method to model the best estimated probability of treatment assignment, and then assess differences in primary study outcomes including overall SVR and SVR in patients with genotype 1 infection2 by treatment arm (ECHO vs. UNM HCV clinic).

Methods

We note that if the treatment effect were related to a covariate, then an imbalance in the mean of this covariate between the two arms of the study will tend to cause a biased estimate of the treatment effect. Propensity score analysis attempts to eliminate the bias in the covariate and thereby give an unbiased estimate of the treatment effect. Propensity scoring computes weights (to be used later in weighted analyses) to modify the multivariate distribution of covariates in the treatment and control arms in an attempt to balance all covariates in mean simultaneously. These weights are computed by considering treatment versus control arms as a binary outcome and using logistic regression to compute, as predicted values, the probabilities that the outcome = treatment arm given the covariate values. With this conceptual/explanatory discussion given, a technical description follows.

First, we conducted multiple logistic regression to build a non-parsimonious model with the treatment versus control arms as the outcome using all data on factors and covariates that could potentially influence the treatment and possibly the outcome3,4 as the independent variables. A total of 27 variables were included: age; sex; minority group (race/ethnicity); marital status; employment status; housing status; HCV viral load, body mass index (BMI); HCV genotype; creatinine, hematocrit, aspartate aminotransferase, alanine aminotransferase, alkaline phosphatase, total bilirubin, total protein, albumin, white blood cell counts, platelet, absolute neutrophil counts; red-cell distribution width; mean corpuscular volume; treatment completed status, treatment duration, follow up time after treatment, and total follow up time. From this model, we calculated the propensity score as the reciprocal of the probability of receiving the treatment that was actually received, Inverse Probability of Treatment Weight (IPTW), and the Stabilized Inverse Probability of Treatment Weight (SIPTW).5 For our study we use SIPTW, which is an improvement over IPTW to reduce potential weights of subjects with either low or high propensity scores.6 Parameter estimates for this model were obtained by using the penalized likelihood to avoid and minimize the chance of overfitting.7

The second step in the reanalysis was to apply the weights to modeling the treatment outcomes (SVR and SVR in genotype 1 infection): adjusting by the SIPTWs to account for any treatment selection bias. We use the area under the receiver operating characteristic (ROC) curve to assess the discrimination power of the non-parsimonious model and Hosmer-Lemeshow test to evaluate its calibration.8 To examine the assumptions of common support and balance we explore both the distribution of the propensity scores9,10 and SIPTWs for the treatment (ECHO) and control (UNM HCV clinic) respectively.

Results

The non-parsimonious model exhibited a high discrimination power between the two treatment groups (ECHO vs. UNM HCV clinic) with an area under the ROC curve of 0.880 (Figure 1). It also showed good calibration according to the Hosmer-Lemeshow statistics (χ2 = 9.948, df = 8, P = 0.269). The common support and balance assumptions were satisfactory for this problem. Specifically, the SIPTWs exhibited similar distributions and the distributions of propensity scores between the two treatments completely overlapped between the two treatments with means close to one (ECHO: M=1.0, SD=0.8; UNM HCV clinic: M=1.2, SD=4.7), suggesting no misspecification of the used model.11

Figure 1

Left panel: ROC curve for classifying ECHO vs. UNM HCV clinic. Right panel: distribution of SIPTWs by treatment arm.

Our results show that after adjustment for SIPTW, HCV treatment outcomes were significantly better for the ECHO patients compared to the UNM HCV clinic patients (Table 1). Higher proportions of patients treated by primary care providers achieved SVR and SVR with genotype 1 compared to those treated at UNM HCV clinic with 15.1% and 16.3% absolute differences, respectively.

Table 1

HCV Treatment Outcomes By ECHO Vs. UNM HCV Clinic: Original Analyses Vs. Analyses Adjusted For SIPTW

	Unadjusted (Previous Analysis)			Adjusted By SIPTW*
ECHO	UNM HCV clinic	P-value	ECHO	UNM HCV Clinic	p-Value
Proportion with SVR	58.2%	57.5%	0.890	58.1%	43.0%	0.003
Proportion of patients with HCV genotype 1 with SVR	49.7%	45.8%	0.572	48.2%	31.9%	0.008

Note: *Stabilized Inverse Probability of Treatment Weight.

Discussion

Our results indicate that by using propensity scoring with measured covariates in this observational study, we reduced bias and achieved improved precision of the difference in HCV treatment results for HCV infection in ECHO compared to UNM HCV clinic patients. And importantly, the results show that patients treated by ECHO providers had significantly better outcomes than those treated by specialty providers at the academic medical center. These results indicate that previously published results (showing no differences) were biased, and resulted in an underestimation of the treatment effect of ECHO on HCV treatment. Analyses to further explore the sources of this bias showed three factors that primarily contributed to the direction of the adjusted findings: male sex, creatinine levels and white blood cell count. For example, there were many more males in ECHO (72%) than in UNM (45%) and males had a somewhat lower SVR rate; this is a selection bias that artificially caused ECHO to have a lower SVR rate than would have been observed in a randomized clinical trial.

This analysis benefitted from a large enough sample size (406 patients: 261 were treated at ECHO sites and 146 were treated at UNM HCV clinic) to implement inverse probability weighting. While there are several strategies to building the multiple logistic regression models to calculate the propensity scores, we used strategy which utilizes a non-parsimonious model as recommended by most authors.2,12–18 We had similar findings with better outcomes in ECHO patients when we used a parsimonious model using methods described by Shadish et al,14 wherein 19 covariates and factors were used instead of 27.

This previously published study is an important example of how propensity scoring can (and should) be used in statistical analysis in applied medicine, where treatment assignment is not randomized. Further these results confirm previous findings showing that the ECHO model is an effective way to deliver HCV treatment to in underserved communities. This research in this study was conducted when interferon-based treatment was standard of care and treatment success rates were much lower than with current direct-acting antiviral (DAA) treatments, which are effective in over 95% of patients.19 As the ECHO model has now been expanded into new clinical areas and populations and with the new highly effective DAA treatment, ongoing evidence will still be needed to measure effectiveness and impact. Using propensity scoring for further observational analyses has important potential provide unbiased and better estimates of these outcomes.