Impact of a Provider Tele-mentoring Learning Model on the Care of Medicaid-enrolled Patients With Diabetes.

BACKGROUND: Project ECHO (Extension for Community Healthcare Outcomes), a tele-mentoring program for health care providers, has been shown to improve provider-reported outcomes, but there is insufficient research on patient-level outcomes.
OBJECTIVES: To evaluate the impact of primary care provider (PCP) participation in Project ECHO on the care of Medicaid enrollees with diabetes. RESEARCH
DESIGN: New Jersey Medicaid claims and encounter data and difference-in-differences models were used to compare utilization and spending between Medicaid patients seen by PCPs participating in a Project ECHO program to those of matched nonparticipating PCPs.
SUBJECTS: A total of 1776 adult Medicaid beneficiaries (318 with diabetes), attributed to 25 participating PCPs; and 9126 total (1454 diabetic) beneficiaries attributed to 119 nonparticipating PCPs. MEASURES: Utilization and spending for total inpatient, diabetes-related inpatient, emergency department, primary care, and endocrinologist services; utilization of hemoglobin A1c tests, eye exams, and diabetes prescription medications among diabetics, and total Medicaid spending.
RESULTS: Participation in Project ECHO was associated with decreases of 44.3% in inpatient admissions (P=0.001) and 61.9% in inpatient spending (P=0.021) among treatment relative to comparison patients. Signs of most other outcome estimates were consistent with hypothesized program effects but without statistical significance. Sensitivity analyses largely confirmed these findings.
CONCLUSIONS: We find evidence that Project ECHO participation was associated with large and statistically significant reductions of inpatient hospitalization and spending. The study was observational and limited by a small sample of participating PCPs. This study demonstrates the feasibility and potential value of quasi-experimental evaluation of Project ECHO patient outcomes using claims data.

Affecting nearly 13% of US adults in 2018, diabetes is a common chronic condition with prevalence expected to be as high as 21% by 2050.1–3 Diabetes is responsible for significant public health, social, and economic costs, imposing significant burden on public payers including Medicaid.2,4,5 Even though quality of care for patients with diabetes has improved over the past 2 decades, diabetes remained the seventh leading cause of death in 2017, with many patients failing to meet recommended care guidelines.2 Racial and ethnic minorities and low-income populations are disproportionately affected by diabetes and experience lower quality of care and higher rates of complications.2,6–11 Diabetic patients often rely on primary care providers (PCPs) for care, but PCPs may face challenges in providing optimal care to patients with complex care needs.12–16 The role of PCPs is especially important considering research highlighting potential shortages of endocrinologists.17 With the advent of newer therapeutics and practice innovations, there are significant opportunities to address unmet need for patients with diabetes in primary care settings.14,16,18,19

Project ECHO (Extension for Community Healthcare Outcomes) is a tele-mentoring program for health care providers introduced in 2003. Subsequently, tele-mentoring programs have been applied globally to a broad range of health conditions in nearly 900 individual programs.20,21 Tele-mentoring programs have been shown to improve provider-reported outcomes including satisfaction, confidence, self-efficacy, and behaviors, with more limited evidence supporting efficacy in patient-level outcomes related to hepatitis C, chronic liver disease, chronic pain management and opioid addiction, and geriatric care.22 A limited number of studies have examined outcomes for patients in Project ECHO programs addressing complex endocrinology (hereafter “EndoECHO”).23,24 One study of 2 PCP organizations showed clinically significant improvement in glycemic control for 39 Veterans Health Administration patients with hemoglobin A1c (HbA1c)  > 9.0%.22,25 Researchers evaluating an EndoECHO intervention in 10 New Mexico health centers for Medicaid enrollees with complex diabetes found increased outpatient visits and higher use of beneficial medications, but also documented increased emergency department (ED) utilization and no impact on inpatient admissions.26

Given the scant evidence on patient outcomes among Project ECHO and similar programs generally, and the limited and mixed evidence for patients with diabetes specifically, there is need for additional patient outcome evaluation of Project ECHO.24 This observation was underscored in a report to Congress which concluded that the evidence for improved patient outcomes from ECHO and ECHO-like models was limited.24,27 This article seeks to address this research gap by evaluating an EndoECHO project in New Jersey (NJ) organized by Rutgers Robert Wood Johnson Medical School (RWJMS) to engage PCPs serving Medicaid enrollees to improve care for patients with complex endocrine and metabolic conditions.

This article draws on 5 years of NJ Medicaid data to compare utilization and spending outcomes among Medicaid patients attributed to PCPs participating in the EndoECHO program to those attributed to matched nonparticipating providers. Difference-in-differences (DD) models at the patient level are utilized to examine program effectiveness. We hypothesized that participation in EndoECHO will be associated with reduced total spending; ED, inpatient, and ambulatory endocrinology utilization and spending; increased PCP visits and spending; and higher use of evidence-based care. We also examine patterns of diabetes medication prescribing patterns because they may mediate other utilization and spending outcomes.

METHODS

Intervention

Fifty-three providers, including 48 PCPs (physicians and nurse practitioners) and 5 pharmacists, enrolled in the RWJMS EndoECHO program, which involved weekly 1-hour tele-mentoring sessions between February 2017 and July 2018. Supplemental Digital Content 1 provides further program details (http://links.lww.com/MLR/C400). The curriculum covered topics related to endocrine and metabolic disorders, with nearly half of sessions focusing on diabetes management. The weekly sessions were intended to promote collaborative problem solving, address behavioral and social barriers to better outcomes, and empower PCPs to address complex endocrinology scenarios. The program was led by a faculty comprising endocrinologists, diabetes educators, pharmacists, and social workers, and involved participant case presentations and discussions of evidence-based guidelines.

Data

Study measures are calculated using NJ Medicaid Management Information System (MMIS) data for calendar years 2015–2019, including enrollee information (eg, enrollment dates, demographics, eligibility category) and comprehensive managed care encounter records and associated provider payments as well as fee-for-service claims and payment amounts. EndoECHO participant rosters and attendance information were provided by RWJMS. We used National Provider Identifiers (NPIs) to identify EndoECHO participating PCPs in the MMIS. The study was approved by a university Institutional Review Board.

Intervention and Comparison Groups

Based on discussions with the RWJMS program faculty and leadership before initiation of the evaluation, we excluded PCPs who completed fewer than 4 of the 64 EndoECHO sessions. We attributed adult (age 18 or older) patients with at least 300 days of Medicaid enrollment in 2016 to participating and nonparticipating PCPs based on the plurality of their primary care visits, following the approach used in Medicare provider incentive programs.28,29 We then created a comparison cohort of up to 5 nonparticipating PCPs based on physician and patient panel characteristics using exact and statistical distance matching. Exact matching was based on provider specialty (Family Medicine, Internal Medicine, OB/GYN, Nurse Practitioner) and place of service most associated with the provider (private office, hospital outpatient, or clinic including federally qualified health centers).

We then employed Mahalanobis minimum distance matching using the “mahapick” procedure in STATA 16.0, based on: size of attributed Medicaid patient panel, size of the attributed panel of patients with diabetes, distributions of patient demographics (sex, race/ethnicity, age), share of attributed patients in the Affordable Care Act Medicaid expansion population, share of attributed patients with dual Medicare-Medicaid enrollment, and mean Chronic Illness and Disability Payment System (CDPS) patient risk classification score. To ensure thorough measurement of outcomes, patients with <300 days of Medicaid enrollment in any study year were excluded.

Outcomes and Analysis

We contrasted trends in outcomes between patients attributed to EndoECHO participating and nonparticipating PCPs over periods spanning January 2015 to December 2016 (before initiation of the EndoECHO intervention), January 2017 to June 2018 (during the intervention period), and July 2018 to December 2019 (following its conclusion). Although the intervention took place February 2017 to July 2018, data available do not permit a more precise alignment of dates. We divided study patients into 3 groups for analysis based on diagnostic codes in 2016: (1) patients with diabetes, (2) those without diabetes but with another endocrine or metabolic condition that was addressed in the EndoECHO curriculum, and (3) all other patients.

For each of these groups, we calculated outcomes in 5 domains, measured quarterly: total Medicaid spending and utilization and spending for ED visits, inpatient admissions, primary care visits, and ambulatory visits to endocrinologists. In addition, among patients with diabetes, we calculated trends in outcomes reflecting the adequacy of diabetes management, also measured quarterly: admissions and spending for short-term complications of diabetes, admissions and spending for any complication of diabetes or uncontrolled diabetes without complications, receipt of recommended HbA1c testing, and receipt of recommended eye exams. Finally, in addition to the above noted utilization and spending measures, we compared trends in prescriptions by class of diabetes medication. To ensure stable estimates of prescribing patterns, we compared average prescription rates for each drug class in the preintervention to postintervention periods.

The population with a diagnosis of diabetes was identified in 2016 data using specifications of the Healthcare Effectiveness Data and Information Set (HEDIS).30 Diagnoses of nondiabetes endocrine or metabolic conditions addressed in the EndoECHO curriculum used to classify the second patient group were identified by co-authors S.A. and J.H.F., and are listed in Supplemental Digital Content 2 (http://links.lww.com/MLR/C401).

Primary care visits were defined as evaluation and management visits to a PCP in an ambulatory setting with HCPCS/CPT codes for outpatient or preventive care visits or consultations (99201-99215; 99241-99245; or 99381-99397). Inpatient diabetes short-term complications and the composite diabetes admission measures were calculated using Agency for Healthcare Research and Quality (AHRQ) Prevention Quality Indicators (PQI-01 and PQI-93, respectively).31 Adherence with recommended HbA1c and retinal eye examination testing followed HEDIS technical specifications.32 We followed the diabetes medication classification method used by Flory et al33; and hospitalizations for endocrine and related conditions were classified using AHRQ clinical classification software (this measure is not available for 2015).34

Our matching strategy accounted for potential provider selection into the program that may be correlated with panel characteristics and outcomes.35 We assessed the quality of the match by examining standardized differences between the treatment and matched comparison group in matching attributes. For each outcome measured quarterly we estimated DD models to test for program effects. While the use of a matched comparison group assures the accuracy of estimators even if preintervention trends are not parallel,35,36 we tested for nonparallel trends and, where appropriate, adjusted estimators for differential pretrends.37 We conducted linear probability modeling for binary outcomes such as annual tests or ordinary least square for continuous and count variables. Standard errors are clustered at the provider level to account for non-independence of observations. For the analysis of changes in prescribing patterns, we compared average rates in the preintervention to postintervention periods and applied χ2 tests.

We conducted supplemental analyses to test for variations in program effects and confirm the veracity of our models. Our main DD models measure outcomes among patients with diabetes. Supplementary models test for program effects on spending and utilization variables that are not specifically diabetes related in 2 other groups of patients, that is those with other program-related endocrine conditions and all other patients. While it is possible that some skills acquired during program participation may lead to better care for patients without diabetes, for example, addressing social determinants of health, we did not expect to find substantial program effects in these supplemental analyses. That is, we consider them to be “falsification tests” where findings of significant program effects would raise suspicions of unmeasured selection bias (eg, providers disproportionately motivated to improve are more likely to enroll) or some other methodological artifact.

Finally, to test whether program effects vary by level of PCP participation we estimated an additional model with separate DD parameters for participants attending fewer (between 4 and 19) weekly sessions and those attending more (up to 64 sessions). A finding of a positive “dose response” would strengthen our inferences about program impact.

RESULTS

Twenty-five of the 53 enrolled participants were available for the study. Two were excluded because no NPI was available, 4 because they did not attend at least 4 EndoECHO sessions, 18 had no Medicaid claims in 2016, 2 were not PCPs, and 2 had no primary care claims. On average, participants attended under a third of the sessions (19 of the 64).

Each of the 25 EndoECHO participating PCPs were matched to 5 comparison providers, and among these 125 matches there were 122 unique PCPs (ie, 3 nonparticipants matched to >1 participant). After we applied the criterion of minimum Medicaid enrollment of 300 days for attributed patients in each study year, 119 comparison PCPs remained for available for analysis. A sample disposition chart is provided in Supplemental Digital Content 3 (http://links.lww.com/MLR/C402).

Our final patient sample includes 1776 patients attributed to participating PCPs, including 318 with diabetes, and 9126 attributed to the comparison providers, including 1454 with diabetes. Table 1 shows that intervention and comparison cohorts are well balanced, with just 1 variable, the proportion of patients of Black race, exceeding a standardized difference of 0.10. Although matching was based on the characteristics of all patients attributed to participating and comparison providers, we observed wider standardized differences for some patient characteristics when calculated based on diabetic patients alone (Supplemental Digital Content 3, http://links.lww.com/MLR/C402). Nevertheless, the use of DD models ensures unbiased estimation of policy effects based on relative changes in outcomes between participating and comparison populations.

TABLE 1

Intervention and Comparison Group Provider Panel Characteristics

Provider Panel Characteristics	EndoECHO Group, Mean (SD)	Comparison Group, Mean (SD)	Standardized Difference
Mean panel size	167.30 (210.42)	158.00 (158.97)	−0.05
Mean diabetic panel size	23.44 (27.13)	21.97 (21.64)	−0.06
Mean patient age	45.59 (10.69)	45.58 (8.77)	0.001
Mean CDPS score	2.28 (1.62)	2.15 (1.25)	−0.09
Proportion of patients
Male	0.33 (0.17)	0.32 (0.14)	−0.05
Black (non-Hispanic)	0.33 (0.30)	0.27 (0.22)	−0.23
Hispanic	0.20 (0.22)	0.22 (0.18)	0.09
Dual Medicare-Medicaid eligible	0.18 (0.27)	0.17 (0.19)	0.02
Eligible under ACA expansion*	0.38 (0.22)	0.37 (0.15)	−0.08

Category also includes a small number of General Assistance patients who were eligible before the expansion.

ACA indicates Affordable Care Act; CDPS, Chronic Illness and Disability Payment System risk adjustment.

Most DD estimates for our primary utilization and spending outcomes among patients with diabetes (Table 2) have signs consistent with hypothesized program effects, with measures of inpatient admissions and spending achieving statistical significance. We found 3.1 fewer inpatient admissions per 100 patient-quarters from the preprogram to postprogram period, a 44.3% reduction, among Medicaid enrollees attributed to EndoECHO participants compared with those of nonparticipating PCPs (P=0.001). From the preprogram to postprogram periods, the number of admissions for “endocrine, nutritional, and metabolic” diseases increased by 19.2% (from 26 to 31) among patients attributed to EndoECHO PCPs but they more than doubled (119.1%; from 89 to 195) among comparison group patients (χ2=4.641, P=0.098). Inpatient spending was $327 lower per patient of intervention PCPs per quarter relative that of nonparticipating PCPs, a reduction of 61.9% (P=0.021). Our estimate of the program effect on total Medicaid spending is close to the magnitude of the reduction in hospital spending but did not achieve statistical significance.

TABLE 2

Difference-in-Differences Estimates for Patients With Diabetes

Outcome	Baseline Mean	Estimate (SE)	P
General utilization and spending
ED visits	0.288	−0.036 (0.030)	0.227
ED spending ($)	91.396	−6.697 (13.218)	0.613
Inpatient admissions	0.070	−0.031 (0.009)***	0.001
Inpatient spending ($)	528.446	−327.366 (140.543)**	0.021
Primary care visits	1.395	0.073 (0.080)	0.360
Primary care spending ($)	48.134	7.499 (6.532)	0.253
Visits to endocrinologists	0.101	−0.015 (0.024)	0.533
Endocrinologist spending ($)	5.350	0.655 (2.056)	0.751
Total spending ($)	3788.778	−259.925 (425.595)	0.543
Diabetes-related utilization and spending
Diabetes short-term complication hospitalizations	0.003	−0.002 (0.003)	0.340
Diabetes short-term complication hospitalization spending ($)	18.962	−9.911 (13.019)	0.448
All diabetes hospitalizations	0.007	−0.006 (0.004)	0.138
All diabetes hospitalization spending ($)	55.347	−60.589 (33.392)*	0.072
Recommended HbA1c testing	0.627	−0.036 (0.031)	0.239
Recommended eye exam	0.006	−0.011 (0.006)*	0.094
Prescriptions of diabetes-related medications
Metformin	0.713	−0.064 (0.048)	0.182
Insulin	0.322	−0.046 (0.021)**	0.031
Sulonylurea	0.354	−0.017 (0.027)	0.524
Dipeptidyl peptidase-4 (DPP4)	0.222	−0.053 (0.039)	0.177
Glucagon-like peptide 1 receptor agonists (GLP1-RA)	0.055	−0.002 (0.021)	0.927
Sodium glucose cotransporter 2 inhibitors (SGLT2-I)	0.062	−0.008 (0.014)	0.562
Combination medications	0.089	−0.008 (0.029)	0.782

N=28,352 all diabetes hospitalizations and 35,440 for all other outcomes.

Three drug classes (Thiazolidinedione, Meglinitinde, and Alphaglucosidase inhibitors) that represented <0.03% of prescriptions in the baseline or postintervention periods are not shown.

ED indicates emergency department; SE, standard error, reported in parenthesis.

P<0.1.

P<0.05.

P<0.01.

Source: New Jersey Medicaid Management Information System.

Among other utilization and spending measures that are not diabetes-specific, only the DD estimate for endocrinologist spending is not in the hypothesized direction. None of the estimates reflecting adequacy of diabetes care management achieved significance at the P<0.05 level. Estimates for the measures of potentially avoidable diabetes-related hospital admissions and spending are signed in the expected direction, but recommended care estimates are not. DD estimates for prescribing of 6 classes of diabetes medications showed no significant changes, except for insulin which decreased more among EndoECHO patients than the comparison patients; this change may reflect improved diabetes management using noninsulin agents.38

Variations in Outcomes by Level of PCP Participation

Table 3 shows outcomes by whether EndoECHO PCPs attended 4 to 19 or >19 of the 64 program sessions. About half (53%) of patients with diabetes in the sample were attributed to 8 PCPs with greater attendance. Nine of the 15 DD estimates for our primary outcomes evince stronger program effects in the group attending more sessions, albeit most without achieving statistical significance. Notably, the “dose response” pattern in the hypothesized direction was evident for utilization and spending in the ED and short-term and total diabetes-related admissions, as well as for inpatient utilization. We estimated 4.5 fewer inpatient stays per 100 patient-quarters of high-attendance PCPs (P<0.001) but only a 1.8 stay reduction per 100 patient-quarters of low-attendance PCPs (N.S.), relative to the comparison group. The composite measure of admissions for uncontrolled diabetes and diabetes complications shows a similar pattern, with a reduction of 1.3 admissions per 100 patient-quarters in the high-attendance group (P=0.017) in contrast to no significant change in the low-attendance group. Estimates of program impacts on primary care utilization and spending and recommended HbA1c testing and eye exams were not consistent with hypothesized program effects among high-attendance PCPs.

TABLE 3

Difference-in-Differences Estimates for Patients With Diabetes by Number of EndoECHO Sessions Attended

Outcome	Baseline Mean	Estimate (SE)	P
General utilization and spending measures
ED visits
4–19 sessions	0.288	−0.015 (0.037)	0.688
20 or more sessions		−0.057 (0.041)	0.167
ED spending ($)
4–19 sessions	91.396	0.571 (17.904)	0.975
20 or more sessions		−13.875 (16.268)	0.395
Inpatient admissions
4–19 sessions	0.070	−0.018 (0.012)	0.129
20 or more sessions		−0.045 (0.008)***	0.000
Inpatient spending ($)
4–19 sessions	528.446	−568.511 (173.445)***	0.001
20 or more sessions		−331.090 (171.868)*	0.056
Primary care visits
4–19 sessions	1.395	0.155 (0.126)	0.221
20 or more sessions		−0.007 (0.084)	0.934
Primary care spending ($)
4–19 sessions	48.134	15.869 (10.541)	0.135
20 or more sessions		−0.766 (6.281)	0.903
Visits to endocrinologists
4–19 sessions	0.101	−0.028 (0.030)	0.354
20 or more sessions		−0.002 (0.032)	0.944
Endocrinologist spending ($)
4–19 sessions	5.350	3.558 (3.198)	0.268
20 or more sessions		−2.212 (2.135)	0.302
Total spending ($)
4–19 sessions	3788.778	451.397 (454.455)	0.323
20 or more sessions		−962.355 (524.307)*	0.069
Diabetes management related measures
Diabetes short-term complication hospitalizations
4–19 sessions	0.003	0.002 (0.001)*	0.054
20 or more sessions		−0.007 (0.004)*	0.058
Diabetes short-term complication hospitalization spending ($)
4–19 sessions	18.962	7.873 (13.440)	0.559
20 or more sessions		−27.473 (17.333)	0.116
All diabetes hospitalizations
4–19 sessions	0.007	−0.0001 (0.005)	0.988
20 or more sessions		−0.013 (0.005)**	0.017
All diabetes hospitalization spending ($)
4–19 sessions	55.347	−21.840 (27.826)	0.434
20 or more sessions		−98.854 (49.180)**	0.047
Recommended HbA1c testing
4–19 sessions	0.627	−0.021 (0.044)	0.630
20 or more sessions		−0.051 (0.035)	0.142
Recommended eye examination
4–19 sessions	0.006	−0.019 (0.009)**	0.036
20 or more sessions		−0.013 (0.007)*	0.085

N=28,352 all diabetes hospitalizations and 35,440 for all other outcomes.

Estimates for Inpatient Spending and Recommended Eye Exam are adjusted for nonparallel preintervention trends.

ED indicates emergency department; SE, standard error reported in parenthesis.

P<0.1.

P<0.05.

P<0.01.

Source: New Jersey Medicaid Management Information System.

Falsification Analysis

We repeated the DD analyses of general utilization and spending outcomes in 2 additional groups of patients: those without diabetes but with another endocrine and metabolic condition that was addressed in the EndoECHO curriculum, and patients with neither diabetes nor any curriculum-related conditions. With the plurality of the EndoECHO curriculum addressing topics in diabetes management, we hypothesized that significant program effects in patients would be found mainly among patients with diabetes. While the program plausibly could lead to better care for patients with other curriculum-related conditions, this group is heterogeneous, thus we expected that we would not find significant evidence of program impacts. Finally, we did not expect to find any program effects among patients with neither diabetes nor other curriculum-related conditions.

Overall, the program effects among diabetic patients were not identified in these 2 groups, with 2 exceptions (Supplemental Digital Content 4, http://links.lww.com/MLR/C403). We found a comparatively large ($623 per patient-quarter) and statistically significant (P=0.006) reduction in total Medicaid spending among patents with curriculum-related conditions, an effect about two-thirds of that found among diabetic patients of PCPs attending 20 or more program sessions. Among patients with neither diabetes nor other curriculum-related condition, we estimated that primary care spending was about $6 lower per patient-quarter (17.9%, P=0.044) among patients attributed to EndoECHO PCPs than among comparison group patients.

DISCUSSION

Project ECHO and similar tele-mentoring programs for health care providers have been adopted widely in the United States and other countries to improve care for patients with a variety of complex conditions, including diabetes and other endocrine diseases.20,21,23 Studies to date have shown that the model has promising impacts on outcomes relating to participating providers, such as indicators of provider knowledge and reports of confidence and self-efficacy, but few studies have linked provider participation to improved patient outcomes.22,23 The analysis in this article applies a novel methodology for evaluating the impact of a Project ECHO program seeking to improve care delivered by PCPs to Medicaid-enrolled patients with diabetes in New Jersey.

The NJ EndoECHO program, hosted by the Robert Wood Johnson Medical School, enrolled 53 providers, of which 25 met study inclusion criteria. The program addressed topics related to management of diabetes and other complex endocrine conditions (Supplemental Digital Content 2, http://links.lww.com/MLR/C401) and spanned 18 months starting January 2017. Program sessions were led by a faculty of endocrinologists, diabetes educators, pharmacists, and social workers in which evidence-based care guidelines and participant case presentations were discussed. PCP participants eligible for the study attended an average of 19 of 64 program sessions.

To evaluate patient outcomes, we first attributed Medicaid patient panels to NJ PCPs based on the plurality of visits in the year before the EndoECHO intervention. We then established a comparison group by matching nonparticipating PCPs to participants based on provider and patient panel characteristics.

Overall, our analysis provides modest evidence that EndoECHO led to better care for Medicaid patients with diabetes. Results showed impressive reductions in inpatient admissions and spending that exhibited evidence of dose-response effects. If replicated, the reduction in inpatient admissions by 3 per 100 patient-quarters would be clinically and operationally important, comparable to the impacts of notably successful policy interventions.39 Our limited sample size means that clinically important benefits went undetected. For example, our nonsignificant point estimate reduction of 3.6 ED visits per 100 patient quarters would certainly be operationally important if one imagines reducing an ED workload by that margin. The absence of findings consistent with program effectiveness among patients without diabetes or other conditions related to the EndoECHO curriculum strengthens confidence that findings among patients with diabetes were the result of their PCPs’ participation in the program.

While potentially subject to unmeasured selection effects, the finding of a dose-response is consistent with the hypothesis that higher PCP participation may have led to greater improvement in patient care. Notably, session attendance in our study was modest, averaging just under 1-in-3 of the sessions offered. Concentrating the most important program content in fewer sessions would reduce demands of participation and potentially improve outcomes. In fact, RWJMS revised its curriculum and reduced the number of sessions in subsequent iterations of their EndoECHO program. Program sponsors might also consider experimenting incentives for higher attendance, for example, by linking continuing education credits to participation in a minimum number of core sessions.

While findings overall support a conclusion that EndoECHO contributed to improved care for diabetic patients of participating PCPs, many results lacked statistical significance. Signs on DD estimates related to utilization and spending for ED, primary care, and endocrinologist services, and total Medicaid spending among patients with diabetes were consistent with program effectiveness but lacked statistical significance. This was also true for measures of hospitalization for short-term complications of diabetes and total diabetes-related admissions but not HbA1c testing and eye examination rates. We observed only one change in prescribing patterns in the DD analysis, reduced use of insulin in the EndoECHO group relative to the comparison group. This change is unlikely to have mediated the observed reduction in hospitalization rates in the EndoECHO group relative to comparison patients.

Findings from this study must be interpreted in light of its limitations. First, this is a single site study of a newly developed program, which may limit its generalizability to other practice contexts or more experienced ECHO hubs. Second, the sample of participating PCPs was small (N=25) and, among these, only 8 attended more than one-third (>20) of the 64 program sessions (although nearly half of study patients were attributed to these 8 PCPs). Third, the sample of patients was limited to those with at least 300 days of Medicaid enrollment during each study years. While near-continuous enrollment is necessary for calculation of reliable utilization and spending outcomes, outcomes for patients with more intermittent enrollment may differ. Finally, in spite of using models that control for differences between treatment and comparison groups, we cannot rule out unmeasured selection bias. However, in DD analysis, unmeasured selection factors that are independent of the timing of the intervention should not influence results.

The analyses presented provides modest evidence of EndoECHO effectiveness, but it also demonstrates the feasibility and value of applying quasi-experimental DD analysis in claims data for the evaluation of provider-level interventions like Project ECHO. Future studies on larger groups of providers with more consistent program participation have potential to provide needed evidence about the effectiveness of such programs.

Supplemental Digital Content is available for this article. Direct URL citations appear in the printed text and are provided in the HTML and PDF versions of this article on the journal's website, www.lww-medicalcare.com.