Applying embedded program evaluation for care delivery transformation: An analysis of a home-based urgent care program.

Background: In 2014, Mass General Brigham, formerly Partners HealthCare, launched a novel urgent home-based medical care program to provide rapid medical evaluation and treatment to homebound patients and older adults with frailty or limited mobility named the partners mobile observation unit (PMOU) program.
Methods: We conducted a pragmatic, embedded evaluation assessing the impact of PMOU on postreferral utilization and total medical expenditure (TME). We used propensity weighting and logistic regression to estimate the 30-day adjusted odds ratios (ORs) of emergency department (ED) utilization and inpatient medical hospitalization for patients enrolled in PMOU (891 episodes of care) relative to those who were referred but not enrolled in the program (57 episodes of care) during the period of April 2017 to June 2018. We additionally conducted a difference-in-differences analysis assessing program impact on TME, comparing claims data 30 days pre/post referral.
Results: Despite positive trends, there were no statistically significant differences between the two groups with regard to postreferral ED visits or hospitalizations, with an OR of 0.83 (p = 0.56) and OR of 0.64 (p = 0.21), respectively. There was no statistically significant difference in pre/post referral TME for intervention relative to control episodes (p = 0.64). In post hoc analysis of control episodes, 75% received care elsewhere within 14 days of referral.
Conclusion: Although the results suggested positive trends, this analysis of this relatively mature program was unable to identify statistically significant reductions in ED visits, hospitalizations, or TME associated with the PMOU program. Future efforts to build home-based urgent care programs or related programs targeting older adults with frailty or limited mobility should aim to improve patient targeting and identify opportunities to improve program operations and generate meaningful reductions in healthcare utilization and spending.

INTRODUCTION

Given growing national pressure and the recent growth of Accountable Care Organizations (ACOs), healthcare provider organizations have sought to develop innovative approaches to deliver high‐quality care and contain the costs of care. Shifting the site of service to lower‐cost settings has been one approach used to accomplish this objective. Previous studies have estimated that 14% to 27% of all emergency department (ED) visits could be managed in alternative settings, such as retail clinics or urgent care centers. Shifting the site of service in these situations could result in as much as $4 billion in cost savings annually. Characterized by physician oversight and execution of a medical care plan, the rise of home‐based medical care delivery models, such as home based palliative care (HBPC), which provide comprehensive, longitudinal primary care for homebound or older adults with frailty through home visit‐based programs, has also been shown to result in fewer hospitalizations and ED visits as well as reduced rates of healthcare spending. , , However, there is presently limited available evidence in the literature evaluating the potential impact of programs providing urgent home‐based medical care targeting older adults or those with significant physical barriers to accessing usual facility‐based care.

In 2016, Mass General Brigham (MGB), formerly Partners HealthCare, without any home‐based urgent medical program offerings to its more than 220 primary care practices, 1000 primary care physicians, and multiple sites of care in the greater Boston area, launched the partners mobile observation unit (PMOU) as its first home‐based medical care program. This stand‐alone program uses advanced practice providers (APPs), nurse practitioners, and physician assistants, to provide episodic home‐based urgent medical care to patients with acute complaints or conditions. This program targets community‐dwelling, older patients with frailty, and limited mobility who have symptoms or changes in condition warranting rapid evaluation or treatment. PMOU aims to enroll homebound patients with care needs that require a timely evaluation and would otherwise require a higher level of care, such as in the ED or hospital, or if left unaddressed, could result in serious illness. Primary care providers across the MGB network can refer patients to the PMOU program. Since the program's launch, PMOU has provided care to nearly 2000 individuals.

To assess the impact of this stand‐alone program and the potential value of targeted urgent home‐based medical care, we conducted an evaluation of the PMOU program, which we present here. Hypothesizing that timely in‐home evaluation and treatment of this frail, homebound older population with urgent clinical care needs would result in lower ED and hospital utilization and costs, this evaluation compared utilization and healthcare expenditures between patients treated in the program to patients that were referred but were not enrolled due to capacity or service area limitations.

METHODS

Intervention

PMOU provides home‐based urgent care by APPs to patients with symptoms or conditions considered by the referring provider to require an evaluation and/or treatment within a 24‐ to 48‐hour time period. All patients who receive care within an MGB primary care practice are eligible for the program. In this program, APPs are centrally dispatched following referral and accordingly provide evaluations and treatment in the patient's home. The goal of the program is to reduce potentially preventable ED visits and hospitalizations. Primary care providers (PCP) are the primary source of referrals, accounting for a large majority of PMOU referrals, but referrals can also be made by affiliated urgent care practices, hospital EDs, and specialists. Indications for referrals are intentionally unrestrained, relying on the clinical judgment of referring providers, who have been educated on the program capabilities and goal of providing urgent, short‐term care in the home. Common conditions for which patients are referred to PMOU include presumed heart failure exacerbations, falls, soft tissue infections, urinary tract infections, and dehydration. Referrals are accepted Monday through Friday 7 a.m.–6 p.m. This program was designed to specifically target adults with frailty, disabilities, or limited mobility. Patient enrollment criteria include living inside the geographic service area (based on prespecified zip codes located within a 60 min drive time), having a MGB PCP, having significant difficulty traveling to the provider's office, and having a condition that would unlikely require immediate hospital admission. Exclusion criteria include residing in a nursing home, having an acute exacerbation of psychiatric condition (specifically suicidal or homicidal ideation), and undomiciled status.

After a patient is referred and enrolled in the program, they are seen in their home on average within 24 hours. PMOU APPs are equipped with basic point‐of‐care laboratory equipment (i‐STAT, urine dipstick, glucometer). They are also able to order additional diagnostic testing (X‐ray, ultrasound, EKG) performed in a patient's home by an external vendor. Typical therapeutic interventions include but are not limited to medication adjustments, initiation of oral antibiotics, administration of intravenous fluids, and loop diuretics. The number of home visits during a PMOU episode of care varies, but in general is limited to one to three home visits over a maximum of 14 days. After the initial APP evaluation, patients can be referred to a higher level of care, such as a hospital ED as required. Documentation for the PMOU encounter occurs in the single enterprise electronic health record used by MGB.

Evaluation design

To evaluate this program, we conducted a pragmatic, embedded evaluation assessing the impact of PMOU on ED utilization and inpatient hospitalization during the 30‐day period following program referral. We analyzed rates of all‐cause hospitalization and inpatient medical hospitalization (i.e., excluding surgical hospitalizations). We chose to analyze medical hospitalizations specifically recognizing that surgical hospitalizations are more commonly planned and less likely to be prevented by short‐term interventions, and thus could potentially dilute the program effect. Medicare Severity‐Diagnosis Related Group recorded on facility inpatient claims were used to subdivide hospitalizations into medical and surgical. As a secondary outcome, we analyzed total medical expense (TME), which was obtained from claims data for patients aligned to MGB Commercial, Medicaid, or Medicare ACOs, measured as per member per month costs comparing the 30 days pre/post referral between the two groups. TME includes medical care reimbursements to qualifying medical providers, including third‐party payers, and patient copayments. ACO supplied claims data include claims for all services reimbursed by the payer for the patients; regardless of whether the service was provided by MGB. The use of these data limited the patient cohort but provided a more complete picture of healthcare utilization. Because of limitations in medical expense data in commercial and Medicaid claims TME was examined only for Medicare beneficiaries.

The intervention group was defined as all patients enrolled in PMOU during the period of April 2017 to June 2018 (see Figure 2). Each episode was treated independently, and an individual patient could accordingly be included more than once. We excluded patients who had been enrolled in the program >3 times during the study period (8.7% of unique PMOU episodes) as these patients: (a) likely had more chronically persistent high healthcare needs that were distinct from patients with fewer PMOU episodes representing an extension of the program's scope beyond its intended episodic design and (b) would potentially be better served by a longitudinal home‐based medical care model. Our control group was defined as referrals to PMOU between April 2017 and June 2018, which did not receive services due to limited program capacity or because the patients lived outside of the program's geographic service area.

Figure 2

Program evaluation consort diagram. Nonrisk: Patients not enrolled in an affiliated ACO or risk contract with available utilization data. ACO, Accountable Care Organization; PMOU, partners mobile observation unit.

Data sources

The evaluation combined data managed by the program team with demographic and Medicare, Medicaid, and Commercial ACO claims stored within a MGB data warehouse. Programmatic data includes referral dates and program dispositions and was used to identify intervention and control cohorts. These data were matched to payer claims data stored in the MGB data warehouse. MGB obtains payer claims data by virtue of accountable care risk contracts with Medicare, Massachusetts Medicaid, and large local commercial payers. Claims data were used to identify emergency room visits and hospital admissions; diagnosis‐related groups on facility claims were used to differentiate medical from surgical hospitalizations. Demographic information and care management enrollment are documented in electronic medical records and likewise housed within the data warehouse. The evaluation was limited to persons with sufficient claims data.

Data analysis

We compared the characteristics, preceding healthcare utilization, and TME of our intervention and comparison cohorts using two sample t‐test for continuous variables and a χ 2 test for categorical variables. We used logistic regression to estimate the 30‐day adjusted odds ratios (ORs) of ED utilization and inpatient hospitalization for patients enrolled in PMOU relative to our control group. In addition, we used a difference‐in‐differences approach to assess any potential impact of the program related to TME, comparing claims data 30 days pre/postreferral. We modeled PMOU engagement propensity for each episode using age at referral, gender, enrollment in high‐risk care management, referral calendar quarter, and annualized outpatient visits, ED visits, and hospitalizations in the 12 months pre‐referral. The MGB high‐risk care management program provides assistance with care coordination, social needs, and promoting health overall, and is made available to patients based on an internal risk assessment algorithm. Propensity weighting reduced the standardized difference in all baseline characteristics except inpatient stays and ED visits to less than 10%. All adjusted analyses applied inverse probability weighting and further adjusted for inpatient visits and ED visits to address residual confounding. Log link models were used to estimate adjusted odds of ED visits and inpatient services in the 30 days post‐PMOU referral. TME per member per month for Medicare ACO beneficiaries was analyzed using the difference in differences models comparing 30 days pre/post referral for intervention versus control episodes. For our analyses, p < 0.05 was selected to indicate statistical significance. This study was conducted using an internal data repository approved by the Mass General Brigham institutional review board for retrospective program evaluation.

RESULTS

During the study period, 891 patient referrals were enrolled in PMOU and met our inclusion criteria. An additional 57 patient referrals not enrolled in PMOU due to capacity or service area‐related factors met our criteria for inclusion in the control group. Referrals were from a total of 212 different providers. The groups were comparatively similar, however, a significantly higher percentage of those included in the intervention group were female (71.2% compared with 59.7%) and a slightly larger percentage of the control group was enrolled in Medicare (94.7% compared with 91.3%) (see Table 1).

Table 1

Patient characteristics and average measures of healthcare utilization and spending during the year before PMOU referral

	Intervention	Control	p Value
	(n = 891)	(n = 57)
Age (mean)	81	80	0.84
Gender (female)	634 (71.2%)	33 (57.9%)	0.03
Active in high‐risk care management	494 (55.4%)	31 (54.4%)	0.88
Payer group			0.06
Medicare	813 (91.3%)	54 (94.7%)
Commercial	57 (6.4%)	3 (5.3%)
Medicaid	9 (1.0%)	0 (0%)
Medicare advantage	12 (1.4%)	0 (0%)
Mean number of ED visits (95% CI)	3.3 (2.9−3.8)	4.6 (2.6−6.5)	0.18
Mean number of hospitalizations (95% CI)	1.8 (1.6−2.0)	3.0 (1.2−4.8)	0.17
Mean number of outpatient visits (95% CI)	11.8 (10.4−13.1)	15.5 (10.8−20.3)	0.18
TME/PMPM during year before PMOU referrala
Mean	$2907	$3290	0.45
Median (IQR)	$1648 (593−3751)	$2766 (683−4753)	0.17

Abbreviations: CI, confidence interval; ED, emergency department; IQR, interquartile range; PMOU, partners mobile observation unit.

Medicare patients enrolled in Mass General Brigham ACO only.

PMOU was associated with lower, but nonsignificant, adjusted odds of 30‐day ED visits (OR: 0.83 [p = 0.56]) and all‐cause inpatient hospitalizations (OR: 0.64 [p = 0.21]) (see Table 2). After adjusting for other variables, 27% of included patients had an ED visit within 30 days of enrollment compared with 31% of controls, and 18% of enrolled patients had an inpatient hospitalization (all‐cause) versus 25% of controls.

Table 2

Adjusted odds ratio of 30‐day postreferral healthcare utilization for patients enrolled in PMOU relative to control patients

	OR (p Value)
ED visit	0.83 (0.56)
Inpatient hospitalization (all‐cause)	0.64 (0.21)
Inpatient hospitalization (medical)	0.6 (0.14)

Abbreviations: ED, emergency department; OR, odds ratio.

Spending increased in both the intervention and control groups pre to post referral. Difference‐in‐differences analysis of patient TME comparing the 30 days pre/post referral found spending increased $1013 less for PMOU episodes relative to control episodes, but this too did not reach the level of statistical significance (p = 0.64) (see Figure 1).

Figure 1

Difference‐in‐differences in total medical expense between patients enrolled in PMOU and control patients. CI, confidence interval; DID, difference‐in‐differences; PMOU, partners mobile observation unit.

DISCUSSION

In this analysis, we found that despite favorable trends, a program designed specifically to deliver episodic home‐based urgent care by APPs was not associated with statistically significant reductions in ED visits, hospitalizations, or TME. The small number of comparison episodes relative to intervention episodes likely impacted power within this embedded analysis of a relatively mature program.

Numerous factors related to program implementation may have contributed to the lack of statistically significant findings in our evaluation. First, the measured impact of the program may have been diluted by the lack of explicit program enrollment criteria and patients being referred to the program that did not necessarily require urgent in‐person care, or who did not require home‐based care. Patient referral criteria were intentionally broad, and administrators were lenient with regard to enrollment criteria to encourage referrals and promote program adoption and growth to achieve scale. This practice may have however resulted in inadequate program targeting as strict referral restrictions could likely discourage both inappropriate and appropriate referrals. These hypotheses are supported by a post hoc analysis of outcomes. Patients referred to PMOU but unable to be accepted into the program were not referred to any specific service. Among control patients referred to but not receiving services from PMOU (n = 57), 3 (5%) visited an ED and were released and 20 (35%) were admitted to a hospital on the day of their PMOU referral. An additional 21 (37%) visited an ambulatory provider (PCP or urgent care provider) within 14 days of PMOU referral. The remaining 14 referrals (25%) who did not receive PMOU services did not seek any care within 14 days of referral.

Second, referring providers may not have understood the clinical capabilities of PMOU. Patients may have been enrolled for conditions that could not be evaluated or managed in the field (e.g., such as potentially angina‐like chest pain) or required treatments unavailable through the program (e.g., IV narcotic pain medication). PMOU APPs may therefore have appropriately recommended ED level care for these patients, resulting in duplication of services rather than substitution. However, examination of the claims data revealed that very few (6.3%) intervention episodes had claims for ED services on the day of enrollment into PMOU.

Lastly, it is also possible that a stand‐alone episodic urgent home‐based medical care program is a less effective model to decrease utilization of acute care services or associated healthcare expenses—particularly for frail, older adults where a holistic, chronic longitudinal approach to care is more appropriate, such as HBPC programs, which have shown reduced acute care utilization and TME. , Other alternative modalities for urgent care delivery, such as retail clinics, have demonstrated limited impact on healthcare utilization and have even been shown to be associated with paradoxical increases in healthcare spending. , , The potential impact of these alternative care modalities is likely moderated by concomitant supply‐sensitive demand and new use by patients who may not have otherwise sought care without such a service available. The PMOU program's in‐home medical delivery model may similarly appeal to patients who would not have otherwise sought emergency or hospital care, although given that patients had to be referred through a provider this is less likely. Still, some effects could have potentially offset associated reductions in utilization or spending. Similarly, PMOU care provision may be subject to induced demand, where service availability and delivery promote further engagement with providers, new utilization of additional services, and an increase in overall spending. Supporting this hypothesis, additional post hoc analyses of PMOU found that patients enrolled in the program were significantly more likely (p = 0.03) to receive certified home healthcare services in the 30 days following initial referral. This outcome may indeed be a benefit of the program, however, as it may have been useful for identifying patients that may benefit from certified home health services.

Our evaluation is subject to several limitations. First, while we chose to include all referral reasons in the evaluation, we did exclude intervention patients with three or more referrals to PMOU during the study period (19% of the excluded intervention episodes), recognizing that this group may represent a unique population with a larger burden of comorbid conditions, restricted mobility, and chronic care needs that might be better served with a longitudinal model such as HBPC. Therefore, our findings cannot be generalized to this population. Also, our comparison group included a small number of patients who had primary care providers within our system but resided outside of the program's geographic service area. It is possible that this group may have differed from the experimental group with respect to access to care, confounding our results. Also, we did not collect data on pre‐existing use of other home care services, so we were not able to assess how this may have impacted the program's effectiveness. Perhaps most notably the number of episodes eligible for our control sample was considerably limited. Given our sample size of 57 control patients, assuming an initial 2:1 ratio, we would have been powered at 80% to detect an effect difference of 0.15 at p < 0.05. While we consider the natural control group design reasonable, and even the strength of our pragmatic evaluation, the relatively small number of control patients limited our ability to understand the precise impact of the program. At the same time, this analysis gave us our first insights into the actual intervention power of this program or lack thereof.

Despite the null outcome, the embedded pragmatic nature of this evaluation and a strong engagement between evaluation and implementation teams led to several program changes after the completion of the analysis. We first used these results to refine our referral criteria to increase the likelihood that the program targeted patients and episodes most likely to benefit from the program, recognizing that nearly a quarter of patients referred but not enrolled did not require any additional care in the subsequent 14 days. Program administrators engaged with stakeholders to explore opportunities to improve upstream patient identification and integration of the program with primary care, particularly given the importance of fragmentation and continuity of care for patients with multiple chronic conditions and older adults. Importantly, faced with the reality of limited resources and competing priorities, these results gave our system enough quantitative insights to make key strategic decisions. Despite some positive signals demonstrated through this evaluation, it was recognized that the PMOU program was not a powerful enough intervention and would be unlikely to offset the operational and opportunity costs of offering this intervention. Ultimately, faced with such insights considerable PMOU program funds were shifted to extend interventions with a more clearly demonstrated impact on patient outcomes and costs of care such as our hospital‐at‐home program.

It is important to recognize, however, that the application of these findings was made within a specific context. There may be other systems, settings, and contexts where home‐based urgent care programs may prove more effective. Indeed, the trends of our data do suggest that there may be some positive impact of a stand‐alone home‐based urgent care but based on our experience it is essential that future efforts using similar models work closely with referring providers to ensure understanding regarding (1) specific program capabilities; (2) target population (i.e., frail, older, and/or homebound adults); and (3) alternative home‐based medical care delivery options (e.g., hospital‐at‐home, HBPC). This would help to ensure that patients are offered the service most appropriate for their condition and needs. One potential option would be to develop a scoring system or algorithm embedded within the EMR to alert the provider to when this service could be offered. Such efforts though must be balanced with ensuring that referral pathways, indications, and restrictions are easy to remember and employ as the risk of increasingly complicating referral pathways is decreased use of the program overall by referring providers.

In addition, we believe that there may be other lower‐cost ways to deploy resources to offer similar in‐home services for urgent evaluation and treatment by employing telehealth and community paramedicine, both of which have seen rapid growth in the past number of years. This addition may aid such programs in achieving cost‐effectiveness and more easily reaching economies of scale. It is also important to note that the period included in our evaluation occurred before the COVID‐19 pandemic. Since the beginning of the pandemic, there have been substantial shifts in the way that care is delivered, which may support a program like PMOU or may negate the need. For example, patient and provider comfort with telehealth has rapidly increased since this program was offered. Also, there has been an increase in the use of mobile integrated health, leveraging paramedics, to offer services similar to those provided by PMOU.

Increasingly as healthcare organizations take on risk and adopt delivery innovations meant to provide more efficient and effective care, designing programs with deeply embedded evaluation processes will be essential to ensuring longevity and supporting management decisions. This requires a balance between pragmatic program design and rigorous investigator‐led evaluation, and each healthcare organization will need to determine its unique criteria and optimal balance to support a learning health system.

CONCLUSION

Overall, despite strong clinician and patient support for the PMOU program, we were unable to identify statistically significant reductions in ED visits, hospitalizations, or TME associated with this stand‐alone urgent home‐based medical care program. The lack of an appreciable effect could be due to a number of factors, including small sample size, suboptimal patient enrollment criteria, mismatches between diagnostic testing, clinical care capabilities and patient needs, and insufficient integration with longitudinal primary care. Future efforts to build services targeting home‐based urgent care programs or similar programs supporting older adults with frailty and patients with limited mobility should aim to improve patient selection and identify other potential changes in program operations that could generate meaningful reductions in healthcare utilization and spending.

AUTHOR CONTRIBUTIONS

Charles T. Pu, Ya Gao, Lisa Nussbaum, and Christine Vogeli developed the study concept and design. Ya Gao, Lisa Nussbaum, and Christine Vogeli acquired and compiled the data for analysis. Lucas C. Carlson, Ya Gao, Lisa Nussbaum, and Christine Vogeli analyzed the data. Lucas C. Carlson, Charles T. Pu, Eden Mark, and Christine Vogeli drafted the manuscript. All authors contributed to the interpretation of the results and made critical revisions to the drafted manuscript.

CONFLICTS OF INTEREST

The authors declare no conflicts of interest.

TRANSPARENCY STATEMENT

The authors confirm that this manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned (and, if relevant, registered) have been explained.