Patient-Reported Outcomes Predict Future Emergency Department Visits and Hospital Admissions in Patients With Stroke.

Background Identification of stroke patients at increased risk of emergency department (ED) visits or hospital admissions allows implementation of mitigation strategies. We evaluated the ability of the Patient-Reported Outcomes Information Measurement System (PROMIS) patient-reported outcomes (PROs) collected as part of routine care to predict 1-year emergency department (ED) visits and admissions when added to other readily available clinical variables. Methods and Results This was a cohort study of 1696 patients with ischemic stroke, intracerebral hemorrhage, subarachnoid hemorrhage, or transient ischemic attack seen in a cerebrovascular clinic from February 17, 2015, to June 11, 2018, who completed the following PROs at the visit: Patient Health Questionnaire-9, Quality of Life in Neurological Disorders cognitive function, PROMIS Global Health, sleep disturbance, fatigue, anxiety, social role satisfaction, physical function, and pain interference. A series of logistic regression models was constructed to determine the ability of models that include PRO scores to predict 1-year ED visits and all-cause and unplanned admissions. In the 1 year following the PRO encounter date, 1046 ED visits occurred in 548 patients; 751 admissions occurred in 453 patients. All PROs were significantly associated with future ED visits and admissions except PROMIS sleep. Models predicting unplanned admissions had highest optimism-corrected area under the curve (range, 0.684-0.724), followed by ED visits (range, 0.674-0.691) and then all-cause admissions (range, 0.628-0.671). PROs measuring domains of mental health had stronger associations with ED visits; PROs measuring domains of physical health had stronger associations with admissions. Conclusions PROMIS scales improve the ability to predict ED visits and admissions in patients with stroke. The differences in model performance and the most influential PROs in the prediction models suggest differences in factors influencing future hospital admissions and ED visits.

area under the curve

intracerebral hemorrhage

modified Rankin Scale

patient‐reported outcome

Patient‐Reported Outcomes Information Measurement System

PROMIS Global Health (generic patient‐reported health status)

Clinical Perspective

What Is New?

Patient‐Reported Outcomes Information Measurement System scales improve the ability to predict future emergency department visits and admissions in patients with stroke.

The Patient‐Reported Outcomes Information Measurement System Global Health scale, which provides summary scores for both physical and mental health, is the single most useful patient questionnaire to predict future emergency department visits and admissions among the multiple patient‐reported scales evaluated in this study.

What Are the Clinical Implications?

Prediction of future healthcare use is another potential use for patient‐reported health measures collected in routine clinical practice of patients with stroke.

Patients with prior stroke are at high risk for subsequent hospitalization after stroke. , These hospitalizations are associated with poor survival, reduced well‐being, and higher healthcare costs, , and are potentially preventable. Risk stratification is often used to identify patients at high risk for poor outcomes and allows more efficient allocation of interventions to reduce the risk of poor outcomes. Examples of interventions that could be implemented in patients identified at high risk of hospital admission include closer follow‐up, patient education to improve self‐management, ensuring continuity of care with physicians, and patient‐centered medical homes. Performance, however, of risk stratification tools for stroke patients and others who are at increased risk of future hospital admission or emergency department (ED) visit has been modest at best. , In addition, a better knowledge of the factors associated with risk for ED visit or admission may provide insights into additional targeted strategies that may be used to mitigate patient's risk.

Patient‐reported outcomes (PROs) aid our understanding of a patient's well‐being by measuring social, emotional, and physical aspects of perceived health. They can improve communication between patient and provider, screen for medical conditions, and inform treatment decisions. Because they provide information not otherwise available in administrative data sets or the electronic health record (EHR), they may also improve the performance of models predicting future admissions and ED use. For example, poor physical function may increase the risk for falls, cognitive difficulties may affect self‐care and medication adherence, and poor sleep and depression may impact functional recovery after stroke , —all of which can contribute to future healthcare use. The routine collection of PROs is becoming more common, with 50% growth anticipated within the next 3 years in many hospitals, and the use of PROs in risk stratification models has become increasingly feasible with the incorporation of PROs into the EHR.

Therefore, the primary aims of our study were to (1) determine the ability of PROs collected as part of routine care to predict all‐cause 1‐year ED visits and hospital admissions when added to other readily available clinical variables; and (2) identify differences in the ability of PROs and other clinical variables to predict 1‐year ED visit versus 1‐year admission. The secondary aims were to determine the ability of Patient‐Reported Outcomes Information Measurement System (PROMIS) PROs and other clinical variables to predict 1‐year unplanned admission and 1‐year unplanned stroke‐related admission.

Methods

Data will be made available to other researchers upon reasonable request.

We performed a retrospective cohort study of patients with ischemic stroke, transient ischemic attack (TIA), intracerebral hemorrhage (ICH), or aneurysmal and nonaneurysmal subarachnoid hemorrhage (SAH) seen in the Cleveland Clinic ambulatory cerebrovascular clinic from February 17, 2015, to June 11, 2018. As previously described, cerebrovascular patients routinely completed PROs using the Knowledge Program data collection system either on electronic tablets at the time of their ambulatory visit or through the EHR patient portal (MyChart, Epic Systems, Madison, WI) before their appointment. Questionnaires are currently in English only. There is an option for proxies to assist with questionnaire completion, in the event patients have language barriers or cognitive or functional limitations preventing them from completing the questionnaires themselves. Clinicians completed the National Institutes of Health Stroke Scale and modified Rankin Scale (mRS) during each visit and recorded the date of the last cerebrovascular event. Inclusion criteria for this cohort study included age >18 years, completion of at least 1 PROMIS scale at ≥1 ambulatory visits during the study period, having a primary care physician within the Cleveland Clinic Health System, and clinical diagnosis of ischemic stroke, ICH, SAH, or TIA. Data for this study were extracted from the EHR and an institutional database containing patient‐reported outcome measures.

Patient‐Reported Outcome Measures

The computer adaptive testing versions of the Quality of Life in Neurological Disorders cognitive function version 1.0 scale, and six PROMIS version 1.0 scales were completed by patients: sleep disturbance, physical function, satisfaction with social roles, pain interference, fatigue, and anxiety. Scores of PROMIS and Quality of Life in Neurological Disorders tools are standardized to the general US adult population on the T‐scale with a mean of 50 and SD of 10. The Patient Health Questionnaire‐9 (PHQ‐9) depression screen was also collected and calibrated to the PROMIS metric providing equivalent PROMIS depression scores. These 8 scales each measure a specific domain of physical, mental, or social health. In addition, PROMIS Global Health (PROMIS‐GH) was completed and provided summary scores for physical and mental health. Scores for all scales are oriented so that higher scores indicate more of the construct being measured.

Emergency Department Visits and Hospital Admissions

Information regarding ED visits and hospital admissions was available from all 10 hospitals and an additional 3 free‐standing EDs within the Cleveland Clinic Health System located throughout northeastern Ohio. All except 1 hospital share the same instance of Epic EHR. The study cohort was limited to patients who had a primary care provider within the Cleveland Clinic Health System to increase the likelihood that admissions and ED visits would occur within our health system. Observation stays (<2 days) were counted as ED visits. Patients who presented to the ED and were subsequently admitted to the hospital were counted as hospital admissions.

Manual chart review of >250 patients was performed to refine the EHR data pulls to ensure accurate delineation of ED visits versus hospital admissions and categorization of admissions as unplanned and stroke related. Approximate household income was estimated using the ZIP code based on 2010 census data. Data on deaths within 1 year of the PRO encounter date were obtained from the Ohio Death Index, Social Security Death Index, and EHR.

Statistical Analysis

Descriptive statistics were computed for the entire cohort and stratified by whether the patient had any ED visits or all‐cause hospital admissions after but within 1 year of the PRO encounter date. Comparisons were made using 2‐sample t‐tests for continuous variables and the chi‐square test for categorical variables. When continuous variables had skewed distributions or had outliers, we used the Mann‐Whitney U test instead. For categorical variables where at least 1 expected cell frequency was <5, we used Fisher's exact test instead.

To determine whether PROs were predictive of 1‐year ED visit or all‐cause hospital admission, separate multivariable logistic regression models were constructed for each PRO to assess each PRO's added predictive capability in isolation from the other PROs. For each model, the dependent variable was ED visit or all‐cause admission after but within 1 year of the PRO encounter date, and the independent variable of interest was the PRO score. Each model was adjusted for the following covariates determined a priori: age, sex, race (White, Black, other), marital status (married, single, divorced, widowed), insurance (Medicaid, Medicare, private/other, self‐pay), median household income, stroke type (ischemic, TIA, ICH, SAH), mRS score (treated as categorical), days since stroke (≤90, 91–365, >365), proxy completion, and whether the patient had an ED visit or hospital admission in 6 months before the PRO encounter date. For all models, missing data were handled using multiple imputation by chained equations, using 100 imputations.

Odds ratio with 95% CIs are presented for an increase of 5 points for each PRO score. Calibration of each model was examined graphically. The area under the receiver operating characteristic curve (AUC) was computed for the model that included only the covariates as well as for each model that included the PRO. The increase in AUC for each model with a PRO was computed by subtracting the value of the AUC for the model that included only covariates. These AUCs were corrected for optimism using bootstrap internal validation, using 200 bootstrap resamples. A brief description of the procedure is as follows: The logistic regression model is fit in the full sample of patients, and the AUC is computed (used in bootstrap analysis). A new data set of the same size is generated by sampling with replacement. The logistic regression model is refit in this bootstrap sample data set. Using the model fit in the bootstrap sample, the AUC is computed in the bootstrap sample and in the original data set. The optimism for this bootstrap sample is computed by taking the difference between these AUC values (AUC computed in the bootstrap sample of patients and AUC computed in the original dataset). The above process is repeated M times, in our case 200 times, and the optimism is averaged over the M samples. The optimism‐corrected AUC is computed by subtracting the averaged optimism from AUC of logistic regression model fit in the full sample of patients.

As a sensitivity analysis, we refit the models in the subset of patients who had ischemic stroke, ICH, or SAH (ie, we excluded patients with TIA from analysis).

Similar models were constructed for 1‐year unplanned admission and 1‐year unplanned stroke‐related admissions. For the model of unplanned admissions, covariates were included as described above. As only 48 patients had an unplanned stroke‐related admission, we created a reduced base model with fewer covariates for this outcome. To do so, we performed a variable selection procedure described by Heymans et al that combines multiple imputation with bootstrapping. For each of the 100 imputed data sets, 200 bootstrap resamples were taken. For each of the 20 000 imputation by bootstrap data sets, we first fit the full model that included all covariates as with the other outcomes. We used backward stepwise selection using P=0.5. Only covariates that appeared in at least 80% of the 20 000 reduced models were included in the final base model for 1‐year unplanned stroke‐related admissions.

In an exploratory analysis, models for ED visits and all‐cause hospital admissions were constructed that included the subset of PROs that were statistically significant predictors when included separately in the respective base models.

Finally, the AUCs of base models (without PRO scores) for 1‐year ED visits, all‐cause admissions, unplanned admissions, and unplanned stroke‐related admissions were recomputed after excluding mRS scale score. Difference in optimism‐corrected AUCs between base models with and without mRS was calculated to determine the effect of mRS on model performance.

All computations were performed in R, version 3.6.1 (R Foundation for Statistical Computing, Vienna, Austria). All tests were 2‐sided. Because we fit separate models for each PRO score, we corrected for multiple testing of the PRO‐specific odds ratios using Holm's method.

This study was approved by the Cleveland Clinic Institutional Review Board. The requirement for informed consent was waived.

Results

There were 1696 patients with ischemic stroke, TIA, ICH, or SAH who completed at least 1 PRO during the study period and were included in the study cohort. The mean age of the patients in the study sample was 62.9 (±14.6) years; 48.8% (n=828) of participants were women, and 73.8% (n=1252) were White. Descriptive statistics for the cohort and stratified by whether the patient had an ED visit or hospital admission are presented in Table 1. Compared with otherwise eligible patients who were excluded because they did not complete PROs, patients in the study cohort were younger, more likely to be White, and married and had lower levels of disability as defined by the mRS (Table S1).

Table 1

Characteristics of Study Cohort and Stratified by Whether Patient Had Readmission After but Within 1 Year of PRO Encounter Date

	All Patients	Admission and/or ED Visit	No Admission	P Value
N	1696	778	918
Age, y, mean (SD)	62.9 (14.6)	64.3 (14.8)	61.8 (14.3)	<0.001
Female, n (%)	828 (48.8)	389 (50.0)	439 (47.8)	0.398
Race, n (%)
White	1252 (73.8)	536 (68.9)	716 (78.0)	<0.001
Black	355 (20.9)	199 (25.6)	156 (17.0)
Other	28 (1.7)	12 (1.5)	16 (1.7)
Missing/unknown	61 (3.6)	31 (4.0)	30 (3.3)
Marital status, n (%)
Married	992 (58.5)	426 (54.8)	566 (61.7)	0.006
Single	351 (20.7)	172 (22.1)	179 (19.5)
Divorced	163 (9.6)	80 (10.3)	83 (9.0)
Widowed	149 (8.8)	85 (10.9)	64 (7.0)
Missing/unknown	41 (2.4)	15 (1.9)	26 (2.8)
Insurance, n (%)
Private/other	690 (40.7)	260 (33.4)	430 (46.8)	<0.001
Medicare	741 (43.7)	385 (49.5)	356 (38.8)
Medicaid	170 (10.0)	109 (14.0)	61 (6.6)
Self‐pay	49 (2.9)	23 (3.0)	26 (2.8)
Missing	46 (2.7)	1 (0.1)	45 (4.9)
Median income (× $1000), mean (SD)	54.5 (18.5)	53.3 (19.0)	55.6 (18.1)	0.010
Stroke type, n (%)
Ischemic	1064 (62.7)	484 (62.2)	580 (63.2)	0.884
Transient ischemic attack	315 (18.6)	143 (18.4)	172 (18.7)
Intracerebral hemorrhage	181 (10.7)	88 (11.3)	93 (10.1)
Subarachnoid hemorrhage	136 (8.0)	63 (8.1)	73 (8.0)
Days since last stroke
Median (IQR)	130 (44–497.5)	114.5 (44–471.5)	151 (45–517)	0.253
≤90, n (%)	624 (36.8)	295 (37.9)	329 (35.8)	0.015
91–36, n (%)5	418 (24.6)	184 (23.7)	234 (25.5)
>365, n (%)	461 (27.2)	193 (24.8)	268 (29.2)
Missing/unknown, n (%)	193 (11.4)	106 (13.6)	87 (9.5)
Any ED or hospital admission in Prior 6 mo, n (%)	977 (57.6)	536 (68.9)	441 (48.0)	<0.001
Modified Rankin Scale score
Mean (SD)	1.2 (1.1)	1.5 (1.2)	1.0 (1.1)	<0.001
Median (IQR)	1 (0–2)	1 (1–2)	1 (0–2)	<0.001
0, n (%)	466 (27.5)	163 (21.0)	303 (33.0)	<0.001
1, n (%)	488 (28.8)	226 (29.0)	262 (28.5)
2, n (%)	290 (17.1)	137 (17.6)	153 (16.7)
3, n (%)	151 (8.9)	95 (12.2)	56 (6.1)
4, n (%)	67 (4.0)	43 (5.5)	24 (2.6)
5, n (%)	3 (0.2)	2 (0.3)	1 (0.1)
Missing, n (%)	231 (13.6)	112 (14.4)	119 (13.0)	<0.001
NIHSS Score
Mean (SD)	1.0 (2.4)	1.3 (2.8)	0.8 (1.8)	<0.001
Median (IQR)	0 (0–1)	0 (0–1)	0 (0–1)	<0.001
Proxy‐completed questionnaires, n (%)
No	1054 (62.1)	447 (57.5)	607 (66.1)	<0.001
Yes	335 (19.8)	177 (22.8)	158 (17.2)
Missing	307 (18.1)	154 (19.8)	153 (16.7)
PHQ‐9 Score, mean (SD)	5.6 (5.6)	6.5 (6.0)	4.8 (5.2)	<0.001
PHQ‐9 on PROMIS Depression Metric, mean (SD)	49.1 (10.3)	50.7 (10.6)	47.7 (9.9)	<0.001
NeuroQOL Cognitive Function T‐Score, mean (SD)	47.3 (10.1)	45.5 (10.2)	48.8 (9.7)	<0.001
PROMIS Anxiety T‐Score, mean (SD)	52.0 (10.1)	53.6 (10.2)	50.7 (9.8)	<0.001
PROMIS Fatigue T‐Score, mean (SD)	52.8 (10.3)	54.7 (10.3)	51.3 (10.1)	<0.001
PROMIS Physical Function T‐Score, mean (SD)	41.7 (10.4)	39.2 (10.3)	43.8 (10.0)	<0.001
PROMIS Pain Interference T‐Score, mean (SD)	52.5 (10.6)	54.2 (10.7)	51.1 (10.3)	<0.001
PROMIS Sleep Disturbance T‐Score, mean (SD)	49.4 (10.2)	49.8 (10.6)	49.1 (9.8)	0.219
PROMIS Social Roles T‐Score, mean (SD)	45.6 (11.1)	43.7 (10.8)	47.1 (11.1)	<0.001
PROMIS‐GH Physical Health Summary T‐Score, mean (SD)	44.4 (9.1)	42.5 (8.8)	45.9 (9.0)	<0.001
PROMIS‐GH Mental Health Summary T‐Score, mean (SD)	46.2 (9.0)	44.8 (8.8)	47.4 (9.1)	<0.001

ED indicates emergency department; IQR, interquartile range; NIHSS, National Institutes of Health Stroke Scale; PHQ‐9, Patient Health Questionnaire‐9; NeuroQOL, Quality of Life in Neurological Disorders; PRO, patient‐reported outcome; PROMIS, Patient Reported Outcome Measure Information System; and PROMIS‐GH, Patient Reported Outcome Measure Information System Global Health.

In the 1‐year following the PRO encounter date, 1046 all‐cause ED visits occurred in 548 (32.3%) patients; 751 all‐cause hospital admissions occurred in 453 (26.7%) patients. Median days to first ED visit was 107 (interquartile range, 43–199), median days to first admission was 79 (interquartile range, =29–169). There were 567 unplanned admissions occurring in 19.6% (333/1696) of patients and even fewer unplanned stroke‐related admissions in the study cohort: 55 admissions occurring in 2.8% (48/1696) of patients. There were 45 deaths (2.6%) in the patient cohort within 1 year of the PRO encounter date; only 1 of these (0.06%) occurred before an ED visit or admission.

Factors associated with ED visit in the base multivariable model (which did not include PRO score) were Black race, being widowed, minimal‐moderate degree of disability (mRS score, 1–3) and previous admissions in the past 6 months (Table 2). These patients were also less likely to have Medicare or private insurance compared with Medicaid insurance, although median estimated household income was not a significant predictor of ED visits. The optimism‐corrected AUC for the base model predicting ED visit was 0.674 (95% CI, 0.659–0.714).

Table 2

Base Multivariable Models of Factors Predicting ED Visit or All‐Cause Hospital Admission Within 1 Year of PRO Encounter Date

Optimism‐Corrected Area Under the Curve (95% CI)	ED Visit		Inpatient Admission
	0.674 (0.659–0.714)		0.628 (0.624–0.682)
	Odds Ratio (95% CI)	P Value	Odds Ratio (95% CI)	P Value
Age (per y)	1.01 (1.00–1.02)	0.124	1.00 (0.99–1.01)	0.489
Male (vs female)	0.88 (0.70–1.10)	0.257	1.21 (0.96–1.53)	0.111
Race (vs White)
Black	1.85 (1.38–2.48)	<0.001	0.81 (0.59–1.11)	0.192
Other	1.47 (0.63–3.43)	0.376	0.78 (0.30–2.00)	0.599
Marital status (vs married)
Single	1.05 (0.77–1.42)	0.766	1.14 (0.83–1.57)	0.412
Divorced	0.97 (0.66–1.41)	0.871	1.32 (0.90–1.92)	0.155
Widowed	1.56 (1.06–2.31)	0.025	0.95 (0.62–1.44)	0.794
Insurance (vs Medicaid)
Medicare	0.55 (0.36–0.84)	0.005	0.66 (0.43–1.01)	0.057
Private/Other	0.43 (0.29–0.63)	<0.001	0.45 (0.30–0.66)	<0.001
Self‐Pay	0.85 (0.43–1.67)	0.629	0.47 (0.22–1.01)	0.052
Median income* (per $10 000)	1.02 (0.95–1.09)	0.580	1.00 (0.93–1.07)	0.982
Stroke type (vs ICH)
Ischemic	0.96 (0.67–1.37)	0.825	0.92 (0.64–1.33)	0.661
SAH	0.83 (0.50–1.37)	0.460	1.05 (0.62–1.78)	0.844
TIA	1.06 (0.69–1.65)	0.781	1.35 (0.86–2.11)	0.191
Modified Rankin Scale (vs 0)
1	1.53 (1.14–2.06)	0.005	1.52 (1.11–2.09)	0.010
2	1.54 (1.09–2.17)	0.014	1.26 (0.86–1.83)	0.234
3	1.70 (1.08–2.68)	0.022	2.89 (1.83–4.57)	<0.001
4	1.62 (0.93–2.84)	0.089	2.41 (1.36–4.27)	0.002
5	1.37 (0.30–6.31)	0.683	2.67 (0.67–10.63)	0.165
Days since stroke (vs ≤90)
91–365	1.04 (0.78–1.39)	0.781	0.96 (0.71–1.30)	0.812
366+	1.32 (0.97–1.78)	0.076	0.99 (0.72–1.36)	0.953
Proxy completion	0.79 (0.57–1.09)	0.144	1.17 (0.84–1.61)	0.349
Admission/ED visit in prior 6 mo	2.76 (2.14–3.58)	<0.001	1.71 (1.32–2.22)	<0.001

Dependent variable in multivariable models were all‐cause ED visits and inpatient admissions within 1 year of office visit. ED indicates emergency department; ICH, intracerebral hemorrhage; PRO, patient‐reported outcome; SAH, subarachnoid hemorrhage; and TIA, transient ischemic attack.

Median household income estimated from ZIP code.

Factors independently associated with all‐cause hospital admission included disability—especially of moderate degree (mRS, 3–4), and a previous ED visit or admission within 6 months of their PRO encounter date. Private insurance was associated with reduced likelihood of hospital admission compared with Medicaid insurance (Table 2). The optimism‐corrected AUC for the base model predicting all‐cause admission was 0.628 (95% CI, 0.624–0.682).

Tables 3 and 4 show results for models of 1‐year ED visit and all‐cause admission, respectively, when PROs were each added separately to the base model. All PROs except sleep significantly predicted ED visits and/or admissions. The PROs measuring domains of mental health—cognition, anxiety, and mental health summary score—had stronger association with ED visits than admissions. In contrast, the PROs measuring physical function and the physical health summary score had stronger associations with admission than ED visits. The pain interference and fatigue scales, which also measure domains of physical health, had predictive value in both the ED use and admission models.

Table 3

Ability of PROs to Predict All‐Cause ED Visit Within 1 Year of the PRO Encounter Date When Added to the Base Prediction Model

	Odds Ratio (95% CI)	Holm‐Adjusted P Value	Optimism‐Corrected Area Under the Curve (95% CI)	Change in Area Under the Curve (95% CI)
Mental health (per 5 points)
PHQ‐9 Score	1.23 (1.11–1.36)	<0.001	0.678 (0.666–0.721)	0.004 (0.00001–0.015)
PHQ‐9 on PROMIS Depression Metric	1.12 (1.06–1.19)	<0.001	0.679 (0.665–0.721)	0.005 (0.0001–0.017)
NeuroQOL Cognitive Function T‐Score	0.84 (0.78–0.90)	<0.001	0.691 (0.680–0.737)	0.017 (0.010–0.039)
PROMIS Anxiety T‐Score	1.15 (1.08–1.22)	<0.001	0.687 (0.669–0.725)	0.013 (0.002–0.024)
Physical health (per 5 points)
PROMIS Fatigue T‐Score	1.12 (1.06–1.19)	<0.001	0.679 (0.666–0.721)	0.006 (0.0003–0.018)
PROMIS Physical Function T‐Score	0.96 (0.89–1.02)	0.403	0.674 (0.660–0.714)	0.000 (−0.003–0.006)
PROMIS Pain Interference T‐Score	1.13 (1.06–1.19)	<0.001	0.683 (0.669–0.723)	0.009 (0.001–0.020)
PROMIS Sleep Disturbance T‐Score	1.02 (0.96–1.08)	0.489	0.672 (0.659–0.714)	−0.002 (−0.003–0.005)
Social health
PROMIS Social Roles T‐Score	0.96 (0.90–1.01)	0.403	0.671 (0.662–0.716)	−0.002 (−0.002–0.010)
Global health
PROMIS‐GH Physical Health T‐Score	0.86 (0.79–0.92)	<0.001	0.685 (0.668–0.723)	0.012 (0.002–0.021)
PROMIS‐GH Mental Health T‐Score	0.88 (0.82–0.95)	0.006	0.681 (0.663–0.718)	0.007 (−0.001–0.013)

Results of separate multivariable models that each included one PRO added to the base model (Table 2). Dependent variable is ED visit within 1 year of PRO encounter date. The last column in the table shows the increase in optimism‐corrected area under the curve that occurs when the PRO is added to the base model (optimism‐corrected area under the curve=0.674 95% CI: 0.659–0.714). Nadmission=548, NNo_admission=1148. ED indicates emergency department; NeuroQOL, Quality of Life in Neurological Disorders; PHQ‐9, Patient Health Questionnaire‐9; PRO, patient‐reported outcome; PROMIS, Patient Reported Outcome Measure Information System; and PROMIS‐GH, Patient Reported Outcome Measure Information System Global Health.

Table 4

Ability of PROs to Predict All‐Cause Hospital Admissions Within 1 Year of the PRO Encounter Date When Added to the Base Prediction Model

	Odds Ratio (95% CI)	Holm‐Adjusted P Value	Optimism‐Corrected Area Under the Curve (95% CI)	Change in Area Under the Curve (95% CI)
Mental health (per 5 points)
PHQ‐9 Score	1.16 (1.05 to 1.29)	0.023	0.636 (0.628 to 0.688)	0.009 (−0.001 to 0.015)
PHQ‐9 on PROMIS Depression Metric	1.08 (1.02 to 1.15)	0.037	0.636 (0.627 to 0.687)	0.008 (−0.002 to 0.013)
NeuroQOL Cognitive Function T‐Score	0.90 (0.84 to 0.97)	0.021	0.635 (0.629 to 0.690)	0.007 (−0.001 to 0.019)
PROMIS Anxiety T‐Score	1.06 (1.00 to 1.13)	0.193	0.631 (0.626 to 0.685)	0.003 (−0.002 to 0.013)
Physical health (per 5 points)
PROMIS Fatigue T‐Score	1.14 (1.08 to 1.21)	<0.001	0.649 (0.639 to 0.698)	0.021 (0.004 to 0.031)
PROMIS Physical Function T‐Score	0.77 (0.72 to 0.83)	<0.001	0.671 (0.657 to 0.716)	0.043 (0.015 to 0.052)
PROMIS Pain Interference T‐Score	1.13 (1.06 to 1.19)	<0.001	0.643 (0.638 to 0.697)	0.015 (0.003 to 0.030)
PROMIS Sleep Disturbance T‐Score	1.01 (0.96 to 1.08)	0.646	0.627 (0.623 to 0.682)	−0.001 (−0.003 to 0.007)
Social health (per 5 points)
PROMIS Social Roles T‐Score	0.87 (0.81 to 0.92)	<0.001	0.641 (0.636 to 0.696)	0.014 (0.002 to 0.028)
Global health (per 5 points)
PROMIS‐GH Physical Health T‐Score	0.80 (0.73 to 0.86)	<0.001	0.668 (0.657 to 0.717)	0.040 (0.016 to 0.054)
PROMIS‐GH Mental Health T‐Score	0.93 (0.86 to 1.01)	0.193	0.635 (0.626 to 0.686)	0.008 (−0.003 to 0.011)

Results of separate multivariable models that each included one PRO added to the same base model (Table 2). Dependent variable is all‐cause inpatient admission within 1 year of PRO encounter date. The last column in the table shows the increase in optimism‐corrected area under the curve that occurs when the PRO is added to the base model (optimism‐corrected area under the curve=0.628, 95% CI: 0.624–0.682). Nadmission=453, NNo_admission=1243. NeuroQOL indicates Quality of Life in Neurological Disorders; PHQ‐9, Patient Health Questionnaire‐9; PRO, patient‐reported outcome; PROMIS, Patient Reported Outcome Measure Information System; and PROMIS‐GH, Patient Reported Outcome Measure Information System Global Health.

Although the AUC of the base admission model was lower than for the ED visit model, the improvement in the AUCs after addition of PROs were generally greater for the models predicting admission than the models predicting ED visits. The largest AUC for the ED models was the one that included Quality of Life in Neurological Disorders cognitive function (AUC, 0.691; 95% CI, 0.680–0.737; increase in AUC, 0.017; 95% CI, 0.010–0.039). The largest AUC for the admission model was the one that included PROMIS physical function as a covariate (AUC, 0.671; 95% CI, 0.657–0.716; increase in AUC, 0.043; 95% CI, 0.015–0.052).

Sensitivity Analyses

In sensitivity analyses, 1‐year all‐cause ED visit and inpatient admission models were replicated after excluding patients with TIA from the cohort. The results of models were similar to the models fit with the full cohort; the odds ratio of PROs and the other covariates and the AUCs did not appreciably change (Tables S2 and S3).

Models for 1‐Year Unplanned Hospital Admission and Stroke‐Related Admission

The optimism‐corrected AUC of the base model of 1‐year unplanned admission was 0.684 (95% CI, 0.670–0.735, higher than the AUC of the all‐cause admission model of 0.628. Significant base model covariates were similar to those predicting all‐cause hospital admissions, with the exception of TIAs predicting more unplanned hospital admissions (odds ratio, 1.71; 95% CI, 1.03–2.83) as compared with ICH) (data not shown). The odds ratios of the PROs added separately to the base model (Table 5) were similar to those in the corresponding all‐cause admission models.

Table 5

Ability of PROs to Predict Unplanned Hospital Admissions Within 1 Year of PRO Encounter Date When Added to the Base Prediction Model

	Odds Ratio (95% CI)	Holm‐Adjusted P Value	Optimism‐Corrected Area Under the Curve (95% CI)	Change in Area Under the Curve (95% CI)
Mental health (per 5 points)
PHQ‐9 Score	1.19 (1.06 to 1.34)	0.014	0.694 (0.676 to 0.741)	0.010 (−0.001 to 0.017)
PHQ‐9 on PROMIS Depression Metric	1.11 (1.04 to 1.18)	0.014	0.692 (0.677 to 0.741)	0.008 (−0.001 to 0.017)
NeuroQOL Cognitive Function T‐Score	0.89 (0.82 to 0.96)	0.014	0.691 (0.675 to 0.741)	0.007 (−0.002 to 0.016)
PROMIS Anxiety T‐Score	1.07 (1.00 to 1.15)	0.105	0.685 (0.673 to 0.738)	0.001 (−0.002 to 0.013)
Physical health (per 5 points)
PROMIS Fatigue T‐Score	1.18 (1.10 to 1.27)	<0.001	0.706 (0.691 to 0.753)	0.022 (0.007 to 0.035)
PROMIS Physical Function T‐Score	0.76 (0.69 to 0.82)	<0.001	0.724 (0.711 to 0.771)	0.040 (0.021 to 0.057)
PROMIS Pain Interference T‐Score	1.13 (1.06 to 1.21)	0.001	0.696 (0.685 to 0.749)	0.012 (0.003 to 0.032)
PROMIS Sleep Disturbance T‐Score	1.04 (0.97 to 1.11)	0.243	0.685 (0.674 to 0.737)	0.001 (−0.002 to 0.013)
Social health (per 5 points)
PROMIS Social Roles T‐Score	0.84 (0.78 to 0.90)	<0.001	0.700 (0.688 to 0.753)	0.016 (0.006 to 0.034)
Global health (per 5 points)
PROMIS‐GH Physical Summary T‐Score	0.76 (0.69 to 0.83)	<0.001	0.723 (0.714 to 0.773)	0.039 (0.022 to 0.061)
PROMIS‐GH Mental Summary T‐Score	0.91 (0.83 to 0.99)	0.105	0.691 (0.676 to 0.740)	0.007 (−0.002 to 0.014)

Results of separate multivariable models that each included one PRO added to the same base model. The dependent variable is unplanned hospital admission within 1 year of PRO encounter date. For each model, covariates included age, sex, race, marital status, insurance, median income, stroke type, mRS score, days since stroke, and proxy completion. The optimism‐corrected area under the curve for the base model (excluded PROs) was 0.684 (95% CI=0.670–0.735). The last column in the table shows the increase in optimism‐corrected area under the curve that occurs when the PRO is added to the base model. Nadmission=333, NNo_admission=1363. NeuroQOL indicates Quality of Life in Neurological Disorders; PHQ‐9, Patient Health Questionnaire‐9; PRO, patient‐reported outcome; PROMIS, Patient Reported Outcome Measure Information System; and PROMIS‐GH, Patient Reported Outcome Measure Information System Global Health.

The final reduced base model for 1‐year unplanned stroke‐related admissions contained only 1 variable: insurance. Among the 20 000 imputed by bootstrap data sets, insurance appeared in 82.5% of the reduced models. All other covariates appeared in <60% of models. Compared with patients on Medicare or Medicaid, patients with private health insurance or who self‐paid had 65% lower odds of having an unplanned stroke‐related admission (odds ratio, =0.35, 95% CI, 0.18–0.69; P=0.003). The optimism‐corrected AUC for this model was 0.612 (95% CI, 0.611–0.619). Better physical function was associated with lower odds of 1‐year unplanned stroke‐related admission. The optimism‐corrected AUC for this model was 0.674 (95% CI, 0.667–0.676), representing an increase of 0.062 (95% CI, 0.050–0.063). None of the other PROs added separately in the base stroke‐related admissions model were significant (Table S4).

In the exploratory analysis, models for 1‐year ED visits and all‐cause admissions included the subset of statistically significant PROs for each (depression, cognition, anxiety, fatigue, pain, and global physical health for ED visits; depression, cognition, fatigue, physical function, pain, social roles, and global physical and mental health for admissions). The optimism‐corrected AUCs for ED visits and hospital admissions including the combination of PROs were 0.698 (95% CI, 0.691–0.699) and 0.672 (95% CI, 0.671–0.676), respectively. These AUCs were <0.01 higher than the corresponding best‐performing model that included a single PRO (eg, Quality of Life in Neurological Disorders cognitive function for ED visits and PROMIS physical function for admissions) (data not shown).

Calibration plots (not shown) revealed good calibration for the ED models across the range of predicted probabilities. Calibration for the admission models were generally good; however, lower predicted probabilities (<0.2) tended to be slightly overestimated. Addition of PROs tended to modestly improve calibration, compared with the base model, for both outcomes. Similar observations were made for the models for unplanned admissions and unplanned stroke‐related admissions.

To determine the effect of adding mRS on model performance, the AUCs of base models (without PRO scores) for the different 1‐year ED visits and hospital admissions were recomputed after excluding mRS scale score. The addition of the mRS score to the base models increased the optimism‐corrected AUC between 0.003 (ED visits model) to 0.041 (unplanned stroke‐related admissions model) (Table S5).

Discussion

ED visits and hospital admissions within 1 year of clinic visit were frequent, occurring at least once in almost half of patients in the study cohort. All PRO measures in this study except for PROMIS sleep disturbance aided in the prediction of future admissions and/or ED visits in models that included clinician‐reported disability and other readily available clinical variables. Inclusion of PROs in a simple base prediction model improved AUCs by 0 to 0.04.

All of the domains measuring components of mental health had greater predictive ability for future ED visits than for hospital admissions. Patient‐reported cognition had the highest predictive value for ED visits than all other PROs. This is consistent with other studies that have found the presence of cognitive deficits to be a significant risk factor for ED visits. Overall, patients with worse self‐reported emotional health may have a lower threshold to use the ED and/or have fewer alternatives for seeking care.

In contrast with the absent association between physical function scores and future ED visits, physical function had the strongest predictive ability for 1‐year all‐cause hospital admissions. The addition of the physical function and PROMIS‐GH physical health summary scores to the hospital admission model had the greatest effect on the AUC of all the PROs in the study, improving the AUC by 0.043 and 0.04, respectively. The anxiety and PROMIS‐GH mental health summary scores, which were strong predictors in the ED model, were not significant in the hospital admission prediction models. A stronger association between patient‐reported physical health as compared with mental health has been seen in other models predicting future healthcare use.

In addition to differences in the strength of association of PRO scales between ED visits and hospital admissions, there were some differences in the associations of other clinical factors between ED visits and hospital admissions. Black race was associated with ED visits but not hospital admissions. Moderate disability (mRS, 3–4) had a stronger association with hospital admissions than with ED visits. Clinician‐reported functional status has been found to have a significant effect on the predictive accuracy of hospital admission for patients recently hospitalized for myocardial infarction, whereas it has been shown to be a risk factor for ED use in only about half of studies that have included this variable. Medicaid insurance was a risk factor for both ED use and hospital admission. This is not surprising and suggests that these patients represent a more vulnerable population from a health and socioeconomic perspective. Healthcare‐related challenges in Medicaid patients with frequent ED use include negative personal experiences with the healthcare system, chronic mental and physical disease burden, and challenges associated with having a low socioeconomic status that make managing health a lower priority. Patients with Medicaid insurance have reported barriers to care such as distance to clinics, clinic availability, and clinic communication. All cohort patients had a primary care physician within the health system and should have the same access to ambulatory care, although the needs of patients with Medicaid insurance may be different, possibly due to complex life situations.

Inclusion of the majority of PROs into regression models improved their predictive accuracy, indicating that PROs provide new types of information to these models. Of note, stroke type was not associated with ED visits or hospital admissions in multivariable models, similar to what has been reported by others. , Including >1 PROM into prediction models did not appreciably improve the AUC. Overall, our findings support the use of PROMIS‐GH as the single most useful PRO to incorporate into prediction models of ED visits and admissions from among the PROMIS scales assessed in this study. The PROMIS‐GH has also been recommended as the outcome measure for patients with stroke by the International Consortium for Health Outcomes Measurement—a nonprofit organization that develops standard sets of outcomes for medical conditions. This scale provides physical and mental health summary scores, which when taken together, resulted in similar or better improvements in model performance compared with individual scales measuring specific physical and mental health domains.

The discrimination of ED visits and all‐cause hospital admission models was modest despite inclusion of PROs and clinician‐reported disability within prediction models. Other than the PROs, the variables were selected for their relative simplicity and ready availability at the time of the ambulatory visit to a stroke clinic. In comparison, models for 30‐day and 1‐year all‐cause hospital admission rates for ischemic stroke that included individual‐level and hospital‐level data from Get With the Guidelines merged with Medicare data had AUCs of 0.59 and 0.62, respectively. Previous prediction models for hospital admissions and/or ED visits in community‐dwelling people have had similarly modest AUCs ranging from 0.60 to 0.70. , The modest level of discrimination in the models in our study suggest that additional unmeasured factors play an important role in future admissions and ED visits. Potential contributors to future admission risk include environmental factors, caregiver support, and other PROs not assessed in our study such as self‐efficacy or health literacy.

Performance of models predicting unplanned admissions was better than the models predicting all‐cause admissions. The prediction of unplanned admissions has greater clinical relevance since these admissions are more likely to be avoidable. With inclusion of the PROMIS‐GH physical health summary score, the optimism‐corrected AUC of the unplanned hospital admission prediction model was 0.723, which is in a range that has been used in clinical practice. , Further research is needed to determine whether implementation of the prediction model for unplanned admissions coupled with targeted interventions can effectively reduce hospitalizations. Disappointingly, none of the PROs or other clinical variables assessed in this study were predictive of unplanned stroke‐related admissions, but this is likely related to low numbers of such admissions in our study cohort.

Some limitations of our study must be acknowledged. First, we did not incorporate death within the analysis. However, the patients in this study, who all attended an outpatient stroke clinic, had a very low mortality rate within 1 year of the PRO encounter date. Second, although our patient cohort was limited to patients with a primary care physician within the health system (making it more likely that patients went to a health system hospital or ED), patients may have been seen at external sites not captured in this study. Third, the variables used in the prediction model were intentionally limited to those that were readily available and would not require extensive programming. Additional predictors of hospital admissions not included in this analysis may improve model performance at the expense of greater complexity of implementation, including markers of medical complexity, more extensive prior use data, and medications. It is unknown if inclusion of PROs in a more extensive model would have provided a similar degree of improvement in model performance. Fourth, although this study included many PROs spanning the dimensions of physical, mental, and social health, it did not include social determinants of health, caregiver support, or other PRO domains, such as self‐efficacy, that may have further improved the prediction of future admissions or ED visits. Fifth, this study was limited to 1 health system and included only the patients with available PRO data, resulting in selection bias, as evidenced by the overall mild degree of disability of the study cohort (median mRS, 1). Finally, this study requires validation in other ambulatory stroke populations.

Summary

PROs improve the ability to predict 1‐year ED visits and hospital admissions. Patient‐reported scales assessing domains of mental health have stronger associations with future ED visits, while scales assessing domains of physical health have a stronger association with future admissions. These findings suggest that there are some differences in the drivers of ED visits and hospital admissions, possibly warranting different approaches to intervention. The PROMIS‐GH scale, which provides summary scores for both physical and mental health, is the single most useful PRO scale to predict future ED visits and admissions among the multiple PROs evaluated in this study. With the inclusion of PROs in these simple models, AUCs reached thresholds that could be clinically useful, especially the model for unplanned hospital admissions. This study demonstrates another potential use of PRO data in patients with stroke—the prognostic utility of PROs for future hospital admissions and ED use support the routine collection of patient‐reported health measures in clinical practice.

Sources of Funding

None.

Disclosures

None.

Tables S1–S5

Click here for additional data file.