Predicting Individual Risk of Emergency Hospital Admissions - A Retrospective Validation Study.

PURPOSE: A high number of hospital admissions may indicate poor general health and less than optimal health care across sectors. To prevent hospital admissions, previous studies have focused on predicting readmissions relating to a defined index admission and specific condition, whereas generic models suited for community-dwelling persons are lacking. The aim of this study was to validate a generic model that predicted risk of emergency hospital admission within the following three months and to investigate regional variation.
MATERIALS AND METHODS: This study is an observational register-based validation study of a prediction model. The prediction model was based on a population of frail elderly, persons with non-communicable diseases, and persons with three emergency hospital admissions using information about diagnoses and hospital contacts. The prediction model consisted of two stages. In the first stage, covariate associations to admissions are estimated from observed data in one year. In the second stage, admissions are predicted in the coming three months based on observed estimations from the first stage. The validity of the model was calculated by comparing predicted and observed admissions from August 1st to October 31st, 2016.
RESULTS: The study included 112,026 persons. In nationwide data, area under the curve (AUC) was 0.7742 (95% CI 0.7698-0.7786), and the positive predictive value was 52% for the 99th percentile (the top 1%). AUC varied between regions from 0.6914 in Southern Denmark (95% CI 0.6779-0.7049) to 0.8224 (95% CI 0.8064-0.8384) in North Denmark. AUC was higher with nationwide data compared to regional.
CONCLUSION: The model performed satisfactorily in predicting individual risk of emergency hospital admission.

Introduction

A high number of emergency hospital admissions may indicate poor general health1 and less than optimal healthcare planning across sectors. Some hospital admissions are caused by ambulatory care sensitive conditions that may be prevented by interventions in primary care or outpatient hospital care.2

Persons with non-communicable diseases such as heart diseases and chronic obstructive pulmonary disease (COPD) are at risk of becoming high-cost users of healthcare resources.3 The incidence of unplanned hospital admissions for ambulatory care sensitive conditions varies with geographical regions, bed availability and quality of primary care.4,5 Being a high-cost user is associated with multimorbidity, mental illness, higher age and low income.6 Number of comorbidities as well as specific combinations thereof are associated with increased risk of hospital admissions,7 and hospital admissions are a substantial driver of healthcare costs.8

Aging populations and increasing healthcare costs call for rigorous prioritization of the delivery of healthcare resources and the recipients. Predicting which persons will become high-cost users and at high risk of needing specialized health care may enable timely preventive measures. Prediction models should accommodate the diversity and complexity of this population.9 Prediction models primarily based on administrative or clinical data tend to perform better than models developed with self-reported data.10 Most models that predict hospital admissions have moderate discriminative ability.10,11 In addition to disease-specific prediction models, generic models to predict hospital admissions have also been developed, but they mostly focus on readmission after a defined index admission, and prediction models without a defined index admission are rare.11,12 A generic model needs to encompass a wide range of conditions and time points in the healthcare trajectory, which may increase complexity but also enable preventive measures for patients with multimorbidity and general frailty. It is pertinent to develop generic prediction models that support personalized treatment and care across diagnoses. Focusing on modifiable risk factors and preventability of admissions may support the use of prediction models in case-finding and prioritizing patients for more intensive supportive measures.

In Denmark, a national randomized controlled trial of telephone-based self-management support entitled Proactive Health Support (PaHS) took place from April 2018 to July 2019.13 The participants in the trial were identified through an algorithm that calculated individual risk of hospital admission within the following three months – the PaHS prediction model. The aim of this article is to validate the PaHS prediction model to predict individual risk of hospital admission and to investigate the extent of regional variation.

Materials and Methods

Design

This nationwide cohort study was based upon two years of information from healthcare registers to predict the risk of hospital admissions. This study describes the validation of the PaHS prediction model alone, whereas model development is outside the scope of this study. The model development was data driven as well as informed by strategic decisions in focusing on predefined preventive admissions and the three target groups described below. This study is reported using the TRIPOD statement.14 Since the PaHS prediction model was used to select participants for a randomized controlled trial and the intervention might affect hospital admissions, we performed the analyses on information predating the trial and the preceding implementation period.15 Therefore, the validated prediction period was from August 1st to October 31st, 2016.

Participants

The target groups to be included in the PaHS prediction model were defined with the following inclusion criteria: 1. The chronic disease group: Persons with a hospital contact within the last year caused by ≥1 of the following diagnoses: heart diseases, connective tissue diseases, pulmonary diseases or diabetes and one or more emergency hospital contacts within 12 months. 2. The emergency group: Persons with three or more emergency hospital contacts in the past six months. 3. The frail elderly group: Persons aged 65 years or more with a preventable hospital admission (admissions caused by the following diagnoses: dehydration, constipation, lower respiratory tract infections, urinary tract infections, gastroenteritis, fractures, nutritional deficiency anemia, social causes, and pressure ulcers) or an emergency readmission (within 30 days regardless of diagnosis). The three target groups were selected because of the frequency of hospitalizations and clinical experiences with earlier versions of PaHS. Persons belonging to either one of the three target groups aged 18 or above were included in the prediction calculation. Persons with a contact within 12 months of the following psychiatric diagnoses were excluded: substance abuse disorder, schizophrenia or dementia and metastatic cancer. Diagnosis codes are included in the .

Data

All data were derived from the Danish National Patient Register (DNPR) and assembled at Statistics Denmark. During analysis, data were pseudonymized and managed according to the requirements of the Danish National Data Protection Agency (Reference Nr. 06046). The dependent variable was a composite score of risk for preventable and acute hospital admissions. Covariates were age, gender, diagnoses, number and type of outpatient visits, admissions, preventable admissions predefined by diagnosis, 30-day readmissions, bed days, referral, acute vs planned contacts, pulmonary and cardiac surgery, any surgery, medical specialty, costs and dates for all contacts and procedures. The full list of covariates can be found in the .

For this study, we included information from the DNPR from August 1st, 2014 to October 31st, 2016. The prediction is calculated for emergency admissions from August 1st to October 31st, 2016.

Analyses

PaHS Prediction Model

The PaHS prediction model predicts risk of emergency hospital admission in the three months following the prediction date based on information from hospital contacts in a period of one year prior to the three months. The impact of each covariate on the risk of emergency hospital admission is estimated from the corresponding data one year prior.

Initially, persons in the three target groups were identified (the chronic disease, emergency and frail elderly groups). In the first estimation step, several generalized linear models are then conducted. The first set of analyses is regressions on number of emergency admissions (within the three months) with a chosen set of variables (based on the prior year). This is done separately for the 57 different covariates presented in . The 57 covariates and their derived forms are modeled separately at first to allow for more flexibility, and the values for each individual are included in the next model, see . The second step combines the results from step one in one regression model, and the third step is a logistic regression of admission (yes/no) based on the values in the second model.

These steps are conducted separately for emergency admissions and preventable emergency admissions as the dependent variable resulting in two scores. Finally, the two scores are combined into one by summing them each weighted by their share of a common sum of both areas under the receiver operating characteristic (ROC) curve. Predictions are made by the same three steps, using results from the estimation part as weight/loadings in each step. In consequence, regression coefficients estimated in the estimation period are combined with register information on the outcome hospital admissions in the prediction period. The samples for the first estimation step and the second prediction step are separated by time and both include 100% of the sample.

Validation

Applying the PaHS prediction model, we calculated a risk score of emergency hospital admission in the three months from August 1st to October 31st, 2016. The predicted risk was then compared with information on emergency hospital admission within the same three months registered in the DNPR. The ROC and the corresponding area under the curve (AUC) were used to evaluate the discriminative ability of the PaHs predicted risk. Sensitivity, specificity, positive and negative predictive values were calculated for the 10th, 25th, 50th, 75th, 90th, 95th and 99th percentiles. The main model discrimination result is AUC. Due to clinical relevance, secondary results are the positive predictive value of the 95th and 99th percentiles since they represent high-risk healthcare users. Model calibration was estimated with the Brier score.

Since precision of the PaHS model may depend on the number of individuals included in the prediction, the calculation of PaHS risk score and validation was conducted for the whole country and for each of the five Danish regions.

Results

There were 112,026 persons aged 18 and over, who were included in at least one of the three target groups on August 1st, 2016. Table 1 presents characteristics of the included persons.

Table 1

Sample Characteristics

Participant Characteristics	Denmark		Region
			North Denmark		Central Denmark		Southern Denmark		Capital		Zealand
Total n=112,026	n	%	n	%	n	%	n	%	n	%	n	%
Gender - Female	58,235	51.98	4700	51.71	9454	49.69	10,853	50.63	23,492	54.21	9736	50.87
Age
18–44	18,277	16.31	1298	14.28	2428	12.76	3185	14.86	9010	20.79	2356	12.31
45–54	10,837	9.67	839	9.23	1652	8.68	2035	9.49	4581	10.57	1730	9.04
55–64	15,159	13.53	1252	13.77	2530	13.30	2961	13.81	5948	13.72	2468	12.90
65–74	28,916	25.81	2411	26.52	5262	27.66	5480	25.57	10,177	23.48	5586	29.19
75–84	25,785	23.02	2161	23.77	4857	25.53	5228	24.39	8840	20.40	4699	24.55
85+	13,052	11.65	1129	12.42	2298	12.08	2545	11.87	4781	11.03	2299	12.01
Chronic diseases*	79,560	71.02	6742	74.17	14,208	74.67	15,983	74.57	29,045	67.02	13,582	70.97
Emergency contacts†	24,565	21.93	1834	20.18	2839	14.92	4137	19.30	12,724	29.36	3031	15.84
Frail elderly‡	32,024	28.59	2109	23.20	6211	32.64	5379	25.10	11,004	25.39	7321	38.25

Notes: *Chronic diseases: Heart, connective tissue, lung and diabetes. †Emergency contacts: ≥ three acute hospital contacts within six months. ‡Frail elderly: ≥ 65 years of age and a preventable admission within 12 months.

Most participants were aged 65 years or older, 52% were women and 71% had at least one predefined chronic disease. The Capital Region differed from the other four regions by having a younger population, more women and fewer participants with chronic diseases compared to the other four regions.

Between August 1st and October 31st, 2016, 11,748 persons had one or more emergency hospital admissions.

Table 2 presents the sensitivity, specificity, positive and negative predictive value and the predicted risk threshold for the 10th, 25th, 50th, 75th, 90th, 95th, 99th percentiles with national data. The positive predictive value for the 99th percentile (top 1%) ranged from 0.251 (95% CI 0.193–0.309) in Southern Denmark to 0.656 (95% CI 0.589–0.723) in Zealand with a national average of 0.517 (95% CI 0.488–0.546) (Table 2). Model performance with regional data is presented in the . For regional data, the positive predictive value for the 99th percentile (top 1%) ranged from not applicable in Southern Denmark to 0.446 (95% CI 0.4–0.493) in the Capital.

Table 2

Model Performance. Sensitivity, Specificity, Positive Predictive Value and Negative Predictive Value for the Following Percentiles: 10th, 25th, 50th, 75th, 90th, 95th and 99th

Model Performance		Percentile
		10th	25th	50th	75th	90th	95th	99th
Denmark	Predicted risk threshold	0.023	0.030	0.041	0.065	0.142	0.265	0.494
	Sensitivity	0.978 (0.975–0.98)	0.945 (0.94–0.949)	0.848 (0.842–0.855)	0.628 (0.619–0.637)	0.298 (0.29–0.307)	0.154 (0.148–0.161)	0.049 (0.045–0.053)
	Specificity	0.109 (0.107–0.111)	0.273 (0.27–0.276)	0.541 (0.538–0.544)	0.794 (0.792–0.797)	0.923 (0.922–0.925)	0.962 (0.961–0.963)	0.995 (0.994–0.995)
	Positive predictive value	0.114 (0.112–0.116)	0.132 (0.13–0.134)	0.178 (0.175–0.181)	0.263 (0.258–0.268)	0.313 (0.304–0.321)	0.324 (0.312–0.336)	0.517 (0.488–0.546)
	Negative predictive value	0.977 (0.974–0.979)	0.977 (0.975–0.978)	0.968 (0.967–0.97)	0.948 (0.946–0.949)	0.918 (0.917–0.92)	0.907 (0.905–0.908)	0.899 (0.898–0.901)
North Denmark	Predicted risk threshold	0.019	0.026	0.034	0.048	0.083	0.135	0.445
	Sensitivity	0.985 (0.977–0.994)	0.963 (0.95–0.976)	0.886 (0.864–0.909)	0.712 (0.68–0.744)	0.453 (0.418–0.489)	0.258 (0.226–0.289)	0.053 (0.037–0.069)
	Specificity	0.108 (0.101–0.114)	0.269 (0.26–0.279)	0.535 (0.524–0.546)	0.792 (0.783–0.801)	0.932 (0.927–0.937)	0.969 (0.965–0.973)	0.994 (0.992–0.996)
	Positive predictive value	0.091 (0.085–0.097)	0.107 (0.1–0.114)	0.148 (0.137–0.158)	0.237 (0.22–0.255)	0.377 (0.346–0.409)	0.429 (0.383–0.474)	0.444 (0.342–0.547)
	Negative predictive value	0.988 (0.981–0.995)	0.988 (0.983–0.992)	0.981 (0.977–0.985)	0.968 (0.964–0.972)	0.949 (0.945–0.954)	0.935 (0.93–0.94)	0.92 (0.915–0.926)
Central Denmark	Predicted risk threshold	0.023	0.031	0.043	0.064	0.117	0.197	0.502
	Sensitivity	0.977 (0.971–0.984)	0.948 (0.938–0.957)	0.861 (0.846–0.876)	0.662 (0.642–0.682)	0.367 (0.347–0.388)	0.205 (0.188–0.223)	0.046 (0.038–0.055)
	Specificity	0.11 (0.105–0.114)	0.275 (0.268–0.281)	0.545 (0.537–0.552)	0.801 (0.795–0.807)	0.933 (0.93–0.937)	0.969 (0.967–0.972)	0.995 (0.993–0.996)
	Positive predictive value	0.12 (0.115–0.125)	0.14 (0.134–0.146)	0.191 (0.183–0.199)	0.293 (0.281–0.306)	0.407 (0.385–0.429)	0.455 (0.424–0.487)	0.516 (0.445–0.587)
	Negative predictive value	0.975 (0.968–0.982)	0.977 (0.973–0.981)	0.969 (0.966–0.973)	0.95 (0.947–0.954)	0.922 (0.918–0.926)	0.907 (0.903–0.912)	0.893 (0.889–0.898)
Southern Denmark	Predicted risk threshold	0.022	0.030	0.042	0.084	0.272	0.461	0.494
	Sensitivity	0.969 (0.959–0.978)	0.921 (0.906–0.936)	0.803 (0.781–0.824)	0.472 (0.445–0.5)	0.167 (0.146–0.187)	0.085 (0.07–0.101)	0.042 (0.031–0.053)
	Specificity	0.104 (0.1–0.109)	0.261 (0.255–0.267)	0.519 (0.512–0.526)	0.764 (0.758–0.77)	0.904 (0.9–0.908)	0.952 (0.949–0.955)	0.992 (0.991–0.993)
	Positive predictive value	0.064 (0.061–0.068)	0.073 (0.069–0.077)	0.096 (0.09–0.101)	0.113 (0.104–0.121)	0.099 (0.087–0.112)	0.102 (0.084–0.12)	0.251 (0.193–0.309)
	Negative predictive value	0.981 (0.976–0.987)	0.981 (0.978–0.985)	0.976 (0.974–0.979)	0.958 (0.955–0.961)	0.945 (0.942–0.948)	0.943 (0.939–0.946)	0.942 (0.939–0.945)
Capital	Predicted risk threshold	0.023	0.029	0.038	0.057	0.112	0.204	0.487
	Sensitivity	0.979 (0.975–0.983)	0.945 (0.938–0.951)	0.85 (0.839–0.86)	0.643 (0.63–0.657)	0.333 (0.32–0.347)	0.184 (0.173–0.195)	0.044 (0.038–0.05)
	Specificity	0.11 (0.107–0.113)	0.274 (0.27–0.279)	0.544 (0.539–0.549)	0.799 (0.795–0.803)	0.929 (0.927–0.932)	0.967 (0.965–0.968)	0.994 (0.994–0.995)
	Positive predictive value	0.121 (0.117–0.124)	0.14 (0.136–0.143)	0.188 (0.183–0.194)	0.285 (0.277–0.294)	0.37 (0.355–0.384)	0.408 (0.387–0.428)	0.49 (0.443–0.537)
	Negative predictive value	0.977 (0.972–0.981)	0.975 (0.973–0.978)	0.967 (0.964–0.969)	0.947 (0.945–0.95)	0.918 (0.915–0.921)	0.905 (0.902–0.908)	0.893 (0.89–0.896)
Zealand	Predicted risk threshold	0.024	0.034	0.049	0.079	0.167	0.293	0.537
	Sensitivity	0.975 (0.969–0.981)	0.942 (0.934–0.951)	0.843 (0.83–0.857)	0.628 (0.61–0.646)	0.327 (0.31–0.345)	0.175 (0.161–0.189)	0.045 (0.037–0.053)
	Specificity	0.113 (0.108–0.118)	0.283 (0.276–0.29)	0.559 (0.551–0.566)	0.815 (0.809–0.821)	0.939 (0.935–0.943)	0.972 (0.969–0.974)	0.996 (0.995–0.997)
	Positive predictive value	0.159 (0.153–0.164)	0.184 (0.178–0.19)	0.247 (0.238–0.255)	0.368 (0.354–0.381)	0.479 (0.457–0.501)	0.514 (0.482–0.545)	0.656 (0.589–0.723)
	Negative predictive value	0.963 (0.955–0.972)	0.966 (0.961–0.971)	0.954 (0.95–0.958)	0.927 (0.923–0.932)	0.891 (0.886–0.895)	0.873 (0.868–0.878)	0.859 (0.854–0.864)

Notes: Discriminatory measures with 95% Confidence Intervals. Emergency hospital admissions predicted from August 1st to October 31st, 2016.

Figure 1 presents the receiver operating characteristic curve. Area under the curve (AUC) was 0.7742 (95% CI 0.7698–0.7786) for the whole country when the model used national data.

Figure 1

Receiver operating characteristics curve for national data.

Table 3 presents AUC for each region when the model ran national and regional data, respectively. The model performed better with national data for estimation. While the AUC for four regions was ≥0.77 regardless of national or regional data, Southern Denmark had AUC of 0.69 (95% CI 0.6779–0.7049) with national data but only 0.64 (95% CI 0.6260–0.6535) on regional data.

Table 3

Area Under the Curve for Each of the Five Regions Predicted with National and Regional Data Respectively

Area Under the Curve		N	AUC*	95% CI
Predicted with national data	Denmark	112,026	0.7742	(0.7698–0.7786)
	North Denmark	9090	0.8224	(0.8064–0.8384)
	Central Denmark	19,027	0.7978	(0.7874–0.8082)
	Southern Denmark	21,434	0.6914	(0.6779–0.7049)
	Capital	43,337	0.7850	(0.7781–0.7920)
	Zealand	19,138	0.7891	(0.7798–0.7984)
Predicted with regional data	North Denmark	9090	0.7658	(0.7470–0.7846)
	Central Denmark	19,027	0.7726	(0.7616–0.7836)
	Southern Denmark	21,434	0.6397	(0.6260–0.6535)
	Capital	43,337	0.7730	(0.7658–0.7803)
	Zealand	19,138	0.7756	(0.7660–0.7852)

Notes: *Area under the receiver operating characteristics curve with 95% Confidence Intervals. Emergency hospital admissions predicted from August 1st to October 31st, 2016.

Table 4 presents model calibration (accuracy) estimated with the Brier score. The Brier score was 0.09 for the national model and no higher than 0.11 in all regions.

Table 4

Model Calibration Assessed with the Brier Score with National and Regional Data Respectively

Model Calibration		Population	Persons Admitted	Brier Score
		n	n (%)
Predicted with national data	Denmark	112,026	11,748 (10.5)	0.090
	North Denmark	9090	757 (8.3)	0.071
	Central Denmark	19,027	2108 (11.1)	0.091
	Southern Denmark	21,434	1277 (6.0)	0.055
	Capital	43,337	4805 (11.1)	0.092
	Zealand	19,138	2801 (14.6)	0.111
Predicted with regional data	North Denmark	9090	757 (8.3)	0.074
	Central Denmark	19,027	2108 (11.1)	0.094
	Southern Denmark	21,434	1277 (6.0)	0.056
	Capital	43,337	4805 (11.1)	0.094
	Zealand	19,138	2801 (14.6)	0.117

Notes: Emergency hospital admissions predicted from August 1st to October 31st, 2016. Model calibration assessed with Brier score.

Discussion

Findings

The PaHS prediction model demonstrated satisfactory predictive value. The positive predictive value and AUC in Southern Denmark were well below the other regions. This may be explained by the relatively low number of emergency hospital admissions in this region, since the discriminatory ability of any prediction model may be reduced when the outcome prevalence is low.16 Furthermore, there may be indeterminable variation in registration practice between the regions such as classifying more hospital admissions as “planned” rather than “emergency”.

Strengths and Weaknesses

The complexity of the model enables prediction in a population that is heterogeneous in disease characteristics as well as point in the healthcare trajectory since there is no index admission. This demanding task is managed by using the estimated associations to calculate the expected number of admissions for each individual for each of the 57 covariates. The PaHS prediction model may identify persons already having many hospital admissions as well as those at risk of becoming consistently frequently admitted. The model design builds on estimation on the two years prior to prediction date and therefore assumes similar patterns of hospital admission in both years, which would account for seasonal variation but not clinical disease progression or recovery. A drawback of the model is the number of covariates that might imply overfitting of the model and might lead to errors in local implementation. The results, however, speak against overfitting since the model performed satisfactorily in four of the five regions. Furthermore, the Brier score was 0.09 overall and no higher than 0.11 in all regions suggesting fair model accuracy.

Similar Studies

The national AUC of 0.77 is comparable to other admission prediction models that rarely exceed 0.80.10 Other models with good discriminatory value have included predictors such as prior healthcare use, multimorbidity or polypharmacy and named medical diagnoses or prescription medication.10 The predictive value of the current model might improve with information about prescription medication. But these data are not available in the Danish National Patient Register, and they are registered with some delay, which would outdate the risk prediction.

Implications

The model is not focused on one specific condition and may be used to identify and prioritize community-dwelling adults for preventive measures. The model was developed for prioritization without a predefined cutoff. Based on the positive predictive value, focusing preventive interventions on the top 1% or top 5% would secure a relevant patient population.

Targeting high-risk persons for interventions includes the risk of the sample regressing towards the mean,17 which may dilute intervention efficacy. For example, an otherwise well-designed trial of highly intensive supportive care for so-called superutilizers turned out negative because the control group also experienced lower hospital admissions and a prediction model was not utilized.18 The PaHS prediction model was employed to select patients for telephone-based self-management support. The impending evaluation of the trial will demonstrate intervention efficacy and investigate whether predicted risk interacts with intervention effects. Since patients have untethered access to their own patient records, documentation of any risk score in a patient record has ethical implications and requires strong communicative skills to avoid stigmatization, ensure patients’ understanding of the risk score and ensure patient involvement in deciding whether to disclose the risk score. The PaHS prediction model was used to invite patients for trial participation. Although the risk score was not intended to have any role in the intervention, the topic might come up since randomization was stratified by risk score. The nurses with patient contact were well trained in communication.13 A qualitative observational study of 22 start-up sessions did not identify the predicted risk as a theme19 and it therefore seems unlikely that participants were burdened from the topic.

Future Research

The present prediction model assigns a risk to each patient at a given date but does not differentiate between stable, fluctuating, decreasing and increasing risk. Furthermore, high-cost healthcare users are heterogeneous and frequent users only account for a small portion of the population.9 Future prediction models could focus on identifying persons that are at risk of developing a high use of healthcare resources, such as trajectory modelling or mapping events that make the overall burden of disease and treatment unmanageable for each patient.

Conclusions

Individual risk of emergency hospital admissions can be satisfactorily predicted.