Utility of models to predict 28-day or 30-day unplanned hospital readmissions: an updated systematic review.

OBJECTIVE: To update previous systematic review of predictive models for 28-day or 30-day unplanned hospital readmissions.
DESIGN: Systematic review. SETTING/DATA SOURCE: CINAHL, Embase, MEDLINE from 2011 to 2015. PARTICIPANTS: All studies of 28-day and 30-day readmission predictive model. OUTCOME MEASURES: Characteristics of the included studies, performance of the identified predictive models and key predictive variables included in the models.
RESULTS: Of 7310 records, a total of 60 studies with 73 unique predictive models met the inclusion criteria. The utilisation outcome of the models included all-cause readmissions, cardiovascular disease including pneumonia, medical conditions, surgical conditions and mental health condition-related readmissions. Overall, a wide-range C-statistic was reported in 56/60 studies (0.21-0.88). 11 of 13 predictive models for medical condition-related readmissions were found to have consistent moderate discrimination ability (C-statistic ≥0.7). Only two models were designed for the potentially preventable/avoidable readmissions and had C-statistic >0.8. The variables 'comorbidities', 'length of stay' and 'previous admissions' were frequently cited across 73 models. The variables 'laboratory tests' and 'medication' had more weight in the models for cardiovascular disease and medical condition-related readmissions.
CONCLUSIONS: The predictive models which focused on general medical condition-related unplanned hospital readmissions reported moderate discriminative ability. Two models for potentially preventable/avoidable readmissions showed high discriminative ability. This updated systematic review, however, found inconsistent performance across the included unique 73 risk predictive models. It is critical to define clearly the utilisation outcomes and the type of accessible data source before the selection of the predictive model. Rigorous validation of the predictive models with moderate-to-high discriminative ability is essential, especially for the two models for the potentially preventable/avoidable readmissions. Given the limited available evidence, the development of a predictive model specifically for paediatric 28-day all-cause, unplanned hospital readmissions is a high priority. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://www.bmj.com/company/products-services/rights-and-licensing/

This is an updated systematic review (2011–2015) of the literature relating to risk predictive models for unplanned hospital readmissions.

This updated systematic review followed rigorous methodology applying comprehensive electronic database search, strict inclusion, exclusion and quality assessment criteria to synthesise current literature on characteristics and properties of risk predictive models for 28-day or 30-day unplanned hospital readmissions.

The outcomes of the predictive models included in this systematic review were restricted to 28-day or 30-day unplanned hospital readmission.

Introduction

Unplanned hospital readmissions cause a disruption to the normality of patients and/or family/carers' lives and result in a significant financial burden on the healthcare system.1 2 In the USA, it has been estimated that 7.8 million (20%) of hospital-discharged patients were readmitted. This accounted for $17.4 billion of hospital payments by Medicare.3 4 In the UK, the figures suggested ∼35% of unplanned hospital readmissions, costing 11 billion pounds per annum (5.3 million admissions in 2010/2011).5

Unplanned hospital readmission rate is considered as a performance indicator to measure a hospital's quality of care.6 7 Unplanned hospital readmission is defined as the percentage of unplanned or unexpected readmission to the same hospital within 28 days of being discharged.8 9 However, the literature has widely used 30 days within the context of measurement of hospital readmissions.1 6 7

One of the strategies to reduce the unplanned hospital readmission rate is the application of predictive models to identify patients at high risk for readmission. Preventive approaches can then be developed and applied to target the identified high-risk patients. A previous systematic review10 was conducted in 2011 on the risk predictive models for adult medical patients' hospital readmissions. A total of 30 studies with 26 predictive models were included, and the overall performance of reviewed models was poor. It is, however, worth noting that studies conducted in developing nations and studies that focused on paediatric patients and adult psychiatric and surgical patients were excluded.

Since 2011, there has been increased interest in either developing new predictive models or validating existing models due to high inpatient demand on the healthcare system.11–15 However, the performance of risk predictive models has varied significantly. The purpose of this systematic review is to update previous systematic review on predictive models for 28-day or 30-day unplanned hospital readmissions and to investigate and assess the characteristics of these models.

Methods

Search strategy and data sources

An electronic database search was carried out using the CINAHL, Embase and MEDLINE to identify studies published between 2011 and 2015. The key search terms included ‘unplanned readmission* or rehospitali*’ AND (‘predict*’ AND ‘model*’) OR ‘ROC or C-statistic*’ OR ‘sensitivity or specificity’ (see online supplementary appendix 1 for full search strategy).

Inclusion/exclusion criteria

Articles eligible for inclusion were those published in English with full-text access from 2011 to 2015. Only peer-reviewed studies were included in this review. The study design of included studies needed to be clearly stated together with details of the performance of the risk predictive model reported. Abstract-only references were excluded. Studies included in the previous systematic review10 were excluded due to overlapping of the search period (1985–August 2011). Studies that included patients discharged from hospital but still receiving treatment, that is, intravenous antibiotics, via ambulatory care or hospital in the home programmes were also excluded.

Study selection and data extraction

Initial literature searches were conducted by HZ and PD. Two authors (HZ and LG) independently screened titles, abstracts and appraised full papers against the inclusion and exclusion criteria. The process of exclusion was relatively straightforward and only a handful of studies warranted discussion between the authors (HZ, LG, SD, PD and PR) and to reach consensus as to whether they met the inclusion criteria.

Data were extracted from the final included studies by three authors (HZ, LG and SD). The data extraction included study characteristics, model performance and key variables of the predictive model. Study characteristics included study setting, population, data source, the timing of data collection, sample size, study design, model name if applicable, model utilisation outcome and readmission rate (table 1). Measures assessing predictive model performance, including discrimination, calibration, cut-off values used to identify patients at high risk of being readmitted to the hospital, sensitivity, specificity, positive predictive value (PPV) or negative predictive value (NPV), were extracted (table 2). Model discrimination is commonly assessed using C-statistic or the area under the receiver operating characteristic curve. Values of the C-statistic measurement range from 0.5 to 1.0. A value of 0.5 indicates that the model is no better than chance at making a prediction of membership in a group, and a value of 1.0 indicates that the model perfectly identifies those within and not within a group. Models are typically considered reasonable when the C-statistic is higher than 0.7 and strong when the C-statistic exceeds 0.8.71 Variables of the readmission risk predictive model were also extracted and presented in table 3. The studies were grouped based on the model utilisation outcome in the three tables. Disagreements between two reviewers about the extracted data were resolved through group discussion.

Table 1

Characteristics of 49 included studies on 28-day or 30-day unplanned hospital readmission (UHR) predictive models

Reference	Model name	Model outcome	Study design/data source	Sample size	Age group (years)	Duration of retrieved data source	Readmission rate
All-cause UHRs (14)
Escobar et al16USA	ED 30Discharge 30 LACE (validation)	30-day all-cause readmissions	Retrospective cohort21 hospitalsElectronic medical records	A total of 360 036 patients179 978 derivation set180 058 validation set	Mean=64.1	1 June 2010–31 December 2013	Derivation: 12.5%; Validation: 12.4%
Yu et al17USA	Institution-specific prediction modelLACE (validation)	30-day all-cause readmission	Retrospective cohort3 hospitals	Hospital 1=2441Hospital 2=26 520Hospital 3=45 785	≥65	Not reported	H1=23%H2=20%H3=18%
Baillie et al18USA	Prediction model	30-day all-cause readmissions	Retrospective and prospective cohort3 hospitals	Retrospective: 120 396 dischargesprospective validation	Not reported—adult	August 2009–September 2012	Retrospective: 14.4%; Prospective: 15.1%
Choudhry et al12USA	ACC Admission and Discharge model	30-day all-cause readmissions	Retrospective cohort8 hospitals	A total of 126 479 patients94 859 derivation set31 619 internal and 6357 external validation	Mean=66.01 (readmission)57.65 (no readmission)	1 March 2010–31 July 2012	7.25%
Gildersleeve and Cooper19USA	Risk of readmission score (RRS)	30-day all-cause readmission	Retrospective cohort1 community hospital	Derivation: 8700 patients	Mean=60.6	2010	14.1%
				Validation: 8189 patients	Mean=65	2011	14.8%
Kruse et al20USA	Unnamed	30-day all-cause readmission	Retrospective cohort91 hospitals—Health Facts Database	463, 351 Index admissions	≥18	1 October 2008–31 August 2010	9.7%
Richmond21USA	Unnamed	30-day all-cause readmission for patients≥65 years	Retrospective cohortstate-level database	4717 patients split into a derivation (80%) and validation sample (20%)	Mean=77.27	January 2010–December 2012	14.4%
Shulan et al22USA	Unnamed	30-day all-cause readmission	Retrospective cohortcentralised database	8718 patientsDerivation (50%)Validation (50%)	Mean=67.04 (UHRs); 66.43 (no UHRs)	2011	16.2%
van Walraven et al23Canada	LACE+ (extension of a validated index)	30-day all-cause readmission	Retrospective cohortcentralised database	499 996 patients/858 410 index hospitalisations	>18	2004–2009	11.8%
Cotter et al13UK	LACE index (validation)	30-day all-cause readmission	Retrospective cohortcentralised database	507 patients	Mean=85	2010	17.8%
	Regression model		Retrospective cohortcentralised database	502 patients (validation cohort)			14.8%
Khan et al24USA	Rehospitalisation Risk Score	30-day all-cause readmission	Retrospective cohort10 hospitals/EMRs	227 patients	Average=79	Single day on 26 January 2011	15%
Lee25Korea	Unnamed	28-day all-cause readmission	Retrospective cohort1 tertiary hospital	11 951 patientsDerivation (70%); Validation (30%)	Ranged from 0 to 70+	2009	28.9%
van Walraven et al26Canada	CMG score (case-mix groups)	30-day all-cause readmission	Retrospective cohort4 health databases	Random 200 000 patients of 3 277 033Derivation: 100 000Validation: 100 000	Mean age ofDerivation: 58Validation: 57.9	1 April 2003–31 March 2009	6.8%
	LACE index (validation)
	Combined CMG score and LACE index
van Walraven et al27Canada	LACE+LACE+ with CMG score	30-day all-cause readmission	Retrospective cohort4 health databases	Random 500 000 of 3 277 033 patients then 1/2 derivation and ½ validation	Mean=57.9 (derivation); 57.9 (validation)	1 April 2003–31 March 2009	14%
Cardiovascular disease-related UHRs including pneumonia (11)
Hebert et al15USA	CHF modelPNA modelAMI modelCombined model	30-day readmission onCongestive heart failure/pneumonia/acute myocardial infarction	Retrospective cohortA tertiary medical centre	A total of 3968 patientsDerivation: 3572	Mean=61	1 August 2009–31 July 2011	16.2%
				Historical validation: 1756		1 August 2008–31 July 2009	17.7%
				Random sample: 396			16.2%
Iannuzzi et al28USA	Vascular surgery readmission risk score	30-day readmission on patients after vascular surgery	Retrospective cohortNational Surgical Database	24 929 patients	Mean=69.5 (UHRs); 69.7 (no UHRs)	2011	10.1%
Keyhani et al29USA	CMS-based model	30-day readmission on patients with stroke	Retrospective cohort114 hospitals	3436 patients	Mean=69.5 (UHRs); 66.9 (no UHRs)	2007	12.8%
	CMS-based model plus social Risk factors
	CMS-based model plus social risk and clinical factors
Rana et al30Australia	Electronic medical record (EMR) model	30-day readmission on ischaemic heart disease of patients after AMI	Retrospective cohortA regional health service—tertiary hospital	1660 AMI admissionsDerivation cohort: 1107Validation cohort: 553	Mean=67.8 (derivation cohort); validation cohort: 68.4	January 2009–December 2011	6.3%
	HOSPITAL score (validation)
	Comorbidities (validation)
Shahian et al31USA	Unnamed	30-day readmission post coronary artery bypass grafting	Retrospective cohortNational Database (846 hospitals)	162 572 admissions	≥65	2008–2010	12.6–23.6%
Shams et al32USA	Potentially avoidable readmission (PAR)	30-day avoidable readmission on pneumonia/HF/AMI/COPD	Retrospective cohortVeterans Health Administration data	5600 admissions	HF: mean=71.3 (PAR); vs 68.6 (no UHRs)AMI: mean=73.3 (PAR) vs 69.3 (no UHRs)	2011–2012	13.09%
			Internal validation
			External validation	478 patients		August and September 2012
	CMS endorsed model (validation)	30-day readmission
Sharif et al33USA	Unnamed	30-day readmission on patients aged 40–64 years with COPD	Retrospective cohortA large national commercial insurance database	8263 patients	Mean=57 (UHRs); no UHRs—age not reported	January 2009–November 2011	8.9%
Lucas et al34USA	Complex all-variable model; parsimonious readmission score	30-day readmissions on patients post general, vascular, and thoracic surgery	Retrospective cohortNational Surgery Database	A total of 230 864 patientsDerivation: 162 159 (70%): Validation: 68 705 (30%)	Median=56	2011	5–16% across surgical specialties
Wallmann et al35Spain	Unnamed	30-day readmission on cardiac-related disease	Retrospective cohort1 tertiary centre	35 531 admissionsDerivation cohort: 24 881Validation cohort: 10 650	Mean=67.9	2003–2009	Derivation: 4.4%; Validation: 4.7%
Wasfy et al36USA	Risk score for 30-day readmission after PCI (parsimonious)	30-day readmission after percutaneous coronary intervention	Retrospective cohortcentralised database	36 060 surviving to discharge	Mean=68.1 (UHRs); 64.3 (no UHRs)	1 October 2005–30 September 30 2008	10.4%
Krumholz et al37USA	Claims model	30-day readmission on acute myocardial infarction (AMI)	Retrospective cohortMedicare Claims Database	Derivation cohort: 100 465Validation cohort: 321 088	Mean=78.7	Half of 2006	18.9%
	Medical record model			Derivation cohort: 130 944Validation cohort: 130 944		2005 and half of 2006	19.96%
Cardiovascular disease-related UHRs including pneumonia—heart failure only (11)
Betihavas et al38Australia	Unnamed	28-day readmission on patients with chronic heart failure	Retrospective cohortMulticentre	280 patients94 (no UHRs); 37 (28-D UHRs)	Mean=69 (no UHRs); 79 (UHRs)	Not reported	13%
Di Tano et al39Italy	Unnamed	30-day readmission on acute HF	Prospective cohortNational Registry Database	1520 patients	Mean=72	Not reported	6.25%
Huynh et al40Australia	The non-clinical modelThe clinical modelThe combined model	30-day readmission on HF	Retrospective cohortstate-wide data linkage	Non-clinical—1537 patientsClinical—977 patients available	Mean=80	2009–2012	25.4%
Raposeiras-Roubin et al41Spain	GRACE risk score	30-day readmission on HF after acute coronary syndrome	Retrospective cohortA single centre	4429 patients	Mean=77 (UHRs); 68 (no UHRs)	2004–2010	1.3%
Sudhakar et al42USA	Readmission Risk score	30-day readmission on patients with CHF	Retrospective cohortA tertiary hospital/chart review	1046 admissions from 712 patients	Mean=65.2	September 2011–August 2013	35.28%
Fleming et al43USA	Unnamed	30-day readmission on patients with HF	Retrospective cohort1 tertiary medical centre	3413 admissionsDerivation: Validation=3:1(2566:847)	Mean=74 (derivation cohort); validation cohort: 74.6	1 October 2007–30 August 2011	24.2% (derivation)
Wang et al44USA	LACE index (validation)	30-day readmission on patients with CHF	Retrospective cohortAn urban public hospital	253 patients	Mean: 57.67 (no UHRs); 56.17 (UHRs)	June 2012–June 2013	24.5%
Eapen et al45USA	Unnamed	30-day readmission on heart failure	Retrospective cohortCenters for Medicare database	33 349 patient70% in derivation cohort30% in validation cohort	Median=80	1 January 2005–31 December 2009	22.8%
Zai et al46USA	The telemonitoring-based readmission model; the psychosocial readmission model (validation)	30-day readmission on heart failure	Retrospective cohortPatients enrolled in the telemonitoring program	100 patients	Average age of 66.8	July 2008–November 2011	38%
Au et al47Canada	Five administrative data-based models: Charlson; CMS KrumholzKeenan; LACE; LACE+	30-day readmission on HF	Retrospective cohort4 health databases	59 652 patients	Mean=76	April 1999 and 2009	19%
Watson et al48USA	The psychosocial readmission model	30-day readmission on HF	Retrospective cohort1 tertiary hospital	729	Mean=71.4	1 October 2007–30 September 2008	13.3% (all female)
Cardiovascular disease-related UHRs including pneumonia—pneumonia only (2)
Mather et al49USA	Hartford Hospital modelCMS Model (validation)	30-day readmission on pneumonia	Retrospective cohortA tertiary hospital	956 index admissions	≥65	January 2009–March 2012	15.5%
Lindenauer et al50USA	Administrative claims model	30-day readmission on pneumonia	Retrospective cohortMedicare enrolment database	Derivation cohort: 226 545Validation cohort: 762 721	Mean=80	Half of 2006	17.4%
	Medical record model			47 429 cases		Half of 2006 and 2005	17.0%
General medical condition-related UHRs (10)
Shadmi et al51Israel	Preadmission Readmission Detection Model	30-day readmission on medical patients	Retrospective cohortClalit Health Services/EMR	Total: 33 639 admissionsDerivation: 22 406Validation: 11 233	Mean=68.2; 67.5 (no UHRs); 72.5 (UHRs)	1 January 2010–31 March 2010	16.8%
Tsui et al52Hong Kong	Unnamed	28-day readmission on elderly medical patients	Retrospective cohort41 hospitals/EMS	Total: 327 529 episodesDerivation: 165 216Validation: 162 313	≥65	Derivation: 2005Validation: 2006	7.8%7.6%
Donzé et al53USA	Unnamed	30-day readmission on medical patients due to end-of-life care	Retrospective cohort1 tertiary medical centre including 3 hospitals	10 275 admissions	Mean=61.5 (no UHRs); 60.8 (potentially avoidable readmissions (PARs)	1 July 2009–30 June 2010	Total:22.3%; 8%—PARs
He et al54USA	Unnamed	30-day readmission on medical patients and chronic pancreatitis (CP)	Retrospective cohortJHH (tertiary centre)BMC (community hospital)	Medical patients: 26 091 (JHH)+16 194 (BMC)	Mean=50.3 (JHH)51.5 (BMC)	Medical patients: January 2012–April 2013;	11.5% (JHH)8.7% (BMC)
				Patients with CP: 3218 (JHH)+706 (BMC)	Mean age: 51.4 (JHH)51.4 (BMC)	CP discharged from January 2007–April 2013	15.6% (JHH)7.8% (BMC)
Taha et al55USA	Readmission Risk Score (RRS)	30-day readmission on general internal medicine services	Retrospective cohort4 teaching and 2 non-teaching general internal medicine services	858 index hospitalisationsDerivation cohort: 613Validation cohort: 245	Mean=54 (derivation); validation cohort: 54	1 April 2010–30 June 2010	16%
Donzé et al14USA	HOSPITAL score	30-day readmissions on general medical patients	Retrospective cohortMulticentre health services	10 731 discharges	Mean=61.3	1 July 2009–30 June 2010	8.5%
Tan et al56Singapore	LACE index (validation)	30-day readmission on general medical patients	RetrospectiveThe largest tertiary general hospital	127 550 patients	≥21	1 January 2006–31 December 2010	4.87–18.43%
Billings et al11USA	PARR-30	30 days readmission on general medical patients	Retrospective cohortcentralised database	576 868 admissions	Adult	1 April 2008 and 31 March 2009	12.2%
Zapatero et al57Spain	SEMI INDEX	30-day readmission on general medical patients	Retrospective cohortNational Health Database	Derivation cohort: 999 089 patients; Validation cohort: 510 588 patients (internal)	Median=70 for two cohorts	January 2006–December 2007	12.4%
						2008	12.5%
Gruneir et al58Canada	LACE index (validation)	30-day readmission on general medical patients	Retrospective cohort6 hospitals	26 045 patients	18–105	2007	12.6%
Medical condition UHRs—cirrhosis only (2)
Singal et al59USA	Unnamed	30-day readmissions on patients with cirrhosis	Retrospective cohort1 large safety-net hospital	A total of 838 patients with 1291 admissionsDerivation: 968Validation: 323	Mean=52.5	January 2008–December 2009	27%
Volk et al60USA	Cirrhosis readmission prediction model	30-day readmission on cirrhosis	Retrospective cohort1 tertiary hospital	402 patients	≥18	1 July 2006–1 July 2009	41%, 22% of which are PARs
Medical condition UHRs—chronic kidney disease only (1)
Perkins et al61USA	Unnamed	30-day readmission on patients with CKD second to HF	Retrospective cohort2 inpatient facilities	607 patients with chronic kidney disease	Mean=72.3 (UHRs); 74.1 (no UHRs)	1 July 2004–28 February 2010	19.1%
Medical condition UHRs—HIV only (1)
Nijhawan et al62USA	Unnamed	30-day readmission on HIV-infected patients	Retrospective cohort1 tertiary hospital	2402 index admissions randomly split (1/2) into derivation vs validation	Mean=43	March 2006–November 2008	24.4%
Medical condition UHRs—acute pancreatitis (1)
Whitlock et al63USA	Unnamed	30-day readmission on acute pancreatitis	Retrospective cohort2 hospitals	Derivation cohort: 248Validation cohort: 198	Mean=51.6 derivationValidation: 52.3	1 June 2005–31 December 20071 January 2008–31 October 2009	19%23%
Surgical condition-related UHRs (6)
Taber et al64USA	30DRA with fixed variable vs 30DRA with fixed variables and dynamic clinical data	30-day readmission on patients following kidney transplantation	Retrospective cohortAn institution	1147 patients Derivation; internal validation using random iteration of 50% sampling	Mean=51 (no UHRs); 52 (UHRs)	2005–2012	11%
Lawson et al65USA	Unnamed(demographic, preoperative and postoperative risk factors)	30-day readmission on patients following colectomy	Retrospective cohortNSQIP	12 981 patients	≥65	2005–2008	13.5%
Iannuzzi et al66USA	Endocrine surgery Readmission Risk Score	30-day readmission on patients following cervical endocrine operations	Retrospective cohortNSQIP—a large national clinical database	34 046 casesDerivation and validation cohort (numbers were not specified)	Mean=54 (no UHRs); 55 (UHRs)	2011–2012	2.8%
Mesko et al67USA	Unnamed	30-day readmission on total hip and knee arthroplasty	Retrospective cohortA readmission database	1291 admissions/1236 patients	Mean=65.6 (UHRs); 68.3 (no UHRs)	1 May 2010–30 April 2011	3.6%
Moore et al68Canada	Unnamed (quality indicator based)	30-day readmission on trauma	Retrospective cohort57 trauma centres	57 524 patients	≥16	1 April 2005–28 February 2010	6.6%
Graboyes et al69USA	Unnamed	30-day readmission on otolaryngology patients	Retrospective cohortA tertiary hospital	1058 patients—1271 hospital admissions	Mean=52 (no UHRs); 56 (UHRs)	1 January 2011–31 December 2011	7.3%
Mental health condition-related UHRs (1)
Vigod et al70Canada	READMIT (41 points)	30-day readmission after discharge from acute psychiatric units	Retrospective cohortNational health data	Derivation: 32 749 patientsValidation: 32 750 patients	Median=41 (UHRs); 44 (no UHRs)	1 April 2008–31 March 2011	8.42–10%

ACS, acute coronary syndrome; AMI, acute myocardial infarction; AP, acute pancreatitis, CHF, congestive heart failure; CKD, chronic kidney disease; COPD, common obstructive pulmonary disease; EMRs, electronic medical records; GRACE, global registry of acute coronary events; HF, heart failure; PCI, percutaneous coronary intervention; PREADM, preadmission readmission detection model; PNA, peptide nucleic acid.

Table 2

Performance of predictive models for 28-day or 30-day unplanned hospital readmissions (UHRs)

Reference	Model name	Discrimination (ROC)	Calibration (H&L)	Threshold (%)	Sensitivity (%)	Specificity (%)	PPV (%)	NPV (%)
All-cause UHRs (14)
Escobar et al16	ED 30	Validation: 0.739	0.40	≥20≥30≥60
	Discharge 30	Validation: 0.756	0.60
	LACE (validation)	Validation: 0.729	0.40
Yu et al17	Institution-specific prediction model	0.74 (hospital 2)0.64 (at admission)0.72 (after discharge)
	LACE (validation)	0.55 (hospital 2)
Baillie et al18	Prediction model	Retrospective: 0.62			40	85	31	89
		Prospective: 0.61			39	84	30	89
Choudhry et al12	ACC Admission Model	Derivation data set: 0.76Internal validation: 0.75Average (500 simulations in derivation data set): 0.76External validation data set with recalibration: 0.76	Derivation data set: 36.0 (p<0.001)Internal validation data set: 23.5 (p=0.0027)External validation with recalibration: 6.1 (p=0.641)	11	70	71
	ACC Discharge Model	Derivation data set: 0.78Internal validation: 0.77Average: 0.78External validation data set with recalibration: 0.78	Derivation: 31.1 (p<0.001)Internal validation: 19.9 (p=0.01)External validation with recalibration: 14.3 (p=0.074)	11	70	71
Gildersleeve and Cooper19	Risk of readmission score (RRS)	Derivation cohort: 0.74	21.6 (p=0.006)	14	74.9	54.4	22.2	92.6
		Validation cohort: 0.70			79.2	55.4	22.6	94.2
Kruse et al20	Unnamed	Derivation set: 0.668Validation set: 0.657
Richmond21	Unnamed	0.60			47	78
Shulan et al22	Unnamed	Derivation cohort: 0.80Validation cohort: 0.70
van Walraven et al23	LACE+ (extension of a validated index)	0.768 (1 hospitalisation per patient)0.730 (all hospitalisations)	H–L χ2 50.3H–L χ2 10 972
Cotter et al13	LACE index (validation)	0.55
	Regression model	0.57		47	54	47
All-cause UHRs (14)
Khan et al24	Rehospitalisation risk score			19	97	28	19	98
				21	58	63	21	90
				27	42	81	27	89
Lee25	Unnamed	ROC was graphically illustrated, but no actual number was reported
van Walraven et al26	CMG Score	0.637	p=0.0079
	LACE index (validation)	0.72	P<0.0001
	Combined CMG Score and LACE	0.743	p<0.0001
van Walraven et al27	LACE+ (validation)	0.743
	LACE+ with CMG score	0.753
Cardiovascular disease-related UHRs including pneumonia (11)
Hebert et al15	CHF modelPNA modelAMI modelCombined model	Derivation cohort: 0.64–0.73;Historical validation: 0.61–0.68;Random sample combined: 0.63–0.76	p>0.05
Iannuzzi et al28	Vascular surgery readmission risk score	Derivation dataset: 0.67Validation dataset: 0.64	0.090.66
Keyhani et al29	CMS-based model	0.636	0.866
	CMS-based model plus social risk factors	0.646	0.462
	CMS-based model plus social risk and clinical factors	0.661	0.856
Rana et al30	EMR model	0.78		5	65	78	21	83.6
	HOSPITAL score (validation)	0.60	62		50	13	78.9
	Comorbidities (validation)	0.53			65	45
Shahian et al31	Unnamed	0.648
Shams et al32	Potentially avoidable readmission (PAR)	Retrospective cohort: 0.836Validation internal: 0.818/external: 0.809			91.95	97.65	86.61	98.65
	CMS endorsed model (validation)	0.63
Cardiovascular disease-related UHRs including pneumonia (11)
Sharif et al33	Unnamed (basic model vs final model)	Basic model (patient characteristics only): 0.677; final model (additional provider-level and system-level factors)Derivation set: 0.717Validation set: 0.73
Wallmann et al35	Unnamed	0.75		4	66	70	10	98
Wasfy et al36	Risk score for 30-day readmission after PCI (parsimonious)	Validation data set: 0.67		>24
Lucas et al34	Complex all-variable model	Derivation data set: 0.721Validation data set: 0.724
	Parsimonious readmission score	Derivation data set: 0.696Validation data set: 0.702		1.2	100	0	8	/
				2.4	99	6	8	99
				4.7	92	28	10	98
				8	77	52	12	97
				11.8	55	73	15	95
				14.6	37	85	17	94
				17.2	21	92	19	93
				20.3	9	97	21	93
				22.2	2	100	22	92
				40	0	100	40	92
Krumholz et al37	Claims model	Derivation cohort: 0.63Validation cohort: 0.62–0.63
	Medical record model	Derivation cohort: 0.58Validation cohort: 0.59
Cardiovascular disease-related UHRs including pneumonia—heart failure only (11)
Betihavas et al38	Unnamed	0.8
Di Tano et al39	Unnamed	0.695
Huynh et al40	The non-clinical modelThe clinical modelThe combined model	0.660.720.76
Raposeiras-Roubin et al41	The GRACE risk score	0.79	p=0.83	37.9	82.5	62.8	5.6	99.1
Sudhakar et al42USA	Readmission Risk (RR) Score	All age group—0.61≥65 years—0.59Random selection—0.58		≥29	33	80	47	69
				≥24	61	52	41	71
				≥21	83	27	38	75
Fleming et al43	Unnamed	Derivation cohort: 0.69Validation cohort: 0.66
Wang et al44	LACE index (validation)			≥10
Eapen et al45		Derivation cohort: 0.59Validation cohort: 0.59
Zai et al46	The telemonitoring-based readmission model	0.21			50	81	61	72
	The psychosocial model (validation)	0.67			87	32	44	80
Au et al47	Five administrative data-based models	0.57–0.61
Watson et al48	The psychosocial readmission model	0.67
Cardiovascular disease-related UHRs including pneumonia—pneumonia only (2)
Mather et al49	Hartford Hospital model	Derivation data set: 0.71Validation data set: 0.67	p=0.96
Lindenauer et al50	Administrative claims model	0.63
	CMS medical record model	0.59
General medical condition-related UHRs (10)
Shadmi et al51	PREADM	Derivation data set: 0.70Validation data set: 0.69
Tsui et al52	Unnamed	Derivation data set: 0.819Validation data set: 0.824	p<0.05
Donzé et al53	Unnamed	0.85

Table 3

Summary of significant variables included in the predictive models for unplanned hospital readmissions (UHRs)

Reference	Model name	Admitting diagnosis	Admitting ward	Blood transfusion	BMI	Comorbidities	Complications before discharge	Daily living score	Demographic/social	Discharge disposition	Discharge hour	Environment	General anaesthesia	Health insurance	Index type of admission	Injury severity score	Laboratory tests	Length of stay	Physical examinations	Postoperative complications	Medications	Number of previous admission	Number of previous ED presentations	Overall prognosis	Procedures at index admission	Substances usage	Symptoms	Use of outpatient clinic	Vital signs
All-cause UHRs (14)
Escobar et al16	ED 30 and Discharge 30					✓				✓							✓	✓				✓
	LACE index (validation)					✓									✓			✓					✓
Yu et al17	Institution-specific prediction model	✓				✓			✓	✓					✓			✓				✓	✓
	LACE index (validation)					✓									✓			✓					✓
Baillie et al18	Prediction model																					✓
Choudhry et al12	ACC Admission and Discharge Model					✓			✓	✓		✓					✓	✓			✓	✓			✓
Gildersleeve and Cooper19	Risk of Readmission Score (RRS)					✓			✓					✓	✓			✓			✓	✓	✓
Kruse et al20	Unnamed						✓										✓	✓				✓			✓
Richmond21	Unnamed	✓				✓			✓												✓	✓		✓
Shulan et al22	Unnamed					✓			✓									✓
van Walraven et al23	LACE+ (validation)	✓				✓			✓						✓			✓				✓	✓		✓
Cotter et al13	LACE index (validation)					✓									✓			✓					✓
	Regression model																						✓
Khan et al24	Rehospitalisation Risk Score	✓				✓		✓	✓					✓								✓
Lee25	Unnamed	✓	✓												✓			✓
All-cause UHRs (14)
van Walraven et al26	CMG score	✓
	LACE (validation)
	Combined CMG and LACE	✓				✓									✓			✓					✓
van Walraven et al27	LACE+ (validation)
	Combined CMG and LACE+	✓				✓			✓						✓			✓				✓	✓		✓
Cardiovascular disease-related UHRs including pneumonia (11)
Hebert et al15	CHF model					✓												✓			✓	✓	✓			✓
	PNA model					✓											✓				✓	✓			✓
	AMI model					✓			✓								✓				✓	✓
	Combined model					✓			✓								✓	✓			✓	✓	✓
Iannuzzi et al28	Vascular surgery readmission risk score					✓			✓	✓					✓						✓				✓
Keyhani et al29	CMS-based Model					✓			✓
	CMS-based Model plus social risk factors					✓			✓																	✓
	CMS-based model plus social risk and clinical factors	✓				✓		✓	✓																	✓
Rana et al30	EMR Model	✓				✓			✓									✓
Shahian et al31	Unnamed					✓			✓
Shams et al32	PARs					✓			✓						✓			✓				✓
	CMS endorsed model (validation)
Sharif et al33	Unnamed					✓			✓									✓			✓							✓
Lucas et al34	Complex all-variable model				✓		✓	✓	✓	✓					✓		✓	✓			✓				✓
	Parsimonious readmission score																	✓							✓
Wallmann et al35	Unnamed					✓			✓													✓			✓
Wasfy et al36	Risk score after PCI (parsimonious)					✓			✓					✓	✓			✓							✓
Krumholz et al37	Claims model (administrative)					✓			✓
	Medical record model					✓			✓								✓		✓								✓		✓
Cardiovascular disease-related UHRs including pneumonia—heart failure only (11)
Betihavas et al38	Unnamed					✓			✓
Di Tano et al39	Unnamed																	✓			✓						✓
Huynh et al40	Non-clinical model					✓			✓		✓							✓
	Clinical model																✓		✓		✓								✓
	Combined model								✓		✓						✓		✓		✓								✓
Raposeiras-Roubin et al41	The GRACE Risk Score					✓			✓								✓								✓				✓
Sudhakar et al42USA	Readmission Risk Score					✓			✓								✓		✓						✓				✓
Fleming et al43	Unnamed								✓		✓						✓					✓
Wang et al44	LACE index (validation)
Eapen et al45	Unnamed					✓			✓					✓	✓			✓			✓	✓	✓
Zai et al46	The telemonitoring based readmission model																				✓						✓
	The psychosocial readmission model (validation)
Au et al47	Charlson (validation)					✓			✓
	CMS Krumholz (validation)
	Keenan (validation)					✓			✓								✓
	LACE (validation)
	LACE+ (validation)
Watson et al48	The psychosocial readmission model					✓																						✓
Cardiovascular disease-related UHRs including pneumonia—pneumonia only (2)
Mather et al49	Hartford Hospital Model					✓	✓		✓								✓					✓
	CMS Model (validation)
Lindenauer et al50	Claims model (administrative)					✓			✓																	✓
	Medical record model					✓			✓								✓				✓					✓			✓
General medical condition UHRs (10)
Shadmi et al51	PREADM				✓	✓			✓									✓				✓						✓
Tsui et al52	Unnamed					✓			✓													✓	✓					✓
Donzé et al (2014)53	Unnamed	✓				✓															✓	✓
He et al54	Unnamed																✓					✓			✓
Taha et al55	Readmission Risk Score (RRS)	✓				✓			✓												✓	✓
Donzé et al (2013)14	HOSPITAL score														✓		✓	✓				✓			✓
Tan et al56	LACE index (validation)
Billings et al11	PARR-30					✓			✓						✓								✓
Zapatero et al57	SEMI INDEX					✓			✓									✓
Gruneir et al58	LACE index (validation)
Medical condition UHRs—cirrhosis only (2)
Singal et al59	Unnamed								✓					✓			✓					✓		✓
Volk et al60	Cirrhosis readmission prediction model																✓				✓			✓
Medical condition UHRs—chronic kidney disease (1)
Perkins et al61	Unnamed					✓				✓					✓		✓	✓			✓							✓	✓
Medical condition UHRs—HIV (1)
Nijhawan et al62	Unnamed	✓													✓		✓					✓	✓
Medical condition UHRs—acute pancreatitis (1)
Whitlock et al63	Unnamed					✓															✓						✓
Surgical condition UHRs (6)
Taber et al64	30DRA with fixed variable					✓			✓											✓
	30DRA with fixed variable and dynamic clinical data					✓			✓											✓									✓
Lawson et al65	Unnamed	✓			✓	✓		✓	✓											✓					✓	✓
Iannuzzi et al66	Endocrine surgery Readmission Risk Score	✓				✓							✓				✓	✓
Mesko et al67	Unnamed			✓		✓				✓			✓				✓	✓
Moore et al68	Unnamed					✓			✓							✓						✓					✓
Graboyes et al69	Unnamed					✓	✓			✓															✓	✓
Mental health condition UHRs (1)
Vigod et al70	READMIT	✓				✓									✓			✓				✓	✓					✓

BMI, body mass index; ED, emergency department.

Quality appraisal

Six domains of potential bias72 were used to appraise the quality of included studies critically. The assessment of risk for bias was completed by two independent reviewers (HZ and SD). The ratings of ‘yes’, ‘partly’, ‘no’ or ‘unsure’ were given to each domain and then an overall risk of ‘low’ or ‘high’ was assigned to each study.

The six domains are:

Study participation: ‘Was source population clearly defined?’ and ‘Was the study population described?’ or ‘Did the study population represent source population or population of interest?’

Study attrition: ‘Was completeness of follow-up described and adequate?’

Prognostic factor measurement: ‘Did prognostic factors measure appropriately?’

Outcome measurement: ‘Was outcome defined and measured appropriately?’

Confounding measurement and account: ‘Were confounders defined and measured?’

Analysis: ‘Was analysis described and appropriate?’ and ‘Did analysis provide sufficient presentation of data?’

Data synthesis

Pooling of quantitative data was not possible as the included studies were not homogeneous. Therefore, the included studies were qualitatively synthesised and presented in narrative form.

Results

Literature search result

The initial electronic database search produced 7310 records. After removal of 1798 duplicates, a total of 5512 references of potential relevance to this systematic review remained. Titles and abstracts were then appraised and excluded 5333 records due to irrelevance. Of the remaining 179 relevant references, 98 were excluded as they were conference abstracts. A total of 81 references were reviewed as full text and a further 21 were excluded against selection criteria. A total of 18 of the 21 excluded studies developed and/or validated risk predictive models for the 48-hour73 or 72-hour74 intensive care unit readmissions or the 3-month to 1-year unplanned hospital readmissions.75–90 One study focused on participants who were discharged to a hospital in the home–hospital programme receiving intravenous antibiotics.91 The other study,92 which had been included in the previous systematic review,10 was also excluded. It was also found that the same result was published in two articles;32 therefore, the later year article32 was excluded. A hand search of reference list of the remaining 60 articles was also conducted and no additional studies were identified. Finally, a total of 60 studies were included in this systematic review. Figure 1 is a flow chart as per the Preferred Reporting Items for Systematic Reviews and Meta-Analyses of the screening process of the database search results. The overall risk of bias of the 60 studies was low when evaluated against the six domains of potential bias. All studies described the population of interest adequately for key characteristics, the response rate information was clearly stated, adequate proportion of the study population had complete data on all independent variables, the outcome variable readmission was measured with sufficient accuracy and the method of statistical analysis was appropriate for the design of the study.72

Figure 1

Flow chart for the search and study selection process (PRISMA). PRISMA, preferred reporting items for systematic reviews and meta-analyses.

Study characteristics

Table 1 summarises the characteristics of the final included studies of this systematic review. The 60 studies were conducted in several countries: USA (n=41), Canada (n=7), Australia (n=3), Spain (n=3), and one from Hong Kong, Korea, Israel, Italy, Singapore and the UK. Of the included studies, the majority employed retrospective data except two. One study18 used retrospective and prospective data and the other39 collected prospective data. Fifty-seven included studies accessed healthcare data of either tertiary hospital, centralised or national health information databases. The remaining three studies used community hospital data.19 44 54 The duration of retrieved data source ranged from 1 single day across 10 hospitals24 to 10 years47 of four healthcare databases. All included studies were based on adult patients’ (aged ≥18 years) healthcare data and the mean age, if reported, ranged from 43 to 85 years.

The 60 included studies reported unique 73 predictive models for 28-day or 30-day unplanned hospital readmissions. A total of 68 of the unique 73 predictive models were developed between 2011and 2015 and 5 were existing models, which were further validated or applied to compare with other developed/existing models. The model utilisation outcome included all-cause admissions (14 studies),12 13 16–27 cardiovascular-related disease including pneumonia (24 studies,15 28–50 of which 11 studies focused on heart failure only), medical/internal medicine conditions (15 studies),11 14 51–63 surgical conditions (6 studies)64–69 and mental health conditions.70 A total of 17 models were based on administrative data and the remaining models were derived or validated using administrative and/or clinical/medical records data. The sample size varied from 100 patients46 to nearly a million57 patients. The unplanned hospital readmission rate ranged from 2.8%66 (n=34 046) to 38%46 (n=100).

Performance of predictive models for 28-day or 30-day unplanned hospital readmissions

Table 2 displays the measures of all included predictive models. Multivariable logistic regression model was used in all included studies. In logistic regression, the outcome variable is the log of the odds of the event (probability of readmission/(1−probability of readmission)). Once the final model is determined, the multivariable logistic regression allows for the calculation of probability of readmission for cohort studies. The predicted probabilities of the final multivariable logistic model are also used for computing the receiver operating characteristic (ROC) curve and the calculation of the ROC, a measure of model discrimination.

Overall, 56 of the 60 included studies reported model discriminative ability (C-statistic), ranging from 0.2146 to 0.88.63 The area under curve for validation studies ranged from 0.5330 to 0.83,63 being slightly lower than those for the derivation study, 0.2146 to 0.88.63 For all-cause unplanned hospital readmission models, the C-statistic was reported by 14 studies ranging from 0.5513 to 0.80.22 Among 16 developed models and 2 existing models, 8 new models and 2 existing models had a C-statistic value >0.70.12 16 17 19 22 23 26 27

Regarding cardiovascular disease-related readmissions (24 studies), the C-statistic ranged from 0.2146 to 0.83632 across 32 developed models and 5 existing models. Of those, only nine developed models had a C-statistic value >0.70.30 32 34 35 38 40 41 49 50 In particular, 13 of the 17 models (12 developed and 5 existing) from 11 studies with the special focus on heart failure-related readmissions were presented with C-statistic <0.70.39 40 42–48 For surgical-related readmissions (6 studies), the C-statistic ranged from 0.5967 to 0.8569 among 7 developed models. Three of the seven models showed moderate-to-high discrimination ability.64 65 69 Patients with heart failure in the telemonitoring program were less likely to be admitted, with the reported C-statistic being 0.21.46 This indicates that the telemonitoring program was effective in identifying and intervening in patients who were reporting symptoms and thus reduced the likelihood of readmission.

However, 10 of 13 developed models and 1 existing model for medical condition-related readmissions (15 studies) were found to have consistent moderate discrimination ability. Four developed models also demonstrated high discrimination ability with C-statistic exceeding 0.80.53 52 57 63

This updated systematic review also identified one study on mental health condition-related unplanned hospital readmission. A predictive model, READMIT <(R) Repeat admissions; (E) Emergent admissions; (D) Diagnoses, and unplanned Discharge; (M) Medical comorbidity; (I) prior service use Intensity; and (T) Time in hospital>, was derived and validated using a 3-year Canadian National Health Database with a C-statistic of 0.63.

One existing predictive model, the LACE index, although validated by eight studies, demonstrated inconsistent model performance. The LACE index was first developed by van Walraven et al93 in 2010 to predict the risk of unplanned readmission or death within 30 days after hospital discharge in medical and surgical patients. The model was derived and validated based on administrative data with a C-statistic of 0.684. The model includes the length of hospitalisation stay (L), acuity of the admission (A), comorbidities of patients (C) and number of emergency department visits in the 6 months before admission (E). Five studies validated the LACE index model using healthcare data of Canada, Singapore, the UK and the USA to predict all-cause readmission (4),13 16 17 26 heart failure readmission (1)44 and general medical condition-related readmission (2).58 56 The discriminative ability of the model (C-statistic), reported by six studies, varied from 0.51 to 0.72.13 16 17 26 56 58

An extension of the LACE index to predict early death or all-cause 30-day urgent hospital readmission was further derived using administrative healthcare data and named as LACE+ index by van Walraven et al27 in 2012. The LACE+ index, in addition to four predictive variables, included patient age and sex, teaching status of the discharging hospital, acute diagnoses and procedures performed during the index admission, number of days on alternative level of care during the index admission and number of elective and urgent admissions to hospital in the year before the index admission. The LACE+ index had a C-statistic of 0.771, which exceeded the performance of LACE index. The LACE+ index was further validated by two large Canadian retrospective studies. The performance of the model was 0.6147 for patients with heart failure and 0.7323 for patients with all-cause hospital readmissions.

A Canadian study compared the performance of different models within the same population for 30-day readmission or death due to heart failure. A total of 59 652 patients' admission information was retrieved from four health databases over a 10-year period. Five models were examined in the study,47 namely Charlson, CMS Krumholz, Keenan, LACE index and LACE+. The five models had the C-statistic of 0.57–0.61. In terms of types of data sources used to develop or validate the 73 unique predictive models, administrative healthcare data were used for 17 models but were found/identified with inconsistent discriminative ability. A total of 13 of the 17 models reported C-statistic between 0.55 and 0.7, and the remaining four models reported C-statistic between 0.7 and 0.876. Similarly, the performance of the remaining 56 models using clinical/medical data varied between 0.21 and 0.88 (C-statistic).

Only two models32 53 were developed targeting the potentially avoidable/preventable unplanned hospital readmissions. The outcome measure of the models focused on the end-of-life patients53 and pneumonia, heart failure, acute myocardial infarction and chronic obstructive pulmonary disease.32 Both models had C-statistic >0.8 (0.85 and 0.83, respectively).

Sensitivity and specificity were calculated by 16 of the 60 included studies. The sensitivity of the predictive model ranged from 5.4% (PARR-30 model, Patients at Risk of Re-admission within 30 days)11 to 91.95% (potentially avoidable readmission (PAR) model),32 while specificity values were between 22% (Rehospitalisation Risk Score)24 and 99.5% (PARR-30 model).11

A total of 14 of the 60 included studies reported the PPV (5.641–86.61%32) and NPV (19.161–99.1%41) of the readmission risk predictive model. Similarly, only 17 studies calibrated the developed predictive models and mostly presented as p value, except one study68 that reported the model calibration as the value of intercept and slope.

Predictive risk of readmission was assessed in all included studies, but only 14 of the included 60 studies specified thresholds for risk categories. Thresholds ranged from 4%35 to 80%.61

Key variables included in the readmission risk predictive model

A total of 28 types of significant variables were extracted from the 73 unique predictive models for unplanned hospital readmissions as shown in table 3. Overall, the top 10 significant variables included in the 73 risk predictive models are comorbidities (n=54), demographic/social (n=45), length of stay (n=29), number of previous admissions (n=29), laboratory tests (n=25), medications (n=21), index type of admission (n=17), procedures at index admission (n=16), admitting diagnosis (n=14) and number of previous emergency department presentations (n=14) (refer to figure 2). The key demographic/social variables consisted of age (n=26), gender (n=25), living arrangement (n=12), race (n=8) and marital status (n=6).

Figure 2

Pareto chart of significant variables included in the predictive models. BMI, body mass index; ED, emergency department.

The variables ‘comorbidities’, ‘length of stay’ and ‘number of previous admissions’ remained as the most frequently cited predictive risk variables against all utilisation outcomes. However, the variables ‘laboratory tests’ and ‘medication’ were more commonly included in the predictive models for cardiovascular disease-related and medical condition-related unplanned hospital readmissions compared with all-cause, mental health and surgical condition-related unplanned hospital readmissions.

Discussion

A total of 60 studies with 73 unique risk predictive models for 28-day or 30-day unplanned hospital readmissions were included in this systematic review. The discrimination ability (C-statistic) of the 73 models varied largely from 0.21 to 0.88. Inconsistent performances were found among models for all-cause readmission, cardiovascular disease-related readmission and surgical-related readmission. However, most of the predictive models for the general medical condition-related readmission exceeded C-statistic of 0.7. In comparison, Kansagara et al10 included 26 models with the focus of adult medical patients only. A total of 13 predictive models measured 30-day readmissions; of these, 10 models performed poorly and only 3 models reported C-statistic >0.70. The outcome measures of the other 13 models ranged from 41-day to 4-year unplanned hospital readmission; as a result of the vast difference in the time frame, the C-statistic also varied from 0.53 to 0.75.

This updated systematic review has certain limitations. The studies included in this systematic review were limited compared with studies that were published in English with full-text access. The outcomes of the predictive models included in this systematic review were also restricted to 28-day or 30-day unplanned hospital readmission. A meta-analysis is not permitted in this systematic review as the included studies were heterogeneous due to diversity of cohort of population, duration of retrieved data source, sample sizes and geographical locations. It was noted that the sample size was reported in different units, that is, (index) admission/hospitalisation, cases, patients or discharges, as shown in table 1. The lack of standardised calculation could also contribute to the broad range of readmission rates (2.8–38%); thus, the results were not comparable. This systematic review also found the sample size is not associated with the model predictive ability. Of the included 73 unique models, Zai et al46 derived a model based on the selected 100 readmitted patients with heart failure and scored the lowest C-statistic of 0.21. In contrast, Whitlock et al63 retrieved around 200 readmitted patients with acute pancreatitis and developed a model with the highest discrimination ability (C-statistic=0.88).

There has been increased recognition that some unplanned hospital readmissions are associated with the diagnosis of the initial hospitalisation and could be potentially prevented or avoided through systematic discharge process. In 2006, a Swiss study94 compared three models (non-clinical model, Charlson-based model and SQLape model, A patient classification system, also designed to adjust for costs and other outcomes) to identify potentially preventable readmission risk on over 60 000 medical patients. The C-statistics of the three models were 0.67, 0.69 and 0.72, respectively, which indicated poor-to-reasonable discrimination ability. In contrast, this systematic review identified two high-performance models32 53 for potentially avoidable/preventable readmissions with C-statistic >0.8. The PAR model32 was also high in other predictive model performance indicators, such as sensitivity (91.95%), specificity (97.65%), PPV (86.61%) and NPV (98.65%). However, the two models were developed based on comparatively smaller sample size of 560032 and 10 27553 using American healthcare data collected over a 12-month period. Overall, the number of potentially preventable readmissions remains unclear due to lack of standardised identification process.95–98

Compared with the previous systematic review,10 there were more studies in this review using clinical medical record data to develop disease-specific predictive models. However, the debate whether a predictive model should be developed using administrative data or clinical/medical records data remains inconclusive. Three key variables extracted from the 73 unique models, ‘comorbidity’, ‘length of stay’ and ‘previous admissions’, were based on administrative data and were consistent with the findings of a previous systematic review.10 The latest evidence has shown that variables based on clinical medical data, that is, ‘laboratory tests’ and ‘medications’, were also valued in models for predicting cardiovascular-related and medical condition-related readmissions. Of note, ineffective communication in transitions of care is reported as a major contributing factor to adverse events that directly risk patient safety.99 100 Poor communication at discharge also leads to preventable unplanned readmissions and frequent problems with the continuity of medication management.101–103 None of the examined 73 models cited the comprehensiveness of discharge information as a predictor to unplanned hospital readmissions.

All included studies in this systematic review were based on adult population. To date, only two paediatric predictive models were identified and both were based on American paediatric populations. One retrospective multicentre study104 retrieved 12-month administrative data from 38 children's hospitals. A model was developed and internally validated with a high discrimination ability (C-statistic=0.81). However, the model outcome measure was 12-month all-cause readmissions. In comparison, a 30-day hospital readmission model105 was developed based on 5376 paediatric patients following plastic surgery procedures. The study accessed prospective medical records, and the model had moderate discrimination ability of C-statistic 0.784.

The performance of the 73 unique predictive models in this review was assessed using a variety of statistical measures. Inconsistency of reported statistical measures was noted in the included 60 studies, of which 2 studies44 58 reported threshold as the only model performance measurement. A US framework for assessing the performance of predictive models106 argued the importance of reporting discrimination and calibration for a risk predictive model. In all included 60 studies, the most reported measure of the risk predictive model is the ROC (C-statistic). The interpretation of the risk predictive model discriminative ability (C-statistic) was inconsistent. For instance, a study47 examined five predictive models and concluded that the models had moderate discrimination ability based on the C-statistic of 0.57–0.6; whereas models are typically considered reasonable when the C-statistic is higher than 0.7 by Hosmer and Lemeshow.71

Conclusion

The risk predictive models which focused on general medical conditions in relation to unplanned hospital readmissions reported moderate discriminative ability. Two models32 53 for potentially preventable/avoidable readmissions showed high discriminative ability. This systematic review, however, found inconsistent performance across the included unique 73 risk predictive models for unplanned hospital readmissions.

The variables ‘comorbidities’, ‘length of stay’ and ‘previous admissions’ were frequently cited across the examined unique 73 models, and ‘laboratory tests’ and ‘medication’ variables had more weight in the models for cardiovascular disease and medical conditions in relation to readmissions. However, comprehensiveness of discharge information was not included in any of the examined models.

This review highlighted the need for rigorous validation of the risk predictive models with moderate-to-high discriminative ability be undertaken, especially the two models32 53 for the potentially avoidable hospital readmissions. There is a need to review and update predictive models. Specifically this is essential for paediatric 28-day all-cause unplanned hospital readmissions as limited evidence was found.

Findings from this updated systematic review revealed an increasing number of developed risk predictive models for specific disease-related unplanned hospital readmission using clinical/medical records data. Findings from this systematic review also confirm the limited applicability of hospital readmission risk predictive models. The performance of the applied existing models was inconsistent. It is, therefore, essential to clearly define utilisation outcomes and the type of accessible data sources prior to determining which risk predictive model to use. For example, most of the models were developed based on healthcare data from the USA, which might not be applicable to patients from other settings.

Table 2

Continued

Reference	Model name		Discrimination (ROC)			Calibration (H&L)	Threshold (%)	Sensitivity (%)	Specificity (%)	PPV (%)	NPV (%)
General medical condition-related UHRs (10)
He et al54	Unnamed	Medical patient	Validation
			Within site	CV on JHH	0.75		21	50	84	29	93
				CV on BMC	0.79		30	50	88	28	95
			Across site	Test on BMC	0.81		9	47	88	27	95%
				Test on JHH	0.78		30	58	76	24	93
		CP	Within site	CV on JHH	0.71		21	50	68	34	84
				CV on BMC	0.65		30	56	79	20	955
			Across site	Test on BMC	0.65		9	85	41	11	97
				Test on JHH	0.73		30	60	71	27	91
Taha et al55	Readmission Risk Score						12			18	95
							16			18	90
							20			20	89
							24			21	86
							28			28	85
							32			38	84
Donzé et al (2013)14	HOSPITAL score		Derivation data set: 0.69Validation data set: 0.71			Derivation data set: p=0.28Validation data set: p=0.15	5.2–18.4
Tan et al56	LACE index (validation)		0.70			13.1 (p=0.107)	16
Billings et al11	PARR-30		0.70				50	5.4	99.5	59.2
Zapatero et al57	SEMI INDEX		0.876			Derivation cohortp=0.247 (≤50 years group)p=0.1 (51–70 years group)p=0.182 (71–90 years group)p=0.227 (>90 years group)Validation cohortp=0.350 (≤50 years group)p=0.1 (51–70 years group)p=0.246 (71–90 years group)p=0.617 (>90 years group)	7.4–22
Gruneir et al58	LACE index (validation)						16

Table 2

Continued

Reference	Model name	Discrimination (ROC)	Calibration (H&L)	Threshold (%)	Sensitivity (%)	Specificity (%)	PPV (%)	NPV (%)
Medical condition UHRs—cirrhosis only (2)
Singal et al59	Unnamed	Derivation cohort: 0.68Validation cohort: 0.66
Volk et al60	Cirrhosis readmission prediction model	0.65
Medical condition UHRs—chronic kidney disease only (1)
Perkins et al61	Unnamed	0.792		20	69	73.4	38.3	90.9
				50	28.5	97.1	70.2	85
				80	1.7	99.8	66.7	19.1
Medical condition UHRs—HIV only (1)
Nijhawan et al62	Unnamed	Derivation: 0.72Validation: 0.70
Medical condition UHRs—acute pancreatitis (1)
Whitlock et al63	Unnamed	Derivation cohort: 0.88Validation cohort: 0.83
Surgical condition-related UHRs (6)
Taber et al64USA	30DRA with fixed variable	0.63	p=0.061	10			57.7	63.8
	30DRA with fixed variable and dynamic clinical data	0.731	p=0.603	10			62.8	73.3
Lawson et al65	Unnamed	0.728
Iannuzzi et al66	Endocrine surgery Readmission risk score	Derivation cohort: 0.676	p=0.083
		Validation cohort: 0.646	p=0.592
Mesko et al67	Unnamed	Derivation data set: 0.59Validation data set: 0.59
Moore et al68	Unnamed	0.651	Intercept, slope 0.000370; 1.0001
Graboyes et al69	Unnamed	0.85
Mental health condition-related UHRs (1)
Vigod et al70	READMIT	Derivation data set: 0.631Validation data set: 0.63	p=0.868

NPV, negative predictive value; PPV, positive predictive value; ROC, receiver operating characteristic.