External validation of the PAR-Risk Score to assess potentially avoidable hospital readmission risk in internal medicine patients.

BACKGROUND: Readmission prediction models have been developed and validated for targeted in-hospital preventive interventions. We aimed to externally validate the Potentially Avoidable Readmission-Risk Score (PAR-Risk Score), a 12-items prediction model for internal medicine patients with a convenient scoring system, for our local patient cohort.
METHODS: A cohort study using electronic health record data from the internal medicine ward of a Swiss tertiary teaching hospital was conducted. The individual PAR-Risk Score values were calculated for each patient. Univariable logistic regression was used to predict potentially avoidable readmissions (PARs), as identified by the SQLape algorithm. For additional analyses, patients were stratified into low, medium, and high risk according to tertiles based on the PAR-Risk Score. Statistical associations between predictor variables and PAR as outcome were assessed using both univariable and multivariable logistic regression.
RESULTS: The final dataset consisted of 5,985 patients. Of these, 340 patients (5.7%) experienced a PAR. The overall PAR-Risk Score showed rather poor discriminatory power (C statistic 0.605, 95%-CI 0.575-0.635). When using stratified groups (low, medium, high), patients in the high-risk group were at statistically significant higher odds (OR 2.63, 95%-CI 1.33-5.18) of being readmitted within 30 days compared to low risk patients. Multivariable logistic regression identified previous admission within six months, anaemia, heart failure, and opioids to be significantly associated with PAR in this patient cohort.
CONCLUSION: This external validation showed a limited overall performance of the PAR-Risk Score, although higher scores were associated with an increased risk for PAR and patients in the high-risk group were at significantly higher odds of being readmitted within 30 days. This study highlights the importance of externally validating prediction models.

Introduction

Potentially avoidable readmissions (PAR) are unforeseen readmissions related to a previously known affliction occurring within a specified time interval [1]. PAR-rates vary between 5 to 79% and are increasingly being used as benchmarks for quality of care, hospital outcomes, and cost reduction measures [2,3]. For Switzerland, the rates vary between 3.8% to 5.6% and generally depend on the level of care the hospital provides [4]. Of the many reasons for hospital readmissions investigated, adverse drug events have been shown to account for 13% of 30-day readmissions to an academic hospital in the US. Of these, 93% were classified as preventable and 49% were caused by inappropriate prescribing [5,6]. Interventions to reduce readmission rates have been explored by focussing on improved discharge planning and reducing adverse drug events, including patient education, telephone follow-up, home visits, and transition coaching [5]. However, no multicomponent intervention program has yet brought consistent evidence to sufficiently reduce readmission rates [5]. Furthermore, they are reported to be time consuming and expensive [7].

To address these issues, readmission prediction models have been developed and validated [7]. These models stratify patients according to their risk of readmission using readily available electronic health care information to calculate a risk score early in the hospitalisation in order to target interventions [8,9]. A recent systematic review on prediction models by Mahmoudi et al. identified 41 studies reporting on prediction models. Of these, 17 predict the risk of readmission on all inpatients while the rest of the models focus on a specific patient group [7]. Only eight studies reported sensitivity and specificity, implementation in the electronic medical record system seemed rare, and no model had been externally validated [10].

In Switzerland, the national Striving for Quality Level and Analyzing of Patient Expenses (SQLape) software [11] is being used to identify PAR within 30 days after discharge. The underlying screening algorithm identifies unplanned readmissions to the same hospital that are related to the initial diagnosis and occur within 30 days of hospital discharge with a specificity and sensitivity of 96% [1]. However, the screening algorithm of SQLape only works in retrospect and, hence, cannot be used for targeted preventive interventions. In 2019, Blanc et al. [12] published the internally validated Potentially Avoidable Readmissions Risk Score (PAR-Risk Score), developed with a dataset of one tertiary university teaching hospital and one regional hospital. The PAR-Risk Score is a 12-items prediction model, where the weights of the regression coefficients were transposed to a simpler scoring system. The internal validation showed a C-statistic of 0.688 (95% CI 0.655 to 0.72), which is close to the models reviewed by Mahmoudi et al. [7] and the already externally validated Swiss HOSPITAL score [13,14].

In this study, we aimed to provide the first external validation of the PAR-Risk Score using data from an older internal medicine patient cohort of a Swiss tertiary teaching hospital.

Methods

Study design and participants

This cohort study used data from hospitalisations of a 360-bed tertiary teaching hospital in Baden, Switzerland [15] between December 2016 and November 2018. As in the development study, we focused on internal medicine patients. The data were derived from routinely collected electronic health records (EHR). The study included patients hospitalised for at least 48 hours and aged 65 years and older, as selected for another study [16]. These dataset characteristics had not been applied to the patient sample in the development study. We applied the exclusion criteria according to the development phase of the model [12], namely: death before discharge, transfer to another hospital, and non-Swiss residents. The study design is visualised in Fig 1.

Fig 1

Cohort design of the external validation of the PAR Risk Score.

We report this study in concordance with the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement [17].

Outcome

The outcome of interest was a 30-day potentially avoidable hospital readmission (PAR), as identified by the SQLape algorithm.

Predictors

The PAR Risk Score assigns points to the following predictors: length of stay longer than four days, admission in previous six months, anaemia, hypertension, hyperkalaemia, opioid prescription during hospital stay, comorbidities such as heart failure, acute myocardial infarction, chronic ischemic heart disease, diabetes with organ damage, cancer, and metastatic carcinoma. The exact scoring system with the individual weights is presented in S1 Table. Comorbidities were defined using International Classification of Disease 10 (ICD-10) codes, as extracted from the EHR, and categorised according to the supplement information of the original publication [12]. For the comorbidity predictor anaemia only ICD-10 codes were used, because haemoglobin values were not available in this dataset. Opioid intake was defined as opioid drugs dispensed during the period of hospitalisation, as identified by the ATC-code N02A and its sub-levels. Medication prescription data was mapped in a semi-automated process to identify non-standardized free-text entries as well. Hyperkalaemia was defined as serum potassium level of >5.5 mmol/L within the last seven days of the hospitalisation.

Missing data

Within individual patients, the most recent serum potassium levels were carried forward. When there was no value available at all, we assumed that those were not missing at random (i.e., the patient had normal serum potassium levels). A sensitivity analysis was performed by setting the hyperkalaemia variable for all patients with missing values to 1. We excluded patients with missing information on dispensed drugs after having performed a sensitivity analysis, as knowledge of opioid intake is needed for the PAR Risk Score. The sensitivity analysis on the 94 excluded patients was performed by setting the opioid variable for all these patients to either 0 or 1.

Statistical analyses

Patient characteristics of the cohort were reported as number and frequencies. We calculated the raw PAR-Risk Score values for each patient as the sum of the predictor variables present at time of discharge (S1 Table). The raw PAR-Risk Score values were used in a univariable logistic regression to predict PARs within 30 days, as labelled as PAR case by the SQLape algorithm. We calculated C statistics, Brier score, and the ‘le Cessie—van Houwelingen—Copas—Hosmer unweighted sum of squares test for global goodness of fit’ as performance statistics of the univariable logistic regression. Additionally, we visualised the calibration by plotting the observed proportion at risk per PAR Risk Score point versus the predicted risk for each point weighted by the respective number of participants.

To compare the influence of a single predictor variable on the outcome to those of the original study, we analysed the unadjusted association between the single predictor variable and the outcome PAR using univariable logistic regression. Analogously, a multivariable logistic regression was performed to assess the independent association between individual predictors and the outcome [18].

We additionally categorised the patients into the three risk groups: low, medium, and high risk based on the raw PAR-Risk Score values using the original threshold levels of <3, 3–10, and >10, respectively. This grouping was redone with adapted threshold levels, which were re-calculated by grouping the patients into tertiles based on the raw PAR-Risk Score values of our patients, analogously to the development phase. Using the original as well as the adapted threshold levels, we calculated the observed proportion at risk, the predicted risk (S4 Table), and odds ratios (ORs) for PAR. We calculated sensitivity, specificity, positive predictive values, and negative predictive values with which the model classifies patients into the different risk groups by comparing each group to the low risk group.

All analyses were performed in R 3.6.1 [19] with the additional packages: tidyverse [20], lubridate [21], rms [22], pROC [23], and caret [24]. The p-values calculated in this report assume a significance level of .05.

Ethics approval

The Swiss ethics committee approved the protocol for the study for which the data were originally extracted (EKNZ Project ID: 2018–01000). The committee also approved the amendment for the study presented here. The data were extracted anonymously, informed patient consent was not required.

Results

Out of 8,252 patients hospitalised between December 2016 to November 2018, we included 5,985 patients in our study by applying the defined exclusion criteria (Fig 2). Of the eligible patients, 340 patients (5.7%) were identified as having experienced a PAR by the SQLape software, whereas it was 562 (7.7%) in the derivation patient cohort [12]. Patient characteristics are depicted in Table 1. The mean age of the patients was 79.7 (± 7.7) years with a mean of 16.9 (± 7.8) number of drugs of and a mean length of stay of 8.8 (± 6.5) days. The frequency of each predictor of PAR and non-PAR patients is shown in S1 Fig. Due to missing data, we set the PAR-Risk Score variable hyperkalaemia to 0 for 432 patients (7.2%). The 94 (1.5%) patients who were excluded due to missing data on dispensed drugs showed an underrepresentation of PAR cases with just one PAR case.

Fig 2

Dataset generation with applied exclusion criteria.

Table 1

Patient characteristics of the cohort.

Patient characteristics at hospital discharge	n	%	Blanc et al. n (%)
Total number of patients	5,985	-	7,317 (-)
SQLape defined PAR cases	340	5.7	562 (7.7)
Age
65–75 years	4,071	68.0	1,555 (21.3)
≥76 years	1,914	32.0	2,896 (39.6)
Male sex	2,808	46.9	3,993 (54.6)
Length of hospital stay:
≤4 days	1,570	26.2	2,259 (30.9)
>4 days	4,415	73.8	5,058 (69.1)
Admission in previous 6 months	1,360	22.7	2,041 (27.9)
Opioids*	1,589	26.5	1,795 (24.5)
Number of drugs dispensed
<5	115	1.9	1,671 (22.8)
6 to 10	4,687	78.3	2,469 (33.7)
>10	1,183	19.8	3,177 (43.3)
Hyperkalaemia (K+ >5.5 mmol/L)*	16	0.3	685 (9)
Comorbidity
Acute myocardial infarction	275	4.6	1,048 (14.3)
Acute respiratory disease	981	16.4	1,260 (17.2)
AIDS	0	0.0	25 (0.3)
Anaemia	1,235	20.6	2,138 (29.2)
Arrhythmia	2,131	35.6	1,342 (18.3)
Cancer	629	10.5	762 (10.4)
Metastatic carcinoma	408	6.8	280 (3.8)
Cerebrovascular disease	917	15.3	268 (3.7)
COPD/asthma	776	13.0	1,043 (14.3)
Chronic ischemic heart disease	1,512	25.3	497 (6.8)
Cognitive troubles/dementia	741	12.4	201 (2.8)
Connective tissue disease	90	1.5	64 (0.9)
Diabetes with organ damage	467	7.8	152 (2.1)
Gastrointestinal ulcer	137	2.3	100 (1.4)
Hepatic cirrhosis	69	1.2	276 (3.8)
Heart failure	1,361	22.7	1,314 (18.0)
Hypertension	4,165	69.6	1,723 (23.6)
Infectious disease (except pneumonia and sepsis)	1,733	29.0	1,655 (22.6)
Intoxication or adverse drug reactions	87	1.5	918 (12.6)
Mental and behavioural disorders due to alcohol	238	4.0	639 (8.7)
Paraplegia/hemiplegia	197	3.3	84 (1.2)
Peripheral vascular disease	543	9.1	186 (2–5)
Pneumonia	619	10.3	1,353 (18.5)
Renal failure	1,461	24.4	1,678 (22.9)
Sepsis	189	3.2	542 (7.4)

Notes: SQLape = Striving for Quality Level and Analyzing of Patient Expenses software [11]; PAR = Potentially avoidable readmissions; AIDS = Acquired immune deficiency syndrome; COPD = Chronic obstructive pulmonary disease.

* = variables with missing values.

Model specification

The distribution of the raw PAR-Risk Score values across SQLape-defined PAR versus non-PAR patients is presented in the supplement (S2 Fig). Unadjusted and adjusted associations between each predictor variable and the outcome are provided in Table 2. Out of the 12 predictor variables only five were found to be significantly associated with PAR, as identified by SQLape: previous admission within six months, length of hospital stay, anaemia, heart failure, and opioids. Multivariable logistic regression identified the four predictor variables previous admission within six months, anaemia, heart failure, and opioids to be significantly associated with the outcome. Based on the tertiles of our patients, the adapted threshold levels for the three risk categories low, medium, and high risk were PAR-Risk Score values of <12, 12 to 25, and >25, respectively.

Table 2

Unadjusted and adjusted associations between predictor and potentially avoidable readmission (PAR).

Predictor	Non-PAR (n = 5645)		PAR (n = 340)		Univariable analysis	Multivariable analysis
	n	%	n	%	OR (95%-CI)*
Admission in previous 6 months	1,254	22.2	106	31.2	1.59 (1.25–2.01)	1.39 (1.08–1.77)
Length of hospital stay	4,146	73.4	269	79.1	1.37 (1.05–1.8)	1.08 (0.82–1.44)
Anaemia	1,134	20.1	101	29.7	1.68 (1.32–2.13)	1.45 (1.12–1.85)
Heart failure	1,256	22.2	105	30.9	1.56 (1.23–1.98)	1.41 (1.09–1.81)
Hypertension	3,919	69.4	246	72.4	1.15 (0.91–1.48)	1.1 (0.86–1.42)
Acute myocardial infarction	259	4.6	16	4.7	1.03 (0.59–1.67)	0.95 (0.53–1.59)
Chronic ischemic heart disease	1,420	25.2	92	27.1	1.1 (0.86–1.41)	1.04 (0.8–1.35)
Diabetes with organ damage	432	7.7	35	10.3	1.38 (0.95–1.96)	1.18 (0.8–1.69)
Cancer	584	10.3	45	13.2	1.32 (0.94–1.81)	1.08 (0.71–1.59)
Metastatic carcinoma	376	6.7	32	9.4	1.46 (0.98–2.09)	1.23 (0.76–1.94)
Opioids	1,466	26.0	123	36.2	1.62 (1.28–2.03)	1.4 (1.1–1.78)
Hyperkalaemia	14	0.2	2	0.6	2.38 (0.37–8.56)	1.75 (0.27–6.44)

Notes

* = The low risk group was used as reference for comparison with the medium and high risk groups. OR = Odds ratio; CI = Confidence interval.

Model performance

The results of the univariable logistic regression of the raw PAR-Risk Score value on 30-days PAR is presented in the supplement (S2 Table). The univariable regression analysis yielded a C statistic of 0.605 (95% CI 0.575–0.635). The graph of the receiver-operating curve is provided in the supplement (S3 Fig). The Brier score was 0.053, indicating decent accuracy. The calibration plot indicated a lack of fit (Fig 3), which was also supported by the goodness-of-fit test with a p-value of <0.01. A summary of the goodness-of-fit test statistic is provided in the supplement (S3 Table).

Fig 3

Calibration plot of the PAR Risk Score weighted by number of participants.

The sensitivity analysis on patients with missing serum potassium levels showed only small changes in the C statistic: 0.600 (95% CI: 0.570–0.633) by setting hyperkalaemia to one. The sensitivity analysis on the 94 excluded patients with missing content on dispensed medication showed only small changes in the C statistic as well: 0.608 (95% CI: 0.579–0.638) by setting opioids to zero, and 0.607 (95% CI 0.577–0.637) by setting opioids to one.

Table 3 shows the frequency of PAR and non-PAR patients and the association between the odds of having a PAR given the threshold categories. When comparing medium and high risk groups to low risk groups, patients were at higher odds of being hospitalised for the adapted threshold levels. For the original threshold levels, only the high risk group was significantly at higher odds of being hositalised.

Table 3

Contingency table and odds ratios describing the association between risk group and potentially avoidable readmission (PAR).

Threshold category	Thresholds	PAR (n = 340)	Non-PAR (n = 5645)	OR (95%-CI)*
Original threshold
Low	<3	9	293	1
Medium	3–10	127	2826	1.46 (0.74–2.91)
High	>10	204	2526	2.63 (1.33–5.18)
Adapted threshold
Low	<12	72	2044	1
Medium	12–25	116	1818	1.81 (1.34–2.45)
High	>25	152	1783	2.42 (1.82–3.23)

Notes

*The low risk group was used as reference for comparison with the medium and high risk groups. OR = Odds ratio; CI = Confidence interval.

Performance measures for original and adapted thresholds are presented in Table 4. Using the original thresholds, the model classified patients into the high risk group with a sensitivity of 95.8% and a specificity of 10.4% when comparing to the low risk groups. Re-calculating the thresholds generally improved specificity (high to low: 53.4%) whilst reducing sensitivity (high to low: 67.9%). The predicted probabilities for each risk group using the original and the new threshold levels are presented in Table 5. For comparison, the observed proportion at risk and predicted probabilities in the derivation and validation cohort of the original publication are presented as well. The predicted patients at risk was overpredicted, yet comparable to the observed values for the low and medium risk groups, but twice as high for the high risk group (predicted: 15.9%, observed: 7.5%).

Table 4

Performance measures with which the model classifies patients into the different risk groups.

	Low vs medium	Low vs high
Original threshold
Sensitivity (%)	93.4	95.8
Specificity (%)	9.4	10.4
Positive Predictive Value (%)	4.3	7.5
Negative Predictive Value (%)	97.0	97.0
Adapted threshold
Sensitivity (%)	61.7	67.9
Specificity (%)	52.9	53.4
Positive Predictive Value (%)	6.0	7.9
Negative Predictive Value (%)	96.6	96.6

Table 5

Observed versus predicted risk for potentially avoidable readmission (PAR).

	Risk group	Observed proportion at risk (%)	Mean predicted risk (%)
External validation
Original threshold	Low	3.0	3.3
	Medium	4.3	5.7
	High	7.5	15.9
Adapted threshold	Low	3.4	4.7
	Medium	6.0	8.0
	High	7.9	18.6
Original publication [12]
Derivation cohort	Low	2.6	3.1
	Medium	5.2	5.0
	High	12.9	13.1
Validation cohort	Low	2.4	3.5
	Medium	5.7	5.7
	High	13.1	13.9

Discussion

The aim of this study was to externally validate the PAR-Risk Score using retrospective data from a Swiss tertiary teaching hospital. Within this dataset, 340 patients (5.7%) were labelled as having experienced a PAR by the SQLape algorithm. For this dataset, we calculated the C statistic of the PAR-Risk Score to be 0.605, which was reported to be higher for the internal validation (0.687) [12]. The difference in the Brier score, indicating better accuracy in our dataset (Brier score: 0.053 versus 0.064), may be biased by the low number of patients with the outcome PAR [25]. The four predictor variables admission in previous 6 months, anaemia, heart failure, and opioids, showed significant associations with PAR in the multivariable analysis. Of these, anaemia, heart failure, and opioids showed a stronger association in our dataset than in the one used for the internal validation. When using the PAR-Risk Score to categorise the patients into three risk groups (low, medium, high) according to the original thresholds (<3, 3–10, and >10), patients in the high risk group were at statistically significant higher odds (2.63, 95% CI 1.33–5.18) of being readmitted within 30 days compared to low risk patients.

We identified that having an admission in previous 6 months, anaemia, heart failure, and opioids as predictor variables from the PAR-Risk Score that had a significant association with PAR in our dataset. These variables are also part of other hospital readmission models [26-28]. Administrative variables such as previous admission to the hospital and length of stay were found to be frequently associated with hospital readmission [7,8]. Length of stay was not significantly associated with PAR in our study, but this result may have been influenced by dichotomising the variable (i.e., length of stay longer than four days or not). Heart failure as predictor variable has been extensively studied and is often reported as a risk factor for readmission [5,7,8,27], which was again confirmed for our cohort of patients. In contrary, anaemia is less frequently reported as risk factor for readmission, but included as predictor in some studies [7,29]. Anaemia has been shown to be associated with mortality in patients with chronic heart failure [30]. Opioid use is associated with medication-related harm in elderly patients, is a well-known cause for adverse drug events, and is linked to readmission [26,31,32]. This was true for our cohort of patients as well.

Variables included in readmission risk prediction generally include some mix of medical comorbidity data and prior use of medical services [8], with the final model depending on local characteristics and dataset limitations. Illness severity, overall health and functioning, and social determinants of health are frequently disregarded [8], with Herrin et al. showing that readmissions occurring after seven days are associated with non-hospital factors such as geodemographic characteristics and community-related factors [33]. Illness severity, overall health and functioning, and social determinants may be poorly accessible from administrative hospital data. Chin et al. were able to demonstrate that the reason for readmission due to hospital-level quality rapidly declined within the first seven days after discharge, meaning that readmissions after this time period are more susceptible to geodemographic characteristics reflecting social and community-related factors. This puts the 30-day readmission rate as outcome into question [34]. In Switzerland, a readmission of 18 days is important for the hospital’s economical management. Therefore, prediction models considering non-hospital factors and reducing the readmission definition of 30 to 18 days might be more meaningful, and clinically and economically useful.

We assume that overall prediction accuracy would improve when accounting for the low incidence of PARs in the dataset during the model development stage. With the original thresholds, the model achieved a sensitivity of 95.8% and a specificity of just 10.4% to classify patients into the high rather than the low risk group. Adapting the thresholds influenced this ratio but did not markedly affect positive and negative predictive values. In regression-based prediction models, an imbalanced dataset will bias the prediction, leading to high accuracy for the majority group, while the minority group will show poor accuracy [35]. Possible techniques to account for class imbalance are basic resampling techniques (e.g., random over-/undersampling) or more sophisticated techniques such as the synthetic minority oversampling technique (SMOTE) [36].

The HOSPITAL score is another risk prediction model distinguishing low, intermediate, and high risk groups for 30-days PAR, which was derived from 10,731 discharges [13]. The model includes the predictive variables haemoglobin (<12 g/dL), discharge from an oncology service, sodium level (< 135 mEql/L), procedure during the index admission (≥1), index type of admission (emergency admission as opposed to elective), number of admissions during the last 12 months (0–1, 2–5, >5), and length of stay (≥5 days). In its external validation, comprising 117,056 patients from 9 hospitals and 4 countries, the HOSPITAL score showed a discriminatory power of 0.72 (C statistic, 95% CI 0.72–0.72) and a Brier score of 0.08 [14]. In the data subset of Switzerland, comprising 8,971 patients with 524 SQLape-identified PARs (5.8%), the score showed a C statistic of 0.68.

Limitations

Our external validation was conducted in older patients than in the development study, and had a required hospitalisation stay longer than 48 hours. This was a limitation of the available data and could have potentially influenced the model’s performance. However, the main risk group for rehospitalisation are older patients [37], and predictive models need to perform for these patients. Based on clinical considerations, we decided against using imputation techniques for missing potassium values and information on dispensed drugs. If irregularities of potassium levels were to be expected, appropriate lab work would have been ordered. To investigate the impact of our considerations, we performed a sensitivity analysis, only showing small changes in the C statistic. No documented medication dispensing is an unlikely scenario for hospitalised patients over 65 years of age, suggesting incorrect data entry. We, hence, decided on excluding these 94 patients. Sensitivity analyses again confirmed the low impact of this decision on the model’s performance. Another necessary deviation from the development phase was the unavailability of haemoglobin values in our dataset, which limited the definition of anaemia to the concerned ICD-codes [38]. This may have resulted in an underestimation of the prevalence of anaemia in our cohort as compared to the development phase (20.6% vs. 29.2%). Presence of anaemia, however, only assigns 2 points to the total PAR-Risk Score. We believe that adaptations of published prediction models to the local characteristics are frequently necessary. An additional limitation is that SQLape only identifies patients with unplanned readmissions to the same hospital. Hence, there is a possibility for misclassifying patients that went to another hospital for their readmission.

Strengths

The strengths of this study stem from the decision to attempt a replication of the development for the model’s external validation whilst simultaneously investigating the individual predictor variables rather than just the finished model. Staying as close to the development of the model as possible enabled statements about its validity and generalisability. Investigating the individual and combined impact of the predictor variables allowed for insights into the robustness of the predictors for future prediction models on PAR. This study additionally highlights the importance of externally validating prediction models by using a dataset derived from the cohort for which the model is intended to be applied to.

Conclusion

In this external validation of the PAR-Risk Score in an internal medicine patient cohort of a 360-bed hospital and a mean age of 79.7 years, the model’s overall performance was limited. Whilst higher scores were associated with an increased risk for PAR and patients in the high-risk group showed a statistically significant association with a 30-day readmission, the achieved C statistic as measure of discriminatory power was poor. This study confirms previous admission, length of stay, heart failure, and opioid use as potentially generalisable predictor variables of PAR. This study additionally displays the necessity of repeating the validation of published models with a local dataset prior to their use.

Frequency of predictors of PAR vs. non-PAR patients.

(DOCX)

Click here for additional data file.

Comparison of the distribution of raw PAR-Risk Score values in non-PAR and PAR group.

The dashed line indicates the original threshold levels (<3, 3–10, >10). The dot-dashed line indicates the adapted threshold levels (<12, 12–25, >25).

(DOCX)

Click here for additional data file.

Receiver operating curve of the univariable logistic regression.

C-statistic = 0.605.

(DOCX)

Click here for additional data file.

Predictors and points for the calculation of the raw PAR-Risk Score.

(DOCX)

Click here for additional data file.

Results of the univariable logistic regression using the raw PAR-Risk Score values to predict PAR by SQLape.

(DOCX)

Click here for additional data file.

Goodness of fit test statistic of the univariable logistic regression.

(DOCX)

Click here for additional data file.

Coefficients of the multivariable regression of the original study.

The predicted risk was calculated by applying the scoring of the original study to each patient and then calculating the mean predicted risk by group.

(DOCX)

Click here for additional data file.

(CSV)

Click here for additional data file.

18 Aug 2021

PONE-D-21-23537

External validation of the PAR-Risk Score to assess hospital readmission risk in internal medicine patients

PLOS ONE

Dear Dr. Stämpfli,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by Oct 02 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Gianluigi Savarese

Academic Editor

PLOS ONE

Journal Requirements:

1. When submitting your revision, we need you to address these additional requirements.

Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. In ethics statement in the manuscript and in the online submission form, please provide additional information about the patient records/samples used in your retrospective study. Specifically, please ensure that you have discussed whether all data/samples were fully anonymized before you accessed them and/or whether the IRB or ethics committee waived the requirement for informed consent. If patients provided informed written consent to have data/samples from their medical records used in research, please include this information.

3. PLOS ONE has specific requirements for studies that are presenting a new method or tool as the primary focus, including a newly developed or modified questionnaire or scale (https://journals.plos.org/plosone/s/submission-guidelines#loc-methods-software-databases-and-tools.) One requirement is that the questionnaire or scale must be openly available under a license no more restrictive than CC BY. In light of this, before we proceed, please include a copy of your questionnaire or scale as a Supporting Information file (in the original language) or provide a link if it is available through an online repository.

4. We note that the grant information you provided in the ‘Funding Information’ and ‘Financial Disclosure’ sections do not match.

When you resubmit, please ensure that you provide the correct grant numbers for the awards you received for your study in the ‘Funding Information’ section.

5. Please include your full ethics statement in the ‘Methods’ section of your manuscript file. In your statement, please include the full name of the IRB or ethics committee who approved or waived your study, as well as whether or not you obtained informed written or verbal consent. If consent was waived for your study, please include this information in your statement as well.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The authors of this manuscript have externally validated a risk score to assess hospital readmission risk in internal medicine patients. As is also clear from the results, it is extremely important to externally validate risk scores in different populations from where the risk score was created. The manuscript is well written, I have few questions that I hope the authors are able to address.

1. Wouldn't it have been an option to validate AND update the model to fit better with the new population? There has been an adaptation in the risk score cut offs but what about new variables that show better association and discrimination for a risk score?

2. As I understood cantonal is a Swiss healthcare system but might be difficult to understand for those that are not Swiss. Perhaps the authors could use secondary care as it is more widely understood?

3. The authors state that the derivation cohort was a primary care hospital but it is unclear to me what this means. Could the authors elaborate on this?

4. Why did the authors specify the age as above 65 years? 40% of the patients in the derivation cohort were under 65 years old? This was not an exclusion criteria in the initial study and could have potentially introduced bias and an underestimation of the c-statistic.

5. How much missing data was there? In the manuscript it is written that population mean was imputed, however this is an unrealiable form of imputing missing data as it shrinks standard errors and could alter the associations between variables. I would suggest to use multiple imputation or a Bayesian methods for single imputation.

6. How did the authors choose the new cut offs in the adapted risk score?

Reviewer #2: Dr. Higi and colleagues performed an external validation of the Potentially Avoidable Readmission Risk Score to evaluate the potentially avoidable hospital readmission risk in internal medicine patients from a Swiss cantonal hospital.

For that purpose, the authors used a 2-years cohort of internal medicine hospitalizations, longer than 48h and in patients older than 65, using the same algorithm (SQLape) of the derivation cohort, to identify the studied outcome (30-day potentially avoidable readmission – PAR). A total sample of 5985 patients was used considering data from electronic health records, but the authors had to deal with missing data.

It is expectable that the performance of prediction models would be poorer in the new sample than in the development population. However, those models should not be used spready before an established external validity. The work of the authors is of utmost relevance and this kind of analysis should be encouraged so the clinical impact of the previous manuscripts (development of scores) could (or not) be projected to the real-world practice. Still, some details could be enhanced in the reviewer opinion. Please find a detailed list below.

1. Title

a. I suggest adding “potentially avoidable” to the outcome “hospital readmission risk” to clearly represent the outcome that was measured.

2. Introduction

a. The medical context is well-explained but should be shortened.

3. Methods

a. The authors used correctly the TRIPOD check-list;

b. I suggest replacing “population” by “sample” or “participants”;

c. To consider an external validation the patients may be different from the derivation cohort. The main differences between the author’s sample and the development population should be emphasised explaining the reasons for that to constitute a sample for external validation. Also, the shared points: internal medicine patients;

d. The inclusion criteria (age > 65 and hospitalizations longer than 48 hours) were not clearly explained;

e. The age criteria could move the purpose of this external validation not only for the validation in another Centre within the same country where the PAR-Risk Score was derived (geographic validation, same country, other region), but also in a specific subgroup of patients, the elderly – these items should be incorporated at aims;

f. What were the main reasons for having missing values? Why did the authors use different approaches to deal with missing data in serum potassium and medication? Please clarify these topics.

g. A single imputation using sample mean was used to serum potassium levels in 514 (8.6%) patients. As the score uses a cut-off value of 5.5 mmol/L it is probably that all of those patients did not enter with a value which count to scoring 4 points, please discuss.

h. The authors performed missing imputation regarding medication and some sensitivity analyses considering two types of imputation and complete-cases analysis – this information is distributed between missing data and statistical analyses. Please report the type of missing imputation that was done within missing data subsection;

i. Please clarify lines 205-206, “the sensitivity analysis on the 94 excluded patients” or “the sensitivity analysis with/adding the 94 excluded patients”?

j. I’m not sure if any model updating was done or only the creation of new risk subgroups according to PAR-score distribution (please verify and complete 10e from TRIPOD accordingly).

4. Results

a. Please report how many PAR cases were reported on the 94 patients excluded due to missing data on dispensed drugs;

b. Please be cautious when reporting that 5.7% (PAR in the external cohort) is comparable with 7.5% (PAR in the derivation cohort) without any quantitative measure for comparison;

c. Table 1 – Please add the % of SQLape defined PAR cases in the derivation cohort;

d. Table 2 and Table 3 – Please add N by groups Non-PAR and PAR;

e. Figure 2 – Please note that the total exclusions (n=2267) do not correspond to the sum of the 4 exclusion criteria;

f. Figure S3 C-statistic (0.602) does not correspond to the C-statistic reported in the text (0.605) – line 202;

g. Table S3 – add observed value;

h. Regarding the risk stratification, medium and high-risk groups were at higher odds of having a PAR than low risk group considering the adapted threshold; while using the original threshold, only the high-risk group was significantly at a higher odd of having PAR versus low-risk category. Please reformulate lines 210-212;

i. Consider adding a calibration plot to table 5 to simplify reading and interpretation.

5. Limitations

a. The development study was not conducted in an older patient sample (~40% of patients were <65 years old) – please verify line 300;

b. Please consider including the limitations of the missing imputations methods. Would a multiple imputation provide more consistent results?

Other comments:

For PAR outcome and internal medicine patients’ population there are diverse prediction models. Therefore, an external validation of those prediction models on the same cohort can provide a comparison of the predictive performance between those models. Indeed, it is surprising that the HOSPITAL score reached similar discriminatory power and overall accuracy in the original/derivation cohort of the PAR-Risk score. It would be expectable that the new score to be superior since it was optimally designed to fit that data/sample. It would be interesting to check how the HOSPITAL score performs in this external cohort, as the medication seems to be the main difference between the PAR-Risk Score and HOSPITAL score and the missing information on medication data is one of the drawbacks of this external validation.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

1 Oct 2021

The authors would like to send their gratitude to the reviewers and the editor for their positive and constructive feedback. The comments and shared thoughts improved the overall work and added important details to the article.

To ease correspondence, we copied all comments into a table (uploaded as separate file). The reviewers’ comments are on the left side; the authors’ answers are on the right side. Indicated pages and lines refer to the clean version of the revised manuscript (track changes off).

We applied the following color scheme to our answers: Green states implementation, yellow states implementation which may be regarded as only partial, and orange states currently no implementation into the main body/only further clarification.

Submitted filename: R1_Answers_PAR_PLOS.docx

Click here for additional data file.

28 Oct 2021

External validation of the PAR-Risk Score to assess potentially avoidable hospital readmission risk in internal medicine patients

PONE-D-21-23537R1

Dear Dr. Stämpfli,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Gianluigi Savarese

Academic Editor

PLOS ONE

15 Nov 2021

PONE-D-21-23537R1

External validation of the PAR-Risk Score to assess potentially avoidable hospital readmission risk in internal medicine patients

Dear Dr. Stämpfli:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Gianluigi Savarese

Academic Editor

PLOS ONE