Predicting hospital readmission risk: A prospective observational study to compare primary care providers' assessments with the LACE readmission risk index.

PURPOSE: This study aims to determine if the primary care provider (PCP) assessment of readmission risk is comparable to the validated LACE tool at predicting readmission to hospital.
METHODS: A prospective observational study of recently discharged adult patients clustered by PCPs in the primary care setting. Physician readmission risk assessment was determined via a questionnaire after the PCP reviewed the hospital discharge summary. LACE scores were calculated using administrative data and the discharge summary. The sensitivity and specificity of the physician assessment and the LACE tool in predicting readmission risk, agreement between the 2 assessments and the area under receiver operating characteristic (AUROC) curves were calculated.
RESULTS: 217 patient readmission encounters were included in this study from September 2017 till June 2018. The rate of readmission within 30 days was 14.7%, and 217 discharge summaries were used for analysis. The weighted kappa coefficient was 0.41 (95% CI: 0.30-0.51) demonstrating a moderate level of agreement. Sensitivity of physician assessment was 0.31 (95% CI: 0.22-0.40) and specificity was 0.80 (95% CI: 0.77-0.83). The sensitivity of the LACE assessment was 0.42 (95% CI: 0.25-0.59) and specificity was 0.79 (95% CI: 0.73-0.85). The AUROC for the LACE readmission risk was 0.65 (95% C.I. 0.55-0.76) demonstrating modest predictive power and was 0.57 (95% C.I. 0.46-0.68) for physician assessment, demonstrating low predictive power.
CONCLUSION: The LACE index shows moderate discriminatory power in identifying high-risk patients for readmission when compared to the PCP's assessment. If this score can be provided to the PCP, it may help identify patients who requires more intensive follow-up after discharge.

Introduction

Readmission after hospitalization is common and costly [1, 2]. It affects almost one in ten of all hospitalizations and up to one-third may be avoidable [1, 3, 4]. Many jurisdictions have identified 30-day hospital readmissions as a key quality indicator [5]. Similarly, strategies to reduce rates of readmissions have become a priority in many countries [6, 7].

Several studies provide support for the effectiveness of early follow-up post-hospital discharge in the primary care setting to reduce readmission [8-11]. Patients aged 65 and over who do not have physician follow-up within 30 days of discharge are three times more likely to be re-admitted [8]. Follow-up post-discharge was associated with a 19% lower chance of readmission for patients with congestive heart failure [9], and a 15% reduction in readmission to hospital after high-risk surgery associated with complications [10]. Follow-up within 7 days post-discharge from hospital in the primary care setting has been used in primary care quality frameworks [11].

A number of readmission risk assessment tools have been developed to identify high-risk patients who may benefit from early post-discharge follow-up [12-15]. The LACE index is a validated score that assesses the risk of death or unplanned readmission after discharge from hospital [12]. This tool has been suggested to help identify patients needing post-discharge interventions [16]. However, some information needed to calculate the score, such as emergency room visits in the previous 6 months, may not be readily available to primary care providers (PCPs) at the time an assessment regarding the need for early post-discharge follow-up is being made.

Primary care providers who possess detailed clinical and social knowledge of patients in the community gained over time may be able to predict which patients are at higher risk for hospital readmission. This study aims to determine if the primary care provider assessment of readmission risk is comparable to the validated LACE tool at predicting readmission to hospital within 30 days, for patients who have been discharged from hospital within the last 14 days.

Methods

Study setting and design

This study took place in Toronto, Canada between September 2017 and June 2018. Three family practice clinics of varying size participated. The sites included 31 physician faculty and a total of approximately 27,500 rostered patients in the sampling frame of the study. This study received ethics approval from the Mount Sinai Hospital Research Ethics Board (MSH REB #17-0173-E). Written consent was obtained for all participating clinicians.

This was a prospective observational study of recently discharged adult patients clustered by PCPs to observe agreement between PCP risk assessment of readmission compared to the LACE tool against a criterion standard of observed hospital readmission within 30 days in any hospital in Ontario, Canada.

Determination of PCP and LACE risk estimates for readmission

To determine PCP estimate of the risk for an individual patient of being readmitted, hospital discharge summaries from the patient’s electronic medical record or faxed to the clinic from the hospital were identified from the period of September 2017 to June 2018. Only discharge summaries for adults 18 years of age or older cared for by a consenting study physician, and received within 14 days of hospital discharge, were eligible.

A survey was attached to each hospital discharge summary that asked the PCP to rate their risk of readmission for that patient within 30 days as low, moderate or high, as well as the factors that led him/her to that decision (S1 File).

The LACE tool provides a risk estimate of readmission based on clinical and administrative data elements to provide a risk score from 0–19. The variables comprising the score include length of stay (“L”); acuity of the admission (“A”); comorbidity of the patient (measured with the Charlson comorbidity index score) (“C”) [17]; and emergency department use (measured as the number of visits in the six months before admission) (“E”) [12]. The Charlson comorbidity index is determined by assigning a score for each comorbid condition depending on the risk of dying associated with each condition. In the original study, the LACE score for each patient was classified into (i) low risk (with LACE score 0–8) corresponding to < 10% risk of readmission, (ii) medium risk (with LACE score 9–13) corresponding to 10–20% risk of readmission and (iii) high risk (with LACE score 14–19) corresponding to >20% risk of readmission [12].

Ontario has universal health care coverage for visits to doctors in emergency departments, clinics, and for hospital admissions under the Ontario Health Insurance Plan (OHIP). These visits are captured using a unique identifier for each eligible resident of the province and all analyses are considered population based. This administrative data is available for research upon request through ICES (www.ices.on.ca, n.d.). Discharge summaries were linked to the administrative data through a combination of patient identifiers which included hospital number, patient name, hospital of admission and date of admission. Administrative data was also used to determine the number of emergency department visits in the prior 6 months, as well as whether the patient was readmitted within 30 days. This was then linked to the data from the hospital discharge summary and physician risk estimate. Anonymized data was sent back to the research team through a secure portal. Data on length of stay, acuity and co-morbidities was retrieved manually from the discharge summary.

Statistical analysis

Agreement between the LACE and physician risk assessment was quantified using the weighted kappa statistic to account for differences on ordinal scale of predicting 30-days readmission (low, medium, high). The weighted kappa statistic can be interpreted as indicating slight agreement (0.01–0.20), fair agreement (0.21–0.40), moderate agreement (0.41–0.60), substantial agreement (0.61–0.80), and almost perfect agreement (0.80–0.99) [18]. The association between the readmission rate derived from physician risk assessment and LACE tool was assessed using two-sided Cochrane-Armitage trend test.

Sensitivity and specificity for predicting a high risk for hospital readmission was estimated by collapsing the “low risk” and “medium risk” categories into a single group for the PCP assessment, as indicated by the treating PCP. This was done to be consistent with the LACE categorization of high-risk readmissions as per the original paper. We felt it would be most prudent to identify and closely follow-up those patients deemed to be high-risk for readmission. These diagnostic measures were estimated for the LACE tool and physician assessment.

The area under receiver operating characteristic curves (AUROC) was used to estimate the predictive power of 30-days readmission for LACE score and physician assessment. An AUROC value of 0.5 suggests no discriminatory predictive power, 0.7 to 0.8 as acceptable discriminatory predictive power and greater than 0.8 as excellent discriminatory predictive power [19]. We compared the AUROC of LACE tool and physician assessment tool using the non-parametric approach, as further described by DeLong, Delong and Clarke-Pearson [20].

Multiple logistic regression models were fitted to describe the relationship for LACE score and physician assessment with respect to the 30-days readmission, while adjusting for patient’s age and gender [21]. We conducted complete case analysis when fitting the multivariable logistic regressions.

All analyses were conducted using SAS v9.4.

This study contracted ICES Data and Analytic Services (DAS) and used de-identified data from the ICES Data Repository, which is managed by ICES with support from its funders and partners: Canada’s Strategy for Patient-Oriented Research (SPOR), the Ontario SPOR Support Unit, the Canadian Institute of Health Research and the Government of Ontario. The opinions, results and conclusions reported are those of the authors. No endorsement by ICES or any of its funders or partners is intended or should be inferred.

Results

Twenty-one of 31 eligible physicians (67.7%) agreed to participate in the study. A total of 257 discharge summaries were collected across the 3 sites during the study period. Eight summaries were excluded as they included pediatric or obstetric cases. Physicians completed survey questionnaires for 238 (96.6%), of which 21 were excluded as the provider was aware of the readmission prior to completing the survey leaving 217 on which analysis was completed. The LACE score was able to be calculated for 208 patients and physician assessments were available for 202 patients (Fig 1).

Fig 1

Number of patients included in the study.

The overall rate of readmission within 30 days was 32/217 (14.7%). There was no statistically significant difference in proportion of readmission when comparing patients less than 64 years of age with more than 65 years of age, males with females or low-income group with high income group (Table 1).

Table 1

Characteristics of the patients involved in the study.

Readmission within 30 days after hospital discharge (with respect to hospital discharge)
	No	Yes	Total
Age group (year)
18–64	80 (82.5%)	17 (17.5%)	N = 97
≥ 65	90 (85.7%)	15 (14.3%)	N = 105
Gender
Female	92 (85.2%)	16 (14.8%)	N = 108
Male	78 (83%)	16 (17%)	N = 94
Income group *
High	115 (83.3%)	23 (16.7%)	N = 138
Low	55 (85.9%)	9 (14.1%)	N = 64

1categorized using income quintiles where [1, 2, 3] is low income and [4, 5] is high income.

Both the LACE tool and physician assessment identified a higher proportion of older patients (age 65+) than younger patients, a higher proportion of male than female patients and a higher proportion of low income group than high income group to be at high risk of readmission.

Patients with at least one emergency visit in the previous 6 months had 18.6% readmission rate while patients with no emergency visit had 9.1% readmission rate (p = 0.05) (Table 2).

Table 2

Risk of readmission associated with increased co-morbidity, acuity of admission and previous number of emergency department visits.

Category	Readmission rate
Two or more co-morbidities vs. 1 or less	18.2% vs. 11.2% (p = 0.16)
Admitted for 2 or more days vs. 1 or less	15.2% vs. 14.0% (p = 0.84)
Acute vs. elective admission	17.1% vs. 7.8% (p = 0.11)
One or more vs. no ED visits in the previous 6 months	18.6% vs. 9.1% (p = 0.05)

The LACE tool categorised 97/208 (46.6%) as low risk, 60/208 (28.8%) as medium risk and 51/208 (24.5%) as high risk. Physicians estimated the risk of readmission low for 101/202 (50.0%), medium for 58/202 (28.7%) and high for 43/202 (21.3%).

Agreement between physician risk assessment and LACE instrument estimates

The weighted kappa coefficient was 0.41 (95% CI: 0.30–0.51) demonstrating the presence of statistically significant agreement between the LACE readmission risk tool and physician assessment demonstrating a statistically significant moderate level of agreement.

When LACE score deemed patient to be low risk of readmission, there was 76% agreement with physician assessment. In contrast, when LACE score deemed patient to be high risk, there was 43% agreement with physician assessment. Overall, there was 57.8% agreement (i.e. (71+20+21)/194) between physician risk assessment and LACE risk groups (Table 3).

Table 3

Agreement between physician risk assessment and LACE tool.

	Physician Risk Assessment
LACE risk group1	Low	Medium	High	Total
Low	71	16	7	94
Medium	19	20	12	51
High	9	19	21	49
Total	99	55	40	194

1categorized using the LACE score (0–9 = low risk; 10–13 = medium risk; 14–19 = high risk).

The odds of being readmitted within 30 days increased by 1.16 with every one point increase in LACE score (95% CI: 1.04–1.28; p-value = 0.004). The AUROC estimated for the LACE readmission risk estimate was 0.65 (95% C.I. 0.55–0.76) demonstrating modest predictive power. The AUROC estimate for the physician risk for readmission suggested a lower level of discrimination with a value of 0.57 (95% C.I. 0.46–0.68) (Fig 2). However, the AUROC of LACE readmission was not statistically significant at the nominal level of 5% to the AUROC of physician assessment (Chi-square = 3.11, p = 0.08).

Fig 2

Receiver Operating Curve (ROC) for LACE score and physician assessment.

When adjusting for age range and gender, the odds of 30-days readmission among patients labelled in high risk group by LACE score was 3.61 times higher than the odds of readmission among patients labelled in low risk group by LACE score (95% CI: 1.37–9.55; p-value = 0.009).

Accuracy of physician and LACE estimates of risk of readmission

There was a 9% rate of readmission within 30 days when LACE score was low risk, 15% risk of readmission when LACE score was medium risk and 25% rate of readmission when LACE score was high risk, indicating statistically significant trend in the increasing rate of re-admission with respect to LACE score (P-value = 0.009) (Table 4). Similarly with physician assessment, there was a 12% readmission rate within 30 days when physician assessment was deemed to be low risk, 14% risk of readmission when physician assessment was medium risk and 21% risk of readmission when physician assessment was high risk, indicating statistically insignificant trend in the increasing rate of re-admission with respect to physician assessment (P-value = 0.18) (Table 4).

Table 4

Readmission rate according to LACE and physician assessment score.

Readmission within 30 days (with respect to hospital discharge)
	No	Yes	Total
LACE risk 1
Low	88 (90.7%)	9 (9.3%)	N = 97
Medium	51 (85%)	9 (15%)	N = 60
High	38 (74.5%)	13 (25.5%)	N = 51
Physician risk
Low	89 (88.1%)	12 (11.9%)	N = 101
Medium	50 (86.2%)	8 (13.8%)	N = 58
High	34 (79.1%)	9 (20.9%)	N = 43

1categorized using the LACE score (0–9 = low risk; 10–13 = medium risk; 14–19 = high risk).

The diagnostic estimates for the physician risk assessment were: (1) sensitivity = 0.31 (95% CI: 0.22–0.40) and (2) specificity = 0.80 (95% CI: 0.77–0.83). The diagnostic estimates for LACE group were: (1) sensitivity = 0.42 (95% CI: 0.25–0.59) and (2) specificity = 0.79 (95% CI: 0.73–0.85).

Discussion

Our study of 21 PCPs at three sites evaluated 194 patients recently discharged from hospital. We found a moderate level of agreement between the LACE tool and physician assessment for predicting the risk of hospital readmission within 30 days after being discharged. The LACE index performed slightly better than physician assessment with moderate predictive power, although this difference was not statistically significant.

The overall rate of readmission in our study was 14.7%. The original LACE study had a readmission rate of 8% [12] which was comparable to population-based data [1]. This may be because the clinics involved in this study were mostly affiliated with tertiary care hospitals where patients may be more complex or because of secular changes over time.

A number of studies assessing the LACE tool have been performed, but these are often in certain population groups e.g. cardiovascular disease [22], chronic obstructive pulmonary disease [23] or in older people [24] and they show conflicting results. The results of this study compare with the Damery study [25] where increasing LACE score and certain components of the LACE index were independent predictors of readmission. A trade-off between sensitivity and specificity can be observed with increasing sensitivity and decreasing specificity as the threshold for LACE score to classify patients as high risk for 30 days readmission is decreased.

Both methods performed sub-optimally; the PCP assessment of readmission risk may not be as effective as the LACE tool. There was overlap of the 95% C.I. for both AUROC deeming the difference not statistically significant. Studies looking at clinician assessment of readmission risk have shown varied results with one demonstrating an AUC derived from ROC of 0.689 for the risk assessment completed by the discharging attending on a hospital team vs. 0.620 for the LACE tool as a predictor [26]. Allaudeen et al. found poor ability of inpatient teams (physician, nurse and case manager) to discriminate between readmissions vs. non-readmissions (AUC from ROC of 0.5786) [27]. However, PCPs have long term patient relationships that may affect their assessment differently as a result of their knowledge of a patient’s social conditions and overall medical status. To the best of our knowledge there are no studies looking at the PCPs’ judgment of readmission, but the results of our study are comparable to the Allaudeen study which showed poor discriminatory power of clinicians in assessing readmission risk. This may indicate that the relationship and contextual knowledge a primary care provider has with a patient does not improve assessment of readmission risk and that the information in the EMR is not readily accessible to the PCP to allow for pertinent decision making.

If the LACE score can easily be calculated and provided to the PCP by the discharging physician it may be helpful in conjunction with physician assessment in deciding who requires more intensive follow-up after discharge. Future directions may include calculating the sensitivity and specificity of a score that combines the LACE tool with physician assessment and to observe whether implementation of such a tool can reduce readmission rates.

Our study has limitations that merit emphasis. ICES data was used to collect data on emergency department use in order to calculate the LACE score. Currently in our healthcare system emergency room visits are not comprehensively reported to PCPs. As a result, it may be more practical for the hospital discharge team to provide the PCPs with a readmission risk score. The analysis was performed without taking into account the unique structure of patient-physician hierarchy (i.e. clustering) due to small sample size of this study. As there are no significant results, further adjustments for clustering would not change these results. During the study period there was a change in the process with regards to how discharge summaries were received resulting in some physician assessments being lost. Furthermore, only readmissions where discharge summaries were received by the PCP were included, and there is a possibility that some discharges were not included. This may have weakened or underestimated the sensitivity and specificity; however this would apply to both the LACE estimates as well as the PCP estimates. We did not assess the quality of the discharge summary, the instructions provided for follow-up care or if this impacted their decision making. However physician response to the survey indicates the factors most likely to influence their assessment were familiarity with the patient and the hospital diagnosis. Finally, this study did not observe effects of implementing an intervention using the LACE tool or physician risk assessment on readmission rates but could be an opportunity for future direction.

Conclusion

The LACE index shows moderate discriminatory power in patients at increased risk for readmission after discharge, and may be superior when comparing to PCP’s clinical judgement of readmission risk. Obtaining all information needed to calculate the LACE score such as emergency department visits may be difficult to obtain in primary care. Consideration should be given to providing the LACE score in the hospital discharge summary sent to the PCP to facilitate identification and early follow up of patients at high risk for hospital readmission. Future studies can include the effectiveness of combining physician assessment with elements of the LACE tool in predicting hospital readmission.

Re-admission risk assessment survey.

(DOCX)

Click here for additional data file.

28 Aug 2021

PONE-D-21-06391

Predicting hospital readmission risk: A prospective observational study to compare primary care providers’ assessments with the LACE readmission risk index

PLOS ONE

Dear Dr. Walji,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

The manuscript has been evaluated by three reviewers, and their comments are available below.

All reviewers recommend greater clarity in the reporting of this manuscript, both in the Methods section and Results. Specifically, the reviewers note the need for greater detail and quantification of the primary tool used in the study, as well as additional analyses.

Could you please carefully revise the manuscript to address all comments raised?

Please submit your revised manuscript by Oct 11 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Avanti Dey, PhD

Staff Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Please include additional information regarding the survey or questionnaire used in the study and ensure that you have provided sufficient details that others could replicate the analyses. For instance, if you developed a questionnaire as part of this study and it is not under a copyright more restrictive than CC-BY, please include a copy, in both the original language and English, as Supporting Information.  If the original language is written in non-Latin characters, for example Amharic, Chinese, or Korean, please use a file format that ensures these characters are visible.

3. Please state whether you validated the questionnaire prior to testing on study participants. Please provide details regarding the validation group within the methods section.

4. We note that the grant information you provided in the ‘Funding Information’ and ‘Financial Disclosure’ sections do not match.

When you resubmit, please ensure that you provide the correct grant numbers for the awards you received for your study in the ‘Funding Information’ section.

5. Thank you for stating the following in the Competing Interests/Financial Disclosure* (delete as necessary) section:

“I have read the journal's policy and the authors of this manuscript have the following competing interests: Dr. Bell is a medical consultant to the Ontario Ministry of Health. The other authors have no competing interests to declare”

.We note that one or more of the authors are employed by a commercial company: Ontario Ministry of Health

a.Please provide an amended Funding Statement declaring this commercial affiliation, as well as a statement regarding the Role of Funders in your study. If the funding organization did not play a role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript and only provided financial support in the form of authors' salaries and/or research materials, please review your statements relating to the author contributions, and ensure you have specifically and accurately indicated the role(s) that these authors had in your study. You can update author roles in the Author Contributions section of the online submission form.

Please also include the following statement within your amended Funding Statement.

“The funder provided support in the form of salaries for authors [insert relevant initials], but did not have any additional role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript. The specific roles of these authors are articulated in the ‘author contributions’ section.”

If your commercial affiliation did play a role in your study, please state and explain this role within your updated Funding Statement.

b. Please also provide an updated Competing Interests Statement declaring this commercial affiliation along with any other relevant declarations relating to employment, consultancy, patents, products in development, or marketed products, etc.

Within your Competing Interests Statement, please confirm that this commercial affiliation does not alter your adherence to all PLOS ONE policies on sharing data and materials by including the following statement: "This does not alter our adherence to  PLOS ONE policies on sharing data and materials.” (as detailed online in our guide for authors http://journals.plos.org/plosone/s/competing-interests) . If this adherence statement is not accurate and  there are restrictions on sharing of data and/or materials, please state these. Please note that we cannot proceed with consideration of your article until this information has been declared.

Please include both an updated Funding Statement and Competing Interests Statement in your cover letter. We will change the online submission form on your behalf.

6. In your Data Availability statement, you have not specified where the minimal data set underlying the results described in your manuscript can be found. PLOS defines a study's minimal data set as the underlying data used to reach the conclusions drawn in the manuscript and any additional data required to replicate the reported study findings in their entirety. All PLOS journals require that the minimal data set be made fully available. For more information about our data policy, please see http://journals.plos.org/plosone/s/data-availability.

"Upon re-submitting your revised manuscript, please upload your study’s minimal underlying data set as either Supporting Information files or to a stable, public repository and include the relevant URLs, DOIs, or accession numbers within your revised cover letter. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories. Any potentially identifying patient information must be fully anonymized.

Important: If there are ethical or legal restrictions to sharing your data publicly, please explain these restrictions in detail. Please see our guidelines for more information on what we consider unacceptable restrictions to publicly sharing data: http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions. Note that it is not acceptable for the authors to be the sole named individuals responsible for ensuring data access.

We will update your Data Availability statement to reflect the information you provide in your cover letter.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Partly

Reviewer #3: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: N/A

Reviewer #3: No

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: No

Reviewer #3: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The article is well written and addresses one of many challenges with hospital medicine - the rate of readmission to the hospital within 30 days of discharge and how to best predict patients who are at higher risk of readmission. This would allow high risk patients to received additional attention with the goal of preventing avoidable hospital readmissions.

Abstract well structured and clear. Reflects the content of the article.

Background - Sufficiently broad and up to date, discusses limitations and utility of current prediction systems (LACE) and frames the research question clearly.

Methods - Clearly presented, ethical approval documented. Statistics are conventional and are without controversy.

Some clarity would be useful - are the patients from one hospital or for the entire province of Ontario? Some basic information like size of hospital and location (100 beds, rural area in the Province of Ontario) or if the entire province some information about the number of hospitals and the population in the service area. Also, would be useful to document if hospital admissions at different hospitals within Ontario or in other provinces would be detected.

Results - Clearly presented and well organized. Tables are clear.

Discussion - Discuss readmission detection as mentioned in the methods section - this may be a strength or a limitation for this study. Results framed in context of other knowledge about readmission prediction.

Conclusions - Not overstated or generalized. Reasonable given methods and results.

Reviewer #2: This is an interesting study comparing the use of a PCP clinical assessment of readmission risk in comparison to a previously validated tool (the LACE index). The results if this study could be used to identify pragmatic tools in defining high risk patients for readmission.

I have the following recommendations to make:

1. My biggest concern in this study is the ambiguity of the PCP assessment. The authors describe what seems to be a qualitative tool in evaluating readmission, but the analysis made was purely quantitative. I suggest further quantifying the PCP assessment tool by showing methodically how PCPs evaluated the risk. In addition, the survey used by PCP should be included in the study and described in detail. Alternatively, I suggest using qualitative research tools to better define the PCP judgment.

2. Authors are asked to clarify if patients enrolled in the study were formally evaluated in the clinic by PCP and/or performed chart check only. Also, authors are asked to provide demographic baseline data of the patient population enrolled to enhance the generalizability of the results.

3. The sensitivity and specificity analysis in the study was calculated by summing the low and medium risk groups into one group. Authors are asked to clarify the reason of doing that as it may overestimate the specificity of the results.

4. Sampling methodology in the study is unclear. Did the authors include all the charts in the registry during the study period? If not please describe how sampling was done?

5. In the discussion section the authors confusedly referencing that 31 PCPs were included in the study. Please correct to 21 PCPs and consider omitting the number of patients in the registry (as only a sample was used).

6. Authors need to report the multivariate regression results assessing the relationship between LACE and PCP assessment. Consider including more variables known to increase risk of readmission, if available (polypharmacy, types of comorbidities…etc.).

Reviewer #3: The authors of this study do a great job of comparing the LACE tool to a physicians assessment. The findings of this study have implications for everyday practice and thus, this paper stands to be highly cited in the future. I just have a few minor concerns:

First, the results section of the abstract should contain the number of patients and years of the study. This will help facilitate future systematic reviews that include this paper.

Second, the ROC AUC comparisons, should contain a p-value. Many statistical software applications, such as MedCalc and Stata, make this value readily available. Even if the difference between the AUCs are statistically insignificant, this will still help interpret the findings of the study.

Finally, it may be beneficial to clearly state in the discussion or conclusions, the implications on future practice. For example, if the authors feel that incorporating the LACE score into the electronic medical record will save (or cost) the clinicians time, this should be clearly stated in the discussion or conclusions.

Thank you for the opportunity to review this important study.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: Yes: Tamer Hudali, MD, MPH, FACP

Reviewer #3: Yes: Joshua Parreco

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

27 Oct 2021

Response to Academic Editor:

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming.

Response: We have adjusted the manuscript file to align with PLOS ONE’s style requirements.

2. Please include additional information regarding the survey or questionnaire used in the study and ensure that you have provided sufficient details that others could replicate the analyses. For instance, if you developed a questionnaire as part of this study and it is not under a copyright more restrictive than CC-BY, please include a copy, in both the original language and English, as Supporting Information. If the original language is written in non-Latin characters, for example Amharic, Chinese, or Korean, please use a file format that ensures these characters are visible.

Response: A copy of the survey has now been included, as Supplementary File 1. We have revised page 6 line 119 accordingly.

3. Please state whether you validated the questionnaire prior to testing on study participants. Please provide details regarding the validation group within the methods section.

Response: As the survey was developed for this study, it has not been validated prior to use. The survey was developed by the investigators of the team to assess physician risk assessment of readmission. Response options were “low”, “medium” or high” in keeping with the LACE assessment risk categories.

4. We note that the grant information you provided in the ‘Funding Information’ and ‘Financial Disclosure’ sections do not match. When you resubmit, please ensure that you provide the correct grant numbers for the awards you received for your study in the ‘Funding Information’ section.

We have added information on the grant funding agency in the Revised Funding Statement of the revised cover letter. The funding disclosure was in relation to a competing interest on of the co-authors had, namely that they have an affiliation with the Ontario Ministry of Health. The Ontario Ministry of Health did not have any involvement in the this study and were not included in the funding information section.

5. Thank you for stating the following in the Competing Interests/Financial Disclosure* (delete as necessary) section:

“I have read the journal's policy and the authors of this manuscript have the following competing interests: Dr. Bell is a medical consultant to the Ontario Ministry of Health. The other authors have no competing interests to declare”

We note that one or more of the authors are employed by a commercial company: Ontario Ministry of Health

a. Please provide an amended Funding Statement declaring this commercial affiliation, as well as a statement regarding the Role of Funders in your study. If the funding organization did not play a role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript and only provided financial support in the form of authors' salaries and/or research materials, please review your statements relating to the author contributions, and ensure you have specifically and accurately indicated the role(s) that these authors had in your study. You can update author roles in the Author Contributions section of the online submission form.

Please also include the following statement within your amended Funding Statement.

“The funder provided support in the form of salaries for authors [insert relevant initials], but did not have any additional role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript. The specific roles of these authors are articulated in the ‘author contributions’ section.”

If your commercial affiliation did play a role in your study, please state and explain this role within your updated Funding Statement.

b. Please also provide an updated Competing Interests Statement declaring this commercial affiliation along with any other relevant declarations relating to employment, consultancy, patents, products in development, or marketed products, etc.

Within your Competing Interests Statement, please confirm that this commercial affiliation does not alter your adherence to all PLOS ONE policies on sharing data and materials by including the following statement: "This does not alter our adherence to PLOS ONE policies on sharing data and materials.” (as detailed online in our guide for authors http://journals.plos.org/plosone/s/competing-interests) . If this adherence statement is not accurate and there are restrictions on sharing of data and/or materials, please state these. Please note that we cannot proceed with consideration of your article until this information has been declared.

Please include both an updated Funding Statement and Competing Interests Statement in your cover letter. We will change the online submission form on your behalf.

Response: We agree with the editor that further clarification regarding the funding of the project is required. This study was funded by the University of Toronto’s UTOPIAN program. The commercial affiliated for Dr. Bell should only be listed as a competing interest as the Ontario Ministry of Health did not provide support for this project and did not have a role in this study. We have included a revised funding statement and a revised competing interests statement in the cover letter.

Revised Funding Statement: This study received grant funding from the University of Toronto’s UTOPIAN Ideas to Proposal program. The funder provided support to cover the cost of conducting the study but did not have any additional role in the study design, data collection and analysis, decision to publish or preparation of the manuscript.

Revised Competing Interests Statement: Dr. Chaim Bell, a co-author on this project, is a medical consultant to the Ontario Ministry of Health, however the Ministry of Health had no role in this study. This commercial affiliation does not alter our adherence to PLOS ONE policies on sharing data and materials. There are no additional competing interests to declare.

6. In your Data Availability statement, you have not specified where the minimal data set underlying the results described in your manuscript can be found. PLOS defines a study's minimal data set as the underlying data used to reach the conclusions drawn in the manuscript and any additional data required to replicate the reported study findings in their entirety. All PLOS journals require that the minimal data set be made fully available. For more information about our data policy, please see http://journals.plos.org/plosone/s/data-availability.

"Upon re-submitting your revised manuscript, please upload your study’s minimal underlying data set as either Supporting Information files or to a stable, public repository and include the relevant URLs, DOIs, or accession numbers within your revised cover letter. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories. Any potentially identifying patient information must be fully anonymized.

Important: If there are ethical or legal restrictions to sharing your data publicly, please explain these restrictions in detail. Please see our guidelines for more information on what we consider unacceptable restrictions to publicly sharing data: http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions. Note that it is not acceptable for the authors to be the sole named individuals responsible for ensuring data access.

We will update your Data Availability statement to reflect the information you provide in your cover letter.

Response: We have revised the data availability statement to state the following.

All data underlying the findings described in the manuscript are not fully available/without restriction. The data used within this study was obtained through the Institute for Clinical and Evaluative Sciences (ICES), and as such there are legal restrictions to sharing the dataset. As well, given that the data includes patient health information, there are ethical restrictions in regard to the sharing of data, whereby the lead study institution, Sinai Health System, would require a data transfer agreement to be approved and in place prior to transferring or sharing the data.

Response to Reviewer 1:

1. Some clarity would be useful - are the patients from one hospital or for the entire province of Ontario? Some basic information like size of hospital and location (100 beds, rural area in the Province of Ontario) or if the entire province some information about the number of hospitals and the population in the service area. Also, would be useful to document if hospital admissions at different hospitals within Ontario or in other provinces would be detected

Response: The patients included in this study are rostered to three different primary care practices and admissions to any hospital in Ontario were captured. The total number of patients rostered to these clinics was approximately 27,500. The three different primary care sites included were in the Toronto downtown region and Greater Toronto area (all urban).

2. Discuss readmission detection as mentioned in the methods section - this may be a strength or a limitation for this study. Results framed in context of other knowledge about readmission prediction

Response: We agree with the reviewer that the method used for readmission detection could be a limitation of the study. Only discharge summaries received by the primary care provider were captured in this study. We have included the possibility of readmission detection being underestimated due to how discharge summaries were collected as a possible limitation to the study. Please see lines 315-319 of page 15 of the manuscript.

Response to Reviewer 2:

1. My biggest concern in this study is the ambiguity of the PCP assessment. The authors describe what seems to be a qualitative tool in evaluating readmission, but the analysis made was purely quantitative. I suggest further quantifying the PCP assessment tool by showing methodically how PCPs evaluated the risk. In addition, the survey used by PCP should be included in the study and described in detail. Alternatively, I suggest using qualitative research tools to better define the PCP judgment.

Response: In the methods section (page 6 lines 117-119) we have indicated that the PCP survey is a quantitative measure, with risk of readmission within 30 days being rated as Low, Moderate or High, by the PCP. The decision regarding the category of risk assigned to each patient’s readmission risk score was determined based on the physician’s clinical judgment, which is consistent with what would happen in clinical practice. We chose these risk categories in concordance with the LACE risk categories. To address the ambiguity regarding the PCP assessment, we have included a copy of the survey as a supplementary file. Interviews or focus groups with the primary care providers to understand their experience using the PCP survey as well as to gain a deeper insight into the PCP’s assessment could be an area for future direction.

2. Authors are asked to clarify if patients enrolled in the study were formally evaluated in the clinic by PCP and/or performed chart check only. Also, authors are asked to provide demographic baseline data of the patient population enrolled to enhance the generalizability of the results.

Response: There were no changes to usual process of care for this study, and as such patients who were included in the study were those whose discharge summary was received by the PCP. No change to usual process occurred to best mimic how the PCP assessment would be used in a real-life scenario to assess whether this assessment would be sufficient to identify patients who were high-risk for readmission and thus may benefit from more intensive follow-up post-discharge.

Table 1 (page 9) of the results section includes the demographic baseline data for the study sample. All baseline data that was made available to us has been included in that table.

3. The sensitivity and specificity analysis in the study was calculated by summing the low and medium risk groups into one group. Authors are asked to clarify the reason of doing that as it may overestimate the specificity of the results.

Response: We agree with the reviewer that more clarity is needed regarding the methodological decision to sum the low and medium risk groups. We have including a statement in the methods section (page 7 lines 156-159) explaining that the PCP assessment’s low and medium risk categories were collapsed into a single group in order to be consistent with the LACE assessment’s categorization of high risk readmission. High risk of admission was deemed to be a >20% risk of readmission within 30 days of discharge as per the original LACE study.

4. Sampling methodology in the study is unclear. Did the authors include all the charts in the registry during the study period? If not please describe how sampling was done?

Response: All discharge summaries that were received by the three participating primary care sites during the study period were captured. Only pediatric and obstetric cases were excluded. All other discharge summaries were included.

5. In the discussion section the authors confusedly referencing that 31 PCPs were included in the study. Please correct to 21 PCPs and consider omitting the number of patients in the registry (as only a sample was used).

Response: We have edited page 13 line 262 of the discussion to show that 21 PCPS were included in the study and as suggested by the reviewer, we have omitted the total number of patients in the sampling frame.

6. Authors need to report the multivariate regression results assessing the relationship between LACE and PCP assessment. Consider including more variables known to increase risk of readmission, if available (polypharmacy, types of comorbidities…etc.).

Response: Unfortunately, the available dataset did not contain the appropriate clinical variables for us to consider in the regression framework. We acknowledge that it is necessary to control for such clinical predictors as confounders in order to generate reliable statistical inference. However, the available data tables did not allow us to achieve this task.

Response to Reviewer 3:

1. First, the results section of the abstract should contain the number of patients and years of the study. This will help facilitate future systematic reviews that include this paper.

Response: We have included the number of patients and study years within the results section of the abstract (page 2 lines 46-47).

2. Second, the ROC AUC comparisons, should contain a p-value. Many statistical software applications, such as MedCalc and Stata, make this value readily available. Even if the difference between the AUCs are statistically insignificant, this will still help interpret the findings of the study.

Response: We would like to thank the reviewers for providing this suggestion and our apologies for this oversight. We added the hypothesis test evaluation for two ROC comparisons of LACE tool and physician assessment. We updated the Methods section (page 7-8, lines 164-166) by adding the following text and reference (page 20, lines 416-418):

“We compared the AUROC of LACE tool and physician assessment tool using the non-parametric approach, as further described by DeLong, Delong and Clarke-Pearson (1988).20”

Reference:

20. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the Areas under Two or More Correlated Receiver Operating Characteristic Curves: A Nonparametric Approach. Biometrics. 1988 Sept; 44(3):837–845

We determined the difference between the two ROC metrics as statistically insignificant (evaluated at nominal coverage rate of 5%). The p-value was computed to be 0.08 and this is also described in the results section of the manuscript (page 11-12, lines 229-232):

“However, the AUROC of LACE readmission was not statistically significant at the nominal level of 5% to the AUROC of physician assessment (Chi-square= 3.11, P-value= 0.08).”

3. Finally, it may be beneficial to clearly state in the discussion or conclusions, the implications on future practice. For example, if the authors feel that incorporating the LACE score into the electronic medical record will save (or cost) the clinicians time, this should be clearly stated in the discussion or conclusions.

Response: In the conclusion section (page 15 lines 312-317), we have discussed future considerations in regard to these tools.

Submitted filename: Response to Reviewers.docx

Click here for additional data file.

22 Nov 2021

Predicting hospital readmission risk: A prospective observational study to compare primary care providers’ assessments with the LACE readmission risk index

PONE-D-21-06391R1

Dear Dr. Walji,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Tamer Hudali

Guest Editor

PLOS ONE

Additional Editor Comments (optional):

I participated as a reviewer for the initial evaluation of this manuscript.

Reviewers' comments:

2 Dec 2021

PONE-D-21-06391R1

Predicting hospital readmission risk: A prospective observational study to compare primary care providers’ assessments with the LACE readmission risk index

Dear Dr. Walji:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Tamer Hudali

Guest Editor

PLOS ONE