Development and validation of a new prognostic index for mortality risk in multimorbid adults.

CONTEXT: Multimorbidity is highly prevalent among older adults and associated with a high mortality. Prediction of mortality in multimorbid people would be clinically useful but there is no mortality risk index designed for this population. Our objective was therefore to develop and internally validate a 1-year mortality prognostic index for older multimorbid adults.
METHODS: We analysed data of the OPERAM cohort study in Bern, Switzerland, including 822 adults aged 70 years or more with multimorbidity (3 or more chronic medical conditions) and polypharmacy (use of 5 drugs or more for >30 days). Time to all-cause mortality was assessed up to 1 year of follow-up. We performed a parametric Weibull regression model with backward stepwise selection to identify mortality risk predictors. The model was internally validated and optimism corrected using bootstrapping techniques. We derived a point-based risk score from the regression coefficients. Calibration and discrimination were assessed by the calibration slope and C statistic.
RESULTS: 805 participants were included in the analysis. During 1-year of follow-up, 158 participants (20%) had died. Age, Charlson-Comorbidity-Index, number of drugs, body mass index, number of hospitalizations, Barthel-Index (functional impairment), and nursing home residency were predictors of 1-year mortality in a multivariable model. Using these variables, the 1-year probability of dying could be predicted with an optimism-corrected C statistic of 0.70. The optimism-corrected calibration slope was 0.93. Based on the derived point-based risk score to predict mortality risk, 7% of the patients classified at low-risk of mortality, 19% at moderate-risk, and 37% at high-risk died after one year of follow-up. A simpler mortality score, without the Charlson-Comorbidity-Index and Barthel-Index, showed reduced discriminative power (optimism-corrected C statistic: 0.59) compared to the full score.
CONCLUSION: We developed and internally validated a mortality risk index including for the first-time specific predictors for multimorbid adults. This new 1-year mortality prediction point-based score allowed to classify multimorbid older patients into three categories of increasing risk of mortality. Further validation of the score among various populations of multimorbid patients is needed before its implementation into practice.

Introduction

Multimorbidity is highly prevalent especially in older adults [1] and, due to the population ageing, the number of people with multimorbidity is growing rapidly [2, 3]. Multimorbidity is associated with a high mortality rate [2] and previous research suggests that all-cause mortality risk for individuals with multimorbidity is nearly 2–3 times higher compared with individuals without multimorbidity [4-6].

Many guidelines recommend tailoring preventive care of multimorbid people according to life expectancy, and hence on the mortality risk [7]. Indeed, patients with high short-term mortality might not have the time to benefit from a preventive care intervention. Because multimorbid patients have a relatively short life expectancy, they are at higher risk of not having the time to benefit. It is therefore necessary to have a valid index for mortality prediction in multimorbid patients.

In a systematic review, Yourman et al. have identified 16 mortality prognostic indices for older adults [8]. While several of these prognostic indices were fairly accurate to predict mortality, the authors concluded that none could be recommended for a widespread use. One major limitation is that none of these prognostic indices has been tested prospectively in various samples. Key is that their transportability in other populations is unknown and clinicians cannot use these indices with confidence across different groups of patients [8]. Further, none of these indices has been developed specifically in multimorbid older adults. Current mortality indices do not consider predictors specific to multimorbid older adults such as the severity of comorbidities and functional impairment. Therefore, there is no tool recommended to accurately predict mortality in older multimorbid adults.

Our objective was therefore to develop and internally validate a 1-year mortality prognostic index for older multimorbid adults.

Methods and analysis

Source of data and study design

We used data from 822 participants of an ongoing cohort study in Bern, Switzerland. Participants were originally enrolled in the clinical trial OPtimising thERapy to prevent Avoidable hospital admissions in Multimorbid older people (OPERAM [9, 10]). Participants were assigned to receive either standard care or a medication review by a Systematic Tool to Reduce Inappropriate Prescribing (STRIP) with observation of the primary outcome of drug-related hospital admission (DRA) over 1 year.

For the current study, we used data collected at baseline (December 2016-October 2018) and up to 1 year after baseline (until October 2019). The local Ethics Committee in Bern, Switzerland, approved the study protocol with the project number 2018–00784. Study nurses collected baseline data by a personal interview with the participant and from medical files. The follow-up was conducted via phone calls. Phone interviews were held with participants or relatives, otherwise with a proxy or with the general practitioner, when the participants were not reachable or not able to answer.

We developed and validated the mortality prognostic index following the Prognosis Research Strategy (PROGRESS) framework [11], and reported it following the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement [12, 13]. We further followed the recommendations of Moons et al. [14, 15] for risk prediction models. This study is part of a research project whose protocol has been published previously [10].

Participants

Participants (N = 822) were enrolled at the time of a hospitalization in the Inselspital, University Hospital, Bern, Switzerland. Inclusion criteria were age of 70 years or older, multimorbidity (3 or more chronic medical conditions), and polypharmacy (use of 5 drugs or more for >30 days).

Written informed consent by patients themselves or, in the case of cognitive impairment by a legal representative, had already been obtained before enrolment. Patients planned for direct admission to palliative care (<24 hours after admission), or patients undergoing a structured drug review other than the trial intervention, or who had passed a structured drug review within the last 2 months were excluded. Patients for whom it was not possible to obtain an informed consent were excluded.

Outcome

The outcome was time to all-cause mortality over one year of follow-up. Information on death and relative date was collected by study nurses through follow-up calls or primary care physician contact.

Candidate predictors identification

Candidate 1-year mortality predictors were derived from previous research efforts in this field [8, 16], ease and reliability of measurement in clinical setting, and background knowledge on potential associations with mortality. We also considered factors included in the OPERAM dataset that may not be identified from the literature but are specific to multimorbid patients. All candidate predictors were based on baseline characteristics. We included demographic variables (age, sex), clinical characteristics (Charlson-Comorbidity-Index, number of drugs, body mass index, weight loss during the last year), smoking, functional status variables (Barthel-Index, falls, nursing home residence), and hospitalization. The variables about falls and hospitalization reflected the number of events during the last 12 months before index hospitalization. The Charlson-Comorbidity-Index was originally developed to predict mortality, with higher scores indicating higher mortality risk. It was calculated using ICD-10 codes by the adaptation of Quan et al. and implemented in the comorbidity library for the R environment [17, 18]. The Barthel-Index measures performance in activities of daily living (ADL) on an ordinal scale from 0 to 100 with higher scores indicating more independence. Continuous variables were categorised for ease of use at the point of care [19]. We based the categorisation decisions on the analysis of the frequency distribution of the variables and on clinical rationale.

Statistical analysis

We have calculated the required sample size for conducting our multivariable prediction model utilizing the criteria proposed by Riley et al. [20] and implemented in the pmsampsize library for the R environment [21]. The minimum sample size required with 12 candidate predictor parameters, an expected outcome event rate of 0.15 per year, and an anticipated Cox-Snell R2 of 0.126 (C statistic of [16] 0.82, [22]) is 799 with 10 events per predictor parameter. Our sample size of 822 is therefore adequate for this project.

The relationship between candidate predictors and outcome was analysed using a parametric Weibull regression model. We first performed a univariable analysis between each potential candidate predictors and the outcome. We then used multiple imputation (number of multiple imputations, m = 10) for missing values under a missing at random assumption in order to reduce bias and avoid excluding participants from the analysis [14]. We performed stepwise backwards variable selection based on the Akaike Information Criterion (AIC) in each imputed dataset. For automatic selection procedures, backward elimination is recommended in TRIPOD [13, 23]. We started with the full model, sequentially dropping variables to maximally reduce AIC. The final model was formed of predictors for which AIC cannot be minimized further and which appeared in all models of the imputed datasets. The final model was fitted in each imputed dataset and results pooled according to Rubin’s rules. We used the mice library for multiple imputation and pooling, and the MASS library with the stepAIC function for stepwise model selection via AIC in the R environment [24, 25].

We investigated the predictive accuracy of the final model by testing calibration and discrimination. The apparent performance and discrimination of the model was assessed with C statistic [14]. We evaluated potential overfitting and optimism by internal validation with bootstrapping techniques [14, 15]. We performed 500 bootstrap cycles. In each bootstrap sample, we derived a mortality prediction model and the relative risk score, as done in the original sample. We calculated optimism as difference in performance measure (C statistic) between the original sample and the respective bootstrap sample. This was repeated for all bootstrap samples to estimate the average optimism. We evaluated the calibration slope and intercept (calibration-in-the-large). We assessed graphical discrimination with a Kaplan-Meier plot of the risk groups.

Point-based risk score

From the final model, we derived a point-based risk score by assigning points to each risk factor. Each β coefficient was divided by the lowest β coefficient and rounded to the nearest integer. We calculated the total risk score for each participant by summing the points for each risk factor [16].

Sensitivity analyses

We performed a flexible parametric model to assess the robustness of the results obtained with the Weibull regression model. We performed a univariable analysis of the specific items in the Charlson-Comorbidity- Index and Barthel-Index. Further, to develop a score easier to use in practice, we performed the described methods above to develop and test a simplified model without the Barthel-Index and Charlson-Comorbidity-Index, because the assessment of both these indices can be difficult at the point-of-care.

Results

Among 822 participants, 805 were included in the analytical sample. We excluded 17 participants because they left the study, and most data were missing. Baseline characteristics of the participants are reported in Table 1. The mean (min to max) age of participants was 79.7 (70 to 99) years and 42% were women. During a mean (min to max) follow-up of 12.2 months (11.0 to 17.1), 158 participants (20%) had died. The proportion of missing data ranged from 0% to 10% in the predictor variables (S1 Table). The univariable analysis showed that age, Charlson-Comorbidity-Index, number of drugs, BMI, number of hospitalizations, and Barthel-Index were associated with 1-year mortality (S1 Table).

Table 1

Baseline characteristics of all participants and of those who died during follow-up.

Variables		Total (%)	Deaths (%)
		n = 805a	n = 158
Age	70–79	421 (52)	69 (44)
	80–99	384 (48)	89 (56)
Sex	Female	338 (42)	68 (43)
	Male	467 (58)	90 (57)
CC-Index	0–2	319 (40)	35 (22)
	≥3	486 (60)	123 (78)
Drugsb	<10	359 (45)	55 (35)
	≥10	446 (55)	103 (65)
BMI	<30	611 (76)	126 (80)
	≥30	164 (20)	22 (14)
Weight loss§	Yes	255 (32)	59 (37)
	No	545 (68)	98 (62)
Smoking	Yes	69 (9)	14 (9)
	No	733 (91)	144 (91)
Hospitalizationsc	0	386 (48)	63 (40)
	≥1	416 (52)	95 (60)
Barthel-Index	<21	34 (4)	18 (11)
	21–60	144 (18)	42 (27)
	61–90	237 (29)	53 (34)
	>90	377 (47)	42 (27)
Falls§	0	445 (55)	78 (49)
	1	174 (22)	31 (20)
	>1	182 (23)	49 (31)
Nursing home residence	Yes	70 (9)	15 (9)
	No	735 (91)	143 (91)

a17 participants excluded (from 822 to 805) due to premature study end.

bbefore index hospitalization.

cduring last 12 months.

The final risk prediction model included the predictors age, BMI, Charlson-Comorbidity-Index, hospitalizations, drugs, Barthel-Index, and nursing home residence (Table 2). We generated 10 imputed datasets by multiple imputation for missing data. The final predictor variables were retained in all imputed datasets after stepwise selection. Table 3 showed apparent and internal validation performance statistics of our risk prediction model. After adjustment for optimism with bootstrapping, our final model was able to discriminate participants with and without death within one year with a C statistic of 0.70. The optimism-corrected calibration slope was 0.93.

Table 2

1-year mortality predictors retained in the final model and associated risk score.

Variable		HR (95% CI)	β coefficienta	p-value	Risk score
Age	70–79	Ref	Ref
	80–99	1.30 (0.94–1.79)	0.26	0.01	1
CC-Indexb	0–2	Ref	Ref
	≥3	2.26 (1.55–3.31)	0.82	< .001	4
Drugs	<10	Ref	Ref
	≥10	1.22 (0.87–1.72)	0.20	0.25	1
BMIc	≥30	Ref	Ref
	<30	1.67 (1.08–2.60)	0.51	0.03	2
Hospitalizations	0	Ref	Ref
	≥1	1.28 (0.92–1.77)	0.24	0.15	1
Barthel-Index	>90	Ref	Ref
	61–90	12.00 (1.33–3.01)	0.69	< .01	3
	21–60	3.02 (1.96–4.65)	1.10	< .001	5
	<21	7.79 (4.46–13.61)	1.88	< .001	9
Nursing home residence		1.96 (1.13–3.42)	0.67	0.02	3

aβ coefficient = logHR.

bCharlson Comorbidity Index.

cBody-Mass-Index.

Table 3

Apparent and internal validation performance statistics of the final prediction model (with 95% CI) including C statistic, Calibration slope and Calibration-in-the-large.

Performance measure	Apparent	Average optimism	Optimism corrected
C statistic	0.71	0.02	0.70 (0.69–0.70)
C slope	1	0.07	0.93 (0.92–0.94)
CITL	0	-0.61	0.61 (0.56–0.66)

Sensitivity analyses

We found similar results by applying the flexible parametric model compared to the main analysis (S3 Table). Univariable analysis of the Charlson-Comorbidity-Index and the Barthel-Index showed the potential of including specific items instead of the entire indices in further projects (S4 Table). A simplified risk score without the Barthel-Index and Charlson-Comorbidity-Index was developed. The risk score points assigned to each of the final predictors in this simpler risk score are listed in Table 4. The simplified model had a weaker discrimination performance with a C statistic of 0.59 (Table 5).

Table 4

1-year mortality predictors retained in the simplified model and associated risk score.

Variable		HR (95% CI)	β coefficienta	p-value	Risks score
Age	70–79	Ref	Ref
	80–99	1.41 (1.02–1.94)	0.34	0.04	3
Drugs	<10	Ref	Ref
	≥10	1.53 (1.09–2.14)	0.43	0.01	3
BMIb	≥30	Ref	Ref
	<30	1.61 (1.03–2.50)	0.47	0.04	4
Hospitalizations	0	Ref	Ref
	≥1	1.39 (1.00–1.92)	0.33	0.05	3
Nursing home residence		1.13 (0.66–1.94)	0.13	0.65	1

aβ coefficient = logHR.

bBody-Mass-Index.

Table 5

Apparent and internal validation performance statistics of the simplified prediction model (with 95% CI) including C statistic, Calibration slope and Calibration-in-the-large.

Performance measure	Apparent	Average optimism	Optimism corrected
C statistic	0.60	0.02	0.59 (0.58–0.59)
C slope	1	0.13	0.87 (0.85–0.88)
CITL	0	-0.99	0.99 (0.75–1.24)

Point-based risk score

The risk score points assigned to each of the final predictors are listed in Table 2. A risk score was calculated for each participant by adding the points for each predictor present. For example, a 81-year-old (1 points) woman with a CC-Index of 3 (4 points), taking 5 drugs (0 points), with a BMI of 28 (2 points), 2 hospitalizations in the last 12 months (1 point), a Barthel-Index of 80 (3 points), and living in a nursing home residence (3 points) would have a total risk score of 14 points.

The mean 1-year mortality risk score in our sample was 7.7 (standard deviation 3.9); it ranged from 2 to 21. Based on the derived point-based risk score to predict mortality risk, 7% (95% CI: 6.3–7.7) of patients in the low-risk category (0 to 5 points) died, 19% (11.4–24.6) with moderate-risk (6 to 10 points), and 37% (25.2–44.8) with high-risk of 1-year mortality (>10 points) (S5A Table). Fig 1A showed the Kaplan-Meier plot of the three risk groups and good graphical discrimination.

Fig 1

A. Kaplan-Meier curves in three risk groups to visually assess separation (low risk: 0–5 points (green), moderate risk: 6–10 points (orange), high risk: 11–21 (red)). B. Kaplan-Meier curves of the simplified score in two risk groups to visually assess separation (low risk: 0–7 points (orange), high risk: 8–14 (red)).

Based on the simpler risk score to predict mortality risk, 13% (95% CI: 10.9–15.1) of patients in the low-risk category (0 to 7 points) died, and 25% (20.1–29.9) with high-risk of 1-year mortality (>7 points) (S5B Table). Fig 1B showed the Kaplan-Meier plot of two risk groups of this simplified risk score.

Discussion

We have developed and internally validated a new risk prediction model to estimate the 1-year mortality risk in older multimorbid adults. The final model included the seven predictors age, BMI, Charlson-Comorbidity-Index, number of hospitalizations, number of drugs, Barthel-Index, and nursing home residence. Using this score, we could classify patients into categories of increasing risk of 1-year mortality with a substantial risk difference. A simpler score was able to categorize 1-year mortality risk with a weaker discriminative power.

We used high-quality data from a large prospective cohort study of multimorbid older adults. In contrast to existing mortality risk indices, our index focuses specifically on multimorbid older patients and accounted for the severity of comorbidity (Charlson-Comorbidity-Index) and functional impairment (Barthel-Index) [16]. The interpretation of performance measures such as the C statistic depends on the clinical area. Other prognostic indices for 1-year mortality in older adults show similar discrimination performance, e.g. C statistic of 0.68 or 0.79 [26, 27]. We have applied a particularly robust methodological framework by following the research guidelines in this field, namely the Prognosis Research Strategy (PROGRESS) framework, and the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement. Notably, we ensured adequate sample size, used multiple imputation for missing data, and applied bootstrapping techniques for internal validation [13, 23].

One major limitation is the lack of external validation but we will explore opportunities to test the score in a different dataset of older multimorbid adults. However, we did our best to assess internal validation by using the bootstrap method for internal validation that makes use of the entire sample. This optimism-corrected validation analysis indicates that both calibration slope and intercept deviate from the null value of one (0.93; 95% CI: 0.92, 0.94) and zero (0.61; 95% CI: 0.56, 0.66) (Table 3), respectively. The calibration slope evaluates the spread of the estimated risks and has a target value of 1. A slope < 1 suggests that estimated risks are too extreme, a slope > 1 suggests that risk estimates are too moderate [28]. The calibration intercept, which is an assessment of calibration-in-the-large, has a target value of 0; negative values suggest overestimation, whereas positive values suggest underestimation [28]. Therefore, our results indicate suboptimal prediction accuracy as the risk estimates were considered too extreme (for the calibration slope) and the model underestimating the predicted risk (for the calibration intercept). As the study participants were included at the time of hospitalization, they may not be representative of all patients with multimorbidity, and this could reduce the transportability of our findings to other populations. Another limitation is that indices such as the Barthel-Index and the Charlson-Comorbidity-Index used as predictor may reduce the ease of use of the risk score at the point-of-care. We have therefore developed a simpler score without such variables. This simpler score showed reduced discriminative power compared to the full score. This could highlight the importance of taking the severity of comorbidities and functional impairment into account to improve risk prediction in older multimorbid people. Additional studies will be needed to test the usefulness of both scores in practice. Finally, prognostic information longer than 1-year mortality risk is needed. We will expand this model to 3-year mortality risk once the 3-year follow-up data collection is completed [10]. For this next research project, we might consider nomogram analysis as a graphically intuitive representation [29].

Our results will be useful for both clinical and research activities by helping health care providers to tailor preventive care according to the estimated mortality risk. Eventually, our study can help preventing under- and overuse of preventive care in the growing older population.

Missing data in the predictors in the 805 participants.

(DOCX)

Click here for additional data file.

Univariable analysis of candidate predictors with a Weibull model from information obtained at baseline.

(DOCX)

Click here for additional data file.

Multivariable analysis of candidate predictors with a flexible parametric survival model.

(DOCX)

Click here for additional data file.

Univariable analysis of the specific items in the Charlson Comorbidity Index and Barthel-Index.

(DOCX)

Click here for additional data file.

(DOCX)

Click here for additional data file.

A. Calibration plot of the final prediction model for 1-year mortality. B. Calibration plot of the simplified model for 1-year mortality.

(TIF)

Click here for additional data file.

15 Mar 2022

28 Apr 2022

April, 2022

RE: PONE-D-22-03428 – “Development and validation of a new prognostic index for mortality risk in multimorbid adults”

To the Editor,

We thank you for your interest and reviewing our manuscript. We are grateful for the editor’s and reviewers’ comments which helped us to clarify and improve our manuscript.

We have addressed all comments raised by the editor and the reviewers in the edited version of the manuscript and in our point-by-point responses to the reviewers' comments. All references quoted in our responses are listed at the end of this letter. We have uploaded a “Revised Manuscript with Track Changes” showing revisions using tracked changes.

We hope that the revised version of our manuscript will be suitable for publication in PLOS ONE.

We are looking forward to hearing from you,

Kind regards

Viktoria Gastens, MSc

Institute of Primary Health Care (BIHAM), University of Bern

Population Health Laboratory (#PopHealthLab), University of Fribourg

Email: viktoria.gastens@biham.unibe.ch

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

Authors’ response: We thank the editor for their comments below allowing us to improve our manuscript.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

Authors’ response: We have adapted the style of the manuscript according to the PLOS ONE formatting requirements.

2. Thank you for stating the following in the Funding Section of your manuscript:

"This work was supported by Swiss National Science Foundation grants (320030_188549/01 to AC; 325130_204361 to CDG). This work is part of the project “OPERAM: OPtimising thERapy to prevent Avoidable hospital admissions in the Multimorbid elderly” supported by the European Union's Horizon 2020 research and innovation programme under the grant agreement No 6342388, and by the Swiss State Secretariat for Education, Research and Innovation (SERI) under contract number 15.0137. The opinions expressed and arguments employed herein are those of the authors and do not necessarily reflect the official views of the EC and the Swiss government."

We note that you have provided funding information. However, funding information should not appear in the Funding section or other areas of your manuscript. We will only publish funding information present in the Funding Statement section of the online submission form.

Please remove any funding-related text from the manuscript and let us know how you would like to update your Funding Statement. Currently, your Funding Statement reads as follows:

"This work was supported by Swiss National Science Foundation grants (320030_188549/01 to AC; 325130_204361 to CDG; www.snf.ch). This work is part of the project “OPERAM: OPtimising thERapy to prevent Avoidable hospital admissions in the Multimorbid elderly” supported by the European Union's Horizon 2020 research and innovation programme under the grant agreement No 6342388 (ec.europa.eu/programmes/horizon2020), and by the Swiss State Secretariat for Education, Research and Innovation (SERI) under contract number 15.0137 (www.sbfi.admin.ch/sbfi/en/home.html). The opinions expressed and arguments employed herein are those of the authors and do not necessarily reflect the official views of the EC and the Swiss government. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript."

Please include your amended statements within your cover letter; we will change the online submission form on your behalf.

Authors’ response: We have deleted the funding section in the manuscript. The funding information in the Funding Statement section of the online submission form is correct.

3. We note that you have indicated that data from this study are available upon request. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly. For more information on unacceptable data access restrictions, please see http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions.

In your revised cover letter, please address the following prompts:

a) If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially sensitive information, data are owned by a third-party organization, etc.) and who has imposed them (e.g., an ethics committee). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent.

b) If there are no restrictions, please upload the minimal anonymized data set necessary to replicate your study findings as either Supporting Information files or to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories.

We will update your Data Availability statement on your behalf to reflect the information you provide.

Authors’ response: We have updated the Data Availability statement accordingly: “This study involves human research participant data containing sensitive patient information. In the EU Horizon 2020 grant agreement for the OPERAM study, it had been specified that the data will be made available upon request if the use has been approved by an ethical committee. Therefore, restrictions to make the underlying data directly publicly available are both due to legal and ethical reasons, as health data are sensitive data.

Data for this study will be made available for scientific purposes upon request for researchers whose proposed use of the data has been approved by the OPERAM publication committee. After approval and signing of a data transfer agreement ensuring adherence to privacy and data handling, data and documentation will be made available through a secure file exchange platform. Partially deidentified participant data, a data dictionary and annotated case report forms will be made available. For data access, external researchers can fill in the contact form on https://www.operam-cohort.biham.ch/ or operam-2020.eu.”

4. Your ethics statement should only appear in the Methods section of your manuscript. If your ethics statement is written in any section besides the Methods, please move it to the Methods section and delete it from any other section. Please ensure that your ethics statement is included in your manuscript, as the ethics statement entered into the online submission form will not be published alongside your manuscript.

Authors’ response: We have moved the ethics statement to the Methods section of our manuscript.

5. Please include captions for your Supporting Information files at the end of your manuscript, and update any in-text citations to match accordingly. Please see our Supporting Information guidelines for more information: http://journals.plos.org/plosone/s/supporting-information.

Authors’ response: We have adapted the Supporting Information files and citations accordingly.

6. Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

Authors’ response: We have reviewed our reference list. We have added references to: “Royston and Sauerbrei; Moons 2015”. This has led to a change in the order of references.

Reviewer 1

The authors sought to develop a risk assessment tool to predict 1-year mortality risk for elderly people with multimultimorbidity (i.e., 3+ chronic health diseases) and polypharmacy (i.e., use of 5+ drugs for 30+ days). This is a secondary analysis of 805 relevant participants from the OPERAM study, a European multicenter, cluster randomized controlled trial. Multiple imputation was performed for missing data, ranging from 0% to 10% under the assumption of missing at random. The predictor candidates were predefined and a backward stepwise selection was conducted to identify the optimal predictive model. Internal validation was carried out using bootstrapping techniques. The optimal model had the optimism-corrected C index of 0.70 and calibration slop of 0.93. A simpler model, excluding Charlson Comorbidity Index and Barthel Index with C index of 0.59 (95% CI: 0.58, 0.59) was considered of “satisfactory discriminative power”.

The manuscript is overall informative and well written. The study was well designed and executed, and the analysis was conducted appropriately. There are however several issues the authors might wish to consider in the subsequent submission.

Authors’ response: We thank the reviewer for their comments below allowing us to improve our manuscript.

7. Please clarify why a backward stepwise selection was used to identify the optimal selection model whereas the other methods, including Bayesian model averaging (BMA) method that have been demonstrated to be statistically more robust are widely available. Wang et al (https://pubmed.ncbi.nlm.nih.gov/15505893/) and Genell et al (https://pubmed.ncbi.nlm.nih.gov/21134252/) have found BMA was superior to stepwise method in model selection. Selection of a robust statistical method can be an important discussion issue to convince the findings were rigorous and accurate.

Authors’ response: We agree with the reviewer that stepwise variable selection can have some disadvantages such as instability of selection or biased estimation. However, according to the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) statement, backwards elimination is still recommended as an automatic selection procedure. [Royston and Sauerbrei 2008; TRIPOD 2015] We have added to our Methods section: “If using automatic selection procedures, backward elimination is recommended in TRIPOD.”

8. It is acceptable to report a calibration slope to assess the predictive performance of a prediction model. As a calibration slop evaluates the spread of the estimated risk, the manuscript should also report the calibration intercept, known as the “calibration-in-the-large” index which quantifies the overall calibration performance or the difference between the average predicted risk and the overall event rate.

Authors’ response: We agree with the reviewer and have added the calibration intercept to the Results section in Table 3A and 3B.

9. It is also acceptable to derive a point-based risk score by assigning points to each predictor in the optimal model. However, the authors might wish to consider a more robust and friendly alternative approach, such as a nomogram analysis. Unlike a point-based risk score, a nomogram provides more graphically intuitive and more friendly risk estimation for subjects with specific risk profiles. A nomogram also provides a practical solution for the implementation progress and makes the manuscript more appealing.

Authors’ response: We agree with the reviewer that nomogram analysis is a graphically intuitive representation. However, nomograms represent models with continuous variables. Due to the binary and categorical characteristics of our predictor variables, a nomogram analysis is not applicable. For the next part of this research project (extending to 3-year mortality risk prediction), we plan to keep variables continuous wherever possible and could therefore apply a nomogram analysis. [Gastens 2021]

10. A simpler model without Charson-Comorbidity index and Barthel index is considered satisfactorily discriminative though its C index was only 0.59. Please provide explanation and cite valid references for this statement as it is practically a bit hard, especially for people working in daily clinical practice to consider a discrimination index of 0.59 satisfactory. Further discussion is definitely needed.

Authors’ response: We agree with the reviewer and revised the text as follows: “A simpler score was able to categorize 1-year mortality risk with a weaker discriminative power.” and “This simpler score showed reduced discriminative power compared to the full score.”

11. The authors are expected to expand the Discussion section. For instance, they might wish to discuss (i) the robustness of the methods used in this study in comparison with previous studies or other methods, (ii) the results, including but not limited to a “satisfactory discriminative power” of 0.59, and (iii) potential implication of the model.

Authors’ response: We have expanded our Discussion section about the robustness of the methods: “We have applied a particularly robust methodological framework by following the research guidelines in this field, namely the Prognosis Research Strategy (PROGRESS) framework, and the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement. Notably, we ensured adequate sample size, used multiple imputation for missing data, and applied bootstrapping techniques for internal validation.” We further discuss the results: “In contrast to existing mortality risk indices, our index focuses specifically on multimorbid older patients and accounted for the severity of comorbidity (Charlson-Comorbidity-Index) and functional impairment (Barthel-Index).” We discuss potential implications: “Our results will be useful for both clinical and research activities by helping health care providers to tailor preventive care according to the estimated mortality risk. Eventually, our study can help preventing under- and overuse of preventive care in the growing older population.”

12. Minor issues:

The authors might wish to make their study rationale stronger and more appealing. For instance, they might add a sentence or two to extrapolate the reasons why several of 16 prognostic indices were “fairly accurate to predict mortality, the authors concluded that none could be recommended for a widespread use”.

Authors’ response: We agree with the reviewer and have strengthened our Introduction section: “One major limitation is that none of these prognostic indices has been tested prospectively in various samples. Key is that their transportability in other populations is unknown and clinicians cannot use these indices with confidence across different groups of patients.”

Reviewer 2

Important topic, clear rationale of the study. I would like to congratulate the authors with a clearly written, and concise paper. I do have a few comments:

Authors’ response: We thank the reviewer for their comments below allowing us to improve our manuscript.

13. -the statistical analyses are sophisticated, and can be considered state-of-the-art in the field of health outcome predictions

Authors’ response: Thank you.

14. -I do miss an independent population to assess external validity of the model

Authors’ response: We agree with the reviewer and have stated in the Discussion section: “One major limitation is the lack of external validation but we will explore opportunities to test the score in a different dataset of older multimorbid adults. We did however our best to assess internal validation by using the bootstrap method for internal validation that makes use of the entire sample.”

15. -The paper could be further strengthend by comparing performance of the model with other (simple) models that predict mortality. Is this model really more appropriate than those currently available?

Authors’ response: We are not sure to understand this comment, because we have compared our model with a simplified one and because there are no other models currently available. These points have been discussed in the Introduction and Discussion section of the paper.

16. -as the authors do acknowledge in the discussion, assessing performance in an extended time horizon beyond 1-year would be of interest

Authors’ response: We agree with the reviewer. As discussed in the Discussion section, the follow-up of our cohort study is still ongoing and we are currently working on extending the prognostic model to 3-year mortality risk: “Finally, prognostic information longer than 1-year mortality risk is needed. We will expand this model to 3-year mortality risk once the 3-year follow data collection is completed. [Gastens 2021]”

Comments from authors

During the revision process, we noticed a slight error in the calculation of the Charlson-Comorbidity-Index. We adapted the results accordingly. This did not change the findings in a substantial way or interpretation of the results.

References

Moons, Karel GM, et al. "Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration." Annals of internal medicine 162.1 (2015): W1-W73.

Royston, Patrick, and Willi Sauerbrei. Multivariable model-building: a pragmatic approach to regression anaylsis based on fractional polynomials for modelling continuous variables. Vol. 777. John Wiley & Sons, 2008.

Gastens, Viktoria, et al. "Development and validation of a life expectancy estimator for multimorbid older adults: a cohort study protocol." BMJ open 11.8 (2021): e048168.

26 May 2022

8 Jul 2022

Reviewer 1

I would like to thank the authors for their efforts to address my concerns. There are however two issues that I appreciate their clarification.

1. Please explain and cite relevant references for your response “However, nomograms represent models with continuous variables. Due to the binary and categorical characteristics of our predictor variables, a nomogram analysis is not applicable.”. The authors might find in the current literature that the nomograms with binary or categorical predictor variables are common in many different research fields, including but not limited to oncology (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4465353/), bone (https://pubmed.ncbi.nlm.nih.gov/17370100/) and other research fields (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7906717/; https://pubmed.ncbi.nlm.nih.gov/31132089/ ).

If the authors decide not to present the nomogram which they agree to be graphically intuitive, they might wish to add a sentence or two in the Discussion section to mention the nomogram analysis might be considered in the extended project.

Authors’ response: We agree with the reviewer and have added to the Discussion section “We will expand this model to 3-year mortality risk once the 3-year follow-up data collection is completed. [Gastens 2021] For this next research project, we might consider nomogram analysis as a graphically intuitive representation. [Balachandran 2015]”.

2. The optimism corrected validation analysis using bootstrapping technique indicated both calibration slop and intercept significantly deviated from the null value of one (0.93; 95% CI: 0.92, 0.94) and zero (0.61; 0.56, 0.66) (Table 3A), respectively. The far less accurate calibration was noticed for the simplified model in Table 3B.

These findings indicated the significant suboptimal prediction accuracy of these prediction models as the risk estimates were considered too moderate (for the calibration slop) or the model significantly underestimated the predicted risk (for the intercept) (https://pubmed.ncbi.nlm.nih.gov/31842878/). Regardless of their importance and significant, these results have not been discussed thoroughly in the current manuscript.

The authors are expected to discuss sufficiently about the significantly suboptimal calibration of the prediction models. They might also wish to discuss the potential impact of this suboptimal prediction accuracy on the implementation of the model and future projects. Additionally, the authors might wish to cross check the assumption for calculating the 95% CI of the calibration intercept given its upper limit of 1.24 (Table 3B).

Authors’ response: We agree with the reviewer and have added to our Discussion section: “This optimism-corrected validation analysis indicates that both calibration slope and intercept deviate from the null value of one (0.93; 95% CI: 0.92, 0.94) and zero (0.61; 95% CI: 0.56, 0.66) (Table 3A), respectively. The calibration slope evaluates the spread of the estimated risks and has a target value of 1. A slope < 1 suggests that estimated risks are too extreme, a slope > 1 suggests that risk estimates are too moderate. [Van Calster 2019] The calibration intercept, which is an assessment of calibration-in-the-large, has a target value of 0; negative values suggest overestimation, whereas positive values suggest underestimation. [Van Calster 2019] Therefore, our results indicate suboptimal prediction accuracy as the risk estimates were considered too extreme (for the calibration slope) and the model underestimating the predicted risk (for the calibration intercept).”.

Additionally, we checked the calculations for the 95% CI of the calibration intercept and confirm its upper limit of 1.24 (Table 3B).

3. It is statistically very hard to interpret the simplified prediction model had “slightly weaker discrimination power” as its C index (0.59; 95% CI: 0.58, 0.59) appeared to be significantly lower than the “full” prediction model (0.70; 0.69, 0.70). Similar results were found for both calibration intercept and slop. These metrics instead the simplified prediction model had significantly weaker prediction performance, in terms of both discrimination and calibration power than the full prediction model. As a result, the authors might also wish to revise the interpretation related to the simplified prediction model accordingly and discuss its potential implication further.

Authors’ response: We agree with the reviewer and have adapted our Discussion section to “Another limitation is that indices such as the Barthel-Index and the Charlson-Comorbidity-Index used as predictor may reduce the ease of use of the risk score at the point-of-care. We have therefore developed a simpler score without such variables. This simpler score showed reduced discriminative power compared to the full score. This could highlight the importance of taking the severity of comorbidities and functional impairment into account to improve risk prediction in older multimorbid people.”.

11 Jul 2022

Development and validation of a new prognostic index for mortality risk in multimorbid adults

PONE-D-22-03428R2

Dear Dr. Gastens,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Robert Daniel Blank, MD, PhD

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: I would like to thank the authors for their efforts to address my concerns and make the manuscript better.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

**********

27 Jul 2022

PONE-D-22-03428R2

Development and validation of a new prognostic index for mortality risk in multimorbid adults

Dear Dr. Gastens:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Professor Robert Daniel Blank

Academic Editor

PLOS ONE