Prediction Models for Physical, Cognitive, and Mental Health Impairments After Critical Illness: A Systematic Review and Critical Appraisal.

OBJECTIVES: Improved ability to predict impairments after critical illness could guide clinical decision-making, inform trial enrollment, and facilitate comprehensive patient recovery. A systematic review of the literature was conducted to investigate whether physical, cognitive, and mental health impairments could be predicted in adult survivors of critical illness. DATA SOURCES: A systematic search of PubMed and the Cochrane Library (Prospective Register of Systematic Reviews ID: CRD42018117255) was undertaken on December 8, 2018, and the final searches updated on January 20, 2019. STUDY SELECTION: Four independent reviewers assessed titles and abstracts against study eligibility criteria. Studies were eligible if a prediction model was developed, validated, or updated for impairments after critical illness in adult patients. Discrepancies were resolved by consensus or an independent adjudicator. DATA EXTRACTION: Data on study characteristics, timing of outcome measurement, candidate predictors, and analytic strategies used were extracted. Risk of bias was assessed using the Prediction model Risk Of Bias Assessment Tool. DATA SYNTHESIS: Of 8,549 screened studies, three studies met inclusion. All three studies focused on the development of a prediction model to predict (1) a mental health composite outcome at 3 months post discharge, (2) return-to-pre-ICU functioning and residence at 6 months post discharge, and (3) physical function 2 months post discharge. Only one model had been externally validated. All studies had a high risk of bias, primarily due to the sample size, and statistical methods used to develop and select the predictors for the prediction published model.
CONCLUSIONS: We only found three studies that developed a prediction model of any post-ICU impairment. There are several opportunities for improvement for future prediction model development, including the use of standardized outcomes and time horizons, and improved study design and statistical methodology.

Survivors of critical illness frequently experience new or worsening physical, cognitive, and/or mental health impairments lasting beyond hospital discharge, recognized as the postintensive care syndrome (PICS) (1). There are currently no proven strategies to reduce the prevalence and severity of multiple impairments within PICS (2), yet the population of survivors is growing (3). A recent large, observational study reported that 64% and 56% of their cohort experienced one or more impairments in the domains of PICS, at 3 and 12 months, respectively (4). Such impairments prevent individuals from returning to their previous familial roles and impact return to employment (5, 6). Ensuing morbidity has broad public health and societal implications (7).

There is a growing urgency to recognize and detect these post-ICU impairments. Survivors of critical illnesses encounter fragmented care as they transition from the ICU to posthospital settings and care in the community in the months after hospital discharge (8–10). Due to the complex and multifaceted care needs of ICU survivors, multiple clinicians are involved in their care, with variable knowledge and abilities to recognize, detect, and manage PICS-associated sequelae (11). Therefore, accurate prediction of future problems following discharge from the ICU or hospital is needed to triage patient care and helps ensure survivors receive the right care, by the right clinician, at the right time in their recovery trajectory. It is particularly important to identify those most at-risk of developing long-term and/or severe impairments to inform clinical decision-making and care coordination. However, the ability to predict impairments in the physical, cognitive, and/or mental health domains of PICS is currently unknown.

In May 2019, the Society of Critical Care Medicine (SCCM) convened a 1-day, state-of-the-art meeting with international and interprofessional experts, to evaluate the literature and reach consensus on key questions relating to PICS (see related article—Mikkelsen et al, in press). We report the results of one aim of this meeting to conduct a systematic review of statistical models for predicting impairments in physical, cognitive, and/or mental health status in adult ICU survivors.

MATERIALS AND METHODS

This systematic review was registered on International Prospective Register of Systematic Reviews (CRD42018117255) and conducted and reported according to the Preferred Reporting Items for Systematic Reviews guidelines (12). For the purpose of this review, we distinguished between publications that could be considered “prediction model” studies, rather than “predictor finding” (also called “prognostic factor”) studies (13). This distinction is based on the goal of a prediction model to use multiple variables, in combination, to estimate patient-specific probabilities of the likelihood of a future health state (14). Comparatively, prognostic factor (predictor finding) studies aim to identify individual predictors’ (such as age, disease stage, or biomarkers) independent contribution to the prediction of a diagnostic or prognostic outcome (14). For purposes of this review, we defined a prediction model publication as a study where a model, score, or formula, comprising the combination of two or more predictors, was reported.

Search Strategy and Sources

PROSPERO was initially searched to identify whether a systematic review on this topic was registered. Thereafter, an experienced medical information specialist (B.S.) iteratively developed and tested a systematic and comprehensive literature search strategy in consultation with the research team. The strategy was peer reviewed by another senior information specialist prior to execution using the Peer Review of Electronic Search Strategies (PRESS) Checklist (15). The searches were performed in PubMed and the Cochrane Library (Wiley platform) on December 8, 2018, and updated on January 6, 2019.

For the purposes of this review, we were not focused on predicting PICS but, rather, impairments in its constituent domains (physical, cognitive, and/or mental health). The searches used a combination of controlled vocabulary (e.g., “Intensive Care Units”, “Survivors”, “Mass Screening”) and keywords (e.g., “critical care”, “post-ICU”, “screen”). Vocabulary and syntax were adjusted across databases. There were no language restrictions, but results were limited to the last 10 years to identify contemporary papers, consistent with the introduction of the term PICS in a 2012 publication. Publications that were nonhuman or not original research (e.g., animal studies, editorials) were removed from the PubMed results. Specific details regarding the strategies appear in Supplemental Table 1 (Supplemental Digital Content 1, http://links.lww.com/CCM/F853).

We also identified three related systematic reviews and on January 20, 2019, searched PubMed for the references cited in each, as well as for records related to or citing any of these reviews.

Study Selection and Screening

Titles and abstracts were assessed against eligibility criteria (Supplemental Table 2, Supplemental Digital Content 1, http://links.lww.com/CCM/F853) with each title and abstract undergoing independent review by two of the four reviewers (K.J.H., E.H., N.L., J.M.). Full-text articles were retrieved where the abstract contained insufficient information to assess eligibility. Full-text articles were independently reviewed by two of the four reviewers, against eligibility criteria (Supplemental Table 2, Supplemental Digital Content 1, http://links.lww.com/CCM/F853). Discrepancies were resolved by consensus between the reviewers, with any remaining conflicts resolved by an independent reviewer (M.O.H.).

Data Extraction (Including Risk of Bias)

A prepiloted, standardized form was used for data extraction by a single reviewer (E.H.) and independently cross-checked by a second reviewer (K.J.H.). Data items included the following: 1) study characteristics; 2) outcome measurement (timing of measurement, tools used); and 3) prediction model descriptors.

Three independent assessors (M.S. G.S.C., M.O.H.) reviewed each article, using the Prediction model Risk Of Bias Assessment Tool (PROBAST) (14) to evaluate risk of bias for the included studies. The PROBAST was designed to assess studies that develop, validate, or update multivariable prediction models for diagnosis or prognosis. The assessment consists of four domains containing 20 signaling questions to facilitate recognition of both the risk of bias and applicability of a study (16). The four domains focus on 1) “Participants”—the data sources used and how participants were selected for enrollment into the study, 2) “Predictors”—definition and measurement of the predictors, 3) “Outcome”—definition and determination of the outcome, and 4) “Analysis”—whether key statistical considerations were correctly addressed. Domain 4 relates to how the studies addressed key statistical considerations related to prediction model development. This domain does not have an applicability section and is only assessed for risk of bias.

For the overall grading for each study, we used the criteria detailed in the PROBAST “Explanation and Elaboration” document (16). Specifically, if the evaluation of a prediction model was judged as low on all domains relating to bias and applicability, the study was rated as “low risk of bias” or “low concern regarding applicability”. If an evaluation resulted in a rating of “high” for at least one domain, it was rated as “high risk of bias” or “high concern regarding applicability” accordingly. If the prediction model evaluation was unclear for one or more domains, but rated as low in the remaining domains, it was rated as “unclear risk of bias” or “unclear concern regarding applicability.”

RESULTS

Study Selection

The search retrieved 8,806 citations. After removal of duplicates, 8,549 titles and abstracts were screened for eligibility, with 1,049 meeting eligibility criteria. After screening the full-text articles, three studies were eligible and included (Fig. ). The top three reasons for exclusion were as follows: No prediction of impairments in physical, cognitive, or mental health (n = 551); study design did not meet our inclusions (n = 189); or classified as a prognostic factor study, as defined above (n = 125).

Preferred Reporting Items for Systematic Reviews flow diagram—selection of articles.

Description of Included Studies

Characteristics of the three included studies, along with the outcome measurement used, are presented in Tables and and Supplemental Table 3 (Supplemental Digital Content 1, http://links.lww.com/CCM/F853). Two of the studies predicted physical function after critical illness, whereas the remaining study predicted mental health. Notably, none of the included studies predicted cognitive function after hospital discharge.

Milton et al (17) in 2018 conducted a prospective, multinational study (n = 572) across 10 general ICUs in secondary and tertiary care hospitals in Sweden, Denmark, and the Netherlands, in a mixed cohort, of middle-aged patients. The prediction model they developed focused on the mental health domain of PICS—specifically anxiety, depression, and posttraumatic stress. The model was developed to predict individual patients’ risk for psychologic problems at 3 three months after ICU discharge. The final predictors in their model included the following: age, lack of social support, traumatic memories, and symptoms of depression assessed at ICU discharge. The intended time point of application for the model was ICU discharge to identify high-risk patients that might require post-ICU follow-up.

Detsky et al (18) in 2017 conducted a prospective, multisite study (n = 303) across three university-affiliated hospitals in Philadelphia, United States, in medical and surgical cohorts of mostly male, middle-aged patients. The prediction model developed focused on the physical domain of PICS and identified important predictors of return to baseline function (6 mo status, compared with pre-ICU admission). In the model, predictors for not returning to baseline function included age, medical (vs surgical) patient, non-White race, higher Acute Physiology and Chronic Health Evaluation III score, hospitalization in prior year, and past history of cancer, liver disease, neurologic condition, or any type of transplantation. The model used easily accessible factors that could be obtained on day 1 of the ICU admission.

Schandl et al (19) in 2014 conducted a prospective, quasiexperimental, single site study (n = 258) in Stockholm, Sweden, in a mixed cohort of middle-aged patients. The predictive model focused on physical disability as the PICS domain of interest. In the final model, the four predictors included are low educational level, impaired core stability, fractures, and an ICU length of stay of more than 2 days. The intended timepoint for application of this model was ICU discharge to help clinicians identify patients at risk of ongoing physical disability and who might warrant ongoing support

Risk of Bias and Applicability (PROBAST) Assessment

The overall and domain specific gradings for risk of bias and applicability are reported in Table . All studies received a rating of high risk of bias, principally due to issues in the analysis domain 4, related to the statistical models used to develop the prediction model.

Domain 1: Participant Selection

All studies selected a diverse critically ill population, and the PROBAST assessment led to the classification of all three studies having a low risk of bias and low concern regarding applicability related to the participants that were selected.

Domain 2: Predictors

All studies had a low risk of bias rating related to the predictors used in the prediction model development. We chose the rating of “high” concern for applicability for the study by Milton et al (17) as there was concern about the ability to capture the modified patient health questionnaire (PHQ)–2 as a predictor at the time of ICU discharge. Indeed, in the supplement they report that the PHQ-2 total score was missing in 11% of responders.

Domain 3: Outcome

All studies had a low concern or risk regarding their methodology in addition to the applicability of the prediction model outcome for use in clinical practice and/or other studies. However, only the study by Milton et al (17) received a low ranking for risk of bias. One concern about the outcome used by Schandl et al (19) and Detsky et al (18) was the reliance on self-reported measures of physical function (e.g. activities of daily living [ADL]), via the patient or caregiver, without inclusion of performance-based measures performed under standardized conditions (20–22). Additionally, the latter study only evaluated a high-level mobility activity (the ability to ambulate up 10 stairs independently) and one basic ADL (toilet independently). These functional activities require different physical skills; furthermore, a range of disability within these two extremes may not be captured.

Domain 4: Analysis

All studies received a rating of high concern for risk in the analysis domain. The key concern in all studies related to the sample size and the ratio of the number of outcome events to the total number of candidate predictors, termed “events per variable” (EPV). There is ongoing discussion about when this ratio becomes a concern, as discussed in section 4.1 of PROBAST “Explanation and Elaboration” document (16). Studies with EPVs lower than 10 may be considered at high risk of creating an “overfitted” model or failing to include important predictors (16). All studies had a EPV ratio that resulted in a concern for potential bias: Milton et al (17) EPV equals to 4.4 (80 participants with psychologic problems/18 candidate predictors), Detsky et al (18) EPV equals to 4.1 (91 participants returned to baseline/22 candidate predictors), and Schandl et al (19) EPV equals to 3.0 (69 participants had new-onset physical disability/23 candidate predictors). Each study had additional signaling questions that supported the ranking of high risk of bias (supplemental materials, Supplemental Digital Content 1, http://links.lww.com/CCM/F853).

DISCUSSION

Prediction of ICU survivors’ outcomes is important to patients, families, clinicians, and for healthcare delivery. However, based on our systematic review, the availability of rigorous prediction models is limited, and more work is required to advance this field. Specifically, our systematic review identified only three studies that developed a model to predict long-term impairments. Of the three, one predicted mental health, two predicted physical function, and none predicted cognitive function after critical illness. Further, the development of the prediction model in each study was deemed to be at a high risk of bias, and to our knowledge, these models have not been externally validated at the time of this review (although from personal communication with the authors, an external validation has been completed for Schandl et al [19], and is under peer review). We also identified issues related to the definition and measurement of outcomes and the reliance on self-reported measures recognizing that performance-based measures are also important to capture health states in critical care research, although both can be prone to measurement error.

Several aspects of clinical care could benefit from future advancements in the prediction of postdischarge impairments. At discharge from the ICU, patients and families might be better informed about potential impairments across the arc of care. A current challenge is that patients and families are often underprepared for what to expect post-ICU discharge, with a recent survey reporting that approximately one third of ICU physicians rarely formally discuss post-ICU challenges with patients and families (23). The reasons for this finding are likely multifactorial; however, a lack of evidence regarding the prediction of PICS outcomes may be a contributing factor.

PICS-related problems are commonly considered within post-ICU programs such as peer support groups (24, 25) and follow-up clinics (26–28). However, one of the common barriers to implementing these programs is identifying which patients are likely to benefit from attending (29). Given issues with the cost and sustainability of such programs, it is crucial to identify the most suitable target population. Once these patients have returned to their communities, they are likely to be managed by their primary care doctor who may have variable exposure to and knowledge of PICS-related problems. Yet, little is known about how to best transition survivors back to primary care (30). Further, patterns of recovery can be variable among survivors where some might experience an initial improvement in their health, followed by a later decline (31).

This review, designed as part of the SCCM’s International Consensus Conference on Prediction and Identification of Long-Term Impairments after Critical Illness (32), has important clinical implications for bedside clinicians. First, patients and families care greatly about post-ICU impairments such as physical function and often want to know an estimate of long-term functional outcomes following critical illness (31, 33). This review highlighted the limited evidence-base with which to predict such outcomes. When providing anticipatory guidance to patients and their family in the ICU, and thereafter, clinicians should acknowledge the uncertainty with which post-ICU impairments can be predicted currently. Although there is no one strategy to reliably predict PICS-related problems, clinicians can use the approach recommended by the consensus conference to risk-stratify and screen survivors at high risk for long-term cognitive, mental health, and physical impairments after critical illness. Therein, factors before (e.g., pre-exiting functional impairments), during (e.g., delirium), and after (e.g., early psychologic symptoms) can be used to identify patients at high risk for long-term functional impairments.

Second, there is growing interest in the critical care community to improve ICU survivorship and address post-ICU impairments. Despite the lack of supporting evidence (24, 26), one key intervention has been the establishment of post-ICU recovery programs globally (27, 28, 34). This review, along with the international consensus conference recommendations (32), can be used to inform the identification of patients who are more likely to encounter new problems or worsening of pre-existing problems following ICU discharge who may benefit from structured recovery programs. It can also aid the identification of survivors in need of referral to specialist clinicians and services (e.g., rehabilitation). Third, there is a need to develop rigorous prediction models to better predict post-ICU impairments that can be used to more robustly inform clinical decision-making as described above. This review summarizes the current state of the literature and provides impetus to advance the science, clearly setting out the methodological groundwork to do so.

To help centralize guidance in this nascent research area, we have developed a summary table (Table ) to provide researchers a centralized resource to guide the development of future prediction model studies using current best practices. These recommendations borrow heavily from the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis statement (35, 36) and PROBAST statement (14, 16). We have also embedded several references in Table 4 to provide methodological details that could help inform researchers developing prediction study funding applications. We specifically direct researchers to recently published prediction model sample size guidance based on the type of outcome being predicted—continuous (16, 37), binary (38, 39), or time-to-event (38). The concern regarding low EPV in domain 4 of the PROBAST assessment stems largely from too small sample sizes, which may have resulted from a lack of clear sample size guidance at the time of their design.

TABLE 4.

Recommended Approaches and Considerations for Researchers When Planning and Reporting the Development of a New Prediction Model

Domains	Key Design and Reporting Considerations
Data source	• Clarify how the data used to develop the prediction model were collected.
Participants	• Clearly report the inclusion and exclusion criteria for individuals included in the study.
	• Provide descriptive summaries of participant characteristics used for internal (and if relevant, external) validation.
	• The targeted sample size should be determined by considering the number of subjects relative to the number of predictor parameters for potential inclusion in the prediction model (i.e., events per variable). For sample size guidance, we direct readers to methodological papers based on the type of outcome being predicted—continuous (16, 37), binary (38, 39), or time-to-event (38).
Outcome	• Specific details should be provided regarding how the outcome was defined, including what information was used to create the outcome variable, at what precise time the outcome was collected or measured during follow-up, and what differences, if any, in how this information was captured for individuals in the study.
Predictors	• Provide a precise definition of the predictor variables included in the final model, along with the method(s) by which these variables were selected.
Missing data	• Report how much data were missing from the predictors and from the outcome.
	• A plan for management of missing data should be developed prior to model development.
Model development	• Univariable selection using p thresholds should be avoided and in general, data-driven variable selection should be done with some conservatism (e.g., accepting higher p thresholds for variable inclusion).
	• Avoid dichotomizing and categorizing continuous predictors and consider other methods (e.g., restricted cubic splines or fractional polynomial methods) to model nonlinear relationships.
Model performance	• Performance measures (40) should be tailored to the intended purpose of the model but generally be visualized when possible and include a measure of discrimination (e.g., area under receiver operator characteristics curve or area under precision recall curve), calibration slope and intercept (e.g., calibration curve [41, 42]), and clinically relevant performance (e.g. decision curve [43]). Avoid performance measures based on arbitrary chosen thresholds (44).
	• CIs should be reported for performance measures.
Model specification	• Report the model that was used to develop the statistical model (e.g., logistic regression, Cox proportional hazards model).
	• The full model equation should be reported when applicable (e.g., intercept or the baseline hazard function in a time-to-event model).
Model Validation	• Variable selection should be repeated using bootstrap methods.
	• Internal validation is an essential component of model development. When possible, conduct external validation. Random splitting for internal validation is not generally a recommended practice (45). Model performance should be assessed by time, geography, center/site, and other key factors (46).
	• Optimism correction should be conducted and reported.
Additional Considerations	• In prediction studies focusing in individuals who survive a critical care admission, many will die within the selected time horizons (e.g., 3–12 mo following discharge). This creates a problem whereby a patient is at risk for multiple events, but only one of these events is of interest (i.e., competing risks). If mortality is not part of the predicted outcome state, more complex methods may need to be considered when developing a prediction model (16).
	• Authors should consider issues related to measurement error (47, 48), fairness (49–54), and how a model would be assessed in a risk of bias assessment (i.e., Prediction Model Study Risk Of Bias Assessment Tool [16]) when constructing their model.

For additional details, authors are directed to the embedded citations and to the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis publication (35) that contains an accompanying checklist that informed this table. In addition to optimally support prediction model replication and extension, we recommend that authors make their programming code (particularly in the case of machine learning models) available in full. In addition, whenever possible, the data used should be made publicly available to the research community.

There are potential limitations to our study. First, it is possible that our inclusion criteria or search did not identify some prediction model articles that have been published. However, omissions are unlikely to have been systematic or frequent, as we are aided by the knowledge of literature by content experts in the working group. Second, due to resource constraints, our search only included two literature databases. However, we hand searched the reference list of the three articles included, as well as the three identified and related systematic reviews. Finally, we restricted our focus to the prediction of physical, cognitive, and mental health outcomes in adult ICU survivors. There may be prediction models for other outcomes or for pediatric populations that are not examined here.

In conclusion, our systematic review of prediction models for physical, cognitive, and mental health impairments in adult survivors of critical illness identified only three published studies. At the present time, there is no published external validation of these models. The design and evaluation of methodologically rigorous prediction models is important to assess for potential benefit on clinical decision-making, patient triage, and communication with survivors of critical illness and their families.

ACKNOWLEDGMENTS

We thank Kaitryn Campbell, MLIS, AHIP (St. Joseph’s Healthcare Hamilton, Hamilton, ON) for peer review of the PubMed search strategy. We would like to thank Lynn Higgins and Western Health Library Services for their contribution in sourcing full-text articles for this review.

TABLE 1.

Characteristics of Included Studies

References	Location	Design	Cohort, n	Data Source	ICU Type	Age, yr	Male, %
Milton et al (17)	Europe	Prospective cohort	572 enrolled; cases n = 80; noncases n = 319	Multinational	10 mixed	Cases 64 (54–72)a; noncases 65 (56–73)a	Cases 59; noncases 62
Detsky et al (18)	United States	Prospective cohort	303 enrolled; 299 at follow-up	Multicenter	3 medical; 2 surgical	62 (53–71)a	57
Schandl et al (19)	Europe	Prospective quasi-experimental	258 enrolled; 148 at follow-up	Single center	1 mixed	Intervention male 53 (17)b; intervention Female 52 (18)b;control male 52 (17)b; control female 54 (20.5)b	Intervention 65; control 64

aMedian (interquartile range).

bMean (sd).

TABLE 2.

Description of Outcomes Measures in Included Studies

References	Postintensive Care Syndrome–Dependent Variable	Outcome Measure/s	Outcome Measure Score Direction	Timing of Outcome Measurement
Milton et al (17)	Mental health—depression, anxiety, posttraumatic stress disorder	HADS; RAND-36 (Mental Component Summary); PTSS-14	HADS—high score = worse outcome; RAND—36 low score = worse outcome; PTSS-14—high score = worse outcome	3 mo post ICU discharge
Detsky et al (18)	Physical function	Independent ambulation up 10 steps pre hospital. Independent toileting pre hospital	N/A	Prehospital and 6 mo post enrollment
Schandl et al (19)	Physical function	Katz ADL Index; ADL Staircase Questionnaire	High score = worse outcome	Katz ADL Index at 2 wk pre hospital; ADL Staircase Questionnaire at 2 mo post ICU discharge

ADL = activities of daily living, HADS = Hospital Anxiety and Depression Scale, PTSS = Posttraumatic Stress Scale.

TABLE 3.

Results From the Prediction Model Study Risk Of Bias Assessment Tool Assessment

Domain and Definition	Milton et al (17)	Detsky et al (18)	Schandl et al (19)
Overall
A) Risk of bias	High	High	High
B) Applicability	High	Low	Low
Domain 1: Participant selection
A) Risk of bias introduced by selection of participants	Low	Low	Low
B) Concern that the included participants and setting do not match the review question	Low	Low	Low
Domain 2: Predictors
A) Risk of bias introduced by predictors or their assessment	Low	Low	Low
B) Concern that the definition, assessment, or timing of assessment of predictors in the model do not match the review question	High.	Low	Low
	Rationale: selected ICU discharge risk factors may be problematic to capture in practice.
Domain 3: Outcome
A) Risk of bias introduced by the outcome or its determination	Low	High.	High;
		Rationale: concerns around the use of baseline information (that were used as candidate predictors) to define the outcome. High risk of bias due to reliance on self-report from either patients or caregivers, without use of a performance-based measure.	Rationale: The combination of high risk of bias due to reliance on self-report from either patients or caregivers, without use of a performance-based measure and differential verification of the outcome for some individuals.
B) Concern that the outcome, its definition, timing, or determination do not match the review question	Low	Low	Low
Domain 4: Analysis
A) Risk of bias introduced by sample size or participant flow	High.	High.	High.
	Rationale: small EPV, univariable p-based screening of variables for inclusion into model, unclear whether all model building steps were repeated in the bootstrap.	Rationale: Small EPV, no adjustment for overfitting, internal validation incompletely described, and unclear whether all model building steps were repeated in the cross-validation.	Rationale: Small EPV, no adjustment for overfitting, internal validation incompletely described, continuous variables were dichotomized.

EPV = events per variable.

Possible rankings for responses related to risk of bias (indicated as “A”) are low, high, or unclear. Justification is not always necessary when a ranking of low is reported but is necessary when a ranking of high or unclear is provided. Possible rankings for responses related to applicability (indicated as “A”) are the same, although only applies to domains 1, 2, and 3 and not domain 4. The Prediction Model Study Risk Of Bias Assessment Tool (PROBAST) consists of four domains containing 20 signaling questions to facilitate both the risk of bias and applicability assessment, which are detailed in the PROBAST “Explanation and Elaboration” document (16). We have noted why an assessment other than low was determined by the raters.