Development of a multivariate prediction model of intensive care unit transfer or death: A French prospective cohort study of hospitalized COVID-19 patients.

Prognostic factors of coronavirus disease 2019 (COVID-19) patients among European population are lacking. Our objective was to identify early prognostic factors upon admission to optimize the management of COVID-19 patients hospitalized in a medical ward. This French single-center prospective cohort study evaluated 152 patients with positive severe acute respiratory syndrome coronavirus 2 real-time reverse transcriptase-polymerase chain reaction assay, hospitalized in the Internal Medicine and Clinical Immunology Department, at Pitié-Salpêtrière's Hospital, in Paris, France, a tertiary care university hospital. Predictive factors of intensive care unit (ICU) transfer or death at day 14 (D14), of being discharge alive and severe status at D14 (remaining with ventilation, or death) were evaluated in multivariable logistic regression models; models' performances, including discrimination and calibration, were assessed (C-index, calibration curve, R2, Brier score). A validation was performed on an external sample of 132 patients hospitalized in a French hospital close to Paris, in Aulnay-sous-Bois, Île-de-France. The probability of ICU transfer or death was 32% (47/147) (95% CI 25-40). Older age (OR 2.61, 95% CI 0.96-7.10), poorer respiratory presentation (OR 4.04 per 1-point increment on World Health Organization (WHO) clinical scale, 95% CI 1.76-9.25), higher CRP-level (OR 1.63 per 100mg/L increment, 95% CI 0.98-2.71) and lower lymphocytes count (OR 0.36 per 1000/mm3 increment, 95% CI 0.13-0.99) were associated with an increased risk of ICU requirement or death. A 9-point ordinal scale scoring system defined low (score 0-2), moderate (score 3-5), and high (score 6-8) risk patients, with predicted respectively 2%, 25% and 81% risk of ICU transfer or death at D14. Therefore, in this prospective cohort study of laboratory-confirmed COVID-19 patients hospitalized in a medical ward in France, a simplified scoring system at admission predicted the outcome at D14.

Introduction

In January 2020, the World Health Organization (WHO) declared the outbreak of coronavirus disease 2019 (COVID-19) to be a Public Health Emergency of International Concern [1]. This outbreak started in China (Wuhan), from where most of the data is available to now. Clinical presentation varies widely among individuals. Although population-based data are lacking, up to one third of patients might be asymptomatic [2, 3]. Among the symptomatic ones, more than 80% develop a mild disease, while only a minority presents the severe form of severe acute respiratory syndrome coronavirus 2 (SARS-CoV2) infection [4]. Intensive care unit (ICU) admissions range from 5% to 16%, depending on characteristics of the studied population [5, 6]. Also, Chinese retrospective studies reported an inpatient mortality rate of 17.6–28.2%, with median time to death between 15 and 18.5 days [7, 8]. Different prognostic factors emerge in this context, such as age and comorbidities [9, 10]. After Asia, Europe was quickly and severely affected by the epidemic. First in Italy then in France, the outbreak rapidly overwhelmed the public health system and ICUs were filled. As of May 12th 2020, France had already confirmed 177.547 cases with 26.646 deaths [11].

Currently, there are no validated treatments for COVID-19 and huge efforts have allowed designing and implementing very rapidly randomized controlled trials. Also, predictive prognostic factors are critical to improve management of high-risk COVID-19 patients. It is crucial to early identify those at risk of worsening for (i) an optimized management of patients' flow and to (ii) to define the population to treat, ensuring healthcare quality [12]. At this time, very limited prospective data is available on outcome and prognostic factors of COVID-19 patients among European population. Our objective through this French single-center prospective cohort study of 152 COVID-19 patients was to develop and validate multivariable predictive models for the patient status at day 14, i.e. (i) major clinical worsening (death or ICU transfer by day 14), (ii) severe status at day 14 (remaining with non-invasive or mechanical ventilation, or death, at day 14), and (iii) favorable hospital outcome (discharge alive by day 14), in adult patients requiring initial hospitalization in a medical ward.

Methods

Study population

This is a prospective single-center observational cohort study of 152 COVID-19 adult patients admitted from March 16th 2020 till the 4th of April in the Internal Medicine and Clinical Immunology Department, at Pitié-Salpêtrière’s Hospital, in Paris, France, a tertiary care university hospital. Included patients were those older than 18 years with initial requirement for hospitalization in medical ward, and diagnosed with COVID-19, defined as positive SARS-CoV-2 real-time reverse transcriptase–polymerase chain reaction (RT-PCR) assay from nasal swabs. Hospitalization criteria in medical ward was either the need for oxygen support (oxygen mask or non-invasive ventilation, but not mechanical ventilation) with hemodynamic stability, or a high-risk comorbidity profile that would need close follow-up according to emergency room judgement.

All patients benefitted from current standard COVID-19 care at the time. The study followed the Strengthening Reporting of Observational Studies in Epidemiology (STROBE) and the TRIPOD reporting guideline for cohort studies [12] We received local ethical committee approval (Comité d’éthique de la recherche Sorbonne University, CER-2020-14), and our study is registered as (NCT04320017).

All data were prospectively collected in a standardized form from the medical files of the patients. At baseline (i.e., hospital admission), we assessed demography and epidemiology features, comorbidity profile, previous treatments, clinical presentation along with the laboratory, chest computed tomography (CT) scan and echocardiogram data. Routine blood examinations included full blood count, glycaemia, renal and liver function tests, creatine kinase, lactate dehydrogenase, C-reactive protein (CRP), procalcitonin, fibrinogen, D-dimer, troponin, ferritin and interleukin-6 (IL-6). CT scan imaging results were reported according to the predominant pattern of lesions and the extent of the lesions. The first administered treatments and clinical course during hospitalization were recorded.

Patients were categorized using the WHO clinical improvement Scale [13] on day 1 (D1) and day 14 (D14). This 9-point ordinal scale measures illness severity over time as follows: 0, uninfected; 1, ambulatory, no limitation of activities; 2, ambulatory, limitation of activities; 3, hospitalized, no oxygen therapy; 4, hospitalized, oxygen by mask or nasal prongs; 5, hospitalized, oxygen by non-invasive ventilation or high-flow; 6, intubation and mechanical ventilation; 7, ventilation with additional organ support (i.e., vasopressors, dialysis, extracorporeal membrane oxygenation); 8, death. All data were collected and reviewed by three physicians (AH, GM and MV). Patients discharged from hospital before D14 were contacted by phone to assess their status at that time point.

Eligibility criteria for the validation cohort was the same used for the development cohort, being carried out in another hospital close to Paris, in Aulnay-sous-Bois, Île-de-France. The outcome was defined and assessed in a similar way to that of development cohort. Data were collected from medical hospitalization records, which included the date of admission and, as appropriate, date of hospital discharge, date of ICU transfer, date of ICU discharge, date of invasive ventilation initiation and withdrawal, date of death. From those dates, outcomes at day 14 of admission were derived, as defined for the analyses.

Non opposition to participate was obtained from each participant, and a dated non opposition form was collected and included in their medical hospitalization records, following French legislation for observational studies on standard of care data.

Definitions of study endpoints

The study endpoints were defined as the occurrence of ICU transfer or death within 14 days of admission (main endpoint), the need for non invasive or mechanical ventilation, or death, at day 14 after hospital admission, and being discharged alive within 14 days of admission.

Statistical analysis

The sample size (number of individuals, n = 152) consisted in all consecutive eligible patients hospitalized at the study center, during the first weeks of the 2020 SARS-CoV2 outbreak in Paris, France. For descriptive analyses, categorical variables are reported with counts (percent) and quantitative variables with median [interquartile range]. The association between groups and variables was evaluated using Fisher’s exact test for categorical variables, and with Wilcoxon’s rank sum test for quantitative variables. Categorical variables were compared using Fisher’s exact test and quantitative variable with Wilcoxon’s rank sum test. Analyses were performed on complete cases. Quantitative predictors were considered as continuous variables (except for age) and qualitative as binary or dummy variables, for model development. A set of predictors was defined after checking for redundancy among candidate predictors based on clinical expertise, as well as and multicollinearity, and accounting for an acceptable number of degrees of freedom given the limiting number of events. We considered predictors that would be available in most medical wards, in routine practice, representing patients status at baseline, both clinically and biologically. The predictor variables used were age, CRP level, lymphocyte count, and respiratory presentation presented as WHO score. These data are measured at the initial presentation of the patient. Poor respiratory presentation is defined as WHO score equal or superior to 5, oxygen by NVI or high flow oxygen (more than 6 L/min). No statistical-based variable selection was performed. The multivariable models of the endpoints of interest were evaluated using logistic regression models, with maximum likelihood. Validation was performed in two stages. Internal validation of the models was first performed using 1000 bootstrap resamples [14]; we estimated models performances, corrected for over-optimism (see S1 File). The models were further evaluated on an external validation sample from another French hospital close to Paris, in Aulnay-sous-Bois, Île-de-France (see S1 Table in S1 File). We defined a tentative simplified scoring system, for the main endpoint (ICU transfer or death within 14 days of admission); to that aim, continuous variables were to be dichotomized (for simplified field risk-assessment) and a unit coefficient was allocated to each of the model variables (see S1 File). The simplified score was validated internally using a resampling approach by bootstrap (number of bootstrap sample, N = 1000), and on the external cohort. For each variable, missing data was described with count. For model development, we used routinely obtained predictors (no missing data). All statistical tests were two-sided at a 5%-significance level. Analyses were performed on R statistical platform, version 3.5.3.

Results

A total of 152 consecutive eligible patients were hospitalized in the ward and included in the study. The main baseline features are presented in Table 1. Median age was 77 years [60-83], male sex and Caucasian origin were predominant, and 80.9% of the patients had comorbidities. By the time of arrival, 28 (18.4%) patients reported angiotensin-converting-enzyme inhibitors as continuous-use medication, while 16 (10.5%) had taken nonsteroidal anti-inflammatory drugs. Dyspnea was the most frequently symptom, followed by fever and dry cough. On admission, 44 patients (28.9%) had a WHO score of 3, 89 patients (58.6%) had a WHO score of 4, and 19 patients (12.5%) had a score of 5. Half of the patients presented with lymphopenia, with values below 800 cells/mm3. Chest CT scan showed that ground glass opacities were the most frequent lesions with an extent greater than 50% of the parenchyma evidenced in 24.7% of patients. IL-6 level was 31.8 pg/mL [14.8–56.0] and higher levels (161.1 pg/mL [32.7–237.8]) were observed in patients with extensive lung opacities (> 50%) as compared to those with a non-extensive lung involvement (31.7 pg/mL [15.4–51.6], p = 0.022). At admission, 129 (84.9%) patients received antibiotics, 68 (45%) hydroxychloroquine and 6 (3.9%) tocilizumab.

Table 1

Demographic, clinical, laboratory findings of patients and treatments on admission.

	All	ICU-free and alive	ICU or death	P value
Total sample‡	152	100 (68%)	47 (32%)
Demographics
Male patients	91 (59.9%)	59 (59%)	31 (66%)	0.47
Age at admission (years)				0.014*
≤ 60	41 (27%)	34 (34%)	7 (15%)
61–74	28 (18%)	14 (14%)	14 (30%)
≥ 75	83 (55%)	52 (52%)	26 (55%)
Caucasian	90/140 (64.3%)	57/90 (63%)	28/45 (62%)
Comorbidities
Smoking	10 (6.6%)	9 (9%)	0 (0%)	0.058
Hypertension	82 (53.9%)	52 (52%)	25 (53%)	1
Diabetes	37 (24.3%)	25 (25%)	12 (26%)	1
Dyslipidemia	50 (32.9%)	31 (31%)	17 (36%)	0.57
Ischemic heart disease	35 (23%)	21 (21%)	13 (28%)	0.41
Cancer	30 (19.7%)	20 (20%)	9 (19%)	1
Chronic obstructive pulmonary disease	12/151 (7.9%)	7/99 (7%)	4 (9%)	0.75
Ambulatory oxygen therapy	3 (2%)	0 (0%)	3 (6%)	0.031
Baseline on-going medications
ACE inhibitor	28 (18.4%)	19 (19%)	6 (12.8%)	0.48
NSAIDs	16 (10.5%)	12 (12%)	4 (8.5%)	0.78
Corticosteroids	16 (10.5%)	11 (11%)	5 (10.6%)	1
Signs and symptoms on admission
Days from first symptoms to admission	5 (2;8)	5 (2;9)	5 (2;8)	0.95
Fever ≥ 38.8°C	38 (25%)	23 (23%)	13 (28%)	0.54
Respiratory rate ≥ 24 breaths per minute	85/151 (56%)	49 (49%)	32/46 (70%)	0.031
SpO2 on room air, %†	93 (90–96)	94 (91–96)	91 (89–93)	0.0001
Oxygen therapy on admission	110 (72.4%)	65 (65%)	42 (89%)	0.003
SpO2 on oxygen therapy, %	96 (95–98)	98 (95–99)	95 (94–97)	0.0009
Oxygen flow, L/min	2 (2–4)	2 (2–3)	3 (2–9)	0.0008
Anosmia	17/150 (11.3%)	13/99 (13%)	3/46 (7%)	0.39
Dry cough	68/151 (45%)	43 (43%)	23/46 (50%)	0.48
Dyspnea	102/150 (67.5%)	58 (58%)	41/46 (89%)	0.0001
Myalgia	32/150 (21.3%)	27/99 (27%)	5/46 (11%)	0.031
Fatigue	70/150 (46.7%)	50/99 (51%)	20/46 (43%)	0.48
WHO clinical scale	4 (3;4)	4 (3;4)	4 (4;5)	<0.0001
Laboratory findings
Neutrophils, /mm3	4350 (2948–6962)	4155 (2722–6145)	5240 (3465–9120)	0.020
Eosinophils, /mm3	0 (0–22)	10 (0–30)	0 (0–10)	0.014
Lymphocytes, < 800/mm3	73 (48%)	39 (39%)	30 (64%)	0.008
C-Reactive protein, mg/L	74.5 (30.9–135.1)	56.6 (24.0–110.6)	112.0 (66.2–212.9)	<0.0001
Interleukine-6				0.002
≥ 30 pg/mL	31/55 (56.4%)	17/38 (45%)	13/14 (93%)
Procalcitonin, ng/mL†	0.2 (0.1–0.5)	0.1 (0.1–0.3)	0.4 (0.1–0.9)	0.0001
Ferritin, μg/L†	913 (341–1612)	786 (318–1348)	1482 (758–2682)	0.004
Troponin, ng/mL†	18.6 (9.4–39.7)	16.5 (7.9–31.1)	24.4 (14.2–47.7)	0.020
Lactate dehydrogenase, U/L†	364 (284–444)	349 (272–418)	404 (311–498)	0.044
D-Dimer, μg/L†	890 (570–1775)	830 (510–1270)	1550 (825–2305)	0.022
Imaging Studies
No.	105	70	32
Signs of SARS-CoV2 pneumonia	101/103 (98%)	66/68 (97%)	32 (100%)	1
Stage				0.009*
No lesions	2/103 (2%)	2/68 (3%)	0 (0%)
Ground-glass opacity	48/103 (47%)	36/68 (53%)	10 (31%)
Consolidation	36 /103(35%)	24/68 (35%)	11 (34%)
Bilateral pulmonary infiltration	17/103 (17%)	6/68 (9%)	11 (34%)
More than 50%	25/103 (24%)	10/61 (15%)	13 (41%)	0.009
Echocardiograhy
No.	63	46	15
Left ventricular ejection fraction, %†	65 (60–65)	65 (65–65)	65 (52–65)	0.065
Medications received during hospitalization
Hydroxychloroquine	68 (45%)	48 (48%)	20 (43.5%)	0.72
Tocilizumab	6 (4.1%)	2 (2.1%)	4 (8.7%)	0.087

ACE, angiotension-converting enzyme; NSAIDs, nonsteroidal anti-inflammatory drugs; SARS-CoV-2, severe acute respiratory syndrome coronavirus 2; WHO, World Health Organization; SpO2, peripheral capillary oxygen saturation. Data are median (IQR), n (%) or n/N (%). P values were calculated by Mann-Whitney U test, Fisher’s exact test, as appropriate.

‡5 of the 152 patients has incomplete follow-up for ICU transfer or death within 14 days

*Fisher’s exact test comparing all subcategories.

†: number of missing values for quantitative variables: SpO2 n = 13, Procalcitonine n = 21, Ferritin n = 59, Troponin n = 22, LDH n = 46, D-Dimer n = 74, LVEF n = 2 among patients with echocardiography [for categorical variables, in case of missing values, the denominator in the table indicates the number of complete cases].

The study's flow-chart represents all patients’ outcomes (Fig 1). Complete 14-day follow-up was available for 146 patients. During their clinical course, 56 (38.3%) patients experienced respiratory worsening, with 49 of them requiring an oxygen flow over 6 L/min at some point. As of day 14 (D14), 17 (11.6%) had been transferred to ICU, 5 to the semi-intensive unit, and, eventually, 32 (21.9%) patients had died, and 84 (57.5%) had been discharged alive from the hospital. For those who died, median time to death from symptom onset or hospital admission were 11 (6.75–14.5) and 6 (4–9) days, respectively. The estimated probability of ICU transfer or death by D14 was 32% (95% CI 25–40), the estimated probability of still needing non-invasive ventilation (NIV) or mechanical ventilation (MV), or being dead, at D14 was 27% (95% CI 20–35), while the estimated probability of being discharged alive by D14 was 58% (95% CI 49–66).

Fig 1

Flow chart of COVID-19 patients’ outcome.

Maximum follow-up = 14 days. *at the time of analysis, 6 patients were still followed up for the study endpoint, 5 hospitalized in the medical ward and 1 in the ICU.

In univariable analysis, age at admission, chronic respiratory failure, respiratory rate ≥ 24 breaths per minute, peripheral capillary oxygen saturation (SpO2) on room air, oxygen therapy on admission, SpO2 on oxygen, dyspnea, myalgia, WHO clinical scale, neutrophilia, eosinopenia, lymphopenia, CRP level, IL-6 level, procalcitonin, fibrinogen, serum ferritin, high-sensitivity cardiac troponin T, lactate dehydrogenase (LDH), D-dimer, and chest CT scan were associated with ICU transfer and/or death within 14 days (Table 1). For adjusted model development, the limiting number of events was 47 patients with ICU transfer or death within 14 days in the original sample. The multivariable model included age (≤ or > 60 years), respiratory baseline presentation (assessed by WHO scale levels from 3 to 5), CRP level and lymphocytes count. Older age (OR 2.61, 95% CI 0.96–7.10), poorer respiratory presentation (OR 4.04 per 1-point increment on WHO scale, 95% CI 1.76–9.25) and higher CRP level (OR 1.63 per 100mg/L increment, 95% CI 0.98–2.71) were associated with an increased risk of ICU requirement or death, while lymphocytes count were associated with better outcome (OR 0.36 per 1000/mm3 increment, 95% CI 0.13–0.99) (Fig 2, S2 Table in S1 File). Fig 2 shows a forest plot of the multivariable models of COVID-19 patient’s outcomes. Internal and external validation of the model was performed: the C-index (equivalent to AUC) was 0.80, 0.78 after correction for over-optimism by resampling, and 0.78 on the external cohort (see S1 File for further details and S1 Table in S1 File for description of the external cohort).

Fig 2

Forest plot of multivariable analysis of COVID-19 patients’ outcome [black squares: Model for ICU transfer or death within 14 days of admission (main endpoint, analysis on 147 observations); gray triangles: Model for hospitalization status at day 14 (analysis on 146 observations); gray circles: Model for detailed status at day 14 (analysis on 146 observations)].

A tentative simplified scoring system was defined for the main endpoint (ICU transfer or death within 14 days of admission), for routine clinical field practice. To that aim, based on the linear predictor and the coefficients of the multivariable model, in an additive manner, 1 point was allocated for age above 60 years old; 1 point for oxygen therapy by nasal prongs or mask (WHO scale level 4); 3 points for high flow oxygen or NIV (WHO scale level 5); 1 point if 10 ≤ CRP plasma level ≤ 75 mg/L, 2 points if 75 ≤ CRP ≤ 150 mg/L, 3 points if CRP ≥ 150 mg/L; 1 point if lymphocytes count below 800/mm3 (See S1 File). Fig 3 displays stratified risk according to each score. Therefore, we defined three risk groups: low (score 0–2), moderate (score 3–5), and high (score 6–8). Cumulative incidence for each of these groups is shown in Fig 4. Overall, the estimated sensitivity of a score greater than 2 (moderate and severe risk groups) was 97% (95% CI 94–100), and the specificity of a score lower than 6 (low and moderate risk groups) was 94% (95% CI 89–98) for the main outcome. The positive predictive value for a high-risk score was 76% (95% CI 61–91), while the negative predictive value for a low risk score was 94% (95% CI 82–100).

Fig 3

Proportions of ICU transfer or death within 14 days after admission by risk score.

Left panel A: development cohort. Right panel B: external validation cohort.

Fig 4

Cumulative incidence of ICU transfer or death by risk score.

At day 14, a total of 40 patients were still treated with NIV (n = 1) or MV (n = 7) ventilation, or had died (n = 32), out of 146 evaluable patients. In univariable analysis, age at admission, weight, chronic respiratory failure, respiratory rate ≥ 24, SpO2 on room air, Oxygen therapy on admission, SpO2 on oxygen, dyspnea, myalgia, WHO clinical scale, neutrophils, eosinophils, lymphocytes, platelets, CRP level, IL-6 level, procalcitonin, serum ferritin, high-sensitivity cardiac troponin T, D-dimer, and chest CT-scan were associated with WHO scale ≥ 5 within day 14. Multivariable analysis is represented in Fig 2.

Eighty-four patients had been discharged by day 14, out of 146 evaluable patients. In univariable analysis, age at admission, respiratory rate < 24, SpO2 on room air, Oxygen therapy on admission, ageusia, dyspnea, WHO clinical scale, neutrophils, eosinophils, lymphocytes, platelets, CRP level, IL-6 level, procalcitonin, fibrinogen, serum ferritin, high-sensitivity cardiac troponin T, LDH, D-dimer, and chest CT scan were associated with discharge alive within 14 days. Multivariable analysis is represented in Fig 2.

Discussion

The natural history and outcome of the COVID-19 patients initially hospitalized in a medical ward remain unpredictable. Currently, the main existing medical information stem from China and prognostic factors of COVID-19 among European population are lacking. The most striking conclusions drawn by this study are (i) up to 35% of the COVID-19 patients hospitalized in a medical ward were transferred to ICU or died at day 14, (ii) we defined high-risk group of ICU transfer or death using a simplified scoring system from the multivariable models including age, CRP level, lymphocytes count and WHO scale and (iii) we highlighted correlation between IL-6 level and extensive lesions in CT scan.

A clear and strong age gradient in death risk has been identified, increasing dramatically after 60 years [15]. Besides older age, comorbidities are also highlighted as key factors associated with death [7, 8, 16]. Compared to the present study, retrospective Chinese cohorts population were younger (from 51 to 56 years) and had less comorbidities (up to 48%) [7, 16]. Even with a median age of 77 years and more than 80% of comorbidity, our reported 21.9% mortality rate lies within the 17.6–28.2% range extracted from other cohorts [7, 8]. In contrast, the median time from symptoms onset to death in our population (11 days) is shorter than the 18.5 days previously reported [7], which can be ultimately the consequence of the higher risk profile of patients in the present study. Additionally, our ICU transfer rate (11.6%) was lower than the 26% described in Chinese cohorts [7, 16]. In this regard, we must underline that our patients presented with less severe infection at baseline [7, 16]. In addition, they were less eligible to ICU admission, due to age and comorbidities. Beyond demographic and clinical characteristics, several laboratory features have been linked to a higher mortality. Studies identified a positive correlation with mortality for neutrophilia, lymphopenia, troponin, LDH and D-dimer levels [7, 16]. Additionally, high levels of serum CRP, procalcitonin, and ferritin have also occasionally been associated with mortality [16, 17]. In our cohort, two simple biomarkers from routine practice, lymphocytes count and CRP level, are independently associated with a worse prognosis. CRP level higher than 75 mg/L and lymphopenia below 800/mm3 increased by two fold the odds of being transfer in ICU or death.

Herein, we provided for the first time a simplified scoring system which allows stratifying COVID-19 patients initially hospitalized in a medical ward, at low, intermediate, or high risk of ICU transfer or death. The score was validated with calibration evaluated both with an internal resampling approach and by external validation on a cohort sample from a different hospital. Based on the linear predictor of the multivariate model, age above 60 years, WHO scale, CRP level (10–75, 75–150, or > 150 mg/L), and lymphocytes count below 800/mm3 were included in the scoring system. A score equal or greater than 6 at baseline had a predicted probability of more than 60% to be transferred to ICU or dead by D14. In our regard, this high-risk patient profile should be monitored more closely and eventually considered for more aggressive treatment protocols than a patient with a score of less than 3. In a systematic review of the prediction models for diagnosis and prognosis of COVID 19 patients, Wynants et al identified ten prognostic models proposed by different Chinese teams [12]. By the time of this article writing, all these models were only available in pre-print and had not been peer-reviewed. They were exclusively based on small retrospectives cohorts, with most of them lacking an external validation cohort, or presenting a non-comparable small validation cohort. Nguyen et al [18] developed a 7-point score based on a retrospective analysis of 279 hospitalized patients but without external validation. The strengths of the score presented here are its prospective nature and its external validation. In addition, its readily accessible variables make it easily reproducible in clinical practice.

Apart from CRP level and lymphocyte count, other significant findings from our study could be further used to refine the score. Chest CT scan is a useful diagnostic tool, especially for RT-PCR negative patients, but its role as a prognostic instrument is still unclear [19]. Herein, we pointed out that parenchymal involvement greater than 50% on chest CT scan at admission was associated with ICU transfer or death in 41% of cases. In parallel, high levels of serum IL-6 have been reported in moderate to severe cases of COVID-19 pneumonia [7, 17]. IL-6 may result in increased alveolar-capillary blood-gas exchange dysfunction, especially impaired oxygen diffusion, and lead to pulmonary fibrosis and organ failure [20]. We were able to establish for the first time the correlation between IL-6 level and extensive parenchymal involvement on chest CT scan for ICU transfer or death.

Our study has some limitations. We presented models with both internal and external validation. Discrimination of the model and of the simplified score for the main endpoint was consistent in the external cohort. Calibration assessment showed a slightly overestimated risk of event in the external cohort for those with higher scores. The external sample consisted of patients from a regional non-university hospital, which could explain the differences on catchment area and patient recruitment. In the acute context of the first SARS-CoV-2 epidemic wave in France, we relied on a sample prospectively defined by consecutive eligible patients in the study center. Overall, the limited sample sizes of both development and validation samples require caution in interpreting results. Ideally, a sample size calculation at planning stage of the study should ensure sufficient collected data for predictive model development and validation; approaches have been proposed to that aim [21, 22]. Further external validation on larger prospective cohorts with planned sample sizes will be useful.

To our knowledge, this is the first prospective European cohort of COVID-19 non-critical inpatients and one of the largest standardized studies describing short term patients outcome. We provided a very simple and easily accessible score to estimate the risk of ICU transfer or death by day 14. In the context of the pandemic, this tool can help the management of patient flow, and also clinical trial design and therapeutic management.

Supplementary material.

(DOCX)

Click here for additional data file.

Transfer Alert

This paper was transferred from another journal. As a result, its full editorial history (including decision letters, peer reviews and author responses) may not be present.

20 Jul 2020

PONE-D-20-14374

Multivariable prediction model of ICU transfer and death: a French prospective cohort study of COVID-19 patients.

PLOS ONE

Dear Dr. Maalouf,

Your submission has now been peer-reviewed by two experts in the field and myself. I agree that the manuscript would benefit from being revised according to the suggestions following and encourage you to do so.

I have read with great interest your manuscript and firmly believe that it contributes to a better understanding of the Covid situation. However, I should point out that several methodological flaws exist and would suggest revising accordingly.

Title:

I would emphasize the target population and change slightly to " Development a multivariate prediction model of intensive care unit transfer or death: A French prospective cohort study of hospitalized COVID-19 patients"

Abstract:

The setting where the study took place should be reported for both the development and validation datasets;

Describe the statistical methods used, particularly the model calibration for both the developmental and validation datasets;

Please describe the number of events (for the primary outcome) and their respective percentual proportion for both the development and validation datasets;

Methods

IRB protocol number is missing;

Describe eligibility criteria for participants fo both the development and validation datasets;

Describe the source of data for both the development and validation datasets. Identify any difference from the development data in setting, eligibility criteria, outcome, and predictors. Differences or similarities in definition with the development study should be described;

Clearly define the primary outcome and the outcome that will be the target of analysis in the clinical risk score validation.

Describe how the outcome was assessed in the validation cohort

Describe all the predictor variables used in developing or validating the multivariable prediction model, including how and when they were measured. What do you mean by poor respiratory presentation?

Describe how missing data were handled with details of any imputation method used. If missing data were imputed, a description of which variables were included in the imputation procedure should be given. Could you mention the proportion of variables that had missing values?

Could you specify all measures used to assess model performance in both developing and validation datasets?

Discussion

Discuss in your limitations, the inability to accurately estimate the sample size needed when developing a clinical prediction model for binary or time-to-vent outcomes. Ideally, the sample size should be large enough to minimize model overfitting and target sufficiently precise model predictions.

==============================

Please submit your revised manuscript by Sep 03 2020 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

We look forward to receiving your revised manuscript.

Kind regards,

José Moreira, MD, MSc

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Thank you for including your ethics statement:

'The study received local ethical committee approval, and is registered as (NCT04320017).'

(a) Please amend your current ethics statement to include the full name of the ethics committee/institutional review board(s) that approved your specific study.

(b) Once you have amended this/these statement(s) in the Methods section of the manuscript, please add the same text to the “Ethics Statement” field of the submission form (via “Edit Submission”).

For additional information about PLOS ONE ethical requirements for human subjects research, please refer to " ext-link-type="uri" xlink:type="simple">http://journals.plos.org/plosone/s/submission-guidelines#loc-human-subjects-research."

3. Please provide additional details regarding participant consent.

In the ethics statement in the Methods and online submission information, please ensure that you have specified (i) whether consent was informed and (ii) what type you obtained (for instance, written or verbal, and if verbal, how it was documented and witnessed).

If your study included minors, state whether you obtained consent from parents or guardians.

If the need for consent was waived by the ethics committee, please include this information.”

4. In your Methods section, please provide additional information about the participant recruitment method and the demographic details of your participants.

Please ensure you have provided sufficient details to replicate the analyses such as:

a) the recruitment date range (month and year),

b) a description of any inclusion/exclusion criteria that were applied to participant recruitment,

c) a description of how participants were recruited, and

d) descriptions of where participants were recruited and where the research took place.

5. Please include additional information regarding the survey or questionnaire used in the study and ensure that you have provided sufficient details that others could replicate the analyses.

For instance, if you developed a questionnaire as part of this study and it is not under a copyright more restrictive than CC-BY, please include a copy, in both the original language and English, as Supporting Information.

6. We note that you have indicated that data from this study are available upon request. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly. For information on unacceptable data access restrictions, please see http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions.

In your revised cover letter, please address the following prompts:

a) If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially identifying or sensitive patient information) and who has imposed them (e.g., an ethics committee). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent.

b) If there are no restrictions, please upload the minimal anonymized data set necessary to replicate your study findings as either Supporting Information files or to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. Please see http://www.bmj.com/content/340/bmj.c181.long for guidelines on how to de-identify and prepare clinical data for publication. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories.

We will update your Data Availability statement on your behalf to reflect the information you provide

7. One of the noted authors is a group or consortium; DIMICOVID.

In addition to naming the author group, please list the individual authors and affiliations within this group in the acknowledgments section of your manuscript.

Please also indicate clearly a lead author for this group along with a contact email address.

8. Please amend either the abstract on the online submission form (via Edit Submission) or the abstract in the manuscript so that they are identical.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: I Don't Know

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Thank you for the opportunity to review this work. I believe there is currently a great demand for prognostic models for COVID-19. The authors have developed a novel prediction score with a population of 152 adults with COVID-19. They conducted validation internally using bootstrapping, and with an external cohort of 131 patients recruited at another hospital. The authors list three outcomes of interest, but their main focus appears to be ICU admission or death within 14 days.

A key concern is the number of events described. In their development sample, 47 people died or were admitted to ICU, and in the validation sample 36. Collins et al. recommend at least 100 and ideally 200 to validate a prognostic model (https://doi.org/10.1002/sim.6787). It would be ideal if the authors were able to validate their score in a sample of this size. If not, the discussion should draw readers’ attention to this issue, and advise caution before this score is used.

I would like to see more details in the main manuscript on the methods used to develop the score. Particular issues are:

1. In univariable analyses, 21 variables are significantly related to ICU admission or death. How were the four included in the multivariable analysis chosen?

2. How were the variables in the multivariable model converted into the score? Of especial note is that CRP is responsible for 3 out of 8 possible points in the score but was statistically non-significant in the multivariable analysis for the outcome of ICU admission or death.

3. There are three different models described in the supplement for the three different outcomes. I think the manuscript needs to be clear what outcome the score is intended to predict.

4. For the calibration curves, it is unclear which model the predicted values come from.

Oxygen requirement, particularly non-invasive ventilation, is included as a risk factor and as a component in the prediction score, and is also treated as an outcome. While I appreciate that these are at different timepoints, I think this needs to be addressed in the discussion.

Could the authors provide more detail about the validation cohort please? Were data collected for this validation or for another purpose? Are there any differences between the cohorts (e.g. different size hospital, recruited at different time in the epidemic) or differences in data collection and follow-up? Any missing data in the validation cohort? It might help to add a second panel to Figure 1 with details of the validation cohort.

The methods state that missing data were described with a count, but these do not appear to be reported.

Figure 1. It would be clearer to have a box showing excluded patients, then the number included in the analysis, rather than the footnote. There are 6 patients who do not have 14 days of follow-up. The one admitted to ICU appears to be included in the analysis but the other five not.

Figure 2. The caption for Figure 2 describes a forest plot, but no forest plot is included in the PDF.

Figure 4 shows time-to-event details, but no time-to-event analyses are discussed. I think it would be clearer to show the calibration plots here, or possibly a receiver operating characteristic curve.

The discussion references other prognostic scores for COVID-19. It would help readers to describe how this novel score compares to those. It would also be useful to see how the authors anticipate the score being used.

They have identified three risk levels. It would be useful to see positive- and negative-predictive values for these thresholds. I would also like more details in the discussion of how the authors anticipate that these levels should be used (keeping in mind the initial concern regarding validation with a larger dataset).

Minor comments:

1. The WHO scale described is a 9-point scale, not 8.

2. The text on page 8 states ‘more than half showing values below 800 cells’, but the table gives this proportion as 48%

3. Typo in Table 1 ventricule for ventricle

4. Typos on page 14: presente for present (twice)

5. Typo halfway down page 15 ‘probability N of’

6. Typo of VNI for NIV in the supplementary table

7. Could the authors please define ‘high flow oxygen’ as a flow rate so this is unambiguous for international readers who might want to apply the score.

Reviewer #2: Major comments

1. It is well-known that, in prognostic cohort studies, it is important to have an adequate sample size when developing a prediction model. A larger sample size will yield more findings of high reliability. Although what constitutes an “adequate” sample size is a difficult question, some results there exist in the literature. From this point of view the authors should verify if their study meets the rule(s) of thumb suggested in:

-- Peduzzi P, Concato J, Kemper E, Holford TR, Feinstein AR. A simulation study of the number of events per variable in logistic regression analysis. J Clin Epidemiol 1996; 49:1373-1379

2. The authors should discuss the important multicollinearity issue.

3. The authors say that missing data were described with count, but the way they are dealt with in the regression analysis is unclear. The authors should discuss this aspect.

4. In addition to calibration and discrimination, the authors should evaluate the strength of the predictions from the model. In these terms, measures of performance (not of association) are needed. For binary outcomes, recently proposed R2 or Brier score can be used to present overall performance measures; see

-- Nagelkerke NJ. A note on a general definition of the coefficient of determination. Biometrika 1991; 78:691-692

-- Tjur T. Coefficients of determination in logistic regression models—A new proposal: the coefficient of discrimination. Am Stat 2009; 63;366-372

-- Rufibach K. Use of Brier score to assess binary predictions. J Clin Epidemiol 2010; 63:938-939

Minor comments

1. The symbol “n” for the sample size is used but never defined. Moreover, it is not in italic. Please check.

2. The authors write: “Categorical variables were compared using Fisher’s exact test and quantitative variable with Wilcoxon’s rank sum test.” I think it should be more correct to say that: “The association/dependence between variables was evaluated using Fisher’s exact test for categorical variables, and with Wilcoxon’s rank sum test for quantitative variables.”

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

23 Sep 2020

Dear Editor and reviewers,

We thank you for your valuable comments and advices on our manuscript “PONE-D-20-14374, ‘‘Multivariable prediction model of ICU transfer and death: a French prospective cohort study of COVID-19 patients.”

You will find in this letter our response, point by point, to your comments and the revised changes that were done on the manuscript according to your recommendations.

We hope that we adequately addressed all the required changes.

We stay at your disposal for any further necessary modifications.

Yours Truly,

Dr Georgina Maalouf

Editor comments:

Title:

• I would emphasize the target population and change slightly to " Development a multivariate prediction model of intensive care unit transfer or death: A French prospective cohort study of hospitalized COVID-19 patients”

The title has been revised following your suggestion.

Abstract:

• The setting where the study took place should be reported for both the development and validation datasets

This information has been added in the revised abstract.

• Describe the statistical methods used, particularly the model calibration for both the developmental and validation datasets;

Statistical methods have been detailed in the abstract according to the reviewer’s recommendation.

• Please describe the number of events (for the primary outcome) and their respective percentual proportion for both the development and validation datasets;

This information has been added to the revised abstract.

Methods

• IRB protocol number is missing

The IRB protocol number is CER-2020-14.

The IRB number and the name of the ethics comity “Comité d’éthique de la recherche Sorbonne University “ were added to the Methods section.

• Describe eligibility criteria for participants fo both the development and validation datasets;

In the study population, we defined the eligibility criteria as follows:

‘Included patients were those older than 18 years with initial requirement for hospitalization in medical ward, and diagnosed with COVID-19, defined as positive SARS-CoV-2 real-time reverse transcriptase–polymerase chain reaction (RT-PCR) assay from nasal swabs.’

We added: Hospitalization criteria in medical ward was either the need for oxygen support (oxygen mask or non-invasive ventilation, but not mechanical ventilation) with hemodynamic stability, or a high-risk comorbidity profile that would need close follow-up according to emergency room judgement.

Eligibility criteria for the validation cohort was the same as the development cohort.

This information has been added in the revised manuscript to the Methods section.

• Describe the source of data for both the development and validation datasets.

Data were collected from the digitalized medical files and record of the patients in each hospital .

This information has been added in the revised abstract to the Methods section, P7.

• Identify any difference from the development data in setting, eligibility criteria, outcome, and predictors. Differences or similarities in definition with the development study should be described;

Eligibility criteria for the validation cohort was the same used for the development cohort, being carried out in another hospital close to Paris, in Aulnay-sous-Bois, Île-de-France. The outcome was defined and assessed in a similar way to that of development cohort.

This information has been added in the revised manuscript to the Methods section, P8.

Furthermore, a description of patients’ demographics, outcomes and predictors is provided in Supplementary Materials, table S1. Patients in the external validation cohort were younger (p0.0001) and had overall greater baseline lymphocytes counts (p=0.0003). There were no other significant differences in outcomes. It illustrates the difference in catchment area of the two centers.

• Clearly define the primary outcome and the outcome that will be the target of analysis in the clinical risk score validation.

This information has been added in the revised manuscript to the Methods section, P9:

In the definition of study endpoints in the methods, we added main endpoint after “death or ICU transfer at Day 14”.

In the Statistical Analysis, P10, we added : We defined a tentative simplified scoring system, for the main endpoint (ICU transfer or death within 14 days of admission).

• Describe how the outcome was assessed in the validation cohort

The outcome was defined and assessed in the validation cohort similarly to the development cohort. Patients’ outcomes were collected from medical hospitalization records, which included the date of admission and, as appropriate, date of hospital discharge, date of ICU transfer, date of ICU discharge, date of invasive ventilation initiation and withdrawal, date of death. From those dates, outcomes at day 14 of admission were derived, as defined for the analyses.

This information has been added in the revised manuscript to the Methods section, P8.

• Describe all the predictor variables used in developing or validating the multivariable prediction model, including how and when they were measured. What do you mean by poor respiratory presentation?

The predictor variables used in developing and validating the multivariable prediction model were age, CRP level, lymphocyte count, and respiratory presentation presented as WHO score. These data are measured at the initial presentation of the patient. Poor respiratory presentation is defined as WHO score equal or superior to 5, oxygen by NVI or high flow oxygen (more than 6 L/min).

This information has been added in the revised manuscript to the Methods section, P10.

• Describe how missing data were handled with details of any imputation method used. If missing data were imputed, a description of which variables were included in the imputation procedure should be given. Could you mention the proportion of variables that had missing values?

Analyses were performed on complete cases samples. Sample sizes for each analysis are indicated in table 1 (in case of missing data: number of complete cases for qualitative variables is mentioned; for quantitative variables, number of missing data is stated in the table footnote). The sample size used for the prognostic model has been clarified and is now available in the title of figure 2 [n=147 for the primary endpoint, n=146 for hospitalization status at day 14]. Of note, variables with more than 20% missing data were not considered for the prognostic model (eg IL6, LDH, ferritin, etc.). We considered that variables which could not be easily and routinely collected at baseline were not relevant to elaborate the prognostic model and score.

• Could you specify all measures used to assess model performance in both developing and validation datasets?

For all outcomes, models performances were assessed with the following procedures, with internal (for over-optimism correction using 1000-boostrap resampling on the development dataset) and external validation: Brier score, for discrimination: Somers’ Dxy (C-index), R², for calibration: calibration intercept and slope (shrinkage) with estimation of the calibration curve via non parametric loess smoothing, and estimation of mean absolute calibration error. These aspects are available and detailed, with obtained results, in the Supplemental material.

Discussion

• Discuss in your limitations, the inability to accurately estimate the sample size needed when developing a clinical prediction model for binary or time-to-vent outcomes. Ideally, the sample size should be large enough to minimize model overfitting and target sufficiently precise model predictions.

We agree with the reviewer that sample size is an important aspect in developing predictive models. We have modified the discussion section accordingly to discuss this aspect in the limitations, P20.

The study was conducted in the acute context of the first SARS-CoV-2 epidemic wave in France; we relied on a sample prospectively defined by consecutive eligible patients during the first weeks of COVID-19 activity in the study center. However, ideally, a sample size calculation at planning stage of a study should ensure sufficient information is collected for predictive model development and validation; approaches have been proposed to that aim [1-2]. More precisely, sample size (accounting for the number of events in our cases of binary endpoints) should be set to minimize model overfitting and target sufficiently precise estimation of the risk of event [2]. Such approaches should be used in planning and conducting further analyses for COVID-19 prognosis.

[1] Riley RD, Snell KIE, Burke D et al. Minimum sample size for developing a multivariable prediction model: Part I – Continuous outcomes. Statistics in Medicine 2019;38 :1262-75

[2] Riley RD, Snell KIE, Ensor J et al. Minimum sample size for developing a multivariable prediction model: Part I – binary and time-to-event outcomes. Statistics in Medicine 2019;38 :1276-96

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

We have addressed these aspects in the revised submission.

2. Thank you for including your ethics statement:

'The study received local ethical committee approval, and is registered as (NCT04320017).'

(a) Please amend your current ethics statement to include the full name of the ethics committee/institutional review board(s) that approved your specific study.

The full name of the committee is Comité d’éthique de la recherche Sorbonne University.

The IRB number is “CER-2020-14”.

(b) Once you have amended this/these statement(s) in the Methods section of the manuscript, please add the same text to the “Ethics Statement” field of the submission form (via “Edit Submission”).

This has been added accordingly in the Methods section P7 and in the submission form.

3. Please provide additional details regarding participant consent.

In the ethics statement in the Methods and online submission information, please ensure that you have specified (i) whether consent was informed and (ii) what type you obtained (for instance, written or verbal, and if verbal, how it was documented and witnessed).

Non opposition to participate was obtained from each participant, and a dated non opposition form was collected and included in their medical hospitalization record, following French legislation for observational studies on standard of care data. This information has been added in the revised manuscript to the Methods section.

We updated our statement on P 23 and in the submission form

4. In your Methods section, please provide additional information about the participant recruitment method and the demographic details of your participants. Please ensure you have provided sufficient details to replicate the analyses such as:

a) the recruitment date range (month and year),

The patient were included from March 16th 2020 until the 4th of April 2020. This has been added to the Methods section P 7.

b) a description of any inclusion/exclusion criteria that were applied to participant recruitment,

The inclusion criteria are detailed in the study population in the methods section, on page 7.

“Included patients were those older than 18 years with initial requirement for hospitalization in medical ward, and diagnosed with COVID-19, defined as positive SARS-CoV-2 real-time reverse transcriptase–polymerase chain reaction (RT-PCR) assay from nasal swabs. Hospitalization criteria in medical ward was either the need for oxygen support (oxygen mask or non-invasive ventilation, but not mechanical ventilation) with hemodynamic stability, or a high-risk comorbidity profile that would need close follow-up according to emergency room judgement”

c) a description of how participants were recruited, and where the research took place.

Patient included were those consecutively admitted to the department of Internal Medicine and Clinical Immunology Department, at Pitié-Salpêtrière’s Hospital, in Paris, France, a tertiary care university hospital., during the inclusion period and who were eligible for the study and unopposed to participating.

This information is available in the study population in the methods section, P7

5. Please include additional information regarding the survey or questionnaire used in the study and ensure that you have provided sufficient details that others could replicate the analyses.

There was no specific survey or questionnaire used in the study.

6. We note that you have indicated that data from this study are available upon request. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly. For information on unacceptable data access restrictions, please see http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions.

We apologize, but, after double checking with the institution, unfortunately, data from this study cannot be made available (either publicly or upon request), for ethical and legal restrictions (data contain potentially indirectly identifying or sensitive patient information as per French regulation).

This information was updated in our manuscript on P23 and in the submission form.

7. One of the noted authors is a group or consortium; DIMICOVID.

In addition to naming the author group, please list the individual authors and affiliations within this group in the acknowledgments section of your manuscript.

DIMICOVID is the department of internal medicine and clinical immunology that managed COVID -19 patient in the Pitié-Salpêtrière Hospital , APHP, Paris Sorbonne. All the doctors that worked in the department during this period are cited in the acknowledgments on page 22.

Please also indicate clearly a lead author for this group along with a contact email address.

Yves Allenbach MD, yves.allenbach@aphp.fr

8. Please amend either the abstract on the online submission form (via Edit Submission) or the abstract in the manuscript so that they are identical.

Both abstracts were updated with the revised version .

Reviewer #1:

Thank you for the opportunity to review this work. I believe there is currently a great demand for prognostic models for COVID-19. The authors have developed a novel prediction score with a population of 152 adults with COVID-19. They conducted validation internally using bootstrapping, and with an external cohort of 131 patients recruited at another hospital. The authors list three outcomes of interest, but their main focus appears to be ICU admission or death within 14 days.

A key concern is the number of events described. In their development sample, 47 people died or were admitted to ICU, and in the validation sample 36. Collins et al. recommend at least 100 and ideally 200 to validate a prognostic model (https://doi.org/10.1002/sim.6787). It would be ideal if the authors were able to validate their score in a sample of this size. If not, the discussion should draw readers’ attention to this issue, and advise caution before this score is used.

I would like to see more details in the main manuscript on the methods used to develop the score. Particular issues are:

1. In univariable analyses, 21 variables are significantly related to ICU admission or death. How were the four included in the multivariable analysis chosen?

As outlined by the reviewer, the main endpoint was ICU transfer or death within 14 days from admission. Given the limited sample size and distribution for this endpoint of interest (47 ICU transfer or death vs 100 alive without ICU transfer), we chose to allow on 4 degrees of freedom in the model predictor. We chose to spend these 4 degrees of freedom including clinically relevant variables and avoiding the unreliability of selection procedures.

We first evaluate all candidate variables for redundancy and collinearity, to reduce the candidate set. We also restricted to variables with limited missing values, aiming to consider only variables that are routinely collected and available upon admission, for external validity. Then we chose to include variables summarizing relevant information at admission; age (as it had been identified as a relevant prognostic factor for COVID-19), the baseline respiratory status, biological inflammation (CRP) and lymphocytes.

These aspects are reported in the supplementary material, section 2.

2. How were the variables in the multivariable model converted into the score? Of especial note is that CRP is responsible for 3 out of 8 possible points in the score but was statistically non-significant in the multivariable analysis for the outcome of ICU admission or death.

We computed the tentative simplified score based on the linear predictor and the coefficients of the main multivariable model. Unit points were allocated as follows:

- 1 point for age above 60 y.o.;

- 1 point for WHO scale at 4, 3 points for WHO scale 5;

- CRP level was treated as:

o 1 point if 10 ≤ CRP ≤ 75 mg/L,

o 2 points if 75 ≤ CRP ≤ 150 mg/L,

o 3 points if CRP ≥ 150 mg/L;

- 1 point if lymphocytes count below 800/mm3.

In the multivariate model, CRP is included as a continuous variable. Although it was not statistically significant at the 5%-level for the main endpoint, the estimate of CRP association with the endpoint, log(HR), is greater than 0, with a 95% confidence interval clearly asymmetrical around 0. This was consistent across the 3 endpoints we examined.

3. There are three different models described in the supplement for the three different outcomes. I think the manuscript needs to be clear what outcome the score is intended to predict.

We thank the reviewer for pointing this out. The manuscript has been modified to make that clearer, in the main text and in the supplementary material.

4. For the calibration curves, it is unclear which model the predicted values come from.

This has been clarified in the figure’s legends in the supplementary material

Oxygen requirement, particularly non-invasive ventilation, is included as a risk factor and as a component in the prediction score, and is also treated as an outcome. While I appreciate that these are at different timepoints, I think this needs to be addressed in the discussion.

We thank the reviewer for raising this point. The population of the study was patients hospitalized for COVID-19 management in medical wards, excluding patients who were directly hospitalized in intensive care. Oxygen requirement is directly involved in the third endpoint: NIV, ICU or death at day 14. We considered baseline (admission=day 0) respiratory status as predictor for the outcome at day 14. We found that 15/18 patients with NIV at baseline were still receiving NIV at day 14. The initial need for NIV upon admission possibly represented a marker of disease severity and risk of prolonged need for respiratory support.

Could the authors provide more detail about the validation cohort please? Were data collected for this validation or for another purpose? Are there any differences between the cohorts (e.g. different size hospital, recruited at different time in the epidemic) or differences in data collection and follow-up? Any missing data in the validation cohort? It might help to add a second panel to Figure 1 with details of the validation cohort.

Data was collected specifically for this validation from patients hospitalization records. They were collected at the same time of the epidemic, during the first wave in Paris region, France. Furthermore, a description of patients demographics, outcomes and predictors is provided in Supplementary Materials, table S1. Patients in the external validation cohort were younger (p0.0001) and had overall greater baseline lymphocytes counts (p=0.0003). There was no other significant differences in outcomes. It illustrates the difference in catchment area of the two centers.

The methods state that missing data were described with a count, but these do not appear to be reported.

Analyses were performed on complete cases samples. Sample sizes for each analysis are indicated in table 1 (in case of missing data: number of complete cases for qualitative variables is mentioned in the table as denominator for the percentage computation; for quantitative variables, the number of missing data, if any, is now stated in the table footnote).

Figure 1. It would be clearer to have a box showing excluded patients, then the number included in the analysis, rather than the footnote. There are 6 patients who do not have 14 days of follow-up. The one admitted to ICU appears to be included in the analysis but the other five not.

Figure 1 has been modified to detail excluded patients excluded from analyses due to incomplete information in D14 status.

Figure 2. The caption for Figure 2 describes a forest plot, but no forest plot is included in the PDF.

We apologize for the missing forest plot. It is included in the submitted revision documents, should now be visible, and is referred to in the manuscript.

Figure 4 shows time-to-event details, but no time-to-event analyses are discussed. I think it would be clearer to show the calibration plots here, or possibly a receiver operating characteristic curve.

We agree with the reviewer that figure 4 relies on a time-to-event approach to represent patients’ outcome, whereas main analyses and models considered the outcome as a binary endpoint. Nevertheless, the cumulative incidences were computed using a competing event framework, with “discharge alive” and “ICU or death” as competing events; therefore these estimates are consistent with the proportions for the binary endpoint at day 14. A barplot for direct illustration of the binary endpoint is available in figure 3.

Of note, as indicated in the supplementary material, a sensitivity analysis using the Fine and Gray model accounting for the time to the event (ICU transfer or death between day 1 and 14, with discharge alive without ICU stay as a competing event; discharge alive between day 1 and 14, with hospital death as a competing event) yielded similar results on the association of the variables with the outcomes distribution.

The discussion references other prognostic scores for COVID-19. It would help readers to describe how this novel score compares to those. It would also be useful to see how the authors anticipate the score being used.

We agree with the reviewer on this point , we added in the discussion on P20 a comparison with another prognostic score to highlight the power of our score :

Nguyen et al [20] developed a 7-point score based on a retrospective analysis of 279 hospitalized patients but without external validation. The strengths of the score presented here are its prospective nature and its external validation. In addition, its readily accessible variables make it easily reproducible in clinical practice.

Furthermore , we added in the discussion on P19 , an example of how the score can be used:

“A score equal or greater than 6 at baseline had a predicted probability of more than 60% to be transferred to ICU or dead by D14. In our regard, this high-risk patient profile should be monitored more closely and eventually considered for more aggressive treatment protocols than a patient with a score of less than 3.”

1. Yann Nguyen, Félix Corre, Vasco Honsel, Vinciane Rebours, Bruno Fantin Adrien Galy, et.al A nomogram to predict the risk of unfavourable outcome in COVID-19: a retrospective cohort of 279 hospitalized patients in Paris area, Annals of Medicine, 52:7, 367-375,

They have identified three risk levels. It would be useful to see positive- and negative-predictive values for these thresholds.

Assuming the included sample is representative of the prevalence of ICU transfer or death in the population of interest (estimated prevalence 32%), the estimated PPV and NPV for the thresholds defining the 3 risk levels are reported in the table below, for each risk group, as well as Sensitivity and Specificity, with their 95% confidence intervals estimated by bootstrap resampling.

Sensitivity Specificity PPV NPV

Score ≥ 3 97% (94;100) 16% (9;24) 35% (33;38) 94% (82;100)

Score ≥ 6 40% (28;55) 94% (89;98) 76% (61;91) 77% (73;82)

These estimates should be interpreted with caution since the sample corresponds to a sample included during the beginning of the first wave of SARS-CoV2 epidemics in Paris, France. The manuscript, in the discussion section, has been revised on these aspects.

I would also like more details in the discussion of how the authors anticipate that these levels should be used (keeping in mind the initial concern regarding validation with a larger dataset).

We agree with the reviewer and this point was discussed with more details in the revised manuscript on P19: “A score equal or greater than 6 at baseline had a predicted probability of more than 60% to be transferred to ICU or dead by D14. In our regard, this high-risk patient profile should be monitored more closely and eventually considered for more aggressive treatment protocols than a patient with a score of less than 3.”

Minor comments:

1. The WHO scale described is a 9-point scale, not 8. We thank the reviewer for pointing out this issue. We have corrected the manuscript accordingly.

2. The text on page 8 states ‘more than half showing values below 800 cells’, but the table gives this proportion as 48% We thank the reviewer for pointing out this issue. We have corrected the manuscript accordingly.

3. Typo in Table 1 ventricule for ventricle, We thank the reviewer for pointing out this issue. We have corrected the manuscript accordingly.

4. Typos on page 14: presente for present (twice) We thank the reviewer for pointing out this issue. We have corrected the manuscript accordingly.

5. Typo halfway down page 15 ‘probability N of’ We thank the reviewer for pointing out this issue. We have corrected the manuscript accordingly.

6. Typo of VNI for NIV in the supplementary table of’ We thank the reviewer for pointing out this issue. We have corrected the manuscript accordingly.

7. Could the authors please define ‘high flow oxygen’ as a flow rate so this is unambiguous for international readers who might want to apply the score. of’ We thank the reviewer for pointing out this issue. High flow oxygen was defined in statistical analysis on P6 : high flow oxygen (more than 6 L/min).

Reviewer #2: Major comments

1. It is well-known that, in prognostic cohort studies, it is important to have an adequate sample size when developing a prediction model. A larger sample size will yield more findings of high reliability. Although what constitutes an “adequate” sample size is a difficult question, some results there exist in the literature. From this point of view the authors should verify if their study meets the rule(s) of thumb suggested in:

-- Peduzzi P, Concato J, Kemper E, Holford TR, Feinstein AR. A simulation study of the number of events per variable in logistic regression analysis. J Clin Epidemiol 1996; 49:1373-1379

We agree with the reviewer that sample size is an important aspect in developing predictive models.

The study was conducted in the acute context of the first SARS-CoV-2 epidemic wave in France; we relied on a sample prospectively defined by consecutive eligible patients during the first weeks of COVID-19 activity in the study center. Keeping that in mind and given the size of the obtained sample and number of events for the main analysis (47 out of 147), we chose to allow only 4 degrees of freedom to our main model and avoid variables selection procedures (beside clinical relevance, redundancy, collinearity) [1]. Indeed, ideally, a sample size calculation at planning stage of a study should ensure sufficient information is collected for predictive model development and validation; approaches have been proposed to that aim [2-3]. More precisely, sample size (accounting for the number of events in our cases of binary endpoints) should be set to minimize model overfitting and target sufficiently precise estimation of the risk of event [3]. Such approaches should be used in planning and conducting further analyses for COVID-19 prognosis.

We have modified the discussion section accordingly to discuss this aspect in the limitations on page 20.

[1] Harrell FR Jr. Regression Modeling Strategies. Second Edition, Springer International Publishing Switzerland 2015

[2] Riley RD, Snell KIE, Burke D et al. Minimum sample size for developing a multivariable prediction model: Part I – Continuous outcomes. Statistics in Medicine 2019;38 :1262-75

[3] Riley RD, Snell KIE, Ensor J et al. Minimum sample size for developing a multivariable prediction model: Part I – binary and time-to-event outcomes. Statistics in Medicine 2019;38 :1276-96

2. The authors should discuss the important multicollinearity issue.

We thank the reviewer for raising this point. For each endpoint, potential collinearity among the predictor variables was examined using a correlation matrix. There was no evidence of major multicollinearity among the predictors (all estimated correlation coefficient lower than 0.5, see table below). This has been added in the supplementatry material.

Age WHO scale CRP

Age

WHO scale 0.097

CRP 0.020 0.456

Lymphocytes count -0.137 -0.193 -0.239

3. The authors say that missing data were described with count, but the way they are dealt with in the regression analysis is unclear. The authors should discuss this aspect.

Analyses were performed on complete cases samples. Sample sizes for each analysis are indicated in table 1 (in case of missing data : number of complete cases for qualitative variables is mentioned; for quantitative variables, number of missing data is stated in the table footnote). The sample size used for the prognostic model has been clarified and is now available in the title of figure 2 [n=147 for the primary endpoint, n=146 for hospitalization status and detailed status at day 14]. Of note, variables with more than 20% missing data were not considered for the pronostic model (eg IL6, LDH, ferritin, etc.). We considered that variables which could not be easily and routinely collected at baseline were not relevant to elaborate the prognostic model and score.

These aspects have been clarified in the manuscript and the supplementary material.

4. In addition to calibration and discrimination, the authors should evaluate the strength of the predictions from the model. In these terms, measures of performance (not of association) are needed. For binary outcomes, recently proposed R2 or Brier score can be used to present overall performance measures; see

-- Nagelkerke NJ. A note on a general definition of the coefficient of determination. Biometrika 1991; 78:691-692

-- Tjur T. Coefficients of determination in logistic regression models—A new proposal: the coefficient of discrimination. Am Stat 2009; 63;366-372

-- Rufibach K. Use of Brier score to assess binary predictions. J Clin Epidemiol 2010; 63:938-939

We agree with the reviewer and the Nagelkerke R² and Brier score for all models are available in Supplementary material, table 2.

Minor comments

1. The symbol “n” for the sample size is used but never defined. Moreover, it is not in italic. Please check.

The notation “n” is now defined in the Statistical Analysis section and is typed in italic.

2. The authors write: “Categorical variables were compared using Fisher’s exact test and quantitative variable with Wilcoxon’s rank sum test.” I think it should be more correct to say that: “The association/dependence between variables was evaluated using Fisher’s exact test for categorical variables, and with Wilcoxon’s rank sum test for quantitative variables.”

We have revised the sentence following the reviewer recommendation.

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

The authors prefer not to publish the peer review history.

Submitted filename: response to the editor about Data Sharing.docx

Click here for additional data file.

2 Oct 2020

Development of a multivariate prediction model of intensive care unit transfer or death: A French prospective cohort study of hospitalized COVID-19 patients

PONE-D-20-14374R1

Dear Dr. Maalouf,

Thank you for submitting your article to PLoS One, which I have pleasure in accepting for publication as an original research paper. Well done and many congratulations! Your manuscript has now been formally accepted for publication. Please see important details concerning the publication process below. Your efforts during the process of revision are acknowledged and I hope you also are pleased with the final result. We appreciate being able to publish your work and look forward to seeing your paper online as soon as possible.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

José Moreira, MD, MSc

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

9 Oct 2020

PONE-D-20-14374R1

Development of a multivariate prediction model of intensive care unit transfer or death: A French prospective cohort study of hospitalized COVID-19 patients

Dear Dr. Maalouf:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. José Moreira

Academic Editor

PLOS ONE