Performance analyses of prognostic scores in critical COVID-19 patients: think outside the numbers.

To the Editor,

A recent multicentre retrospective cohort study by Martın-Rodrıgueza et al. [1] compared the quick-COVID-19-Severity-Index (qCSI) and National-Early-Warning-Score (NEWS) effectiveness in COVID-19 patients transferred by ambulance to the emergency department (ED), showing that NEWS provides a better prognostic capacity for both early- and long-term mortality [1].

During the pandemic many pre-existing assessment tools, pneumonia or sepsis-specific systems, early warning scores, and new COVID-19 models were designed to optimize clinical management. However, none can detect all patients at high risk of poor outcomes [2-4].

To highlight the scoring systems’ role, we briefly reported the performances of qCSI, NEWS, Systemic-Inflammatory-Response-Syndrome (SIRS), Rapid-Physiology-Score (RAPS), Rapid-Emergency-Medicine-Score (REMS), quick-Sequential Organ-Failure-Assessment (qSOFA), Acute-Physiology-and-Chronic-Health-Evaluation-II (APACHE II), Confusion-Urea nitrogen-Respiratory rate-Blood pressure-age ≥ 65 (CURB-65) and the Pneumonia-Severity-Index (PSI) in 106 consecutive patients with COVID-19 acute respiratory failure admitted to an Acute-Medical-Unit (March/April 2020) in a tertiary care hospital in Lombardy, Italy. At admission 75.5% required non-invasive ventilation, 7.5% were transferred to the intensive care unit (ICU) for mechanical ventilation, and the in-hospital mortality rate was 28.3%.

The more accurate scores for in-hospital mortality were those that included physiological data and age: CURB-65 (area under the receiver operator characteristic, AUROC, 0.73, 95% CI 0.63–0.82), REMS (AUROC 0.77, 95% CI 0.67–0.86), APACHE II (AUROC 0.80, 95%CI 0.71–0.88) and PSI (AUROC 0.83, 95%CI 0.75–0.91), where the last two scores also considered comorbidities. Notably, these four scoring systems for mortality performed the worst for ICU transfers. Except for SIRS (AUROC 0.72, 95%CI 0.56–0.86) and NEWS (AUROC 0.73, 95%CI 0.62–0.83), all the scores performed poorly in predicting ICU transfers (Table 1).

Table 1.

Area under the receiver operator characteristic.

	AUROC (95% CI)In-hospital mortality	AUROC (95% CI)ICU transfers
qCSI	0.64 (0.52–0.76)	0.68 (0.49–0.86)
NEWS	0.68 (0.55–0.80)	0.73 (0.62–0.83)
SIRS	0.60 (0.48–0.72)	0.72 (0.56–0.86)
RAPS	0.58 (0.47–0.69)	0.53 (0.37–0.70)
REMS	0.77 (0.67–0.86)	0.50 (0.29–0.70)
qSOFA	0.63 (0.53–0.74)	0.68 (0.52–0.84)
APACHEII	0.80 (0.71–0.88)	0.46 (0.30–0.62)
CURB65	0.73 (0.63–0.82)	0.53 (O.34–0.72)
PSI	0.83 (0.75–0.91)	0.51 (0.33–0.69)

Area Under the Receiver Operator Characteristics (AUROC, quick-COVID-19-Severity-Index (qCSI), National-Early-Warning-Score (NEWS), Systemic-Inflammatory-Response-Syndrome (SIRS), Rapid-Physiology-Score (RAPS), Rapid-Emergency-Medicine-Score (REMS), quick-Sequential Organ-Failure-Assessment (qSOFA), Acute-Physiology-and-Chronic-Health-Evaluation-II (APACHE II), Confusion-Urea nitrogen-Respiratory rate-Blood pressure-age ≥ 65 (CURB-65), Pneumonia-Severity-Index (PSI), Intensive Care Unit (ICU).

None of the scores reached acceptable AUROC (>0.7) for both (in-hospital mortality and ICU transfer) outcomes.

Interpretation of COVID-19 studies should consider selection biases, confounders, clinical characteristics, clinical settings, stress on health-care systems and COVID-19 prevalence over time. ICU admission criteria (which may change due to capacity and evolving medical knowledge), non-standard definition of respiratory decompensation, clinical settings (EDs, general-inpatients-wards, Intermediate-Care-Units, or ICUs) and lack of validation (internal/external as for qCSI/NEWS [1]) could explain contradictory results. Moreover, the selection of outcome timeframe is crucial: NEWS, as a track-and-trigger system, should be considered for early mortality [1]. Its performance is not affected by age, sex, or comorbidities up to two days, but all confounding factors relate to longer mortality time [1]. Expanding outcomes would improve the prognostic power of any score, and the change in cut-offs would increase the usability as a screening tool at the cost of specificity, while considering that relatively low event rate from small populations risks incompatibility. All standard indices of accuracy should be considered to limit risk of under- or over-triage, especially in times of limited health-care resources.

A comprehensive acute patient assessment should include evaluation of clinical severity and clinical complexity, optimizing the applicability of the derived models in real life. Scores are useful to share objective data in a common language among practitioners in different settings. However, it is necessary to change the “one-size-fits-all” approach and consider that numbers can support but not replace clinical judgement, emphasizing the importance of skilled evaluation in clinical practice.