Comment on: Developing a risk assessment score for patients with cancer during the coronavirus disease 2019 pandemic.

To the editor,

We have read with interest the article “Developing a risk assessment score for patients with cancer during the coronavirus disease 2019 pandemic” [1]. The authors propose a score to evaluate the risk of patients with cancer of developing complications if infected by COVID-19 and provide patient management recommendations in accordance with the resulting risk stratification. The idea is interesting and could potentially benefit patients. However, we have substantial concerns regarding the methodology used to produce this score and its applicability.

Risk models refer to any model that predicts the risk that a condition is present (diagnostic) or will develop in the future (prognostic) based on clinical and/or nonclinical characteristics [2]. These models can help physicians to take decisions that are based on combining information from multiple predictors observed or measured from an individual. A prospective cohort, nested case-control, or case-cohort design is commonly recommended for the development and validation of these models. The correct design of the study (i.e. statistical power and effective sample size consideration), the precise definition of the participants (i.e. eligibility criteria), a clear definition of the outcome that ensures reproducibility and the description of the statistical modelling techniques (including handling of missing data) are critical steps at the very early stage of developing a new model [3]. As part of the model development, the statistical performance, namely the discrimination and the calibration, should also be assessed [4]. Unfortunately, none of these essential premises were followed in the development of the ‘Milano-Policlinico ONCOVID Score’ [1].

The decision to include a certain predictor in the model should be made carefully. First, candidate predictors should be selected from prospective or retrospective data sets, where all data or a portion of the data are used to develop the model. Predictors are thereafter selected by multivariable analysis (either by backward elimination or forward selection), and the assignment of mutually adjusted weights to each variable (e.g. 0, 1 or 2) needs to be accurately calculated from the regression analysis. Second, the performance of any model needs to be validated in a patient data set different from the one used for the model development (independent or external validation). Third, before and after developing any prediction model, a critical appraisal and synthesis of the literature is essential to select the model variables with a sound biological and scientific rationale.

In the ‘Milano-Policlinico ONCOVID Score’, a clear definition of the outcome is missing; therefore, the aim of the score remains confusing – whether it predicts the risk of death by SARS-COV2 or of developing COVID-19 infection complications (also not defined). Moreover, in the development of the proposed score, the selection of the variables was not the result of a systematic analysis of the existing evidence, but a “thorough review” of the literature. The methodology of said review is not described by the authors, and hence the variable selection process remains opaque and nonreproducible. For instance, two of the “Patient characteristics” variables (performance status score and use of corticoids) seem to have been arbitrarily added to the ones selected through the initial literature review. Similarly, arbitrary is the choice of laboratory parameters, which the authors seem to have selectively chosen from the work by Guan et al. they referred [5].

The variables were categorized into two or three subcategories and arbitrarily assigned a scoring system between 0 and 2. In general, categorising continuous variables is not recommended in prediction models as it creates vast information loss. For instance, the age cutoff at 70 years chosen by the authors should be carefully justified. Three risk categories (low, intermediate and high risk) were defined based on random cutoffs of less than 4, 4 to 6 and 7 or more points, respectively.

For each risk category, the authors describe a list of recommendations, which are, under our perspective, confusing and not solidly sustained. For instance, ‘variations in laboratory values may indicate subclinical changes’ – is this only applicable for high-risk patients, and if yes, why? ‘Avoiding unnecessary procedures’ – should not this be applicable to all patients, regardless of their risk group? Because the inclusion criteria are not described, the targeted population is equivocal. For example, is this scoring system applicable to patients with haematological malignancies? Is it applicable to paediatric patients with cancer?

There have been efforts to produce scoring systems to evaluate the risk of developing severe/critical illness in hospitalised patients with COVID-19 [6] or the risk of death [7], but none has reached global acceptance yet. Moreover, reputed institutions such as The Centre for Evidence-Based Medicine in Oxford have stated that no reliable, applicable or usable scoring system is currently available to predict outcomes for patients with COVID-19 [8].

Methodological errors that are made in the development of prediction models [9] can ultimately lead to wrong decisions when applied into clinical practice. It is therefore our collective responsibility to develop well-designed prediction models that follow strict statistical principles [10]. This is of crucial importance in this critical episode of our history. The COVID-19 outbreak has tested national health systems’ capacity to unprecedented limits and has demonstrated how fragile they can be. Resources are limited and it is imperative to make an optimal use of them by means of systems that accurately evaluate the need of our patients and allocate the appropriate resources to them.

We urge the authors to reconsider the concept and design of this scoring system before using it in daily practice and before starting to collect prospective data for validation. We believe the clinical utility of this scoring system to be severely hampered by its current design. Patients could be equivocally assigned to risk categories, leading to potentially deleterious management decisions such as interruption or modification of cancer treatments, with unforeseeable consequences.

Funding

There was no specific funding for this study.

Conflict of interest statement

The authors declare that they do not have any potential conflict of interest.