Tocilizumab for reduction of mortality in severe COVID-19 patients: How should we GRADE it?

To the Editor,

A recent systematic review/meta‐analysis of randomized trials (RCTs) of tocilizumab (plus standard of care [SoC] vs. SoC w/wo placebo) in severe COVID‐19 patients was a pleasure to read owing to a clear presentation of a thorough approach to data. Authors assigned high quality GRADE levels to the evidence of efficacy in reduction of mortality overall (10 RCTs) and in patients without MV at baseline (data from nine RCTs), and reduction of incident MV (10 RCTs). The grading was based on fixed‐effect pooling, likely owing to low inconsistency index (I2) and practically identical fixed‐effect and random‐effects estimates —as observed also in some other meta‐analyses of tocilizumab RCTs in this setting, for example, a previous study. It is this point that deserves a few comments. Conceptually, fixed‐effect meta‐analysis of RCTs in medicine is rarely justified since the underlying assumption is practically inevitably violated—trials are typically considerably clinically heterogeneous. In the present case, the authors described a range of differences in trial designs (e.g., one/repeated tocilizumab dose, more/less corticosteroid use, different proportions of subjects on MV). When variance of effects across trials is low, fixed and random‐effects estimates are numerically close/identical, but the conceptual differences remain. Conceptually, the random‐effects method is preferred (regardless of numerical closeness of fixed/random estimates) and the choice should not be based on the heterogeneity estimates. At this point, the issue of the choice of the variance (τ2) estimator should be mentioned—the variance estimate directly reflects on the effect estimate. A number of estimators have been explored: performance depends on the nature of the outcome, may vary across trial sizes, depends on the differences in size of included trials, and is problematic when the number of studies is low, for example, previous works. , , , While no τ2 estimator is ideal, , , , it has been suggested that the Paule–Mandel (PM) estimator performs better than the common DerSimonian–Laird estimator for binary outcomes. , Another point to consider is the method to calculate confidence intervals (CIs) around the pooled estimate. While not without certain limitations, the Hartung–Knapp–Sidik–Jonkman (HKSJ) method has been repeatedly shown (under different scenarios) to result in more adequate coverage probability than the standard method. , Figure 1A re‐creates meta‐analysis (data presented by the authors ) on mortality across the 10 RCTs (all subjects)—it is only that it uses PM variance estimator and HKSJ correction: random‐effects estimate suggests that the mean of the distribution of the effects is 0.88 (as reported ), but the CIs extend to 1.04 suggesting that it includes also effects somewhat above unity. The 95% prediction interval (the best illustration of heterogeneity , ) is even wider (Figure 1A). When viewed from the present standpoint, data indicate a non‐trivial imprecision of the pooled estimate and numerical heterogeneity of effects across trials. This appears to be in line with the actual state of the matter—the authors themselves reported differences (mortality reduction vs. no reduction) between estimates based on RCTs with a high proportion vs. low proportion of patients concomitantly treated with corticosteroids (or those generated accounting only for corticosteroid‐treated vs. not treated patients, but such data were very scarce ): so, there is apparent numerical inconsistency of the estimates across different clinical settings. As re‐created in Figure 1B,C, there was a tendency of reduced mortality in trials with a high proportion of patients co‐treated with corticosteroids (variable regimens), but with quite some imprecision and heterogeneity; and no such tendency with “low corticosteroid use.” Similarly, in patients not on MV at baseline, there was a consistent reduction in mortality across trials with a high proportion of steroid co‐treated patients, but not in trials with a low proportion of co‐treated patients (Figure 1D,E). There was also a consistent reduction of risk of incident MV in trials with a high proportion of corticosteroid co‐treated patients (Figure 1F), whereas the estimate in trials with “low steroid use” is burdened with heterogeneity and imprecision (Figure 1G).

FIGURE 1

Re‐creation of the published meta‐analysis using data provided in the published figures: the difference is in that the present estimates are generated using the Paule–Mandel variance estimator (Q‐profile method for variance estimate confidence intervals) instead of the DerSimonian–Laired method available in the RevMan software used by the authors, and Hartung–Knapp–Sidik–Jonkman correction for random effects (see text for explanation). Panel A corresponds to published Figure 1, panels B and C correspond to published supplemental Figure S4. Published meta‐analysis does not include figures that would correspond to panels D‐G. Panels E and G are reduced to summaries for brevity. Note that in all meta‐analyses point‐estimates of I2 and τ2 were low, but the upper limits of their confidence intervals were rather high, particularly when only four RCTs were included (except in panel F with highly consistent results across trials). “High%” or “low %” steroid use refers to trials (as presented in the published meta‐analysis ) in which >50% or <50% of the patients were co‐treated with corticosteroids. Meta‐analyses were performed using package meta in R. MV—mechanical ventilation; RCT—randomized controlled trial; SoC—standard of care

A “high quality” level of evidence (GRADE) implies a high certainty that the estimated and true effects overlap. Regarding the above elaboration, if one were to grade evidence of benefit of tocilizumab in severe COVID‐19 patients based on the 10 addressed RCTs, alternatives appear more reasonable: (a) considering (indiscriminately) all 10 RCTs (and all patients), certainty about reduced mortality (and the extent of the effect) is closer to “low/moderate” (imprecision, heterogeneity/inconsistency) than to “high”; (b) considering only data pertaining to patients who were (still) not on MV and were in high proportion treated with corticosteroids, estimates of reduced incident MV and of reduced mortality were consistent and precise—but to which specific daily clinical setting do they “translate”? To severely ill (but pre‐critical) patients in whom immunomodulation (corticosteroids) is already in place? In respect to this specific scenario, the existing data are indirect and hypothesis‐generating rather than confirmatory, and it seems more reasonable to be at least “somewhat uncertain” than to be “highly certain” about the existence and extent of such an effect.

CONFLICT OF INTEREST

The author has no conflict of interest to declare.