Double-zero-event studies matter: A re-evaluation of physical distancing, face masks, and eye protection for preventing person-to-person transmission of COVID-19 and its policy impact.

In a recent timely systematic review, Chu et al. [1] assessed the effectiveness of face masks, eye protection, and physical distancing for preventing COVID-19. Because the sample sizes are not large, especially in some studies of COVID-19, this review contains a considerable number of studies with zero counts of infection events, creating challenges in estimating effect sizes. If zero counts appear in both groups, this double-zero-event study (DZS) is omitted from the analyses, as implied in the forest plots in Chu et al. [1] Specifically, at least 9 out of 44 studies in this review are DZS with 1784 subjects. An omission of information about the rare outcome in DZS or artificial correction of the zero counts could impact the conclusions. [2], [3], [4], [5].

We re-analyzed the meta-analyses containing DZS in Chu et al. [1] using a bivariate generalized linear mixed-effects model in SAS version 9.4. [6] It includes zero counts by modeling with the binomial likelihood. [7,8] Appendix A presents the detailed methods.

Key findings

Some conclusions of the meta-analyses of prevention measures for COVID-19 changed noticeably after including double-zero-event studies.

What this adds to what is known?What is the implication and what should change now?

The inclusion of double-zero-event studies can be implemented using a generalized linear mixed-effects model.

Double-zero-event studies in meta-analyses should be properly taken into consideration.

Double-zero-event studies are frequently removed from meta-analyses, but they contribute important information to evidence synthesis.

Sensitivity analyses with and without double-zero-event studies are recommended to assess the robustness of synthesized evidence.

Table 1 compares the results of our re-analyses including DZS with those excluding DZS presented in Figures 2, 4, and 6 and Appendix 6 in Chu et al. [1] These results are also visualized in Appendix B. Inclusion of DZS changed the statistical significance and RR point estimates of several interventions. For example, in the subgroup of MERS, the relative risk (RR) of physical distancing was nonsignificant when excluding DZS, while it became significant after including DZS. In contrast, the RR of eye protection changed from significant to non-significant. Such changes were also observed when including DZS in two meta-analyses presented in Appendix 6 in Chu et al., [1] which focused on combinations of different prevention approaches. By including DZS, with physical distance of 2 m, the effect of eye protection on reducing infection risk was no longer statistically significant, and the reduced risk of physical distancing might not be statistically significant when using N95 respirators. The main changes of the RR point estimate after the DZS inclusion are described in Appendix B.

Table 1

Comparison between the synthesized intervention effects with and without double-zero-event studies in meta-analyses on physical distancing, face masks, and eye protection.

Meta-analysis	No. of subjects in DZS (percentage of total sample size)	No. of DZS (percentage of all studies)	RR (95% CI) with DZS excluded	RR (95% CI) with DZS included
Risk with physical distancing
MERS (n=1158, N=8)	201 (17.4%)	4 (50.0%)	0.23 (0.04, 1.20)	0.18 (0.03, 0.99)
SARS (n=9101, N=17)	41 (0.5%)	1 (5.9%)	0.35 (0.23, 0.52)	0.40 (0.27, 0.58)
COVID-19 (n=477, N=7)	204 (42.8%)	2 (28.6%)	0.15 (0.03, 0.73)	0.07 (0.01, 0.98)
Overall (n=10736, N=32)	446 (4.2%)	7 (21.9%)	0.30 (0.20, 0.44)	0.32 (0.21, 0.49)
Risk with face masks
Healthcare setting (n=9445, N=26)	358 (3.8%)	6 (23.1%)	0.30 (0.22, 0.41)	0.34 (0.23, 0.51)
Overall (n=10170, N=29)	358 (3.5%)	6 (20.7%)	0.34 (0.26, 0.45)	0.37 (0.25, 0.54)
Risk with eye protection
MERS (n=1056, N=4)	34 (3.2%)	1 (25.0%)	0.24 (0.06, 0.99)	0.22 (0.01, 4.41)
SARS (n=2581, N=8)	134 (5.2%)	2 (25.0%)	0.34 (0.21, 0.56)	0.36 (0.24, 0.55)
Overall (n=3713, N=13)	244 (6.6%)	4 (30.8%)	0.34 (0.22, 0.52)	0.35 (0.21, 0.59)
Risk with eye protection comparing different physical distances
0 m (n=881, N=4)	102 (11.6%)	1 (25.0%)	0.21 (0.13, 0.34)	0.23 (0.15, 0.32)
1 m (n=1786, N=6)	118 (6.6%)	3 (50.0%)	0.53 (0.39, 0.72)	0.55 (0.35, 0.85)
2 m (n=1084, N=5)	62 (5.7%)	2 (40.0%)	0.24 (0.06, 0.99)	0.21 (0.01, 7.09)
Overall (n=3756, N=15)	288 (7.7%)	6 (40.0%)	0.34 (0.22, 0.52)	0.35 (0.21, 0.50)
Risk with physical distancing for different face masks
N95 respirators (n=1213, N=12)	370 (30.5%)	6 (50.0%)	0.38 (0.16, 0.91)	0.51 (0.22, 1.15)
All non-N95 masks (n=9531, N=20)	78 (0.8%)	1 (5.0%)	0.27 (0.17, 0.43)	0.27 (0.16, 0.46)
Overall (n=10744, N=32)	448 (4.2%)	7 (21.9%)	0.30 (0.20, 0.44)	0.33 (0.22, 0.50)

n, number of subjects; N, number of studies; SARS, severe acute respiratory syndrome; MERS, Middle East respiratory syndrome; RR, relative risk; DZS, double-zero studies; CI, confidence interval.

These changes were related to the proportions and sample sizes of DZS in the corresponding meta-analyses (Table 1). In the subgroup of MERS, four out of eight (50%) studies were DZS, containing 201 out of 1,158 (17.4%) subjects. In the subgroup of COVID-19, two out of seven (29%) studies were DZS, containing 204 out of 477 (42.8%) subjects. The DZS contained considerable information in these subgroups; thus, the RRs produced by excluding and including DZS had noticeable differences.

Given that high-quality evidence about COVID-19 is in great demand and some intervention effects can be changed by including DZS, we suggest future systematic reviews and meta-analyses on COVID-19 properly consider the impact of including DZS. Sensitivity analyses that include DZS are recommended. If including DZS does not substantially change the conclusions, the meta-analyses can be used as reliable and robust evidence by policymakers. Researchers who wish to include DZS in future meta-analyses can find our computer code in Appendix C.

Financial support

This research was supported in part by the U.S. National Institutes of Health/National Library of Medicine grant R01 LM012982. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Declaration of Competing Interest

None.