Spurious early ecological association suggesting BCG vaccination effectiveness for COVID-19.

BACKGROUND: Several ecologic studies have suggested that the bacillus Calmette-Guérin (BCG) vaccine may be protective against SARS-CoV-2 infection including a highly-cited published pre-print by Miller et al., finding that middle/high- and high-income countries that never had a universal BCG policy experienced higher COVID-19 burden compared to countries that currently have universal BCG vaccination policies. We provide a case study of the limitations of ecologic analyses by evaluating whether these early ecologic findings persisted as the pandemic progressed.
METHODS: Similar to Miller et al., we employed Wilcoxon Rank Sum Tests to compare population medians in COVID-19 mortality, incidence, and mortality-to-incidence ratio between countries with universal BCG policies compared to those that never had such policies. We then computed Pearson's r correlations to evaluate the association between year of BCG vaccination policy implementation and COVID-19 outcomes. We repeated these analyses for every month in 2020 subsequent to Miller et al.'s March 2020 analysis.
RESULTS: We found that the differences in COVID-19 burden associated with BCG vaccination policies in March 2020 generally diminished in magnitude and usually lost statistical significance as the pandemic progressed. While six of nine analyses were statistically significant in March, only two were significant by the end of 2020. DISCUSSION: These results underscore the need for caution in interpreting ecologic studies, given their inherent methodological limitations, which can be magnified in the context of a rapidly evolving pandemic in which there is measurement error of both exposure and outcome status.

Introduction

Since the identification of SARS-CoV-2 in December 2019 in Wuhan, China, COVID-19 has spread to humans in every country in the world, resulting in over 250 million infections and five million deaths as of this writing [1]. Throughout the pandemic, scientists have considered the possibility of repurposing existing preventive and therapeutic agents, including the century-old [2] bacillus Calmette-Guérin (BCG) vaccine. Currently, 157 countries mandate universal BCG vaccination, and 19 countries (e.g., the United States, Italy, Belgium) only vaccinate high-risk groups [3]. In 2020, laboratory analyses found that BCG vaccination induces specific immunity against the envelope protein of SARS CoV-2 [4].

However, to date, most of the evidence for BCG protection against COVID-19 has come from a series of ecologic studies, including a highly-cited pre-print by Miller and colleagues in March 2020 [5]. In that paper, Miller et al. found that middle/high- and high-income countries that never had a universal BCG policy experienced higher COVID-19 incidence and mortality compared to countries that currently have universal BCG vaccination policies. In addition, this analysis suggested that the older the universal BCG policy (and thus presumably the larger the proportion of the population vaccinated), the lower the national COVID-19 mortality rate. Since this early work, there have been several observational studies and a meta-analysis of four ecologic studies that yielded a statistically significant pooled negative correlation between the percentage of the population vaccinated with BCG and COVID-19 mortality at the country level (random effect pooled r: -0.48 [95% confidence interval: -0.61 to -0.35]) [6].

In April 2020, the WHO announced that, in the absence of positive results from RCTs (the gold standard for evaluating vaccine efficacy), there was insufficient evidence of a causal relationship between the BCG vaccine and COVID-19 infection [7]. As of March 2021, there were more than two dozen clinical trials underway across the globe to evaluate whether BCG protects against COVID-19. Completed assessments to date include a case-control study [8] that did not demonstrate effectiveness and an unpublished RCT [9] conducted in elderly patients in Greece claiming effectiveness that is limited by small sample size and high loss to follow-up. In the interim, we aimed to evaluate whether the early ecologic findings that BCG appeared to protect against COVID-19 persisted as the pandemic progressed.

Methods

To assess the robustness of the Miller et al. findings, we replicated their analysis using identical statistical methods for each month in 2020 subsequent to their March 2020 analysis. Those authors examined the relationship between BCG and COVID-19 morbidity and mortality from January 1, 2020 –March 21, 2020 using COVID-19 data from Google News. Because Google News does not provide historical data by month, we utilised data on COVID-19 incidence and mortality from the COVID-19 Data Repository at Johns Hopkins University. This database compiles data from several reliable sources, including the World Health Organization, several country-specific Centers for Disease Control and Prevention and ministries of health, and provides historical data by month [1].

A comparison between the Miller et al. March 2020 dataset and the March 2020 dataset generated by the Johns Hopkins Repository showed these two sources yield similar results, with the Hopkins Repository providing on average 1.07 [range: 0.00 to 2.45] and 1.09 [range: 0.50 to 2.33] times higher mortality and incidence counts, respectively, for each country. In addition, replicating the Miller et al. analyses with the Hopkins Repository data for March provided similar statistical results at the p = 0.05 level.

Like Miller et al., we evaluated three outcomes: per capita COVID-19 mortality (number of deaths from COVID-19 per 100,000 population), incidence (number of COVID-19 cases per 100,000 population), and mortality-to-incidence ratio (the number of deaths divided by the number of cases). Like Miller et al., we extracted data on country-specific BCG vaccination policies from the BCG World Atlas [3], restricting our analyses to countries that 1) have a population of over one million people, and 2) are classified as middle/high- or high-income per World Bank categories [10]. Wilcoxon Rank Sum Tests were then used to compare population medians between countries with universal BCG policies and those that never had such policies. Pearson’s r correlations were also computed to evaluate the association between year of BCG vaccination policy implementation and COVID-19 outcomes. Statistical tests were considered significant if p<0.05 (2-sided).

Results

Relationship between policies and outcomes

Miller et al. examined 55 middle/high- and high-income countries that had universal BCG polices in place and five that had never had a universal BCG vaccination policy. In March 2020, those countries with universal vaccination policies had statistically significantly lower COVID-19 mortality, incidence, and mortality-to-incidence ratios than those that never had such BCG policies using Wilcoxon Rank Sum Tests. When we repeated their analyses for subsequent months, we initially found that COVID-19 mortality rates in countries with no history of BCG policies continued to significantly exceed those in counties with current BCG policies, with the magnitude of that difference essentially stable after May 2020 (Fig 1). In contrast, differences between these groups with respect to COVID-19 incidence were not significant from May to October, although they became statistically significant again in November and December. For COVID-19 mortality-to-incidence ratios, statistically significant differences were present through July, but not thereafter.

Fig 1

Median COVID-19 mortality rate per 100,000 population (A), incidence rate per 100,000 population (B), and mortality-to-incidence ratio (C) by BCG policy status, March-December, 2020.

* Statistically significant at p<0.05 using Wilcoxon Rank Sum Test.

Relationship between duration of policies and outcomes

Miller et al. also examined the correlation between the number of years of implementation of the BCG policy (for the 45 countries for which data on year of policy implementation were available) and the same three COVID-19 outcomes. Twenty-eight countries had current universal BCG policies, while 17 previously had them.

For those with current BCG policies (Fig 2A), a statistically significant positive association was present for mortality rate in March 2020 but not thereafter; Pearson correlation coefficients for incidence rate and mortality-to-incidence ratio were not statistically significant at any period but the correlation coefficients were lower in all months than in March 2020 (and sometimes negative suggesting that more recent BCG policy implementation was associated with lower burden of COVID-19). For countries with previous universal BCG vaccination policies (Fig 2B), statistically significant findings for mortality and incidence in March 2020 were no longer apparent by April and May 2020, respectively, and remained non-significant throughout the study period. The correlation coefficient for mortality-to-incidence ratio was never statistically significant. Except for a small increase in the correlation coefficient for incidence in April 2020 among countries that no longer have universal BCG policies, all correlation coefficients were lower in every month after March 2020 for all three outcomes.

Fig 2

Pearson correlation coefficients between years of implementation of BCG Policy and COVID-19 outcomes among countries with current universal BCG vaccination policy (A) and among countries that previously had universal BCG vaccination policies (B), March-December, 2020.

* Statistically significant Pearson correlation at p<0.05.

Discussion

Our analysis demonstrates that the differences in COVID-19 burden associated with BCG vaccination policies that Miller et al. observed in March 2020 generally diminished in magnitude and usually lost statistical significance as the pandemic progressed. While six of nine analyses were statistically significant in March, only two were significant by the end of 2020, including none of the six correlations between year of BCG policy implementation and COVID-19 burden, three of which were originally considered statistically significant.

Our case study illustrates that the timing of the initial analysis was an important factor in the apparent detection of associations, as we found that the indices measured by Miller et al. varied substantially and typically diminished in magnitude over time. This finding was also observed in a more limited confirmatory analysis, in which Lindestam Arlehamn et al. repeated the analysis of a different ecologic study [11] and found that certain associations apparent in March 2020 were no longer present in August 2020 [12].

Ecologic studies have provided valuable initial understandings of the effects of various exposures on population-level disease outcomes. Innovative ecologic studies of lactation on breast cancer [13], salt intake and blood pressure [14], and male circumcision on HIV transmission [15], have provided important insights into causal hypotheses at the individual-level. Other advantages of these studies includes the ability to efficiently examine trends over time, similar to the analysis here where the BCG-COVID relationships were reassessed as the pandemic progressed. Further, ecologic studies are a critical part of the methodological toolbox for evaluating the impact of policy changes and interventions by examining pre and post population-level outcomes.

However, the limitations of ecologic studies have also been well-described and many are magnified here. First, the ecologic fallacy describes the hazards of using population-level data to make inferences at the individual level [16]. Second, neither dichotomising countries by national BCG vaccination policies nor the number of years policies were in effect captures the substantial variability in actual vaccination coverage [3, 17]. Third, particularly in the early phases of the pandemic, there was wide variation in testing rates, case definitions, and case reporting. Recent modeling estimates indicate that regions with generally high BCG vaccination rates have estimated COVID-19 deaths rates 10 to 14 times greater than reported estimates while regions with low BCG vaccination rates have estimated mortality rates 1.4 to 2 times greater than reported estimates [18], a bias that would tend to produce a spurious finding of vaccine efficacy [19]. Fourth, countries have, to varying extents, implemented a range of interventions to limit the spread of SARS-CoV-2 and ecologic studies to date have not been able to fully adjust for this variation. Fifth, previous studies have yet to standardise for different distributions of effect modifiers (e.g. age, sex, risk factors). Finally, analyses that are snapshots in time do not account for the dynamic nature of the pandemic as countries have reached peaks in COVID-19 burden at different times and have typically experienced a series of sub-epidemics within their borders.

Subsequent ecologic studies of the relationship between BCG vaccination and COVID-19 attempted to overcome these biases. For example, several studies attempted to account for potential exposure misclassification by using vaccine coverage rather than policy as the exposure of interest [11, 20–28]. Others used multivariable frameworks to account for confounding [11, 22, 23, 25, 27–35], but important confounding variables such as health care access, health care quality, and health consciousness were typically not addressed. Moreover, potential differential outcome misclassification bias, coexistence of COVID-19 mitigation policies, and the dynamically changing nature of the pandemic remain and have yet to be fully accounted for in these analyses.

Conclusion

Although ecologic relationships may point the way for additional research, our findings suggest that significant caution is in order in their interpretation. We agree with the WHO’s recommendation that, in the context of COVID-19, the BCG vaccine should only be used in RCTs. While we await those data, the present study underscores the need for caution in evaluating COVID-19 ecologic analyses, and highlights the limitations of this study design more broadly.

Input data used for analyses.

(XLSX)

Click here for additional data file.

24 May 2022

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Wenping Gong, Ph.D.

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Thank you for stating the following financial disclosure:

ML and PL receive funding from the Sydney E. Frank Foundation

Please state what role the funders took in the study.  If the funders had no role, please state: "The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript."

If this statement is not correct you must amend it as needed.

Please include this amended Role of Funder statement in your cover letter; we will change the online submission form on your behalf.

3. Thank you for stating the following in the Acknowledgments/Funding Section of your manuscript:

This work was supported by the Sidney E Frank Foundation.

Please note that funding information should not appear in the Acknowledgments section or other areas of your manuscript. We will only publish funding information present in the Funding Statement section of the online submission form.

Please remove any funding-related text from the manuscript and let us know how you would like to update your Funding Statement. Currently, your Funding Statement reads as follows:

ML and PL receive funding from the Sydney E. Frank Foundation

Please include your amended statements within your cover letter; we will change the online submission form on your behalf.

4. We note that you have stated that you will provide repository information for your data at acceptance. Should your manuscript be accepted for publication, we will hold it until you provide the relevant accession numbers or DOIs necessary to access your data. If you wish to make changes to your Data Availability statement, please describe these changes in your cover letter and we will update your Data Availability statement to reflect the information you provide.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Partly

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: No

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: No

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: This is a useful contribution to the literature.

In the Conclusions, the authors frame the value of ecologic studies as hypothesis generating and findings should be interpreted with caution.

Ecologic studies certainly have value and indeed may be the most appropriate data for helping to answer certain questions (eg the impact of policy changes on population level outcomes). I would suggest moderating these claims.

Reviewer #2: Although authors referred to the Miller´s paper as the main point for the analysis, there are several other reports to be taken in consideration that authors omitted. Then a paper with only 14 references for such an important topic. It would recommended to work on those papers to enrich the discussion and conclusions.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: Yes: Enrique Teran

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

3 Jul 2022

Please see separate file "Response to reviewers"

Submitted filename: Response to Reviewers (1).docx

Click here for additional data file.

7 Sep 2022

Spurious early ecological association suggesting BCG vaccination effectiveness for COVID-19

PONE-D-21-36915R1

Dear Dr. Mark Lurie ,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Wenping Gong, Ph.D.

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #2: N/A

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #2: Thank you for take care of all suggestion given, particularly to increase the number of references. No further changes are necessary from my side.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #2: Yes: Enrique Teran

**********

12 Sep 2022

PONE-D-21-36915R1

Spurious early ecological association suggesting BCG vaccination effectiveness for COVID-19

Dear Dr. Lurie:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Wenping Gong

Academic Editor

PLOS ONE