COVID-19 vaccine misinformation in English-language news media: retrospective cohort study.

OBJECTIVES: To describe COVID-19 vaccine misinformation and track trends over time in traditional news media.
DESIGN: Retrospective cohort study of a large database of online articles, July 2020-June 2021.
SETTING: English-language articles from 100 news outlets with the greatest reach. MAIN OUTCOME MEASURES: Numbers and percentages of articles containing COVID-19 vaccine misinformation over the study period. Further analysis by misinformation themes and whether articles included primary misinformation, fact-checking or simply referred to misinformation.
RESULTS: 41 718 (3.2% of all COVID-19 vaccine articles) contained at least one of the vaccine misinformation themes based on the Boolean string developed for this study. The volume of such articles increased beginning in November 2020, but their percentage of all articles remained essentially stable after October 2020. 56.2% contained at least one mention of a safety theme, followed by development, production, and distribution (26.6%), and conspiracies (15.1%). Of 500 articles through January 2021 randomly selected from those identified by the Boolean string, 223 were not relevant, and 277 included either fact-checking (175 articles), refers to misinformation (87 articles) or primary misinformation (15 articles). In eight study weeks, the reach of these 277 articles (defined as visitors to the sites containing the articles) exceeded 250 million people. Fact-checking accounted for 69.6% of all reach for these articles and the number of such articles increased after November 2020. Overall, approximately 0.1% (95% CI 0.05% to 0.16%) of all articles on COVID-19 vaccines in our sample contained primary misinformation.
CONCLUSIONS: COVID-19 vaccine misinformation in traditional news media is uncommon but has the capacity to reach large numbers of readers and affect the vaccine conversation. Recent increases in fact-checking may counteract some of the misinformation currently circulating. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Use of a very large dataset to identify articles including COVID-19 vaccine misinformation.

Close read of a random subset of identified articles to confirm presence of misinformation and analyse further.

Automated approach to data analysis, with resultant false-positives.

Relative rarity of misinformation in close-read subset, precluding various cross-tabulations.

Inability to generalise our findings beyond the top 100 outlets or to non-English-language sources.

Introduction

Vaccines are among the most successful medical interventions, with immunisation estimated by WHO to prevent 4–5 million deaths per year.1 Antivaccine sentiment is as old as vaccines themselves, and a resurgence in false beliefs about vaccines has been a substantial and worrying aspect of the COVID-19 pandemic,2 particularly in certain developed countries.

Even before the declaration of the COVID-19 pandemic in March 2020, the WHO had identified vaccine hesitancy as one of the top 10 threats to global health.3 Surveys among adults in the USA and Canada have found likely COVID-19 vaccine refusal rates of 20%–25%, high enough to threaten the achievement of herd immunity.4 Conspiratorial beliefs related to COVID-19 vaccines can be common. One study of several Arab countries found that 28% of those surveyed expressed a belief that vaccines are intended to inject microchips into recipients, while 23% believed vaccines could cause infertility.5 Belief in conspiracy theories about COVID-19 and vaccinations is also associated with resistance to disease control measures such as mask-wearing and lockdowns, and reduced willingness to be vaccinated.6

Experimental research has shown higher refusal rates for COVID-19 vaccines among people exposed to recent misinformation about them.7 And misinformation is highly prevalent. The Center for Countering Digital Hate investigated 409 English-language antivaccination social media accounts and found that together they had 58 million followers.8 Misinformation has also circulated about specific classes of COVID-19 vaccines, such as the false claim that mRNA platform vaccines might genetically modify the recipient’s cells.9 Vaccine misinformation is also sometimes promoted on social media by malign state actors with an interest in undermining liberal democracies.10

We are not aware of any published work that exclusively quantifies the extent of vaccine misinformation in traditional news media in the context of the COVID-19 pandemic. Traditional news media can have greater readership and potentially more influence on public attitudes than social media because of the perception that they are more authoritative11; moreover social media postings frequently link to traditional news media sources. Accordingly, this study aims to describe COVID-19 vaccine misinformation and track trends over time in English-language traditional news media, referred to as traditional media moving forward.

Methods

To examine the magnitude of vaccine misinformation in traditional media (Professionally Generated Content, not User Generated Content), we used Cision’s NextGen Communications Cloud platform12 for the period 27 July 2020 (the day that phase 3 trials for the Moderna and Pfizer vaccines began) through 30 June 2021. The NextGen platform is a commercial platform that aggregates online content (including licensed print and traditional media articles via LexisNexis), news-themed blogs, podcasts, television and radio, sourced via webcrawlers and third-party content providers. In total, excluding social media sources, it encompasses a network of over 7 million global sources of print, broadcast and online content. The database aggregates global coverage, with the largest volume of English-language results, to which this study is restricted, coming in descending order from the USA, UK, India, Ireland, Australia and New Zealand, with African and other Asian nations also represented.

Due to the enormous amount of COVID-19 vaccine coverage, the study was restricted to the 100 online news outlets and news-focused blogs with the largest ‘reach’ with respect to COVID-19 vaccines (see online supplemental material 1). Reach data were tracked and obtained from Similarweb13 and are based on desktop users. Reach figures represent the total number of unique visitors to an online publication (not to a specific article) in a given month. To generate a list of outlets with the greatest reach, we first developed a Boolean string14 to identify articles on COVID-19 vaccines, yielding 27 122 380 articles during the study period. We then identified the 10 000 articles in outlets with the greatest reach on the 15th and 16th of the middle month in each quarter in Calendar Year 2020 and combined them into a single file, which included 5815 unique outlets. From these, the 100 outlets with the highest reach for Calendar Year 2020 were identified. Non-news sites such as Good Housekeeping, weather.com and Psychology Today were excluded, based on a consensus of six of the authors, and replaced with the outlet with the next highest reach. These outlets included 1 298 054 articles on COVID-19 vaccines during the study period.

We next sought to identify which of those articles referred to misinformation on COVID-19 vaccines. We built a list of ‘subthemes’ by reviewing the publicly available materials from three vaccine-focused organisations (Planned Parenthood, the Adult Vaccine Access Coalition and the Immunisation Action Coalition). We supplemented these by conducting Google searches using search terms such as “COVID-19 AND conspiracy” or “coronavirus AND misinformation” and reviewing relevant content. Articles associated with each subtheme were generated by Boolean strings. For each subtheme, we randomly selected 20 articles and refined the Boolean string in iterative fashion until all articles not consistent with that subtheme were eliminated. A complete list of the 17 subthemes is provided in table 1, along with definitions for each subtheme. These subthemes were combined into six themes, which also appear in table 1, and together comprised the study’s Boolean search string (see online supplemental material 2). To establish the sensitivity of this approach, we identified 50 articles known to contain misinformation; 49 of these (98%) were identified by the search string.

Table 1

COVID-19 vaccine misinformation themes and subthemes

Theme	Subtheme	Definition
Development, production and distribution	Developed too quickly	Discourse around unverified claims that the development process was rushed
	Not tested properly	Misinformation about the validity of COVID vaccines based on testing history
	Mandates	Conversation about unverified or untrue claims relating to forced vaccinations
Safety	Cause death	False claims that COVID vaccines cause death rather than prevent infection
	DNA/RNA alteration	Claims that COVID-19 vaccines alter DNA, usually referencing novel RNA processes
	Cause autism	Claims that COVID-19 vaccines will cause autism
	Chemicals	Theories that claim COVID vaccines contain dangerous chemicals, including high levels of mercury, bleach, etc.
	Cause COVID	Claims that COVID-19 vaccines will cause COVID-19
	General side effects	Ambiguous or outlying misinformation claims that COVID-19 vaccines cause unverified side effects not mentioned in other themes
Conspiracy	Microchips	General theories about microchip implantation via COVID-19 vaccines
	Population control	Theories that COVID vaccines will be used for population control
Political and financial	Election conspiracies	Misinformation surrounding unverified or false claims that politicians or governments influenced vaccines for election gains
	Doctor kickbacks	False claims that doctors or hospitals are incentivised to issue vaccines because they receive financial kickbacks
Alternatives	Miracle cures	Unfounded claims that various alternative remedies such as hydroxychloroquine or colloidal silver are preferable to vaccines
	Natural immunity	Discourse around misinformation that assumes or claims that ‘natural immunity’ works better than vaccines
Morality and ethics	Fetal remains	Misinformation surrounding the use of fetal or child remains in vaccines
	Human experiments	Claims that COVID vaccines are human experiments, or that unethical/unlawful human experimentation produced them

To determine whether the articles identified by the Boolean misinformation string really contained misinformation and to examine those that did in greater detail, we randomly selected 500 articles from the period 27 July 2020 to 19 January 2021. A random unique identification number was inserted into each record and the 500 with the highest identification numbers were selected. We used Excel’s random number generator to identify a random set of 200 of these articles and these were read by two coders who followed a predetermined coding rubric to tag each article according to the authors’ guidelines. For the coding process, coders used an Excel workbook designed by the authors to categorise and bucket each article by assigning tags that corresponded to both the category of misinformation content and all tracked themes in the primary text. Articles were characterised as either (1) primary misinformation (ie, articles promoting misinformation); (2) fact-checking (ie, articles concerned with determining whether certain assertions were accurate) or (3) refers to misinformation (ie, made reference to misinformation but did not advance it or referenced the COVID-19 vaccine ‘infodemic’ without providing specific fact-checking). Definitions of these terms and example articles are found in online supplemental material 3. Articles were also assigned in a non-mutually exclusive fashion to misinformation themes and subthemes, including designation to a ‘general’ subtheme when elements of the article were specific enough to be included in the parent theme but not specific enough to be included in a subtheme. The coders met with one of the authors (JA) who helped resolve any disagreements and refine coding practices based on the coding rubric. The remaining 300 articles were then coded by one of the two original coders.

We used descriptive statistics to describe the number of articles from the full sample that contained each of the themes and subthemes as well as their trends over time. We carried out qualitative assessments of articles published at the time of peaks in reach and described what the outlets studied were publishing during those times. For the 500-sample subset, we used similar techniques to describe numbers and trends for the three categories of misinformation and selected themes. We also described the reach of the randomly selected vaccine misinformation articles, including their trends over time. To calculate the 95% CIs surrounding the percentage of the random subset containing primary misinformation, we used the Clopper and Pearson method.15

Patient and public involvement statement

Patients or the public were not involved in the design, or conduct, or reporting, or dissemination plans of our research. This study does not involve human participants.

Results

Of the 1 298 054 COVID-19 vaccine articles that appeared between 27 July 2020 and 30 June 2021 in the 100 outlets with the greatest reach, 41 718 (3.2%) contained at least one of the vaccination misinformation themes based on our Boolean string (table 1 and figure 1). In total, 23 448 articles (56.2%) contained at least one mention of a safety theme), followed by development, production and distribution (hereinafter ‘development’; 11 114 articles or 26.6%), conspiracies (6289 articles or 15.1%), morality and ethics (1806 articles or 4.3%), political and financial (1731 articles or 4.1%) and alternative treatments (771 articles or 1.8%). Figure 2 provides the numbers of articles by subtheme. The most common subthemes were general side effects (9693 articles or 23.2%) and developed too quickly (5565 articles or 13.3%), followed by cause COVID-19, not tested properly and chemicals (all with about 3500 articles or 8.3%). Articles referring to election conspiracies, mandates causes autism and natural immunity all had fewer than 300 mentions. The USA was the country of origin for 60% of these articles, followed by the UK (9%), Canada (7%), India (5%) and Australia (4%).

Figure 1

Sources used in study. *Includes English only; excludes social media.

Figure 2

COVID-19 vaccine misinformation subthemes by volume (vaccine misinformation subset July 2020–June 2021).

The volume of vaccine articles was essentially stable from July through early November, after which there was an approximately threefold increase in the number of articles, with a slight decline after March (figure 3A). The number of articles containing misinformation peaked in February (figure 3B). However, viewed as a percentage of all vaccine articles, misinformation-containing articles were a higher percentage from July to mid-October 2020, always exceeding 4% and ranging as high as 8.7%, and have been generally stable at 3% since then (figure 3C).

Figure 3

(A) Volume of COVID-19 vaccine articles, July 2020–June 2021. (B) Volume of COVID-19 vaccine misinformation, July 2020–June 2021. (C) Percentage of misinformation conversation in COVID-19 vaccine articles, July 2020–June 2021.Based on 1 298 054 articles in 100 ooutlets with greatest reach that discussed COVID-19 vaccines.

Trends in the misinformation themes are depicted in figure 4A. The safety theme increased dramatically in November and December (driven by the filing of emergency use authorisations with regulatory agencies) and has remained higher than all other categories ever since, with another peak in early March related to blood clots associated with the AstraZeneca vaccine. Other notable peaks occurred in August–September in development (related to anti-vaccine protests and comments by the US President) and in conspiracy in January–February associated with a pharmacist in Wisconsin, USA who intentionally destroyed COVID-19 vaccines.16

Figure 4

(A) Number of COVID-19 vaccine articles by misinformation theme, July 2020–June 2021. (B) Number of COVID-19 vaccine articles by category of misinformation, July 2020–January 2021. (C) Reach of COVID-19 vaccine articles by misinformation theme, July 2020–January 2021. (D) Reach of COVID-19 vaccine articles by category of misinformation, July 2020–January 2021. (A) Based on 41 718 articles with COVID-19 vaccine misinformation; (B–D) based on 277 individually-coded articles with COVID-19 vaccine misinformation.

Of the 500 articles identified as related to vaccine misinformation by our Boolean string and randomly selected for coder review, 158 (31.6%) were determined to be unrelated to vaccine misinformation (often due to sidebar advertisements or links to other articles) and 65 (13.0%) were unavailable for inspection due to dead links or paywalls.

This left 277 articles (55.4%) that contained at least one misinformation theme. In this latter subset, the most common theme was development (151 articles or 54.5%), followed by safety (124 articles or 44.8%) and conspiracy (67 articles or 24.2%), with the remaining three themes all under 20% (table 2). One hundred and forty-four articles (52.0%) contained more than one theme, most commonly development combined with safety (47.2% of all articles with more than one theme).

Table 2

Summary of COVID-19 vaccine misinformation findings

Source data
Articles on COVID-19 vaccines	27 122 380
Articles on COVID-19 from 100 outlets with greatest reach	1 298 054 (4.8%)
Articles containing misinformation from 100 outlets with greatest reach	41 718 (3.2%)
Close-read articles with misinformation	277/500 (55.4%)
Themes in misinformation conversation (N=41 718)
Safety	23 448 (56.2%)
Development, production and distribution	11 114 (26.6%)
Conspiracies	6289 (15.1%)
Morality and ethics	1806 (4.3%)
Political and financial	1731 (4.1%)
Alternative treatments	771 (1.8%)
Themes in close-read subset (N=277)
Development, production, distribution	151 (54.5%)
Safety	124 (44.8%)
Conspiracies	67 (24.2%)
Political and financial	47 (17.0%)
Alternative treatments	19 (6.9%)
Morality and ethics	18 (6.5%)
Categories in close-read subset (N=277)
Fact-checking	175 (63.2%)
Refers to misinformation	87 (31.4%)
Primary misinformation	15 (5.4%)

Of these 277 articles, the majority (175 articles or 63.2%) were considered fact-checking, followed by refers to misinformation (87 articles or 31.4%), and primary misinformation (15 articles or 5.4%; table 2). This latter category included an article in which a US Senator asserts that China was seeking to ‘hijack or disrupt’ COVID-19 vaccine development17 and another from Australia in which it is asserted that the advent of vaccines will lead to an influx of immigrants.18 The breakdown of these three categories of misinformation was generally similar across the six themes, except political and financial had somewhat more primary misinformation (6 of 47 articles or 12.8%) than other themes.

The most common themes with at least one mention in the 15 primary misinformation articles were development (nine articles), followed by safety and political and financial (six articles each). Seven primary misinformation articles had more than one theme and two had more than three. For both development and safety, the general subtheme contained over half the articles. The three categories of misinformation were generally stable over time, with a notable increase in fact-checking (mostly development and safety) beginning in November (figure 4B).

Overall, of 1 298 054 articles on COVID-19 vaccines, 3.2% contained apparent misinformation according to our search string and 15 articles in our random subset of 500 (3.0%; 95% CI 1.7% to 4.9%) contained primary misinformation. Thus, one can estimate that 0.096% of all COVID-19 vaccine articles (3.2%*3.0%; 95% CI 0.05% to 0.16%) contained primary misinformation. If we remove the 65 articles that were unavailable or behind paywalls, we estimate that 0.11% of all COVID-19 vaccine articles contained primary misinformation (95% CI 0.06% to 0.18%).

Figure 4C depicts the reach of the 277 articles in our random subset. Peaks of approximately 300–450 million unique visitors to the outlets’ websites per week for safety and development were identified in August (driven by several articles on Russian vaccine development on CNN) and September, with additional peaks of almost 350 million for each in the November–December period. These included articles on Bell’s Palsy, US Food and Drug Administration Advisory Committee meetings, and others featuring prominent US Catholic Bishops urging their congregations to avoid the Johnson&Johnson vaccine, claiming it contained fetal components or fetal-derived components. There was also an increase in the conspiracy category to 237 million in the week beginning 11 January, driven by fact-checking regarding claims that the Rockefeller Foundation had planned the COVID-19 pandemic in ‘Operation Lockstep’. Together, articles that had at least one mention of development had 54.8% of all reach in the 277 articles in the study period, while those with safety mentions had 49.6% of reach, with 68.9% of all reach occurring in articles mentioning at least one of these.

Figure 4D examines the reach of these same articles by categories of misinformation, demonstrating that the peaks were largely driven by fact-checking, which accounted for 69.6% of all reach in the study period. Primary misinformation represented 5.5% of all reach, similar to its percentage of articles. The percentage of reach that was related to primary misinformation was higher in the UK (15.6%) than in the USA (8.9%), but the reach of such articles from the USA was larger (346 million vs 68 million).

Overall, in the first half of the study, articles from the USA had 80.8% of reach, compared with 10.8% from the UK, and 2.8% from India. These percentages were generally consistent across themes, except the United Kingdom had a higher share of conspiracy reach (20.2%) and a lower share of alternatives reach (0.2%).

Discussion

Our analysis used a large international database of online traditional media to assess English-language articles on COVID-19 vaccines, allowing us to track trends in misinformation, and particular types of misinformation, as the pandemic unfolded. Of 1 298 054 articles on COVID-19 vaccines in the 100 news-focused outlets with the greatest reach during the study period, approximately 0.1% contained primary misinformation.

However, even these articles had the potential to have substantial reach. In eight particular weeks during the first 6 months of our study, articles among the 277 we identified in our subset analysis as containing misinformation in some form appeared on websites visited by over a quarter of a billion readers. This information has the potential to affect the vaccine conversation and contribute to vaccine hesitancy, particularly if shared avidly among certain groups. Even articles that simply refer to misinformation (31% of the misinformation articles in our close-read subset), but do not refute it, have the potential to influence the conversation, a phenomenon known as the ‘illusory truth effect’.19 Moreover, articles in news outlets with less reach than those included in our study may have a significant impact on the vaccine conversation in local communities.20

The main themes of the articles with misinformation in our close-read subset were development (55% of both articles and reach) and safety (45% of articles and 50% of reach), suggesting that, at least in these more influential outlets, the primary focus was on the process of product approval, as opposed to conspiracy theories, moral/ethical concerns, or alternative therapies. In an encouraging sign, fact-checking outstripped refers to misinformation and primary misinformation articles in most weeks, particularly from November onward, suggesting that traditional media are responding to public concern about misinformation.

A previous study of traditional media by some of the present authors examined all COVID-19-related misinformation and found 1.1 million news articles that disseminated, amplified or reported on misinformation related to the pandemic between 1 January 2020 and 26 May 2020. Much of the misinformation originated with powerful actors, including then-US President Trump, who was the largest driver of the COVID-19 misinformation.21 This study extends that work by including a close reading of a random subset of identified articles.

A study conducted in the early phases of the pandemic found that both mainstream broadcast and print and online media use correlated with accurate beliefs about the nature and lethality of COVID-19, while use of conservative media correlated with conspiracy theories about the pandemic.22 A later study collected hundreds of rumours and conspiracies related to COVID-19 vaccines in traditional and social media but did not estimate the prevalence of misinformation.23

Evidence of the adverse effects of traditional media stories on vaccination rates existed well before COVID-19. In Japan, for example, a dramatic fall in HPV vaccination rates—from 80% in 2013 to less than 1% more recently for girls aged 12–16 years—followed extensive mainstream media coverage of anecdotal ‘vaccine injury’ events.24 In Denmark, the nationwide airing of a controversial television documentary entitled ‘The Vaccinated Girls-Sick and Abandoned’ similarly led to a sharp decline in the uptake of HPV vaccine.13

This study is subject to a number of limitations. First, the size of the database required an automated approach to data analysis, which can lead to imprecision in the estimates generated. However, we also undertook a closer inspection of a random subset of articles identified by the automated approach. This identified a fairly high false-positive rate for our Boolean string, but a separate analysis revealed that its false-negative rate was low. Resource constraints prevented us from closely examining more than 500 articles. The close-read subset was confined to the first part of the study period, although we have no specific reason to think that the false-positive rate would be different in the second part of the study period. Some assurance of generalisability is provided by the generally stable number of articles with each misinformation theme in the full dataset in the first part of the study compared with the second, except for an increase in the safety theme following authorisation of the initial vaccines (figure 4A) and the lack of change in the percentage of all vaccine articles containing misinformation after October 2020 (figure 3C). Second, the close readings are by their nature subjective, although we used multiple coders for a substantial subset of these and used a standardised rubric for all. Third, we cannot generalise our findings beyond the top 100 outlets or to non-English-language sources. Fourth, outlet reach, although a standard measure in media analysis, is based on visitors to particular sites from desktop computers in a given month and excludes mobile device use. While there is no assurance that visitors actually read the articles in question, trends over time are likely still meaningful. Finally, the rarity of primary misinformation in the closely read subset precluded our conducting various cross-tabulations that may have been of interest, including primary misinformation rates by outlet and country, and limit the conclusions that can be drawn from this study.

Policy implications

Public attitudes to COVID-19 vaccines are not static and are responsive to current information and sentiment trends around the vaccines.9 In the UK, vaccine hesitancy dropped from 39% in mid-November 2020 to 11% in mid-April 2021 as the country’s vaccination programme advanced, according to YouGov public opinion polling.25

Given the predominance of the safety and development themes, better collaboration between media outlets and the public health/scientific communities might help ensure that journalists reporting on vaccines understand—and can accurately communicate—the rigorous scientific procedures by which vaccines are developed and tested for safety. Clinicians should be aware of the most prevalent forms of misinformation and be ready to address them in a direct and respectful manner and respected community leaders should be enlisted to counter misinformation.

To mitigate the danger of uncredentialed experts promoting misinformation on COVID-19 vaccines, media outlets could work with public health organisations to generate an inclusive list of vetted science communicators and genuine public health experts who could serve as sources and/or review vaccine-related content to ensure that it is scientifically and medically accurate. The American Association for the Advancement of Science has such a service, called SciLine.26 The WHO’s Vaccine Safety Net also certifies websites that provide reliable information on vaccine safety.27

Similarly, government agencies could conduct trainings for science and medical reporters to improve their understanding about how vaccines are developed, reviewed, and regulated and should address misinformation directly in public education campaigns and through collaborations with community-based organisations and trusted voices within communities. To their credit, governments and other organisations have been trying to do more to meet this need. The US Surgeon General’s report on misinformation provides a model for drawing public attention to this critical issue.28 Governments have held media briefings and health education campaigns, and there has been a proliferation of private sector initiatives such as the US-based Made to Save29 programme and the Mercury Project,30 and the UK-based Vaccine Confidence Project.31 However, more may need to be done to address the effects of misinformation as the pandemic evolves and vaccine development presents new questions, especially in areas where vaccination uptake has slowed.

Media outlets should also be made aware of the various conspiracy theories circulating around vaccines, including the major actors involved in their dissemination, to help them avoid inadvertently sharing misinformation. Media might directly address the various reasons behind vaccine hesitancy in their reports. Reporters should also challenge misinformation shared by public figures and refrain from including misinforming quotes in their coverage without context or fact-checking, preferably in the same report.

Conclusion

COVID-19 vaccine misinformation in traditional media is uncommon but has the capacity to reach large numbers of readers and affect the vaccine conversation. Recent increases in fact-checking may counteract some of the misinformation currently circulating.