Accuracy of health-related information regarding COVID-19 on Twitter during a global pandemic.

This study was performed to analyze the accuracy of health-related information on Twitter during the coronavirus disease 2019 (COVID-19) pandemic. Authors queried Twitter on three dates for information regarding COVID-19 and five terms (cure, emergency or emergency room, prevent or prevention, treat or treatments, vitamins or supplements) assessing the first 25 results with health-related information. Tweets were authoritative if written by governments, hospitals, or physicians. Two physicians assessed each tweet for accuracy. Metrics were compared between accurate and inaccurate tweets using χ 2 analysis and Mann-Whitney U. A total of 25.4% of tweets were inaccurate. Accurate tweets were more likely written by Twitter authenticated authors (49.8% vs. 20.9%, 28.9% difference, 95% confidence interval [CI]: 17.7-38.2) with accurate tweet authors having more followers (19,491 vs. 7346; 3446 difference, 95% CI: 234-14,054) versus inaccurate tweet authors. Likes, retweets, tweet length, botometer scores, writing grade level, and rank order did not differ between accurate and inaccurate tweets. We found 1/4 of health-related COVID-19 tweets inaccurate indicating that the public should not rely on COVID-19 health information written on Twitter. Ideally, improved government regulatory authority, public/private industry oversight, independent fact-checking, and artificial intelligence algorithms are needed to ensure inaccurate information on Twitter is removed.

INTRODUCTION

In December, 2019, a novel coronavirus was identified as the cause of pneumonia in a cluster of patients in Wuhan, China (Zhu et al., 2020). Since this initial outbreak, the identified virus, severe acute respiratory syndrome coronavirus 2 (SARS‐CoV‐2), has spread across the globe with the World Health Organization (WHO) declaring a global pandemic on March 11, 2020 (World Health Organization, 2020a). As of April 13, 2021 there were over 136 million cases and 2.9 million deaths reported worldwide (World Health Organization, 2021). WHO control of this pandemic has involved public health measures including quarantine of exposed individuals, isolation of infected individuals, and social distancing with a heavy reliance on the public to follow recommendations from national and international experts and governments. Subsequently, the dissemination of information has become an important component of regulating behavior and guiding the worldwide response to this disease.

One of the many means of transmitting health information to the public is via social media. Twitter use rose 23% in 2020, largely due to the influence of social distancing and individuals seeking information regarding COVID‐19 (Kafka, 2020). As Twitter use increased, questions have arisen about the spread of COVID‐19‐related misinformation on this platform (Frenkel et al., 2020). During previous infectious disease outbreaks (2012 Middle East respiratory syndrome coronavirus (MERS‐CoV), 2013 H7N9, 2014 Ebola, and 2016 Zika), social media was used positively as an avenue for identifying and tracking infections, gauging a community's response to those outbreaks, and providing information to at‐risk populations (Al‐Surimi et al. 2017; Fung et al., 2016, 2013; Rudra et al., 2018) However, the spread of inaccurate information during past disease outbreaks occurred in 10%–60% of Ebola‐related, 61% of yellow fever‐related, 25% of diphtheria‐related, and 5%–10% of H1NI‐related tweets (Chew & Eysenbach, 2010; Ortiz‐Martinez & Jimenez‐Arcia, 2017; Oyeyemi et al., 2014; Pandey et al., 2010; Porat et al., 2019; Sell et al., 2020) More recently, tweets related to COVID‐19 have contained misinformation in 25%–39% of cases and unverifiable information in 17% of tweets (Brennan et al. 2020; Kouzy et al., 2020).

The International Fact‐Checking Network (IFCN) has labeled COVID‐19 as “the biggest challenge fact‐checkers have ever faced” (Suarez, 2020). Twitter is one of the most common means of spreading public health information with nearly 123 million COVID‐19‐related tweets posted between January 21 and May 15, 2020, the majority of which were posted in English (Chen et al., 2020). The WHO states that in addition to a global pandemic related to COVID‐19, there is a COVID‐19 infodemic or an “excess amount of information (misinformation) about a problem which makes it difficult to identify a solution” (Department of Global Communications, 2020) According to the WHO, an infodemic allows for the “spread of misinformation, disinformation, and rumors during a health emergency,” hampers an effective public health response, and has the potential to create confusion and distrust among populations (Department of Global Communications, 2020).

Because Twitter is so widely used a public source of communication and news, the accuracy, relevancy, and timeliness of information is important. For this reason, we chose to study health‐related information regarding Twitter during this COVID‐19 global pandemic. Our primary goal was to determine the accuracy of health‐related tweets related to COVID‐19. A secondary goal was to determine features associated with accurate tweets.

METHODS

Between March 24 and April 5, 2020, five new Twitter accounts were established with no followers, no accounts followed, default of English language, people (anyone), location (anywhere), and highly sensitive content turned off. Before performing the study, two physician study authors created five sets of search terms to be used. Search terms were chosen to elicit tweets that addressed common issues surrounding the management of COVID‐19. These included “COVID‐19” plus either “cure,” “treat or treatment,” “prevent or prevention,” “emergency or emergency room,” or “supplements or vitamins.” Tweets were categorized as having an authoritative source if they were written by a government, physician/physician group, or hospital/hospital system. Tweets were defined as medical or health‐related if they contained medical or health information presented as a fact, a recommendation, a statement, or an opinion. Tweets were categorized as news if they self‐reported that the author was a news organization or worked for a news organization. Tweets that were primarily political, memes or jokes, or religious without health or medical claims were excluded.

Before initiating the study, a 2‐h training session took place with all study authors (data abstractors) that emphasized definitions, uniform tweet reviews, and coding of information to be placed into an Excel spreadsheet. Two weeks before initiating this study, using “influenza” as a practice search term, all data abstractors simultaneously analyzed 10 non‐COVID‐19‐related tweets to ensure uniform reviews and information collection. During the study, data abstraction, data entry, and coding rules were rereviewed with abstractors by the principal investigator after each 25 tweets. The principal investigator arbitrated all data collection and coding questions on an ongoing basis.

On three separate dates (Friday, April 17; Wednesday, April 22; Saturday, April 25, 2020) between 4:00 and 8:00 p.m. Eastern Standard Time the searches (queries) were performed on Twitter using previously selected terms. Each term was assigned to an individual study author/abstractor during this time period such that only one individual performed each search for only their assigned terms on each given day. To perform their search, five study authors logged into their new/native Twitter account and entered “COVID‐19” into the search box within the Twitter search box plus either “cure,” “treat or treatment,” “prevent or prevention,” “emergency or emergency room,” or “supplements or vitamins” such that there was no overlap of searches. This time of day was selected to coincide with the peak time for retweets and click‐throughs on Twitter across the United States (Sailer, 2019). On each date, the top 25 retained tweets for each search with any health or medical‐related information were copied into a spreadsheet. Twenty‐five tweets were chosen as the search limit since it was estimated that each tweet and accompanying links/figures/pictures could be read within one minute and the average half‐life of a tweet is 18–24 min with engagement and retweets tapering rapidly after this period (Wilson, 2016). For each tweet, the following information was recorded: The tweet, any tweet hyperlinks, the author, author's country listed on their public profile, author's credentials, author's Twitter verification status, the number of followers, number of retweets, number of likes, and the tweet's rank order within that day's search. Duplicate tweets were excluded.

An online tool termed a Botometer developed by the Observatory on Social Media and the Network Science Institute at Indiana University was used to generate a score that estimated whether or not authors of Tweets exhibited bot‐like activity (https://botometer.osome.iu.edu/; Botometer, 2021). Bots are automated programs that generate messages, follow accounts, reply to or share hashtags via automation or machine learning. The Botometer uses machine learning to characterize tweet authors based on user data, temporal features/patterns, content, friends/retweets, networking/links, and sentiment‐related features. According to the Pew Research Center, a Botometer score > 0.43 on a 0–1 scale (>2.15 on a 5‐point scale) is the optimum cutoff for classifying a Tweet as more likely to be written by a bot than a human (Wojcik et al., 2018).

Accuracy

Two physician authors independently reviewed and categorized information within each tweet as accurate/generally accepted or unproven/inaccurate. Before assessment, identifying information (author, affiliations, sponsors, advertisements, videos, nonessential pictures) was removed from each tweet to allow blind assessment. Accurate or generally accepted information was defined as that which agreed with the National Institutes of Health guidelines, Infectious Disease Society of America guidelines, WHO, Centers for Disease Control and Prevention, American College of Emergency Physicians, American Academy of Pediatrics and current major textbooks in emergency medicine, infectious disease, internal medicine, and pediatrics. In addition, reviewers were allowed to search the National Library of Medicine (PubMed) for original articles and the Cochrane Database to analyze the accuracy of tweets. For tweets with more than one health‐related statement, it was predetermined that the presence of any single inaccurate statement would lead to the tweet being categorized as inaccurate. Disagreement between two reviewers were settled using a third physician author.

Statistics

Prior studies found that internet and social media‐based medical information was frequently incorrect with 12%–40% of health‐related tweets described as untrustworthy or inaccurate (Albalawi et al. 2019; Gage‐Bouchard et al., 2018; Kedzior et al., 2019; Love et al., 2013; Shah et al., 2019). Assuming an inaccuracy rate within this range, it was estimated that a sample size of at least 369 tweets would be needed to derive an overall accuracy with 95% confidence that was within 5% of these values.

All data were treated as nonparametric. Categorical data were compared between accurate and inaccurate tweets using chi‐squared analysis or Fisher's exact test. Pairwise comparisons of continuous and ordinal data were made using the Mann–Whitney U test. p Values were adjusted for multiple comparisons using Benjamini and Hochberg's method (McDonald, 2014). Tweets were ranked based on their order within a search. Spearman rank order was used to assess the correlation between tweet accuracy and their respective rank order.

Interrater correlation for initial tweet accuracy was calculated using Cohen's kappa. A kappa coefficient was considered almost perfect at 0.81–1, showed substantial or good agreement at 0.61–0.80, moderate agreement at 0.41–0.60, fair agreement at 0.21–0.40, slight agreement at 0.01–0.20, and less than chance at <0.

Data were analyzed using MedCalc (MedCalc Statistical software, v18.2.1; MedCalc Software Bvba).

RESULTS

There were 375 tweets collected during the study period with 17 duplicates deleted leaving 358 evaluable tweets. Two hundred and sixty‐seven tweets (74.6%, 95% CI: 69.8–78.8) were graded as accurate. Tweets with the search terms “COVID‐19” plus “cure” were more likely to be inaccurate compared to other tweets (Table 1).

Table 1

Accuracy of COVID‐19 tweets for each query‐search

Search term(s)	Number accurate/total (%)	95% Confidence interval
Cure	34/69 (49.3%)	37.1%–61.5%
Emergency room or ER	58/70 (82.9%)	72.4%–89.9%
Prevent or prevention	66/75 (88%)	78.7%–93.6%
Treat or treatment	56/73 (76.7%)	65.8%–84.9%
Vitamins or supplements	53/71 (74.7%)	63.5%–83.3%
Total	267/358 (74.6%)	65.9%–84.1%

Abbreviation: COVID‐19, coronavirus disease 2019.

Authoritative authors wrote 69 tweets (31 government, 25 physicians, and 13 hospital tweets) and nonauthoritative authors wrote 289 tweets. A total of 67 of 69 authoritative tweets were accurate compared to 200 of 289 nonauthoritative tweets (97.1% vs. 69.2%, 27.9% difference, 95% CI: 19.2–33.8). The inaccurate tweets were written by authoritative authors comprised two physician tweets claiming COVID‐19 was cured with vitamin C (Table 2). Of the subset of authoritative authors listed as government or hospitals/hospital systems, 41 (100%, 95% CI: 91–100) tweets were accurate. A total of 64 accurate and 25 inaccurate tweets were authored by self‐reported news organizations (24% vs. 27.5%, −3.5% difference, 95% CI: −14.5 to 6.3).

Table 2

Inaccurate tweets by category

Inaccuracya	Number of tweetsb
Hydroxychloroquine or chloroquine can cure or have cured COVID‐19 c	23
Herbs or supplements can cure or have cured COVID‐19 (most common supplement Artemis followed by coconut oil, garlic, ginger, honey, lemon, lime, melatonin, Peruvian bark, probiotics, turmeric)	18
Vitamin C can cure or has cured COVID‐19	13
Zinc can cure or has cured COVID‐19, zinc deficiency causes COVID‐19	10
Bleach or chlorine dioxide ingestion—includes 3 mentions of injecting bleach and 2 mentions of vaping bleach	10
Big pharmaceutical companies are against a cure (3 mentions of big pharmaceutical companies blocking the use of hydroxychloroquine/chloroquine)	5
Azithromycin	4
Unknown product cures COVID‐19 (uncertain if drug or supplement)	4
Ultraviolet light or sunlight will cure people with COVID‐19 (these tweets do not refer to the use of UV light or sunlight to kill viruses on surfaces)	3
Vaccines weaken the immune system and will worsen COVID‐19	2
States are purposefully undercounting cases to hide real mortality	2
One each of the following items were inaccurate: Cures for COVID‐19 = one each for breast milk, camel urine, cannabis, diet, hand sanitizer, homeopathy, immune globulin (nonspecific, pooled), placental cells, montelukast, vitamin A, vitamin D, whiskey Death panels are the cause of COVID‐19 mortality in the United States	1 Each (total subset 13 tweets)

Abbreviations: COVID‐19, coronavirus disease 2019; UV, ultraviolet.

Tweets that stated that supplements and vitamins (or a particular diet) cured COVID‐19 were labeled as inaccurate. However, if tweets stated they supported or potentially strengthened the immune system, they were not labeled as inaccurate.

Total adds up to more than 91 since multiple tweets listed more than one inaccurate product or statement.

These were only labeled as inaccurate if the tweet stated these products cured COVID‐19. Tweets that stated they hydroxychloroquine/zinc/azithromycin might have antiviral properties without stating they cured COVID‐19 were not labeled as inaccurate. At the time of the study, definitive studies proving the ineffectiveness of hydroxychloroquine had not yet been published.

Overall, accurate tweets were significantly more likely to be written by authoritative sources (25.1% vs. 2.2%, 22.9% difference, 95% CI: 15.6–28.6) and authors verified by Twitter (49.8% vs. 20.9%, 28.9% difference, 95% CI: 17.7–38.2). Authors of accurate tweets had significantly more followers than authors of inaccurate tweets (19,491 vs. 7346; 3446 difference, 95% CI: 234–14,054). The number of likes, tweets authored by news organizations, Botometer scores, retweets, tweet length, and Flesch–Kincaid grade level did not differ between accurate and inaccurate tweets (Table 3). North America was the most common author location, N = 171 (47.8%), with the United States comprising the source for 153 (42.7%) of all tweets (Table 3).

Table 3

Comparison of accurate versus inaccurate tweets

Features a	Accurate N  = 267	Inaccurate N  = 91	Differences in 95% confidence intervals	p Value b
Authoritative source (government, hospital or physician)	67 (25.1%)	2 (2.2%)	22.9% (15.6–28.6)	<0.01
News tweet—author self‐report as news organization	64 (24%)	25 (27.5%)	−3.5% (−15.1 to 6.8)	0.67
News tweet/link—author self‐report or link to news organization	98 (36.7%)	40 (44%)	−7.5% (−19 to 4.1)	0.33
Author's location c				0.23
North America	135 (50.6%)	36 (39.6%)
Africa	21 (7.9%)	7 (7.7%)
Asia	23 (8.6%)	5 (5.5%)
Europe	21 (7.9%)	5 (5.5%)
Australia/New Zealand	4 (1.5%)	2 (2.2%)
South America	1 (0.4%)	0
No location listed	62 (23.3%)	36 (39.6%)
Author verified by Twitter	133 (49.8%)	19 (20.9%)	28.9% (17.7–38.2)	<.01
Number of followers	19,491 (1697–237,214)	7346 (877–71,025)	3446 (234–14054)	0.05
Botometer score > 0.43 (>2.15 on 5‐point scale) d	114/248 (46%)	25/72 (34.7%)	11.3% (−2.5 to 23.6)	0.22
Botometer score > 0.43 (>2.15) in subset labeled news tweet/link d	52/94 (55.3%)	13/31 (41.9%)	13.3% (−8.2 to 33.1)	0.34
Number of retweets (shares)	36 (6–183)	45 (7–146)	0 (−12 to 9)	0.85
Number of likes	85 (10–334)	69 (17–313)	−1 (−23 to 17)	0.87
Tweet length (number of characters)	235 (155–264)	222 (149–265)	−6 (−27 to 12)	0.61
Flesch–Kincaid grade level	10.3 (7.6–14.2)	9.7 (7.1–11.8)	−1.2 (−2.3 to 0)	0.14

Median (interquartile range) for continuous and ordinal data.

p Values were corrected for multiple comparisons using the Benjamini and Hochberg adjustment with a corrected value ≤0.05 considered significant (McDonald, 2014).

No location was listed for 98 authors, with North America comprising 171 tweets (US 153, Canada 18), Africa 28 tweets (Nigeria 14, South Africa 6, Kenya 4, Democratic Republic of Congo 2, 1 each Somalia, Uganda), Asia 28 tweets (India 8, Philippines 7, Pakistan 6, China 2, 1 each Hong Kong, Malaysia, Oman, Saudi Arabia, Thailand), Europe 26 tweets (United Kingdom 19, France 2, Netherlands 2, 1 each Belgium, Romania, Spain), Australia/New Zealand 6 tweets, and South America (Argentina) 1 tweet.

Scores could not be calculated by the Botometer for 19 accurate and 19 inaccurate Tweets, Botometer scores could not be calculated for 13 Tweets in subset of news related/news link tweets.

The median overall rank order of retained tweets was 37 (95% CI: 32–46) with a range of 1–219. The rank order of tweets was not associated with tweet accuracy (Spearman's rho = −0.0164, 95% CI: −0.12 to 0.087).

The interrater reliability for the physicians assessing tweet accuracy was substantial (kappa = 0.77, 95% CI: 0.69–0.84).

DISCUSSION

Our study found 74.6% of COVID‐19 health‐related tweets to be accurate. Accurate tweets were more likely to be written by authoritative authors (physicians, hospitals, or government agencies), authors verified by Twitter, and authors with more followers compared to inaccurate tweets. We found no association between tweet accuracy and number of retweets (shares), number of likes, tweet length (number of characters), or Flesch–Kincaid Grade Level. These findings indicate that medical advice related to COVID‐19 on Twitter should not be trusted when written by nonauthoritative sources. The high number of inaccurate tweets indicate that efforts to improve health‐related information on Twitter regarding global outbreaks are needed.

The most common inaccurate tweets in our study comprised recommendations for using unproven prescription medicines, hydroxychloroquine or chloroquine, to treat or prevent COVID‐19 (e.g., “#Hydroxychloroquine with Zn supplement cures #COVID‐19”). The United States Food and Drug Administration (FDA) approves medicines after their effects have been reviewed by the Center for Drug Evaluation and Research and their benefits are found to outweigh known and potential risks for an intended population (Food and Drug Administration, 2019a). Since neither is approved for use in COVID‐19, tweeted recommendations to use hydroxychloroquine or chloroquine for this disease would be classified as unapproved or off‐label by the FDA. Off‐label drug use is allowed by the FDA when there is no FDA‐approved drug to treat a condition or there is insufficient supply of FDA‐approved drugs for a particular condition (Food and Drug Administration, 2019b). Importantly, recent FDA attempts to limit off‐label drug use have been constrained by court rulings that support a pharmaceutical company's right to “free speech” when promoting such use “as long as their statements are not false or misleading” (Kim & Kapcynski, 2017). In a similar manner, the FDA does not have the authority to regulate the free speech (i.e., tweets) of individuals especially if they have no commercial interest in a product (Kim & Kapcynski, 2017).

Unproven herbs, vitamins, and supplements were another common recommendation within inaccurate tweets (e.g. “there are medicinal plants in Madagascar such as Artemisia, which can cure COVID‐19”) These products, collectively termed dietary supplements, are characterized by containing at least one identified dietary ingredients such as a vitamin, mineral, herb, botanical, amino acid, enzyme, or metabolite. Dietary supplements are not approved for use by the FDA and can be brought to market without having been proven safe or effective (Harris, 2000). Product labeling, although regulated in the United States by the FDA (under the 1994 Dietary Supplement and Health Act), is much less stringently regulated. While specific disease claims are prohibited, the FDA allows for claims regarding the structure and function of these supplements with a required label that these products are “not intended to diagnose, treat, prevent, or cure any disease” (Harris, 2000; Owens et al., 2014). A 2014 study of 1300 dietary supplement retail and nonretail websites found that 20%–38% of websites made disease‐related claims and only 8% of retail websites studied included the required FDA disclaimer regarding disease claims (Owens et al., 2014). This study cited a lack of FDA manpower and resources to adequately enforce labeling requirements for dietary supplements. Based on its limited ability to monitor and regulate both drugs and dietary supplements, it is likely the FDA would need more funding and more regulatory authority before it could meaningfully impact misinformation on social media platforms like Twitter.

The Federal Communications Commission (FCC) regulates interstate and international communications with authority over communications law, regulation, and technological innovation (Federal Communications Commission, 2021). While not directly tasked with verifying the accuracy of the information on social media sites, the FCC has the legal authority to interpret important aspects of communication laws regarding websites and social media companies. Section 230 of the Communications Act of 1934 and an amendment (the Telecommunications Act of 1996) state that “no provider or user of an interactive computer service shall be held liable on account of… any action voluntarily taken in good faith to restrict access to or availability of material that the provider or user considers to be obscene, lewd, lascivious, filthy, excessively violent, harassing, or otherwise objectionable, whether or not such material is constitutionally protected” (Johnson, 2020). In addition to protection from criminal lawsuits, experts have interpreted Section 230 as conferring protection to social media companies from civil lawsuits due to harm caused by third‐party content (i.e., harmful or inaccurate information; Barriott & Wilkens, 2020). As such, social media companies have protection from liability when either illegal or inaccurate information is posted as long as those companies make good faith efforts to remove this material. While there may be bipartisan Congressional support to amend or replace Section 230 such that companies would be required to remove or moderate inaccurate or harmful content, it is undetermined if proposed changes would meaningfully improve the accuracy or validity of health‐related content on social media (Reardon, 2020).

As in our study, others found that unverified Twitter accounts contain more misinformation compared to verified accounts (authors authenticated by Twitter (Kouzy et al., 2020). The 25% rate of inaccurate information in our study is consistent with these prior studies of epidemic‐related tweets (Brennan et al., 2020; Kouzy et al., 2020). These findings indicate that misinformation during disease outbreaks continues to be a common and important problem.

While most tweets listed individual authors and contained no hyperlinks, 44% of inaccurate and 36.7% of accurate tweets were authored by or had links to self‐reported news sites. Knowledge regarding the accuracy of news sites is especially important since up to 67% of Americans get their news from social media (Shao et al., 2018). Inaccurate tweets found in our study included those reported by an affiliate of major news organizations (e.g., NEWS4SanAntonia/NBC promoting melatonin for COVID‐19), international news organizations (e.g., abs‐cbn, “virgin coconut oil could be COVID‐19 cure”), entertainment news, and news sites that appeared to mix advocacy with news (e.g., vaxxter.com, “China cures coronavirus with vitamin C”). Identifying whether or not a reported news story is real and verifying the legitimacy of a news source can be difficult. While there is no single technique for verifying whether or not a story is real (vs. fake), experts recommend reading reputable news sources, reading original studies or sources, looking for verification of stories on multiple sites, ensuring there are author attributions for stories, and using fact check tools (i.e., www.snopes.com, www.factcheck.org, www.politifact.com. www.punditfact.com; Spector, 2020). An international not‐for‐profit organization, Health on the Net/HON, exists for verifying the accuracy and legitimacy of health and medical websites (Boyer et al., 2016) However, HON requires that individual websites request a review. The review provided by HON only analyzes eight principles of information authority (e.g., confidentiality, authorship, source attribution, supporting information, transparency, advertising, objectiveness, and financial disclosure) and not the accuracy of the information within websites (Boyer et al., 2016). It is unlikely that a similar review process could be performed on Twitter which relies on a rapid dissemination of information during news cycles with immediate commentary and feedback by users of this platform. Due to the speed of information dissemination and high volumes of information on social media, automated algorithms (i.e., artificial intelligence) are one method for identifying real versus fake news currently being studied (Lara‐Navarra et al., 2020).

Social media bots or automated programs used to engage social media have been accused of spreading misleading information on Twitter (Shi et al., 2020). Botometer scores were high for both accurate and inaccurate authors of Tweets in our study indicating that a substantial amount of COVID‐19‐related information on Twitter might be spread by these automated programs. Tweet accuracy did not appear to be associated with bot scores. Not all bots are malicious, and many bots can perform legitimate functions. Bots that appropriately follow Twitter rules can include those that update real‐time news and weather. Human Twitter authors also can automate a portion of their account by forwarding Facebook posts, sending Real Simple Syndication feeds, or tweeting/retweeting in the absence of the human user. These bot functions would tend not to disseminate misinformation. Importantly, Botometer scores fluctuate over time leading some experts to question their reliability, validity, and reproducibility (Rauchfleisch & Kaiser, 2020). A recent study using the Botometer also incorrectly misclassified humans as bots in 41%–76% of political tweets (Rauchfleisch & Kaiser, 2020). Subsequently, Botometer scores should be interpreted with caution.

In addition to relaying inaccurate medical information, Twitter has been identified as a source for unfounded conspiracy theories during disease outbreaks. Government conspiracies to hide information, control populations, or hide treatments were rumored during Ebola, Zika, influenza‐H1N1, and MERS outbreaks (Sell et al., 2020; Smallman, 2015; Vijaykumar et al., 2018; Yang & Lee, 2020). With COVID‐19, an unfounded rumor linking the SARS‐CoV‐2 virus to 5G (fifth‐generation mobile phone) networks began in late January 2020 and spread rapidly leading to widespread misinformation and the burning of 5G towers in the United Kindgom (Ahmed et al., 2020). Other COVID‐19 conspiracies include the deliberate release of the virus as a bioweapon and pharmaceutical companies blocking known treatments to boost their own drugs and vaccines (Neil & Campbell, 2020). Experts have noted misinformation and conspiracy theories were amplified and retweeted more than accurate tweets during the Zika outbreak (Brennan et al., 2020; Cinelli et al., 2020; Vosoughi et al., 2018). Tweets describing conspiracies in our study included five linking pharmaceutical companies with attempts to suppress available cures, two stating governments were purposefully hiding cases, and one linking death panels to COVID‐19 mortality in the United States.

Multiple organizations (i.e., WHO, United Nations, Centers for Disease Control, and IFCN) and experts have provided recommendations for combating misinformation on social media (Centers for Disease Control and Prevention, 2020; International Fact‐Checking Network, 2020; United Nations, 2020; World Health Organization, 2020b). Proposals include a sustained coordinated effort by independent fact‐checkers like the IFCN, an independent (nonbiased) news media, tracking of misinformation plus dissemination of accurate, easy to read, information by public health and government authorities/agencies, and censorship by social media companies (Brennan et al., 2020; Garrett, 2020; Limaye et al., 2020; LLewyllen, 2020; Sell et al. 2020; Yang & Lee, 2020). Our study was conducted after  Twitter implemented a policy stating they would delete tweets that run the risk of causing harm by spreading misinformation about COVID‐19. Thus, self‐censorship alone may be an ineffective screen for information accuracy within Twitter (Gadde & Derella, 2020; Hern, 2020).

Limitations

Our study evaluated the content of tweets, not whether individuals acted on this information or were harmed by misinformation. Certain inaccurate tweets (e.g., ingesting bleach can cure disease) are potentially more harmful than other inaccurate tweets (e.g., taking vitamin C can cure disease). Despite this difference, we treated all inaccurate statements equally within our study. Future studies might concentrate on the potential of inaccurate tweets to cause harm when evaluating the accuracy of health‐related information on social media.

We only evaluated the first 25 health‐related tweets for each query/search. Since tweet searches partially use timelines to identify relevant tweets, older tweets may not have been identified during searches. In addition to the timeliness of tweets, other factors used by Twitter to rank results are proprietary and unknown. Tweets are also dependent upon the news cycle and evaluation of tweets on different dates or during different news cycles might yield different results.

Since our goal was to analyze the accuracy of the information on Twitter, we did not evaluate the accuracy or legitimacy of website hyperlinks within tweets. Within accurate and inaccurate tweets, there were links to self‐reported news websites, online magazines, blogs, YouTube videos, published/unpublished studies, and health‐related websites (integrated medicine, homeopathy). For websites describing themselves as news‐related, some appeared to report legitimate news (e.g., a television news website wherein a nurse reporting he was cured of COVID‐19 by taking hydroxychloroquine, vitamin C, and ritonavir, an antiretroviral HIV medicine) while others appeared to have a more commercial news slant (e.g., homeopathyplus.com which described homeopathic COVID‐19 remedies).

While the authoritative tweet categories of hospitals and government agencies are easily identifiable, listing of author qualifications or training is often not apparent when analyzing biographies on Twitter. Thus, physician authors who did not detail their qualifications might have been erroneously labeled as nonauthoritative. It is unknown what effect miscategorization in this manner would have on the accuracy of tweets with authoritative authors.

Each author's country was determined by the country listed within their public Twitter profile. This setting is based upon the country selected by a user, can be changed by users, and cannot be independently verified (Twitter Help Center, 2021). Separately, Twitter uses Internet Protocol/IP addresses plus global positioning satellite/GPS information about wireless networks and cell towers to identify countries associated with users for internal purposes. This internal “country setting is non‐public information,” is used by Twitter to customize content and advertisements, and is not available for study (Twitter Help Center, 2021).

Countries with the highest use of Twitter include the United States and many Western European nations (Chen et al., 2020; Singh et al., 2020). Our study only analyzed English language tweets. Thus, our findings might not be applicable to countries outside these regions and to non‐English speaking countries. Moreover, we only analyzed tweets related to five terms chosen by study authors. It is possible that alternate terms or combinations of terms would yield different results.

In the month before initiating our study, Twitter instituted new measures to limit potentially abusive, manipulative, and inaccurate content (Gadde & Derella, 2020). Part of this strategy involves machine learning and automation. Machine learning requires a large data set evaluated over time to create useful algorithms. Because of this, it is possible that Twitter will be better able to identify and remove this problematic content in the future.

CONCLUSION

We found over one‐quarter of health‐related COVID‐19 tweets to be inaccurate. Authoritative authors of tweets, especially government entities and hospitals/hospitals systems, were more likely to post accurate tweets. These findings suggest the public be wary of COVID‐19 health information posted on Twitter.

Ideally, Section 230 of the amended Communications Act should be updated by Congress to hold social media companies responsible for harm from inaccurate health‐related information on their sites if comprehensive attempts are not made to identify and remove this information. This action would incentivize those companies to ensure information is fact‐checked and removed if potentially harmful or misleading. Increased funding for the FDA would allow enhanced review of supplement companies for removal of improper disease and medical claims within advertisements and on their websites. Legislation that requires supplements to be proven safe with oversight and approval by the FDA would potentially decrease adverse events related to supplements. While not directly addressing misinformation, such legislation would increase surveillance of these companies and potentially improve their adherence to regulations disallowing health and medical claims for their products. Other measures to improve the accuracy of health‐related information on social media include enhanced public/private industry oversight, independent fact‐checking, and effective automated, artificial intelligence algorithms. It is doubtful that any single approach will resolve the “infodemic” of COVID‐19 misinformation on Twitter and a multi‐faceted approach encompassing each of these potential solutions is needed to improve the accuracy of health‐related information on social media.

CONFLICT OF INTERESTS

The authors declare that there are no conflict of interests as per the ICJME guidelines.

AUTHOR CONTRIBUTIONS

All authors (Sarah B. Swetland, Ava N. Rothrock, Halle Andris, Bennett Davis, Linh Nguyen, Phil Davis, Steven G. Rothrock) were involved in the conception of study, design of the study, data collection and abstraction, drafting and revision of the manuscript. All authors analyzed the data and Sarah B. Swetland/Ava N. Rothrock/Steven G. Rothrock performed statistical analyses. All authors take responsibility for the paper as a whole.

ETHICS STATEMENT

The data, models, and methodology used in this research are not proprietary.