Finding the facts in an infodemic: framing effective COVID-19 messages to connect people to authoritative content.

The public's need for timely and trusted COVID-19 information remains high. Governments and global health agencies such as the WHO have sought to disseminate accurate and timely information to counteract misinformation and disinformation that has arisen as part of an 'infodemic'-the overabundance of information on COVID-19-some accurate and some not. In early 2020, WHO began a collaboration with Google to run online public service announcements on COVID-19, in the form of search ads displayed above results of Google Search queries. Web-based text ads can drive online searchers of COVID-19 information to authoritative COVID-19 content but determining what message is most effective is a challenge. WHO wanted to understand which message framing, that is, the way in which ad information is worded for the public, leads searchers to click through to WHO content. WHO tested 71 text ads in English across four COVID-19 topics using a mix of message frames: descriptive, collective, gain, loss, appeals to values and emphasising reasons. Between 11 September 2020 and 23 November 2020, there were 13 million views of the experimental WHO text ads leading to 1.4 million click-throughs to the WHO website. Within the set of 71 ads, there was a large spread between the most effective and least effective messages; for messages on COVID-19, the best performing framings were more than twice as effective as the worst performing framings (18.7% vs 8.5% engagement rate). Health practitioners can apply the messaging tactics WHO found to be successful to rapidly optimise messages for their own public health campaigns and better reach the public with authoritative information. Similar collaboration between big technology companies and governments and global health agencies has the potential to advance public health. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

The public’s need for timely and trusted COVID-19 information remains high, especially as the COVID-19 outbreak and response has been accompanied by an ‘infodemic’—the overabundance of information on COVID-19—some accurate and some not.

Web-based text ads on Google Search provide an opportunity to help users find WHO content on COVID-19 but determining an effective messaging is a challenge.

Optimising message framing in English language text ads informed by behavioural science was shown to be an effective mechanism for connecting millions of users to WHO content.

There was a large spread between the message frames that engaged users more and those that did not. For messages on COVID-19, the best performing framings were more than twice as effective as the worst performing framings (18.7% vs 8.5% engagement rate).

National and global health practitioners may be able to apply the message tactics WHO found to be successful to rapidly test and optimise messages for more successful public health campaigns and better reach the public with trustworthy information.

Introduction

The public’s need for timely and trusted COVID-19 information remains high as cases are still rising in many parts of the world and there is continuing interest in COVID-19-related topics such as COVID-19 vaccines.1 For example, in 2020, COVID-19 was the most searched for topic globally, and the search volume for the term ‘coronavirus vaccine’ increased by 380% between December 2020 and its peak in August 2021.2 3

The COVID-19 pandemic and response have been accompanied by an ‘infodemic’.4 This is an overabundance of information on COVID-19—some accurate and some not—which makes it hard for people to find trustworthy sources and reliable guidance when they need it. Given the extraordinary demand for information, governments and global health agencies such as the WHO have sought to disseminate accurate and timely information to counteract misinformation and disinformation. As part of its approach to disseminate facts and address misinformation, in early 2020, WHO began a collaboration with Google to boost public engagement with WHO’s COVID-19 web content via public service announcements on Google Search.5

The internet offers an opportunity to move beyond historical public health campaign tools, such as posters, public service announcements and billboards, to web-based tools that can better target public health information. WHO, therefore, viewed Google Search ads as an opportunity to reach millions more people with authoritative content about COVID-19. Google provided pro-bono ad grants for WHO to run online public health campaigns, in the form of text ads. Text ads are advertisements that appear on a Google Search results page coupled with a clickable link to the ‘advertiser’s’ site. For instance, users searching for ‘COVID-19’ or related terms on Google Search engine could see text ads from WHO as top results, giving WHO the ability to reach a larger population.

Historically, the wording of public health advertisements has been determined by intuition about what language might be most compelling, often without supporting data.6 The evidence base is still growing on the most effective message framing for public health campaigns, and several systematic and rapid reviews of health message framing on a variety of health topics emphasise the need for clear, repeated, action-oriented messaging from a trusted source.7–10 Positive and gain-framed messages that emphasise a collective or social responsibility message, versus an individual approach and loss-framed approach, may also be more effective for influencing health behaviours. The evidence, however, is mixed.7–10

Further, determining what messaging appeals to which groups of people remains critical.11 For example, a review of vaccine communication found participant characteristics could be moderators of the effect of goal-framed versus loss-framed messaging (eg, perceived risk or loss), but again findings are inconsistent.12 The existing literature does reinforce, however, that adaptation is needed to fit local contexts.13 14 Health messaging should be tailored to reflect the range of influences and drivers of behaviour which may differ across target audiences, based on, for example, gender, age and sociocultural context.12 15 Stakeholder engagement is also important in identifying the most appropriate message framing and medium of message delivery.7

Given the unprecedented scale of the COVID-19 pandemic and influence of social media, previous findings may not directly apply.7 Yet, recent studies of COVID-19 health promotion campaigns have shown that positive framing or messaging that promotes collective responsibility and those noting personal health risks may increase people’s intention to change or adopt behaviours, such as frequent handwashing or social distancing.16–18 These studies are reinforced by others that have promoted a series of strategies and principles based on behavioural science, social marketing and consumer research to inform communication interventions to promote whole population COVID-19 measures, including to motivate social distancing and improve vaccination uptake.19–21

Much of the current evidence describes framing for entire health promotion campaigns, which may differ from trying to direct people to authoritative content. However, informed by this broader context of work, we describe the development of a large-scale digital message optimisation experiment designed to increase traffic to WHO website content. We highlight the findings of the experiment and the learnings for future practice.

WHO COVID-19 global ad campaign

In March 2020, Google announced US$250 million in ad grants to help WHO and over 100 government entities globally spread critical, accurate information on how to prevent the spread of COVID-19 and other measures to help local communities and individuals make better health decisions.5 Text ads on Google are limited to three 30-character headline messages and two brief 90-character descriptions. They are a common part of the experience of the millions of users who search on Google every day. Google text ads appear only when users enter specific related search keywords. The Ads system provides advertisers with standardised, aggregated and anonymised measurement of how many times each ad has been seen (‘impressions’) and clicked on (‘clicks’).

In April 2020, three months after declaring COVID-19 a Public Health Emergency of International Concern (PHEIC), WHO commenced its ad campaign on Google Search to raise credible content and counter misinformation on COVID-19-related information. At the time, the pandemic was in the early stage, and it was unclear when a COVID-19 vaccine might become publicly available. The early headline messages had a particular emphasis on providing generic information on COVID-19 (eg, ‘COVID-19 Worldwide Dashboard–WHO Live World Statistics’) and combating dangerous misinformation (eg, ‘Can masks protect against the new Coronavirus infection?’) that was spreading quickly, often leading to dire health consequences and hindering public health response.22–24 WHO was aware of public interest in disinformation and misinformation around COVID-19 vaccines, as the antivaccination movement had gained momentum during the pandemic. For example, in May 2020, a Pew Research poll found only around 40% of US adults said they would definitely get a COVID-19 vaccine if it were available.25 This suggested there was further opportunity to improve messaging, with a focus on vaccine acceptance, combating misinformation and connecting more users to WHO authoritative COVID-19-related content.

What we did

To increase engagement with WHO content on COVID-19, we—a multidisciplinary team of public health, behavioural economics, medical, advertising and digital marketing experts at WHO and Google—designed 71 new text ads (ie, experimental ads). The text ads covered four COVID-19 topics: COVID-19 general; COVID-19 vaccines; vaccines general; and masks. For each topic, up to three 30-character headline messages were created in English using one of several frames: descriptive, collective, gain, loss, appeals to values and emphasising reasons (see table 1). These frames were selected based on the behavioural science academic literature regarding effective health messaging,9 26 and our various domain expertise. The aim was to understand which message framings appeal to the searcher and lead to a higher click-through to WHO content (see online supplemental figure 1). Given the urgent demand for information about COVID-19 and the increasing need to connect people to authoritative content, we did not undertake user testing during development of the messages.

Table 1

Examples of different headline message framing

Message framing	Description of framing	Examples of headline message
Descriptive frame	Highlights the information that is found on the site	WHO Official UpdatesSymptoms of COVID-19
Collective frame	Highlights the actions of others	Join your communityMillions wear masks
Gain frame	Highlights positive outcomes	Stay safeKeep the spread low
Loss frame	Highlights potential losses	Don’t risk your safetyDon’t miss the latest
Appeals to values	Highlight personal values such as protecting others and exercising personal choice	Protect each otherHeroes wear masks
Emphasising reasons	Highlights that there are reasons supporting	Learn why vaccines workSee what you can do

The 71 text ads were loaded into the Google Ads system to show to Google users worldwide who were searching for COVID-19-related information in English. The ads ran from 11 September 2020 until 23 November 2020. The experimental ads did not change during this period. While these ads were running, WHO was also running existing, non-experimental ads. Ads were triggered by specific search terms, which were consistent across all experimental ads on the same topic (see online supplemental table 1 for common searches that could lead to these ads being shown). Which ad was shown to the user was determined by Google’s automated Ads system. When a user clicked on an ad link, they were taken to relevant, associated content on the WHO website. To measure user engagement with WHO ads, a click-through rate (CTR), defined as total clicks (‘clicks’) that an ad received divided by the total number of times that the ad was shown (‘impressions’), was calculated as the primary impact measure. For search ads, the system reports only total impressions and does not report unique impressions—meaning that the same user may have seen an ad more than once. This ad metric can also be broken down by country and certain inferred demographic factors (eg, gender, age), which are available to all Google Ads advertisers. According to Google documentation,27 when people are signed into their Google account, certain demographics may be inferred from their settings or activity on Google properties, depending on their account status. The data provided by the Ads system are anonymised and aggregated and cannot be traced back to individuals. The CTR was compared between frame types and across individual headlines to identify high and low-performing messages (see online supplemental table 2 for statistical details).

What we found

Between 11 September 2020 and 23 November 2020, there were 13 million views of the experimental WHO text ads in English and 1.4 million click-throughs to the WHO website. India had the largest number of impressions (2.34 million) and clicks (245 744), followed by the UK, USA, Canada, the Philippines and Australia. The CTR varied across countries, reaching close to and exceeding 25% in some cases (eg, Lithuania, Malta and Uganda) and as low as 5%–8% in other countries, such as Argentina and the USA (see online supplemental table 3 for CTR across all available countries). From the inferred demographic factors, we found that males across all age groups had the highest number of impressions (which may reflect the global internet user gap with more men using the internet)28; a CTR between 11% and 13% was similar across genders and age groups (table 2). The 25–34 and 35–44 year age groups had the highest number of impressions and click-throughs to the WHO website.

Table 2

Impressions and clicks by age and gender

Age	Female			Male			Undetermined			Grand total
	Impressions	Clicks	Click- through rate (%)	Impressions	Clicks	Click- through rate (%)	Impressions	Clicks	Click- through rate (%)	Impressions	Clicks	Click- through rate (%)
18–24	781 938	88 693	11	836 878	99 884	12	801	118	15	1 619 617	188 695	12
25–34	956 183	103 446	11	1 577 865	178 603	11	1795	181	10	2 535 843	282 230	11
35–44	673 075	72 710	11	1 271 645	138 969	11	2117	228	11	1 946 837	211 907	11
45–54	533 422	58 989	11	885 388	98 369	11	1154	131	11	1 419 964	157 489	11
55–64	411 169	47 489	12	602 049	67 522	11	844	106	13	1 014 062	115 117	11
65 or more	366 083	42 448	12	495 489	59 400	12	1095	132	12	862 667	101 980	12
Undetermined	236 739	31 464	13	185 571	23 578	13	3 231 836	277 888	9	3 654 146	332 930	9
Total	3 958 609	445 239	11	5 854 885	666 325	11	3 239 642	278 784	9	13 053 136	1 390 348	11

When the Ads system produces a demographic breakdown, it includes two ads that were not a part of the experiment. As a result, the totals in this table differ slightly from other tables.

Within the set of 71 ads there was a large spread between the most effective and least effective messages and frames. Across all message frames, descriptive frames had the highest or second highest overall CTR (table 3), all of which were statistically significantly higher than the average CTR (see online supplemental table 2). The headline that was clicked on at the highest rate was ‘WHO Official Updates: Coronavirus Disease 2020’ (table 4). Other ads across the four topics with high CTRs included descriptive framing such as ‘updates’, ‘information’, ‘current status’, ‘symptoms’ and ‘latest’. These keywords were also frequently searched for and may have matched the information that users were seeking.

Table 3

Ad performance by framing condition

Topic	Framing	Clicks	Impressions	Click-through rate (%)
COVID-19 general	Descriptive	85 507	458 398	18.7
	Gain	28 338	217 450	13.0
	Collective	5266	49 716	10.6
	Emphasising reasons	3360	33 677	10.0
	Appeals to values	3797	40 920	9.3
	Loss	1562	18 433	8.5
Masks	Descriptive	7752	70 062	11.1
	Gain	1606	14 918	10.8
	Appeals to values	6763	67 524	10.0
	Collective	5047	60 575	8.3
	Loss	2836	42 554	6.7
	Emphasising reasons	2219	54 792	4.1
COVID-19 vaccines	Descriptive	649 939	4 306 302	15.1
	Collective	61 583	434 462	14.2
	Emphasising reasons	28 490	223 574	12.7
	Gain	16 431	159 609	10.3
	Loss	30 174	340 421	8.9
	Appeals to values	96 717	1 390 501	7.0
Vaccines general	Gain	12 728	172 174	7.4
	Descriptive	271 401	3 684 051	7.4
	Appeals to values	19 577	307 795	6.4
	Collective	27 862	502 341	5.6
	Emphasising reasons	14 774	266 126	5.6
	Loss	6307	130 704	4.8

Table 4

Highest and lowest performing individual ads by topic and framing

Topic	Highest performing framing	Headline	CTR (%)	Lowest performing framing	Headline	CTR (%)
COVID-19 general	Descriptive	WHO Official Updates: Coronavirus Disease 2020	22.4	Loss	Don’t miss new updates | WHO Official coronavirus info	6.4
COVID-19 vaccines	Descriptive	Official Updates | COVID-19 vaccine updates	17.2	Loss	Don’t miss anything | Coronavirus vaccine updates	3.0
Masks	Appeals to values	Protect each other | Wear a mask. Fight coronavirus	13.4	Emphasising reasons	See how to wear a mask safely | Wear a mask. Fight coronavirus	2.6
Vaccines general	Descriptive	Vaccines are safe | Protect against disease	8.8	Collective	2–3 million lives saved yearly | Get vaccinated	4.0

CTR, click-through rate.

Although descriptive framing performed best across all topics overall, messages that highlighted a positive outcome (ie, gain frame), such as staying informed and keeping the spread low, tended to perform well for general COVID-19, general vaccination and mask topics. Messages that highlighted potential losses tended to perform poorly. For example, the three gain-framed messages were in the top five performing mask ads, while the three loss-framed messages fell in the bottom 50% of mask ads. On average, across all four topics, gain frame had a higher CTR than loss frame.

What we learnt

Our work demonstrates that testing message framing in targeted online search ads can be used to identify messages that can connect large numbers of users to WHO content. We also showed that it is technically possible to launch an online campaign during an infodemic that can bring more people into contact with trustworthy content. Increasing access to WHO’s authoritative content via Google Ads has helped to fill an information gap in countries in the absence of reliable and readily accessible public information.29

After this experiment ended, the top-performing ads were rolled out across other WHO ad campaigns to further increase traffic to the WHO website. In addition to removing the lowest performing ads, additional permutations of high-performing messages were added, and ads were expanded to Spanish, French, Portuguese, Russian and Chinese. Anecdotally, WHO saw an increase of approximately 1.6 million visitors to their website each month as a result of this work. These findings have also informed the message framing of other WHO product headlines, such as its weekly electronic newsletters.

Interestingly, we found that descriptive framing of information often led to higher engagement than other message framings. We suspect that this may be unique to contexts where individuals are actively searching for specific information. As opposed to other types of framing that we examined, descriptive framing may make it easy for users to determine if the hyperlinked information is relevant to their search query, and if so click on it. For example, if a user is searching for recent information on the COVID-19 pandemic, then ‘WHO Official Updates’ may more clearly communicate that the hyperlinked page contains the information that they are looking for than does ‘Don’t miss new updates’. Whether descriptive framing works well in other online contexts, such as social media, should be explored in future work.

Recent research suggests that, although many messaging strategies may be effective across many different countries, they may need to be adapted to local contexts to increase their effectiveness.13 Our findings echo this recommendation. For example, some of the differences that we found between framings were consistent with existing research on effective health communications—such as gain-framed messages performing better on average than loss-framed messages in the mask and vaccine topics, both of these are ‘health-affirming’ behaviours as opposed to ‘illness-detecting’ behaviours.26 However, across topics, countries and demographic factors, the best performing framings sometimes varied. What works best in one language or one country may not be what works best in another language or another county. Governments and global organisations such as WHO will need to consider that focusing on a particular framing might be more or less effective in different languages, countries and communities. To ensure that messages are maximally effective and do not create or exacerbate digital divides, organisations should work closely with local communities and experts to craft or adapt messages for the local context.

There are several limitations to this work. We only examined messages in the context of online search ads in English; our findings may not hold in other online contexts (such as social media), offline contexts or other languages. Additionally, we only examined one outcome—whether or not users clicked on an ad (ie, CTR). There are many other important outcomes that could be examined including whether users understand the information and feel confident about how to use it. Connecting users with authoritative information is only a first step; it is crucial that the information also be accessible, understandable and actionable for a wide audience.30 To maximise the impact early on, organisations may want to initially prioritise connecting individuals to easy read, understandable and actionable information over information that is more difficult for the general public to comprehend or apply in practice.

Another important limitation is that this experiment ran for a little more than 2 months, and we examined message performance for large populations of people. The most effective messaging strategy may vary based on the stage in the pandemic as well as the information needs of different subgroups of a population. For example, highlighting the gains of receiving the COVID-19 vaccine might be a more motivating and effective message now when the vaccine is available in many countries, as compared with this experiment which ran before the vaccine was publicly available. Similarly, effective messages may differ for early adopters of COVID-19 health behaviours (eg, mask wearing, vaccination) versus late adopters or those who are hesitant. When agencies and organisations run health messaging campaigns, it would be beneficial to monitor the performance of different messaging over time and across different subgroups so that they can detect and respond to changing information needs.

There are also limitations due to Google Ads system. Because the artificial intelligence-automated system was largely self-directed, we cannot examine how multiple variables, such as the context of the user search, including the operating system, time of day and day of the week, were combined to optimise the ads. In addition, although online ads are common, executing an online ad campaign has not historically been part of public health training or practice. Therefore, it may not be feasible for some public health teams to launch and manage such a campaign. WHO and other agencies have received ad grants to undertake ad campaigns, and the cost implications may also limit replicability. However, smaller tests of shorter duration could be executed with online ad platforms and enable national or regional public health authorities to optimise messages for their own contexts. In our view, it is also important for agencies and researchers that have the resources to experiment with messaging and share their findings and insights broadly so that they can be used by others.

Historically, online public health ad campaigns were not commonly used by governments or their ministries of health. This has changed somewhat during the COVID-19 pandemic as some governments, such as Australia’s, have adopted them. We believe that even more agencies could do so more often. As part of the Google Ads programme, currently 129 government agencies and other health-related organisations are receiving ad grants to disseminate timely and accurate COVID-19 messages and are applying the messaging tactics we found to be successful to design their ads. As a result of this work, many are using message framing similar to that we found to be effective and, anecdotally, are seeing increases in engagement.

Meaningful partnerships between technology companies, governments and global health agencies have the potential to advance public health. Yet, important challenges and concerns of agencies, such as WHO, exist around working with large technology companies, including that this work may go beyond the ‘normative’ work of the agency, that it may raise ethical concerns such as data ownership and privacy and that it may create real or perceived conflicts of interest. These are important issues that need to be considered and addressed carefully by agencies and companies before working together. In particular, it is important for collaborations to have a high degree of transparency to ensure accountability and help foster confidence in the efforts of public health agencies. If these issues are not appropriately addressed, there may be risks to public perception and trust that may negatively impact the public’s willingness to follow the advice of these agencies.

For this work, we consulted with experts in health policy and ethics to ensure that the work followed standard practice for collecting and examining ad performance data and that there were no ethical concerns. All of the data are owned by WHO, and the metrics used for this work are standard metrics available to all advertisers who use Google Ads. The data are aggregated and anonymised so that no individual can be identified, the data are protected by the Ads system, no additional user data were collected and no data were used for purposes other than examining the performance of these messages. Google provided pro-bono funding for these ads and assisted in the implementation of the experimental ads but otherwise provided no data or non-public information. Discussions are ongoing within WHO to ensure that ethics continue to be managed successfully in future collaborations.

Developments in digital technology, combined with the increased demand for timely and reliable health information during the COVID-19 pandemic, provided WHO with an opportunity to develop a data-driven solution to a pressing public health issue. In doing so, WHO was able to effectively connect more people to its authoritative content than had ever been possible in the past.

Conclusion

WHO is successfully optimising messages as part of a public health ad campaign to increase traffic to its authoritative COVID-19 content, counter misinformation and provide reliable and timely information to those who need it. We have demonstrated that online ads hold promise for national and global health campaigns and offer practitioners a powerful tool to target the public with trustworthy information and to rapidly test and optimise messages for public health campaigns. This way of looking at messaging and changing framing is a major lesson that may also be applicable to other digital platforms, including social media.