Instructional interventions for improving COVID-19 knowledge, attitudes, behaviors: Evidence from a large-scale RCT in India.

Seeking ways to encourage broad compliance with health guidelines during the pandemic, especially among youth, we test two hypotheses pertaining to the optimal design of instructional interventions for improving COVID-19-related knowledge, attitudes, and behaviors. We randomly assigned 8376 lower-middle income youth in urban India to three treatments: a concentrated and targeted fact-based, instructional intervention; a longer instructional intervention that provided the same facts along with underlying scientific concepts; and a control. Relative to existing efforts, we find that both instructional interventions increased COVID-19-related knowledge immediately after intervention. Relative to the shorter fact-based intervention, the longer intervention resulted in sustained improvements in knowledge, attitudes, and self-reported behavior. Instead of reducing attention and comprehension by youth, the longer scientific based treatment appears to have increased understanding and retention of the material. The findings are instrumental to understanding the design of instruction and communication in affecting compliance during this and future pandemics.

RCT Entities:   Population Interventions Outcomes

Introduction

The number of COVID-19 cases worldwide rose to more than 100 million in early 2021 and hit a peak of 5 million new cases per week (WHO, 2021). India has been hit especially hard with 11 million confirmed cases in early 2021 and at its peak in early September there were more than 640,000 new cases per week. Governments around the world are grappling with methods to stop the spread of the virus. Like other public health crises, building a successful response to COVID-19 relies on ensuring broad and informed compliance with health guidelines. Unfortunately, individuals frequently lack accurate knowledge of and/or harbor negative attitudes towards health guidelines, resulting in failures to practice essential hygiene, violations of stay-at-home orders, and other forms of misconduct (Barbanel, 2020; Yan, 2020; Geldsetzer, 2020; Wolf et al., 2020). Misconduct is often more prevalent among disadvantaged populations with less access to reliable information and among young people who myopically believe the risks of non-compliance are low to them and their communities (Wolf et al., 2020; Pirisi, 2000).

To empower individuals to limit the spread of COVID-19, international agencies, governments, and the media have provided a wide range of informational and instructional resources. Potentially limiting their effectiveness, however, these resources are mostly dispersed within and across online sites. Individuals must proactively search for and consolidate accurate information while also filtering out misinformation. Consolidated and targeted instructional interventions might be more effective in improving individual knowledge, attitudes, and behaviors (Bettinghaus, 1986), but due to the newness of this pandemic, little systematic evidence on the effectiveness of such interventions has been documented.

Even when information is targeted and consolidated, the effectiveness of these resources may vary, however, with some creating lasting behavioral change and others failing to do so (Faust and Yaya, 2018). Researchers in the areas of education and public health have demonstrated, for example, that some informational interventions increase knowledge recall, but that this increased recall does not result in behavioral changes (e.g., Davis et al., 2011; Fryer, 2016).

Which factors, according to the literature, affect the effectiveness of informational or instructional interventions? Past studies have shown that interventions affect behavior when messaging is consistent with people's values and worldviews (Bolderdijk et al., 2013), when it appeals to altruistic motivations (Bonafide and Vanable, 2014), or when the messaging is actionable and encourages minor behavioral changes (Dupas, 2011). The efficacy of messaging can also be affected by the amount of information provided. Presenting too much information can confuse or overwhelm individuals. Recent studies confirm that people have limited attention spans and that providing simple and direct instructions substantially improves behavioral outcomes (Beshears et al., 2013; Carvalho and Silverman, 2019; Dizon-Ross, 2019). Furthermore, in information-rich settings such as the COVID-19 pandemic, the nudge of the intervention itself may be more valuable than the actual content (Banerjee et al., 2020).

Information or instruction may also be limited if recipients do not understand the reasoning behind it. Without developing a deeper conceptual understanding, people may become skeptical of information and reluctant to change associated behaviors (Noar et al., 2007; Damgaard and Nielsen, 2018). To our knowledge, few studies systematically explore the important question of whether it is more effective for public health officials to only provide simple and direct information or also provide information that explains underlying concepts. To use an example from the COVID-19 pandemic, are people more likely to wear a mask if they are directly told to do so (and essentially nothing else) or if they told to wear a mask, how the virus is transmitted from one person to another, and how a mask can stop this transmission?

This study tests two hypotheses regarding the optimal approach for designing such instructional interventions. First, we explore whether it is beneficial to concentrate essential COVID-19-related information in an instructional intervention, specifically targeting youth. We hypothesize that providing information on essential facts and instructions of the type provided by international and national public health agencies such as the WHO and CDC, but through a more intensive, concentrated, and targeted instructional intervention, can improve knowledge, attitudes, and behaviors. Second, we explore whether there is a more fundamental problem that information is ineffective because young people do not understand underlying scientific concepts (i.e., the “why”). Therefore, the second hypothesis we test is whether a longer intervention that not only presents the facts but also explains scientific concepts associated with those facts is more effective. Theoretically the answer is not clear: Instruction based on both facts and scientific concepts may deepen recipients' understanding, potentially leading to greater and more lasting improvements in knowledge, attitudes, and behaviors; however, this instruction also increases instructional time and complexity, potentially weakening recipients’ attention.

To test these hypotheses, we designed and implemented a large-scale randomized controlled trial (RCT) with 8376 low-income and lower-middle income youth (ages 15–30) in urban India. Youth were randomly assigned at the individual-level (without strata or blocking) in equal proportions to Facts: the 10-min concentrated and targeted fact-based intervention; Facts plus Concepts: the longer 22-min intervention that provides the same facts plus underlying scientific concepts; or a control group (neither intervention). Estimates from the three possible pair-wise comparisons allow us to test our hypotheses and provide novel evidence on optimal methods for educating disadvantaged youth about COVID-19.

Data and methods

Interventions: Each intervention consisted of a video with English voice-over and subtitles. The Facts intervention consisted of 10 min of video footage with narration. The Facts plus Concepts intervention involved 22 min of video footage with narration. It included the exact same facts, footage, and narration of the Facts intervention (Table S1) but included another 12 min of explanation of related scientific concepts (see Table S2). For both interventions, content consisting of straightforward facts and instructions was largely drawn from COVID-19 instructional materials prepared by the World Health Organization (WHO), the U.S. Centers for Disease Control (U.S. CDC), the Indian Government, and other expert sources (see Table S1). A video production company in India put together the visual and audio content and added the subtitles. We reviewed and piloted the instructional material with staff at the NGO with which we collaborated, who are familiar with the living conditions of the participants. The instructions and recommended behaviors were not only in line with public health recommendations, but were also considered actionable. The videos are publicly available at the following links:

Facts Only: https://www.youtube.com/watch?v=xMN2S3NT3kc

Facts plus Concepts: https://www.youtube.com/watch?v=91Qc-MKFNRU

Sample: The experiment was conducted among low and lower-middle income urban youth (ages 15–30) enrolled in Freedom Employability Academy India (FEA), an eight-year old Indian NGO. FEA offers an intensive, free, year-long training program focused on English language, non-cognitive, and basic computer skills. During normal conditions, FEA operates more than 100 in-person branches in lower and middle-class urban neighborhoods across northern India and accepts any student above the age of 15. Most students at FEA are native Hindi speakers who are studying or have recently transitioned to the workforce. After the COVID-19 lockdown, FEA has been in daily online contact with approximately two-thirds of its students (~17,500), providing instruction and exercises through its online platform. For this study, we focused on online-accessible students that had been at FEA for approximately six months or more and were therefore comfortable with the basic English language level of the instructional interventions in this study.

Survey and Experimental Design: The study took place in four stages. First, from April 15–18, 2020 (near the start of the nationwide COVID-19 lockdown in India), a short baseline survey elicited COVID-19-related knowledge, attitudes, and behaviors as well as background information (age, sex, location, and parental education levels). Since the study took place relatively early to the arrival of the pandemic in India, participants did not yet have much systematic exposure to COVID-19 related information disseminated by the Government of India, apart from widespread exposure to instructions of safe behavioral practices. Similar to individuals in virtually all countries, participants were also exposed to a large amount of misinformation, especially from various social media sources. The baseline survey was conducted through Google Forms and was distributed to all students by FEA's teachers. Students could opt out of the baseline survey for a different online activity and could choose to opt out of the remainder of the study at any point in time. Student performance on the study was not tied to class performance and study participants were not at risk of any other forms of harm.

Second, students included in the study were individually randomized in equal numbers to the treatment arms and control (2792 students in each arm—see Fig. 1 ): Facts, Facts plus Concepts, and control. The authors conducted the randomization using Stata 15.1. Specifically, they generated a uniform random number between 0 and 1 for each individual, ranked the individuals on the random number from lowest to highest, and then assigned the top third to the control, the middle third to Facts and the bottom third to Facts plus Concepts. With power set at 0.8 and alpha conservatively set at 0.01 (Bonferonni-adjusted to test for five two-sided hypotheses at the 5% significance level), and with 2792 individuals randomly assigned to each of the two treatment arms and control, the experiment allowed the estimation of minimum detectable effect sizes (MDESs) of 0.09 SDs for each pair-wise treatment comparison. These power calculations do not account for increased statistical precision gained in the favored specification which controls for covariates (baseline knowledge, attitude, behavioral scores, and basic background characteristics including age (years), female (1/0), attended FEA in Delhi (1/0), mother attended senior high or higher (1/0), father attended senior high or higher (1/0)—estimated R-squared of 0.5), and thus reduce the MDES to 0.07 SDs. Given the short duration of the study and the strong ties of FEA with its students, only a small amount of statistical power was expected to be lost to student attrition from baseline through the follow-up surveys (attrition of approximately 5%). The authors built a platform to ensure that students could only see the videos that they were supposed to see; the platform also hosted both follow-up surveys. FEA's teachers directly encouraged students to go to the video platform and to complete the follow-up surveys. The teachers knew which students were participating in the study, but they were not informed as to which student was assigned to which group.

Fig. 1

CONSORT 2010 flow diagram.

Baseline observables were balanced across treatment arms and the control (see Table S3 for the descriptive statistics for the baseline characteristics). We have been unable to find any representative surveys of COVID-19 knowledge, attitudes, or behaviors for India, but at least one non-representative national study suggests that Indians were broadly aware of desirable behaviors (Freed et al., 2021); our baseline results are in line with this study. Third, immediately after the instructional interventions were administered, an initial follow-up survey elicited COVID-19-related knowledge and attitudes from all students. Students watched the videos and completed the first follow-up survey over two days during a window of April 21–24, 2020. Fourth, one week after the treatment was administered (from April 28-May 1, 2020), a second two-day follow-up survey elicited COVID-19-related knowledge, attitudes, and behaviors. Attrition from baseline to the follow-up surveys was low (7.5% for the first and 3.4% for the second follow-up). Although there was some differential attrition in the first follow-up between the Facts treatment arm and the other two treatment arms (differences of approximately 2%), all background variables were balanced among non-attriters in each pair-wise treatment arm comparison. There was no differential attrition among treatment arms in the second follow-up.

Primary Outcomes: Three outcomes were constructed using data from the first follow-up survey. The first outcome was a total knowledge score composed of the number correct out of 26 questions. The second outcome was an applied knowledge score comprising the number correct out of 9 knowledge questions. The questions required participants extrapolate from facts presented in the interventions to new scenarios. The third outcome was an attitudes score, which was a GLS-weighted index using 6 survey questions (Anderson, 2008). Z-scores were calculated for all three outcomes using means and standard deviations of the control group.

Eight outcomes were constructed from the second follow-up survey. The first was a total knowledge score (percent correct out of 27 questions, z-scored). The second was an applied knowledge score (percent correct out of 11 questions, z-scored). The third was an attitudes score (GLS-weighted index using 6 survey questions, z-scored). The last five outcomes were binary self-reported behavior measures (whether participants: wore masks if they went out, cleaned/disinfected surfaces at least once a day, consistently maintained 2 m distance from others outside the home, washed hands thoroughly using soap or sanitizer, and stayed home during the lockdown except for essential trips).

The bulk of the questions for the baseline and follow-up surveys were initially drawn from Pagnini et al. (2020) and adapted to the local context, with input from FEA staff. See Table S8 for a complete list of the questions used.

Statistical Approach: We estimated treatment effects using the regression:where is a particular outcome of interest measured during follow-ups for student i; and are binary indicators for the Facts and Facts plus Concepts treatments, and is the vector of baseline characteristics (Table S3). In cases of missing values for controls we included a missing value dummy variable in the regression. Since treatment effect estimates with and without baseline controls ( were extremely similar, for the sake of brevity, we present the former. We tested for effects by sex by limiting the sample to the appropriate subsample of students. In all cases, we report heteroskedastic-robust standard errors.

Finally, approximately 14% of students were unable to watch the instructional videos due to technical limitations, usually insufficient internet bandwidth. As such, in addition to reporting intent-to-treat (ITT) effects using the above ordinary least squares (OLS) regression we also report treatment-on-the-treated (TOT) effects using instrumental variable (IV) regressions. As expected, TOT estimates are slightly larger than, but substantively the same as, the ITT estimates.

Results

Both instructional interventions had positive effects on COVID-19-related knowledge immediately after the intervention (Table 1 , columns 1–4). The total knowledge score increased by approximately 0.35–0.40 standard deviations (SDs) for the Facts group (p < 0.001***) and 0.30–0.35 SDs for the Facts plus Concepts group (p < 0.001***). Both groups were able to successfully recall the information they learned. The applied knowledge score also increased by approximately 0.30–0.35 SDs for the Facts group (p < 0.001***) and 0.25–0.30 SDs for the Facts plus Concepts group (p < 0.001***), indicating that both groups had moved beyond recall and successfully extrapolated their knowledge to scenarios not explicitly covered by the interventions. Impacts on knowledge were statistically indistinguishable between the two interventions.

Table 1

Impact of educational interventions on COVID-19-related knowledge (z-score).

	(1)	(2)	(3)	(4)	(5)	(6)	(7)	(8)
	Immediately after Interventions				After One Week
	Total Knowledge Score		Applied Knowledge Score		Total Knowledge Score		Applied Knowledge Score
	OLS (ITT)	IV (TOT)	OLS (ITT)	IV (TOT)	OLS (ITT)	IV (TOT)	OLS (ITT)	IV (TOT)
Facts	0.348***	0.395***	0.297***	0.338***	0.108***	0.128***	0.043*	0.051*
SE	(0.028)	(0.032)	(0.028)	(0.032)	(0.025)	(0.030)	(0.026)	(0.031)
p-values	0.000	0.000	0.000	0.000	0.000	0.000	0.098	0.099
Facts + Concepts	0.306***	0.351***	0.259***	0.298***	0.133***	0.163***	0.107***	0.131***
SE	(0.028)	(0.032)	(0.028)	(0.033)	(0.025)	(0.031)	(0.026)	(0.032)
p-values	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000
(Facts + Concepts) - Facts	−0.042	−0.044	−0.038	−0.040	0.025	0.035	0.064**	0.080**
SE	(0.030)	(0.032)	(0.029)	(0.032)	0.026	0.031	0.027	0.032
p-values	0.151	0.167	0.194	0.214	0.336	0.262	0.016	0.012

Notes.

1. ***p < 0.01, **p < 0.05, *p < 0.10.

2. Heteroskedastic-robust standard errors in parentheses.

3. N = 7743 for columns 1–4, N = 8108 for columns 5–8.

4. All results control for student age (years), sex, attend in Delhi or not, father attended senior high or not, mother attended senior high or not, baseline knowledge, attitude, and behavior composite scores, and baseline missing value dummies.

5. Raw Means and SDs for test score outcomes are as follows: (a) total knowledge score immediately after intervention (mean = 13.4, SD = 2.8, total items = 26); (b) applied knowledge score immediately after intervention (mean = 3.5, SD = 1.5, total items = 9); (c) total knowledge score after one week (mean = 15.0, SD = 3.7, total items = 27); (d) applied knowledge score after one week (mean = 5.1, SD = 1.9, total items = 11).

Positive effects on COVID-19-related knowledge were retained, but not fully, after one week (Table 1, columns 5–8). Relative to the control group, total knowledge increased by approximately 0.11–0.13 SDs for Facts (p < 0.001***) and 0.13–0.16 SDs from Facts plus Concepts (p < 0.001***). Applied knowledge also increased by approximately 0.04–0.05 SDs for Facts (p = 0.10*) and 0.11–0.13 SDs from Facts plus Concepts (p < 0.001***). Impacts on applied knowledge were greater for the Facts plus Concepts group (0.06–0.08 SDs, p < 0.01**).

The Facts plus Concepts intervention ultimately also had positive impacts on attitudes compared to Facts alone or to the control group (Table 2 ). Immediately after the interventions, attitudes improved by approximately 0.05 SDs for Facts plus Concepts (p = 0.08*) and 0.04 SDs for Facts (p = 0.15) relative to the control group. Impacts were statistically indistinguishable between the two interventions. After one week, only Facts plus Concepts retained positive effects on attitudes, with effect sizes of 0.04–0.06 SDs relative either to control or Facts.

Table 2

Impact of educational interventions on COVID-19-related attitudes (z-score).

	(1)	(2)	(3)	(4)
	Immediately After Interventions		After One Week
	OLS (ITT)	IV (TOT)	OLS (ITT)	IV (TOT)
Facts	0.037	0.042	0.003	0.004
SE	(0.026)	(0.029)	(0.026)	(0.031)
p-values	0.152	0.150	0.902	0.903
Facts + Concepts	0.045*	0.052*	0.044*	0.054*
SE	(0.026)	(0.030)	(0.026)	(0.032)
p-values	0.080	0.080	0.092	0.092
(Facts + Concepts) - Facts	0.008	0.010	0.041	0.050
SE	(0.026)	(0.029)	(0.026)	(0.031)
p-values	0.747	0.734	0.121	0.112

Notes.

1. ***p < 0.01, **p < 0.05, *p < 0.10.

2. Heteroskedastic-robust standard errors in parentheses.

3. N = 7742 for columns 1–2, N = 8095 for columns 3–4.

4. All results control for student age (years), sex, attend in Delhi or not, father attended senior high or not, mother attended senior high or not, baseline knowledge, attitude, and behavior composite scores, and baseline missing value dummies.

5. Attitudinal index outcomes are in z-scores with mean = 0 and SD = 1.

Finally, only Facts plus Concepts had a statistically significant effect on self-reported COVID-19-related behaviors (Table 3 ). Self-reported non-compliance in the control group ranged from 28.2% (for wearing a mask whenever the subject leaves home to 38.9% (for maintaining a 2 m distance from others outside the home). Compared to the control condition, Facts plus Concepts increased the likelihood that individuals reported consistently wearing a mask when going outside by 2.2–2.6 percentage points (p = 0.06*). Compared to Facts, Facts plus Concepts increased the reported likelihood of wearing a mask by 2.9–3.5 percentage points. (p < 0.01***) and the reported likelihood of regularly cleaning and disinfecting surfaces at home by 3.4–4.1 percentage points (p < 0.01***). Stated differently, relative to the shorter facts-based intervention, the longer facts-plus-concepts-based intervention decreased the proportion of those reporting not wearing a mask by 10–12 percent and the proportion of those reporting not regularly cleaning/disinfecting surfaces at home by 9–11 percent. Conservatively, adjusting estimates of impacts of the five behavioral measures for multiple hypothesis testing based on what was prespecified in our pre-analysis plan (AEARCTR-0005739), we find that mask-wearing and cleaning surfaces are positively and significantly affected (at the 5% level) by the Facts plus Concepts intervention versus Facts alone. At the same time, for self-reported mask-wearing, the estimated impact of the Facts plus Concepts intervention versus the control does not retain statistical significance.

Table 3

Impact of educational interventions on COVID-19-related behavior after one week.

	(1)	(2)	(3)	(4)	(5)	(6)	(7)	(8)	(9)	(10)
	Wore mask every time		Cleaned surfaces at least once a day		Always kept 2 m distance		Washed hands well		Stayed home
	OLS (ITT)	IV (TOT)	OLS (ITT)	IV (TOT)	OLS (ITT)	IV (TOT)	OLS (ITT)	IV (TOT)	OLS (ITT)	IV (TOT)
Facts	−0.007	−0.009	−0.017	−0.020	−0.004	−0.005	−0.013	−0.015	0.010	0.011
SE	(0.012)	(0.014)	(0.013)	(0.015)	(0.013)	(0.015)	(0.013)	(0.015)	(0.012)	(0.014)
p-values	0.525	0.520	0.201	0.198	0.743	0.742	0.321	0.321	0.410	0.410
Facts + Concepts	0.022*	0.026*	0.017	0.021	0.013	0.016	−0.005	−0.006	0.017	0.020
SE	(0.011)	(0.014)	(0.013)	(0.016)	(0.013)	(0.016)	(0.013)	(0.016)	(0.012)	(0.014)
p-values	0.060	0.060	0.179	0.181	0.310	0.310	0.692	0.693	0.149	0.148
(Facts + Concepts) – Facts	0.029**	0.035**	0.034***	0.041***	0.017	0.021	0.008	0.009	0.007	0.009
SE	(0.012)	(0.014)	(0.013)	(0.015)	(0.013)	(0.015)	(0.013)	(0.016)	(0.012)	(0.014)
p-values	0.011	0.011	0.009	0.009	0.179	0.176	0.552	0.566	0.534	0.510
% Non-compliance Control Group	28.2%		35.5%		38.9%		38.4%		30.0%

Notes.

1.***p < 0.01, **p < 0.05, *p < 0.10.

2. Heteroskedastic-robust standard errors in parentheses.

3. N = 8096.

4. All results control for student age (years), sex, attend in Delhi or not, father attended senior high or not, mother attended senior high or not, baseline knowledge, attitude, and behavior composite scores, and baseline missing value dummies.

5. % Non-compliance Control Group refers to the percent of respondents in the control group that do not comply with the behavior.

We also examined treatment impacts separately for males and females (Table S8). Both educational interventions improved knowledge among males and females, but females experienced larger gains, and unlike males, improved in their applied knowledge score. We also found some evidence that the longer intervention was more effective than the shorter intervention at improving females’ applied knowledge, attitudes and self-reports of specific behaviors (cleaning/disinfecting surfaces and physical distancing).

Conclusions

Instructional messaging is crucial to responding to public health crises like the COVID-19 pandemic. We provide the first experimental evidence on whether concentrated and targeted instructional interventions increase positive knowledge and attitudes and reduce noncompliant behaviors among individuals in the face of the COVID-19 pandemic—over and above the dispersed information dissemination efforts of governments and health organizations. We also provide the first experimental evidence on the optimal method of designing instructional interventions to improve knowledge, attitudes, and behaviors for the COVID-19 pandemic.

Findings from our large-scale randomized trial indicate that both a shorter, concentrated and targeted, fact-based intervention and a longer intervention providing the same facts plus explanation of underlying scientific concepts improved COVID-19-related knowledge. In contrast to the fact-based intervention, the facts-plus-concepts intervention resulted in greater improvements in applied knowledge, attitudes, and behavior. Taken together, the findings indicate that the short, fact-based instructional interventions typically provided by national and international public health agencies are not as effective as interventions which take more time to explain underlying scientific concepts.

As COVID-19 continues to spread around the world and health experts warn of additional waves, these findings provide evidence-based guidance on how to best design and deliver instructional materials for young people. Improving the knowledge, attitudes, and behaviors of young people is paramount to ensuring greater compliance and ultimately reducing the spread of COVID-19. If a longer, concept-based intervention such as the one we studied can change behaviors among 10 percent of non-compliers, that could make significant inroads in slowing the spread of the virus and its health impacts. Moreover, concept-based interventions may deliver even greater returns for low-salience public health issues where compliance is low. Such interventions may be especially important for a densely populated country such as India with limited healthcare infrastructure and resources (Singh, 2020).

More broadly, these findings are fundamental to our better understanding the role of instruction and its method of dissemination in affecting knowledge, attitudes, and behavior. Understanding tradeoffs between shorter interventions that mostly provide facts and longer interventions that also explain scientific concepts (but risk losing the attention of recipients) is important. Tensions between providing shorter versus longer lists of instructions in health education interventions or balancing facts versus concepts in STEM instruction have been discussed but seldom tested in large-scale RCTs (Gallagher, 2020; Wieman and Perkins, 2005; Noar et al., 2007; Seligman et al., 2007). Our findings indicate that the provision of scientific explanations in instructional interventions is necessary for sustained impacts and should be used more broadly in the communication efforts of governments and health organizations.

The study has both strengths and limitations. With regard to strengths, it examines a highly consequential and pressing question concerning the best way to design public communications during the current pandemic as well as other public health campaigns. It tests different designs using a well-executed, large-scale and highly powered RCT. With close to 8400 participants, high treatment compliance, little chance for treatment-control spillovers, and low attrition, estimated effects from the trial are not only detectable at small sizes but also likely unbiased (internally valid). Last, the study examines effects on not only objectively measured COVID-19 related knowledge immediately after the treatments but also on attitudes and behaviors after one week.

The study has at least two potential limitations. First and perhaps most importantly, the data include self-reported, and not objective, measures of attitudes and behaviors. A recent study in Kenya found that reported versus actual mask wearing diverged substanitally (Jakubowski et al., 2021) and we note that estimates based on self-reported data could be biased if misreporting of attitudes and/or behaviors is significantly correlated with treatment assignment. That being said, it is nearly impossible to obtain unbiased measures of the types of behaviors the paper is concerned with: for example, the regular wearing of masks, proper washing of hands, and cleaning of surfaces. Using trackers or video to measure such behaviors on a large-scale, even if it were possible and feasible, would potentially introduce alternative and stronger biases (such as Hawthorne effects) into the results. Moreover, it is unlikely that young adults (the participants in our study) would succumb to social desirability bias from a short instructional intervention. It is especially unlikely that participants would self-report differently between the two COVID-19-instruction treatment arms: the Facts plus Concepts versus the Facts Only arms.

Second, the results of our study are drawn from studying low and middle-income urban youth in one part of India, but are not automatically generalizable to other contexts. The effect estimates, more likely than not, could be a lower-bound since compliance with and enforcement of COVID-19 related guidelines has been relatively high in urban India in the early stages of the pandemic. For example, our baseline data indicate that approximately 70% of participants did not leave their homes even once a week just after guidelines were set in place. Because compliance was already high, we would expect even larger effects if COVID-19 had not been as salient of an issue. We hope that our study will be compared to future instructional-based studies of public health issues in environments with lower information and salience. Furthermore, study participants were still developing their skills in English, the language the intervention and testing materials were provided in. Language constraints may have resulted in our underestimating the impact of the interventions, while the combination of self-selection into FEA and language constraints limit the generalizability of our findings. Undoubtedly, more research is needed to understand the effective design of instructional interventions, especially among diverse populations.