A Non-randomized, Quasi-Experimental Comparison of Effects Between an In-person and Online Delivery of a College Mental Health Literacy Curriculum.

Mental health literacy (MHL) training is essential in college environments. These programs are commonly delivered in-person via workshops or for-credit courses. Campuses now seek high-quality online options. We compare the effectiveness of a for-credit MHL course against a comparison course, focusing on whether online asynchronous delivery was as effective as in-person synchronous delivery. This quasi-experimental pretest/posttest treatment/comparison study included 1049 participants across five semesters (pre-COVID-19) who were 18 years or older and self-selected enrollment in a Mental Health Awareness and Advocacy (MHAA) course (treatment; n = 474) or a general lifespan development course (comparison; n = 575). Using linear mixed effect modeling, changes in MHL were compared across groups and across online/in-person modalities. Students in the treatment group significantly increased their MHL knowledge (β Identifying = .49, p < .001; β Locating = .32, p < .001; β Responding = .46, p < .001) and self-efficacy (β = .27, p < .001), and treatment effects did not differ across modalities. With increased concern regarding mental health issues of isolated college students during the COVID-19 pandemic, this study supports the efficacy of delivering MHL courses online.

Introduction

College students’ mental health issues have increased over the past decade (Duffy et al., 2019b) with recent data suggesting that problems are compounded by the COVID-19 pandemic (Czeisler et al., 2020; Gonzales et al., 2020; Son et al., 2020). Community-based programs are often used on college/university campuses to prevent mental health issues in students from occurring or worsening (Duffy et al., 2019a). Mental health literacy (MHL), defined as the “knowledge and beliefs about mental disorders which aid their recognition, management or prevention” (Jorm et al., 1997, p. 182), is a common training approach leveraged in these programs (Jorm, 2019). While there are numerous evidence-based MHL programs on college campuses (e.g., mental health first aid and QPR), academic institutions have had to shift delivery of in-person trainings into online formats to mitigate the spread of the COVID-19 virus (Ali, 2020; Kitchener & Jorm, 2008; The QPR Institute, 2013). Even if COVID concerns dissipate, institutions may want to maintain online delivery of MHL education/training due to its cost effectiveness, ease of widespread dissemination, and user convenience. The current report uses a non-randomized, quasi-experimental design to examine MHL outcomes when delivered online vs. in-person (students self-select into courses thereby prohibiting our ability to use a randomized control trial). This study will help determine if online MHL is a viable, long-term delivery option for college students, and if so, which areas of MHL are most impacted by the modality of delivery.

Course/program “modality” refers to the method by which instruction is delivered. While there is a variety of viable course delivery options, in this study, we only look at in-person (face-to-face and synchronous) and online (asynchronous) modalities. A recent meta-analysis that compiled the results of 91 comparative studies of online and face-to-face university courses found that only 41% of the studies reported no significant differences. Another 41% reported that online courses were associated with improved learning outcomes while 18% of the studies found that face-to-face courses were associated with improved learning outcomes (Stevens et al., 2021). This speaks to the potential variability in learning outcomes by modality across different subjects or courses. Looking at health and mental health specifically, research generally supports similarities in outcomes when comparing in-person and online modalities in mental health content areas, such as courses on stress management (Fish & Kang, 2014). Initial studies of MHL also support that online delivery is effective (Bond et al., 2015).

Some differences have been identified in student attitudes towards course experiences (Martin & Bolliger, 2018), with some students reporting more positive experiences in face-to-face courses due to the access to a “real person” teaching (Tichavsky et al., 2015). Relatedly, other modality comparisons suggest that results depend upon the type of outcome. For instance, a study of medical students’ found that while modality did not have a significant impact on learning outcomes, those in the in-person course had significantly higher social outcomes than those in the online course (Torda & Shulruf, 2021). Further, much of the research we reviewed assesses very general measures of learning outcomes. Distinguishing more detailed outcomes (e.g., gaining knowledge and gaining skills) is especially relevant in MHL, as each outcome contributes uniquely to MHL processes.

The Mental Health Awareness and Advocacy (MHAA) Curriculum

The Mental Health Awareness and Advocacy (MHAA) curriculum is a semester-length MHL program offered both in-person and online via the Canvas learning management system (Instructure, Inc., 2021) as part of a degree-seeking program or as an elective, at a large western university (Aller et al., 2021b). The MHAA curriculum is influenced by both the health belief model (Becker, 1974; Rosenstock et al., 1988) and social cognitive theory (Bandura, 2005). The health belief model informs MHAA curriculum by emphasizing prevention behaviors while recognizing college students’ susceptibility to mental health issues. Relatedly, social cognitive theory draws attention to the importance of increasing students’ self-efficacy in order to increase the likelihood of students engaging in a prevention behavior, such as referring a friend to a therapist or practicing self-care. The curriculum emphasizes a process-based approach focused on macroprocesses (larger domains/skills) needed in mental health literacy, including (1) identifying signs and symptoms of mental health issues (Identifying), (2) locating evidence-based mental health resources (Locating), and (3) responding effectively and appropriately to mental health issues (Responding; see Fig. 1). The MHAA curriculum fosters these macroprocesses through three learning-based microprocesses within each domain, including acquiring knowledge, building self-efficacy, and applying skills (behaviors). For example, in the broader MHL process of Identifying mental health issues, the learner must acquire knowledge about identification of mental health issues, build self-efficacy in identification, and apply skills behaviorally, specific to Identifying mental health issues. The same learning microprocesses are at work for Locating resources and Responding to mental health issues.

Fig. 1

Process-based model of Mental Health Awareness and Advocacy curriculum (Aller et al., 2021b). Note: The large, outer circles represent macroprocesses or content domains needed for building mental health literacy. These areas inform specific content covered in the Mental Health Awareness and Advocacy (MHAA) curriculum. The smaller center circle represents the three learning microprocesses that participants build as they learn identifying, locating, and responding skills. Declarative knowledge refers to the learning process of acquiring knowledge; self-efficacy refers to the learning processes of building self-efficacy; and behaviors refers to the learning process of applying skills. Microprocesses occur within each macroprocess, i.e., participants acquire knowledge, build self-efficacy, and apply skills related to each step of MHL: identifying symptoms, locating resources, and responding appropriately

Teaching approaches for both modalities were highly consistent and included identical course readings, lecture slides, assignment instructions, multi-media engagement, discussions (albeit in person this was via speaking and online this was written), and feedback on assignments. Notable differences across modalities are related to course delivery capabilities. In-person courses yielded opportunities for more organic conversations, including moments where the instructor could provide detailed and immediate feedback to students’ questions or discussion. Online, this manifested as discussion posts and delayed instructor response, which may have yielded small but important interruptions in the learning microprocesses. Additionally, lectures of course content were prerecorded for the online modality, but not for the in-person modality, theoretically influencing the dosage of information students received (i.e., online students could watch the lecture repeatedly, if they wished).

For five semesters (January 2018–May, 2020), the course was included in a formal research assessment, studying its effectiveness in increasing MHL, compared to a control group (Institutional Review Board protocol #8648). The first two semesters served as a pilot (162 college students; control N = 89; treatment N = 73), exploring initial treatment effects, albeit with less robust analytic methodologies than the current study. Pilot data indicated that students in the MHAA course increased declarative knowledge in identifying mental health issues and locating evidence-based resources and their MHL self-efficacy (Aller et al., 2021b). This preliminary investigation was not adequately powered to explore differences in effects between in-person and online delivery mechanisms. The current study is a continuation of the pilot, incorporating pilot data and new data (N = 1049; + 887) that provides adequate power to replicate pilot findings using novel analytic tools and importantly to address if there are differences between course modalities. The following research questions are addressed:

RQ1. Do students that participate in the MHAA curriculum improve their MHL compared to those in the comparison group?

Hypothesis 1: Students participating in the MHAA curriculum will improve their MHL, specifically their knowledge, self-efficacy, and behaviors above and beyond the effects observed in the comparison group.

RQ2. Do the treatment effects of MHAA vary by course modality (in-person vs. online)?

Hypothesis 2: There will be no observed differences between in-person compared to online delivery of the MHAA curriculum.

Methods

Sample and Procedures

Participants were undergraduates at a large western state university in the USA. Students self-enrolled in either in-person or online modalities of the semester-length MHAA course (treatment group; N = 474; 57.52% of total student enrollment of 824 across all MHAA courses) or an introductory human development course (comparison group; N = 575; we are unable to calculate percentage of total student enrollment data for the comparison group due to instructors having left the university and not having access to student enrollment data from their courses). Students were considered a participant if they completed at least one of the pretest or posttest surveys. Neither course had registration restrictions (e.g., year in school and major) meaning all registered university students could self-select into either group.

Participation in the class was required for those that were registered; however, participation in the research evaluation was optional. Those who participated completed a pretest survey during the first week of class and a posttest survey during the last 2 weeks of the semester via Qualtrics. The first author of this study was the instructor of record for all semesters for both modalities of the treatment course, while the comparison classes were taught by varying instructors. The MHAA instructor was not present during recruitment to the research project and was blind to all deidentified study data until after courses concluded. Each participant received 1% extra credit to their grade for participation. The research evaluation ended in spring of 2019 due to the program objective being completed and expiration of IRB approval.

Measures

Participants reported demographic characteristics and completed the Mental Health Awareness and Advocacy Assessment Tool (MHAA-AT; Aller et al., 2021a), measuring the primary outcome of interest MHL, to evaluate RQ1. The 65-item tool was created by therapists and mental health researchers to evaluate MHL and demonstrates strong psychometric properties and construct validity (Aller et al., 2021a). Specifically, in line with the MHAA curriculum, the MHAA-AT evaluates MHL via three learning microprocess of (1) acquiring knowledge (30 items), (2) building self-efficacy (20 items), and 3) applying skills/behaviors (15 items), within three MHL macroprocesses: (1) identifying mental health issues, (2) locating evidence-based resources, and (3) responding to mental health issues (see Fig. 1). Treatment, mode, and time were all dichotomous variables. Treatment was coded as 0 = control, 1 = treatment; mode was coded as 0 = online, 1 = in-person; and time was coded as 0 = time 1, 1 = time 2.

Analysis

Analyses were performed in the R statistical environment version 4.0.2 (R Core Team, 2020). Using linear mixed effects modeling to account for the repeated measures and the clustering of participants within semesters and controlling for intra-individual variability by using a random intercept, we assessed changes from pretest to posttest for the treatment and comparison group (RQ1), controlling for modality, gender identity, year in school, childhood socioeconomic status, and race/ethnicity. The model can be expressed as:where i is the ith individual, j is the jth semester, and Y is each MHL outcome (knowledge domains 1–3, self-efficacy, and behavior), and the estimate of interest is the showing the interaction between condition and time. The control variables were modality (in-person, online), gender identity, year in school, socioeconomic factors, and race/ethnicity. Null hypotheses were tested via likelihood ratio tests.

Building upon this model, for RQ2, we tested a three-way interaction between group, time, and course modality (online/in-person). As such, the model tested whether changes from pretest to posttest by condition (treatment or control) for each MHL outcome differed by course modality (online or in-person), controlling for gender identity, year in school, socioeconomic factors, and race/ethnicity. Null hypotheses were again tested via likelihood ratio tests.

Using the lme4 package in R, missing observations were removed via listwise deletion at the observation level. That is, if individuals are missing at a single time point, their available data are still used by the maximum likelihood (ML) estimation of the mixed effects models. Ultimately, models used data from 741 (353 in treatment, 388 in comparison) individuals, except for self-efficacy models (n = 727, 346 in treatment, 381 in comparison) and behavior models (n = 721, 344 in treatment, 377 in comparison). Missing values on individual items were uncommon (< 10% on all variables of interest) and unlikely affect the results in meaningful ways, even if the values are missing not at random. All code, data, and output relevant to this report are provided at https://osf.io/j3ew8/.

Results

Descriptive statistics are provided in Table 1 by group and modality. Of note, the treatment group generally had more advanced students (“year in school”) and were somewhat higher in SES. As such, each model controlled for these characteristics (along with gender identity and race/ethnicity). Notably, the majority identify as white and female across all conditions and modalities. Estimates shown are standardize estimates.

Table 1

Key sample characteristics at pretest

	ControlLifespan courseIn-personN = 400	ControlLifespan courseOnlineN = 175	TreatmentMHAA courseIn-personN = 234	TreatmentMHAA courseOnlineN = 240	p-value
Year in school					<.001
Freshman	65%	35%	18%	3%
Sophomore	24%	29%	29%	13%
Junior	8%	24%	32%	33%
Senior	2%	9%	20%	50%
Other	0%	3%	1%	1%
Gender identity					.515
Female	89%	89%	86%	90%
Male	11%	11%	14%	9%
Other	0%	0%	<1%	1%
Race/ethnicity					.572
White	92%	87%	94%	91%
American Indian	<1%	2%	0%	0%
Asian	1%	<1%	<1%	1%
Hispanic or Latino	3%	6%	3%	5%
Black	<1%	<1%	<1%	0%
Bi-racial	2%	3%	2%	2%
Financial stress growing up					<.001
Not at all concerned	47%	30%	48%	36%
Somewhat concerned	43%	51%	40%	42%
Very concerned	10%	19%	12%	22%

While we controlled for the variables that were significantly different between groups, we note that such an approach is unlikely to fully account for all the reasons the groups differed at pretest. p-values are based on chi-square tests of independence and Fisher’s exact tests

RQ1: The treatment group (MHAA students) improved from T1 to T2 at a significantly higher rate than the comparison group on the following: Declarative Knowledge Identifying (β = 0.49, SE = 0.08, p < 0.001), Declarative Knowledge Locating (β = 0.32, SE = 0.07, p < 0.001), Declarative Knowledge Responding (β = 0.46, SE = 0.07, p < 0.001), and Self-efficacy (β = 0.27, SE = 0.08, p < 0.001). Specifically, the aforementioned outcomes increased from pretest to posttest in the treatment group, whereas the comparison group showed either no change or only slight improvement. Differences were consistently moderate for these outcomes. Table 2 provides a summary of the raw mean change across all assessed domains, while Table 3 provides the linear mixed effects model results.

Table 2

Raw means, standard deviations, with available data between conditions and modalities on each outcome measure from the MHAA-AT

Measure		Control		Treatment
		Pre M (SD)	Post M (SD)	Pre M (SD)	Post M (SD)
Declarative Knowledge Identifying	In-Person	4.69 (2.21)	4.88 (2.30)	6.25 (2.06)	7.07 (1.97)
	Online	3.39 (2.13)	4.68 (2.16)	5.98 (2.05)	6.67 (1.92)
Declarative Knowledge Locating	In-Person	4.70 (2.46)	4.70 (2.48)	6.28 (2.44)	6.95 (2.25)
	Online	5.21 (2.34)	5.42 (2.29)	6.60 (2.18)	7.18 (4.77)
Declarative Knowledge Responding	In-Person	3.38 (1.63)	3.79 (1.73)	4.55 (1.83)	4.96 (1.68)
	Online	3.39 (1.69)	3.62 (1.74)	4.40 (1.67)	4.77 (1.72)
Self-Efficacy	In-Person	2.89 (0.95)	3.03 (1.01)	3.53 (1.12)	4.25 (1.09)
	Online	3.00 (0.95)	3.18 (0.97)	3.48 (1.08)	4.12 (1.04)
Behavior	In-Person	15.7 (12.7)	16.1 (13.1)	20.0 (14.4)	20.5 (14.4)
	Online	20.0 (14.9)	19.8 (16.4)	18.3 (14.4)	18.3 (14.5)

All output related to linear mixed models are available at https://osf.io/j3ew8/

Table 3

Standardized model results

	Declarative Knowledge (Identifying)	Declarative Knowledge (Locating)	Declarative Knowledge (Responding)	Self-efficacy	Behavior
RQ1
Treatment group	0.49*** (0.08)	0.32*** (0.07)	0.46*** (0.07)	0.27*** (0.08)	0.15 (0.09)
Time	0.13** (0.04)	−0.81*** (0.05)	−0.77*** (0.05)	0.18*** (0.04)	0.01 (0.03)
Mode	−0.22** (0.07)	0.11 (0.06)	0.02 (0.06)	−0.09 (0.07)	0.07 (0.08)
Treatment × Time	0.28*** (0.06)	−0.22** (0.08)	−0.29*** (0.08)	0.58*** (0.06)	0.00 (0.05)
Constant	−0.44 (0.62)	1.12* (0.55)	0.62 (0.56)	0.51 (0.63)	−0.44 (0.71)
RQ2
Treatment group	0.51* (0.21)	0.44* (0.20)	0.76*** (0.20)	0.49* (0.21)	0.78*** (0.24)
Time	0.10 (0.12)	−0.90*** (0.16)	−0.76*** (0.16)	0.17 (0.11)	0.09 (0.10)
Mode	−0.21* (0.10)	0.11 (0.10)	0.09 (0.10)	0.00 (0.10)	0.32** (0.11)
Treatment × Time	0.38* (0.18)	−0.25 (0.23)	−0.51* (0.23)	0.66*** (0.17)	0.00 (0.15)
Treatment × Mode	−0.01 (0.14)	−0.09 (0.13)	−0.21 (0.13)	−0.16 (0.14)	−0.45** (0.16)
Time × Mode	0.02 (0.08)	0.07 (0.11)	−0.01 (0.11)	0.01 (0.08)	−0.05 (0.07)
Treatment × Time × Mode	−0.07 (0.12)	0.01 (0.16)	0.16 (0.16)	−0.06 (0.11)	0.00 (0.10)
Constant	−0.45 (0.63)	1.11* (0.56)	0.53 (0.57)	0.39 (0.63)	−0.76 (0.71)

Standard errors in parentheses. Both models controlled for gender, year in school, childhood SES, and race/ethnicity

* p < 0.05; ** p < 0.01; *** p < 0.001

RQ2: Results also indicated that the modality of the course (online vs. in-person) did not influence the effectiveness of the program (all p 0.01), with both modalities consistently improving to similar degrees across both the treatment and control groups (see Tables 2 and 3).

Discussion

The MHAA curriculum was effective in improving students’ MHL knowledge and self-efficacy. Specifically, students in the MHAA class improved their ability to accurately identify symptoms of depression and anxiety that were described in clinical vignettes, improved their ability to identify evidence-based resources including community organizations and treatments for specific mental health issues, and improved their knowledge relating to how to respond to MHI and refer others and themselves to high-quality resources. There were also significant increases in MHL self-efficacy (i.e., feeling comfortable in identifying mental health issues, locating evidence-based resources, and responding appropriately). There were, however, no significant changes in students’ MHL-related behavior (which is a replicated finding from the pilot study; Aller et al., 2021b). This could be due to evaluation being too close to the treatment (not enough time to be in a mental health-related situation that needed a response) or possible measurement issues with defining MHL behavioral outcomes, which have been noted in studies using other measures (Spiker & Hammer, 2019; Wei et al., 2017).

The effectiveness of the MHAA curriculum did not differ by modality, suggesting that students MHL improves regardless of in-person or online delivery. Although this result could be due to low power to detect the interaction, we note that the estimated effect sizes were small and unlikely to be meaningful. We also note that careful attention was given to mirror learning processes across both formats (e.g., both modalities included active instructor feedback, facilitated discussions, and social support among students) using guidance of best practices in instructional design (Baldwin & Ching, 2019; Nilson & Goodson, 2021). With advanced instructional design, online formats are established as effective learning modalities; this study extends these findings for knowledge and self-efficacy outcomes in a MHL-specific curriculum. We highlight the need for continued research to investigate how MHL courses can better measure and/or influence behavioral changes in how participants respond to MHIs in others.

Limitations

We note the limited diversity in gender, race/ethnicity, and geographical background as an important limitation of this study as all participants came from a single university in the Western United States. The results of this study are not generalizable to the population as a whole and future research with more diverse samples is warranted. Additionally, it is important to note that for many of the outcome measures, the treatment condition often started with higher scores; while the modeling approach accounted for this, it is unlikely that it is fully accounted for all the various reasons that the students in the treatment condition had higher scores at the beginning.

Conclusion

The MHAA curriculum demonstrates acceptable proof of concept for adapting MHL content to online delivery as a college course. Given the increased need for mental health education and referral, this format could serve hundreds of students each semester in a manner that stays compliant with safety guidelines of the CDC considering the COVID-19 pandemic (Center for Disease Control, 2020). Post-pandemic, colleges/universities can continue to offer MHL in a course format (as opposed to a workshop), and online, to reach a larger number of students than what might be possible in person.

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 36 KB)