Exploring Retention, Usage, and Efficacy of Web-Based Delivery of Positive Emotion Regulation Skills During the COVID-19 Pandemic.

COVID-19 prompted distress and increased reliance on digital mental health interventions, which previously demonstrated low rates of retention and adherence. This single-arm trial evaluated whether self-guided, web-based, positive affect regulation skills (PARK) were engaging and associated with changes in well-being during the pandemic. Over 6 weeks, PARK delivers brief lessons and practices in skills designed to increase positive emotions: noticing positive events, savoring, gratitude, mindfulness, positive reappraisal, personal strengths, and self-compassion. Patient-Reported Outcome Measurement Information System (PROMIS) computer adaptive tests of anxiety, depression, social isolation, positive affect, and meaning and purpose were administered at baseline, post-intervention, and 6 months after baseline. Retention and usage of PARK were measured by the web-based assessment and intervention platforms. The sample (n = 616) was predominantly female, non-Hispanic, white, and well-educated. Of those who completed baseline, only 42% completed a follow-up assessment; 30% never logged into PARK. Among those who did, 86% used at least one skill, but only 14% completed PARK. Across retention and usage metrics, older age predicted more engagement. In multivariable models, people of color and people with greater baseline anxiety were more likely to complete PARK. All well-being indicators improved over time, with greater improvements in anxiety and social isolation among participants who accessed at least one PARK skill compared to those who did not. Retention and usage rates mirrored pre-pandemic trends, but within this select sample, predictors of engagement differed from prior research. Findings underscore the need for additional efforts to ensure equitable access to digital mental health interventions and research. Trials registration: NCT04367922. © The Society for Affective Science 2022.

From the beginning of the COVID-19 pandemic, many recognized the potential for widespread changes in psychological well-being (Galea et al., 2020). Early US data showed that rates of distress increased sharply (Daly & Robinson, 2021), while happiness, life satisfaction, and other elements of flourishing declined (VanderWeele et al., 2020). To prevent deterioration of public mental health, evidence-based recommendations encouraged web-based dissemination of emotion regulation programs including reappraisal, mindfulness, and other coping skills (Park et al., 2021).

Positive psychological interventions (PPIs) specifically aim to increase emotional well-being and include many of these skills (Schueller et al., 2014). According to the Positive Pathways to Health Model, PPIs increase positive emotion, setting off a cascade of benefits such as improved coping, strengthened social connections, and decreased stress reactivity, resulting in better physical and psychological health (Moskowitz, Addington et al., 2019). Meta-analysis has confirmed that multicomponent PPIs decrease stress, anxiety, and depression; increase hedonic and eudaimonic well-being; and improve quality of life (Carr et al., 2020; Hendriks, Schotanus et al., 2019).

We previously demonstrated the efficacy of one such multicomponent PPI for adults facing health-related stress (Carrico et al., 2019; Moskowitz et al., 2017; Moskowitz, Cheung et al., 2019) and adapted it for self-guided, web-based delivery (Addington et al., 2019; Moskowitz et al., 2021). Given the likely need for approaches that would remain accessible during stay-at-home orders and other pandemic precautions, we used this as the basis for the web-based, self-guided Positive Affect Regulation sKills program (PARK), designed to improve coping with stress during the COVID-19 pandemic. PARK aims to increase daily experiences of positive emotion through brief, weekly lessons and daily practice of the following skills: noticing positive events, savoring, gratitude, mindfulness, positive reappraisal, personal strengths, attainable goals, and self-compassion. To evaluate the potential for PARK to improve psychological well-being among U.S. adults during a public health crisis, PARK was made available for a single-arm trial starting May 1, 2020.

Despite the relative low cost and potential for broad dissemination, rates of retention and adherence to digital mental health interventions (DMHI) tend to hover at or below 50% in randomized controlled trials (RCTs), and are even lower for open access sites/apps (Christensen et al., 2009; Fleming et al., 2018). For example, within self-guided apps available in the Google Play store to improve coping or emotional well-being, the median percentage of users who open the app 1 day after installation is around 70% and falls to or below 10% after 1 week (Baumel et al., 2019). In previous reviews, adult users of PPI and DMHI tend to be women, younger (vs. older adults), and people with higher socioeconomic status (SES; i.e., employed, higher education, and income), whereas mental health symptoms can interfere with usage (Borghouts et al., 2021; Christensen et al., 2009; Hendriks, Warren et al., 2019; Onyeaka et al., 2021). Meta-analysis has shown that greater usage of DMHI, in both self-guided and facilitator/therapist-guided formats, predicts better mental health outcomes (Gan et al., 2021). Thus, knowing who is likely to use an intervention is important for predicting who is likely to benefit from it.

The present study addresses this question in an open trial of a web-based PPI launched at the start of a global pandemic that significantly undermined well-being (Daly & Robinson, 2021; VanderWeele et al., 2020). Given homogenous samples in prior studies and low rates of engagement in many PPI and DMHI (Baumel et al., 2019; Christensen et al., 2009; Fleming et al., 2018; Hendriks, Warren et al., 2019), we examined baseline predictors of retention and usage, including demographic factors, perceived impact of the pandemic, and metrics of psychosocial well-being. We also assessed longitudinal change in well-being, according to pre-registered analytic plans (NCT04367922), and conducted exploratory tests of PPI usage as a moderator of well-being over time.

Method

Enrollment and Participation

Recruitment for the single-arm trial of PARK was conducted via online (e.g., ResearchMatch) and social media (e.g., Twitter) advertisements containing a link to an electronic screening form. Participants were eligible if they were age 18 or older, living in the USA, and able to access the internet and read English. People with an invalid email address were excluded. Eligible participants were able to automatically access the consent form, provide electronic documentation of consent, and directly begin the baseline (T1) assessment.

For all consenting participants who completed T1, study staff created their PARK accounts, which initiated automated emails providing access to the PARK website. Over 6 weeks, PARK delivers brief didactic material and practice in skills designed to increase daily experiences of positive emotion (see Table 1). Participants cannot skip ahead, but they can return to earlier skills and practices. The weekly lessons encourage participants to complete the current week’s practices each day, or as often as possible.

Table 1

Overview of PARK intervention

Week	Skill lessons	Goals	Practice
1	(1) Noticing positive events	Recognize positive events and associated emotions	Note one positive event each day
	(2) Savoring	Practice ways to amplify the experience of positive events	Write about thoughts & feelings associated with recalling that day’s positive event
	(3) Gratitude	Learn to practice gratitude	Gratitude journala
2	(4) Everyday mindfulness	Learn and practice the awareness and nonjudgment components of mindfulness	Mindfulness during everyday activities such as washing hands, brushing teeth, walking
	(5) Mindfulness meditation		10-minute breath awareness meditation with guided audioa
3	(6) Positive reappraisal	Understand positive reappraisal and how it can increase positive emotions in the face of stress	Report a relatively minor stressor and list positive reappraisals of it
4	(7) Personal strengths	Identify their personal strengths and how they have used them recently	Name a strength and how it was “expressed” behaviorally
	(8) Attainable goals	Understand characteristics of attainable goals	Set attainable goals and note progress towards them
5	(9) Self-compassion	Recognize that being kind to oneself, rather than harshly self-critical, can increase positive emotions	Name a recent example of self-criticism; then describe a self-compassionate response instead, as if you were talking to a friend who was being self-critical
6	(10) Wrap-up	Review the skills and plan for continued practice	None

aContinues throughout the remainder of the intervention period

Participants receive daily emails asking them to report their levels of positive and negative emotions and positive and stressful events over the past 24 h (see “Self-reported Measures”); these daily emotion reporting emails also contain reminders to complete the skills practice. Additional features designed to increase PARK usage include (1) virtual badges: awards that participants can earn for activities such as logging into the PARK website, accessing the skills, and completing home practices; and (2) discussion boards that allow participants to share and discuss their home practice responses. No monetary incentives were provided for accessing PARK or completing assessments.

Self-reported Measures

Assessments were administered via REDCap at baseline (T1), post-intervention (T2, 8 weeks after baseline), and follow-up (T3, 26 weeks, approximately 6 months after baseline). Participants self-reported socio-demographics (e.g., age, race, ethnicity, and gender) and the impact of the pandemic on their financial status, access to resources, and psychological health (3 items each; rated 1–7; higher scores indicate greater impact). Patient-Reported Outcomes Measurement Information System (PROMIS) computer adaptive tests measured five well-being indicators: positive affect, meaning and purpose, depression, anxiety, and social isolation (Hahn et al., 2014; Pilkonis et al., 2011, 2014; Salsman et al., 2014). All five PROMIS outcomes are presented as T-scores (M = 50, SD = 10 in general population), with higher T-scores indicating more of the construct being measured (PROMIS Reference Populations, n.d.). PROMIS cut points [e.g., within normal limits, mild, moderate, and severe (PROMIS Score Cut Points, n.d.)] allow interpretion of clinical significance of T-scores, while a within-group change or between-group difference of 3 T-score points is accepted to represent a meaningful change or difference (Meaningful Change for PROMIS, n.d.).

During the intervention period, participants also receive daily surveys of emotions and events experienced in the past 24 h. Daily emotions are reported on the modified differential emotions scale, which asks participants to indicate how frequently they felt each of 20 positive and negative emotions (e.g., awe, glad, love; angry, sad, scared), using a scale ranging from 0 (never) to 4 (most of the time; Fredrickson, 2013). Participants additionally completed the 12-item daily inventory of stressful and positive events, where they indicated which events (e.g., being discriminated against; a positive interaction with someone) occurred and rated each one’s impact from 1 (not at all stressful/positive) to 4 (very stressful/positive; Almeida et al., 2002).

Objective Measures of Retention and Usage

Retention was calculated based on completion of REDCap-administered assessments, regardless of engagement with the PARK intervention website. Participants who completed baseline and T2 or T3 were classified as retained. Drop out was defined as completing only T1.

Usage was based on whether or not participants ever logged into the PARK website, number of skills accessed (0–10, see Table 1), number of different practices completed (0–9, see Table 1), and percent of total pages accessed within the PARK website. We defined completers as participants who accessed all 10 skills. We additionally report the mean number of days that participants completed the daily emotion reporting.

Analytic Strategy

These analyses include participants who consented between May 2, 2020, and August 26, 2021. We used independent t-tests and chi-square analyses to compare baseline variables between drop out vs. retained participants. Due to bimodal usage distributions (see “Results”; Figs. 1 and 2), we categorized usage into dichotomous variables: accessed at least 1 skill (yes vs. no), accessed at least 1 home practice (yes vs. no), accessed all 10 skills (yes vs. no), and accessed all 9 home practice assignments (yes vs. no). We examined bivariate relationships between baseline predictors (socio-demographic, COVID, well-being measures) and each of these four usage outcomes using independent t-tests for continuous predictors and chi-square or Fisher’s exact tests for categorical predictors. We then created multivariable logistic models for each usage outcome, adjusting for all baseline predictor variables; results are reported as odds ratios (OR).

Fig. 1

Distribution of the number of skills accessed

Fig. 2

Distribution of the number of home practice exercises accessed

To avoid over-estimation of missing data in the context of low rates of retention in assessments (see “Results”), longitudinal analyses of well-being include only participants who completed PROMIS measures at baseline (T1) and at least one of the follow-up points (T2 or T3). We first examined changes in well-being over time (T1-T2, T1-T3) using unadjusted longitudinal growth models with maximum likelihood estimation. Then we added the effect of usage (dichotomous: 0 vs. > 1 skills) and the interaction of usage and time (categorical, representing the 3 assessment points: 0, 8, 26 weeks). These adjusted models allow the slope to change between time points and examine whether linear change in well-being over time differed between participants who accessed none vs. any of the skills, allowing us to approximate a test of “control” vs. “intervention” effect in this single-arm trial.

Results

Enrollment and Retention

Of n = 1,070 who completed screening, 99% were eligible (Table 2). However, 36% never completed the consent form, 6% declined, and 2% withdrew after consenting. The final sample for this analysis includes n = 616 who consented and completed T1. The sample was predominantly female (74%), non-Hispanic (90%), white (75%), and well-educated (76% with college degree or higher; Table 3). Using PROMIS cut points, mean scores at baseline indicated moderate anxiety, mild depression, social isolation within normal limits, low levels of positive affect, and average levels of meaning and purpose.

Table 2

CONSORT table for the PARK study

Completed screener	1,070
Ineligible	15	1%	of completed screener
Loss to follow-up — consent	388	36%	of completed screener
Eligible	667	62%	of completed screener
“No” to consent	37	6%	of eligible
Consented, completed baseline	630	94%	of eligible
Withdraw	14	2%	of consented, completed baseline
Study sample	616	98%	of consented, completed baseline
Missing both follow-ups	356	58%	of active participants
Completed at least 1 follow-up	260	42%	of active participants
Completed post (8 weeks)	214	82%	of completed at least 1 follow-up
Completed follow-up (26 weeks)	159	61%	of completed at least 1 follow-up
Completed both follow-ups	113	43%	of completed at least 1 follow-up

Table 3

PARK participant characteristics and baseline predictors of retention

Predictors	Total	Drop out = No	Drop out = Yes	p-value
	n = 616	n = 260 (42%)	n = 356 (58%)
Socio-demographics
Gender, n (% with valid data)	599 (97%)
Male	145 (24%)	66 (26%)	79 (23%)	0.46
Female	443 (74%)	188 (73%)	255 (75%)
Non-binary	11 (2%)	3 (1%)	8 (2%)
Race, n (% with valid data)	573 (93%)
White	432 (75%)	189 (76%)	243 (75%)	0.79
Black	57 (10%)	22 (9%)	35 (11%)
Native/Asiana	58 (10%)	26 (10%)	32 (10%)
Multiracial	26 (5%)	13 (5%)	13 (4%)
Ethnicity, n (% with valid data)	596 (97%)
Non-Hispanic	539 (90%)	236 (92%)	303 (89%)	0.31
Education, n (% with valid data)	598 (97%)
High school or less	25 (4%)	9 (3%)	16 (5%)	0.53
Some college	118 (20%)	48 (19%)	70 (21%)
College degree	159 (27%)	76 (29%)	83 (25%)
> College degree	296 (49%)	127 (49%)	169 (49%)
Income, n (% with valid data)	540 (88%)
< $30K	87 (16%)	38 (16%)	49 (16%)	0.84
$30K–$60K	129 (24%)	61 (26%)	68 (22%)
$60K–$100K	125 (23%)	53 (22%)	72 (24%)
> $100K	199 (37%)	85 (36%)	114 (38%)
Marital status, n (% with valid data)	595 (97%)
Married/partnered	320 (54%)	147 (57%)	173 (51%)	0.20
Age, n (% with valid data)	593 (96%)
Mean (sd)	42 (16)	46 (17)	39 (14)	< 0.0001*
COVID impact
Financial, n (% with valid data)	581 (94%)
Mean (sd)	3.6 (2.1)	3.4 (2.1)	3.7 (2.0)	0.09
Resources, n (% with valid data)	582 (94%)
Mean (sd)	3.2 (1.6)	3.1 (1.6)	3.2 (1.7)	0.26
Psychological, n (% with valid data)	581 (94%)
Mean (sd)	4.5 (1.7)	4.3 (1.7)	4.6 (1.6)	0.06
PROMIS well-being
Anxiety, n (% with valid data)	575 (93%)
Mean (sd)	60.0 (8.5)	59.0 (8.2)	60.8 (8.6)	0.01*
Depression, n (% with valid data)	571 (93%)
Mean (sd)	57.3 (8.5)	56.6 (8.1)	57.9 (8.7)	0.08
Social isolation, n (% with valid data)	567 (92%)
Mean (sd)	53.6 (8.5)	53.5 (8.6)	53. 8 (8.5)	0.66
Positive affect, n (% with valid data)	564 (92%)
Mean (sd)	40.5 (8.6)	40.8 (8.4)	40.2 (8.8)	0.42
Meaning & purpose, n (% with valid data)	558 (91%)
Mean (sd)	44. 9 (10.9)	44.4 (11.1)	45.3 (10. 8)	0.35
Usage
Ever accessed, n (% with valid data)	616 (100%)
Yes	430 (70%)	194 (75%)	236 (66%)	0.03*
Accessed ≥ 1 skill, n (% with valid data)	430 (70%)
Yes	368 (86%)	179 (92%)	189 (80%)	< 0.001*
Ever did HP, n (% with valid data)	616 (100%)
Yes	430 (70%)	194 (75%)	236 (66%)	0.03*
Accessed ≥ 1 HP, n (% with valid data)	430 (70%)
Yes	227 (53%)	130 (67%)	97 (41%)	< .0001*
% of total pages accessed	616 (100%)
Mean (sd)	25% (33%)	40% (40%)	13% (20%)	< .0001*
Number of skills accessed (0–10)	430 (70%)
Mean (sd)	3.7 (3.4)	5.5 (3.6)	2.2 (2.2)	< .0001*
Number of skills’ HP accessed (0–9)	430 (70%)
Mean (sd)	2.6 (3.1)	3.9 (3.5)	1.5 (2.1)	< .0001*
Number of daily emotion reports entered	290 (47%)
Mean (sd)	7.3 (9.6)	10.8 (11.2)	3.5 (5.2)	< .0001*

*p-value significant at the alpha level of 0.05. HP, home practice. aNative/Asian includes all participants who selected race American Indian or Alaska Native, Asian or Asian-American, or Native Hawaiian or Other Pacific Islander

More than half (n = 356, 58%) were non-respondents at both T2 and T3. Comparing this drop-out sample with those who completed T2 and/or T3 (Table 3), the retained sample was older and statistically less anxious; however, given that the between-group difference in anxiety was only 1.8 T-score points, this does not represent a meaningful difference. Retained participants demonstrated higher usage of PARK across all metrics.

Usage

Thirty percent of the baseline sample (n = 186 of 616) never accessed the PARK website after study staff created their account. Compared to those who logged in at least once, those who never accessed PARK were more likely to be single, separated, divorced, or widowed, rather than married/partnered (Table 4). They also reported greater pandemic-related impact on their resources, higher depression, higher anxiety, and lower meaning and purpose scores.

Table 4

Results of bivariate analysis of predictors of PARK usage

	Access			Users						Completers
	Log-in to PARK			Skills			HP			Skills			HP
Predictor	Never (0)	Ever (≥ 1)	p	No (0)	Yes (≥ 1)	p	No (0)	Yes (≥ 1)	p	No (< 10)	Yes (10)	p	No (< 9)	Yes (9)	p
Total n (%)	186 (30%)	430 (70%)		62 (14%)	368 (86%)		203 (47%)	227 (53%)		368 (86%)	62 (14%)		381 (89%)	49 (11%)
Gender
Male	37 (21%)	108 (26%)	0.49	21 (35%)	87 (24%)	0.07	55 (28%)	53 (24%)	0.58	92 (26%)	16 (26%)	0.28	97 (26%)	11 (22%)	0.31
Female	136 (77%)	307 (72%)		39 (65%)	268 (74%)		141 (70%)	166 (74%)		261 (72%)	46 (74%)		269 (72%)	38 (78%)
Non-binary	3 (2%)	8 (2%)		0 (0%)	8 (2%)		3 (2%)	5 (2%)		8 (2%)	0 (0%)		8 (2%)	0 (0%)
Race
White	130 (75%)	302 (76%)	0.84	42 (76%)	260 (76%)	0.99	141 (76%)	161 (75%)	0.60	260 (76%)	42 (71%)	0.68	271 (77%)	31 (66%)	0.15
Black	20 (11%)	37 (9%)		5 (9%)	32 (9%)		20 (11%)	17 (8%)		32 (9%)	5 (9%)		31 (9%)	6 (13%)
Native/Asiana	16 (9%)	42 (10%)		6 (11%)	36 (10%)		17 (9%)	25 (12%)		34 (11%)	8 (13%)		33 (9%)	9 (19%)
Multiracial	8 (5%)	18 (5%)		2 (4%)	16 (5%)		7 (4%)	11 (5%)		14 (4%)	4 (7%)		17 (5%)	1 (2%)
Ethnicity
Non-Hispanic	159 (90%)	380 (90%)	0.96	49 (82%)	331 (92%)	0.01*	175 (88%)	205 (92%)	0.17	322 (90%)	58 (94%)	0.37	333 (90%)	47 (96%)	0.20
Education
≤ High school	9 (5%)	16 (4%)	0.62	1 (2%)	15 (4%)	0.01*	7 (4%)	9 (4%)	0.62	16 (4%)	0 (0%)	0.08	16 (4%)	0 (0%)	0.08
Some college	39 (22%)	79 (19%)		21 (36%)	58 (16%)		42 (21%)	37 (16%)		70 (19%)	9 (15%)		73 (20%)	6 (12%)
College	43 (25%)	116 (27%)		16 (27%)	100 (28%)		51 (26%)	65 (29%)		99 (28%)	17 (27%)		103 (28%)	13 (27%)
> College	85 (48%)	211 (50%)		21 (35%)	190 (52%)		97 (49%)	114 (51%)		175 (49%)	36 (58%)		181 (48%)	30 (61%)
Income
< $30K	30 (19%)	57 (15%)	0.38	9 (17%)	48 (15%)	0.37	28 (16%)	29 (14%)	0.86	53 (17%)	4 (7%)	0.21	53 (16%)	4 (9%)	0.31
$30–$60K	42 (27%)	87 (23%)		14 (27%)	73 (22%)		37 (21%)	50 (24%)		75 (23%)	12 (21%)		79 (23%)	8 (18%)
$60–$100K	32 (20%)	93 (24%)		15 (29%)	78 (23%)		43 (24%)	50 (25%)		76 (23%)	17 (30%)		78 (23%)	15 (33%)
> $100K	54 (34%)	145 (38%)		14 (27%)	131 (40%)		69 (39%)	76 (37%)		121 (37%)	24 (42%)		127 (38%)	18 (40%)
Marital status
Married/partnered	84 (47%)	236 (56%)	0.04*	28 (47%)	208 (58%)	0.10	112 (57%)	124 (56%)	0.71	196 (55%)	40 (65%)	0.17	207 (56%)	29 (59%)	0.68
Age	40.8 (14.9)	42.2 (15.9)	0.31	37.4 (15.1)	43.0 (15.9)	0.01*	39.5 (14.5)	44.6 (16.6)	< 0.01*	40.8 (15.4)	50.4 (16.4)	< .01*	41.1 (15.6)	50.6 (15.6)	< .01*
COVID impact
Financial	3.8 (2.1)	3.5 (2.0)	0.05	3.6 (2.1)	3.4 (2.0)	0.52	3.5 (2.1)	3.4 (2.0)	0.46	3.5 (2.1)	3.1 (1.9)	0.09	3.5 (2.0)	3.0 (2.0)	0.11
Resources	3.4 (1.7)	3.1 (1.6)	0.04*	3.3 (1.8)	3.0 (1.6)	0.16	3.1 (1.7)	3.1 (1.5)	0.90	3.1 (1.6)	2.8 (1.4)	0.10	3.1 (1.6)	2.8 (1.5)	0.28
Psychological	4.6 (1.7)	4.4 (1.6)	0.23	4.3 (1.7)	4.4 (1.6)	0.62	4.3 (1.7)	4.5 (1.6)	0.17	4.5 (1.6)	4.0 (1.7)	0.02*	4.5 (1.6)	4.1 (1.7)	0.17
PROMIS well-being
Anxiety	60.8 (8.0)	59.7 (8.6)	0.16	59.1 (8.7)	59.8 (8.6)	0.54	59.4 (9.3)	60.0 (8.0)	0.46	60.0 (8.7)	58.3 (7.7)	0.16	59.7 (8. 8)	59.5 (7.0)	0.85
Depression	58.6 (8.7)	56.8 (8.3)	0.03*	55.6 (7.7)	57.0 (8.4)	0.25	56.4 (8.8)	57.2 (7.9)	0.36	57.1 (8.4)	55.0 (7.5)	0.07	56.9 (8.4)	56.1 (7.8)	0.52
Social isolation	55.3 (8.4)	53.0 (8.5)	< 0.01*	51.8 (7.7)	53.2 (8.6)	0.28	52.1 (8.7)	53.8 (8.3)	0.03*	53.0 (8.5)	52.8 (8.4)	0.86	53.0 (8.5)	53.2 (8.7)	0.85
Positive affect	39.4 (7.9)	40.9 (8.8)	0.06	42.7 (7.9)	40.6 (8.9)	0.09	41.5 (8.9)	40.3 (8.7)	0.16	40.8 (8.9)	41.4 (8.3)	0.56	41.0 (8.8)	39.8 (8.6)	0.37
Meaning & purpose	43.3 (9.2)	45.5 (11.4)	0.02*	47.5 (11.5)	45.2 (11.4)	0.16	46.3 (11.5)	44.7 (11.4)	0.16	45.4 (11.2)	45.9 (13.0)	0.78	45.7 (11.1)	43.9 (13.9)	0.41

Note. Users and completers are determined within the subset of participants who accessed the PARK website at least once (n = 430). HP, home practice. Categorical variables: n (%). Continuous variables: mean (SD). aNative/Asian includes all participants who selected race American Indian or Alaska Native, Asian or Asian-American, or Native Hawaiian or Other Pacific Islander. *Significant at alpha level of 0.05

Usage is summarized in Table 3, and Figs. 1 and 2 show distributions of usage for number of skills accessed (0–10) and practiced (0–9). Of n = 430 who logged into PARK, n = 62 (14%) never accessed the intervention content. The same amount (n = 62, 14%) were completers, accessing all 10 skills. All usage metrics were highly correlated (number of skills and home practices: r = .85, p < .0001; number of skills and daily emotion reports: r = .70, p < .0001).

In bivariate analyses (Table 4), several baseline variables predicted PARK usage. Across metrics, older participants showed higher usage. In addition, non-Hispanic ethnicity and higher education predicted accessing > 1 (vs. 0) skills and higher social isolation predicted accessing > 1 (vs. 0) practice exercises. The only additional predictor of completion was that lower psychological impact of the pandemic predicted skill completion (accessing all 10 vs. < 10 skills).

In the multivariable logistic regression models predicting each usage outcome while controlling for all baseline variables (Table 5), accessing at least one PARK skill was more likely among participants who were married/partnered, but less likely among those without a college degree. Older age was the only significant predictor of accessing at least one practice. People of color and older participants showed higher odds of completing all 10 skills and all 9 practice assignments, while completion of home practice was also more likely among those with greater anxiety at baseline.

Table 5

Results of multivariable logistic regression models predicting each usage outcome among the n = 430 who accessed PARK at least once

	Usera				Completerb
	Skills: Yes (≥ 1)		HP: Yes (≥ 1)		Skills: Yes (10)		HP: Yes (9)
Predictor	OR	p	OR	p	OR	p	OR	p
Baseline well-being
Anxiety	0.98	0.96	1.00	0.97	1.04	0.07	1.08	0.04*
Depression	1.04	0.65	1.00	1.00	0.97	0.20	0.97	0.53
Social isolation	1.04	0.39	1.04	0.05	1.03	0.48	1.01	0.87
Positive affect	0.97	0.19	1.01	0.54	1.01	0.27	1.00	1.00
Meaning & purpose	0.99	0.34	1.00	0.80	1.00	0.87	0.97	0.27
Gender (not male as ref)
Male	0.62	0.24	0.81	0.45	0.91	0.80	0.56	0.19
Age (years)	1.03	0.05	1.02	0.01*	1.04	0.00*	1.05	0.00*
Race (white as ref)
People of color	2.55	0.06	1.58	0.11	2.17	0.04*	3.01	0.01*
Ethnicity (non-Hispanic as ref)
Hispanic	0.80	0.72	0.66	0.36	1.08	0.91	0.44	0.45
Education (≥ college as ref)
No college degree	0.40	0.03*	0.89	0.69	0.67	0.37	0.65	0.40
Income (> $100K as ref)
< $30K	0.67	0.41	1.22	0.54	0.96	0.93	0.64	0.42
$30K–$60K	0.85	0.73	1.28	0.42	1.58	0.25	1.45	0.40
$60K–$100K	1.50	0.54	1.29	0.54	0.75	0.66	0.58	0.48
Marital status (not married as ref)
Married or partnered	2.24	0.04*	1.02	0.95	1.19	0.62	0.92	0.83
COVID impact
Financial	1.04	0.68	0.93	0.27	0.96	0.63	0.95	0.64
Resources	0.83	0.16	1.00	0.97	0.93	0.56	0.94	0.63
Psychological	0.89	0.47	1.12	0.26	0.87	0.30	0.78	0.12

Note. HP, home practice; OR, odds ratio. aResults predict odds of using at least one (vs. 0) skill or home practice, respectively. bResults predict odds of completing all 10 skills (vs. < 10) or all 9 home practices (vs. < 9), respectively. *Significant at alpha level of 0.05

Change in Well-being

In unadjusted longitudinal growth models observing the change in PROMIS scores from baseline to post and from baseline to follow-up (Table 6), all five well-being metrics demonstrated statitistically significant improvements from baseline to post and from baseline to follow-up. Anxiety, depression, positive affect, and social isolation demonstrated meaningful change (> 3 T-score points) from T1 to T3; only meaning and purpose did not. Anxiety and depression were mildly elevated at baseline and fell within normal limits at follow-up. At all time points, social isolation was within normal limits, and both positive psychological outcomes (positive affect, meaning and purpose) were within the average range — although positive affect was just above the low to average threshold at baseline.

Table 6

Model-based estimates for PROMIS well-being T-scores at each assessment

	Total (n = 260)a	Unadjusted modelb	Accessed 0 skills (n = 15)	Accessed ≥ 1 skill (n = 179)	Usage (0 vs. ≥ 1 skill) × time
	M (SE)	p	M (SD)	M (SD)	β	p
Anxiety
T1	59.00 (0.55)	-	59.38 (2.22)	58.20 (0.63)	-	-
T2	56.71 (0.59)	< .0001*	58.70 (2.09)	57.05 (0.59)	−6.04	0.03*
T3	55.39 (0.62)	< .0001*	57.17 (2.56)	54.45 (0.72)	−1.42	0.54
Depression
T1	56.70 (0.53)	-	58.45 (2.05)	55.57 (0.59)	-	-
T2	53.85 (0.57)	< .0001*	57.09 (1.93)	54.36 (0.55)	−3.59	0.14
T3	53.10 (0.53)	< .0001*	54.03 (2.35)	51.64 (0.66)	0.54	0.80
Social isolation
T1	53.52 (0.54)	-	53.81 (2.09)	52.80 (0.60)	-	-
T2	51.40 (0.58)	< .0001*	52.02 (1.95)	51.87 (0.56)	−5.46	0.04*
T3	50.24 (0.62)	< .0001*	48.00 (2.43)	49.50 (0.69)	2.67	0.24
Positive affect
T1	40.67 (0.55)	-	41.37 (2.23)	41.71 (0.64)	-	-
T2	43.31 (0.59)	< .0001*	43.24 (2.07)	42.94 (0.59)	1.99	0.50
T3	44.75 (0.64)	< .0001*	47.45 (2.62)	45.70 (0.74)	−2.09	0.41
Meaning & purpose
T1	44.16 (0.73)	-	44.69 (2.96)	44.89 (0.85)	-	-
T2	45.48 (0.78)	.035*	46.10 (2.83)	45.68 (0.81)	−4.91	0.17
T3	46.86 (0.82)	< .0001*	49.28 (3.35)	47.45 (0.95)	−1.87	0.51

Note. T1 = baseline; T2 = post-intervention (week 8); T3 = follow-up (6 months after baseline). aTotal sample for longitudinal well-being analyses includes participants with PROMIS scores at baseline and at least one other timepoint. bLongitudinal growth models examining change from baseline. *p < .05

To approximate a test of intervention efficacy in this single-arm trial, we examined change in well-being over time among retained participants who accessed at least one skill (“intervention” group) compared to those who did not (“control”). The interaction between this usage group and time was significant only for anxiety and social isolation at T2 (Table 6). As shown in Figs. 3 and 4, participants who accessed at least one skill demonstrated significantly greater baseline to post-intervention decreases in anxiety and social isolation than participants who did not access any skills. There was no evidence that improvement in either of these well-being metrics from baseline to follow-up (T1 to T3) was moderated by usage. Further, we did not find evidence suggesting that improvements in any of the other well-being metrics (depression, positive affect, meaning and purpose) exhibited moderation effects by usage.

Fig. 3

Linear line of best fit for anxiety T-score change over time by artificial treatment delegation

Fig. 4

Line of best fit for social isolation T-score change over time by artificial treatment delegation

Discussion

At first glance, many findings from this single-arm trial of PARK, a self-guided, web-based program of positive emotion regulation skills designed to enhance coping with the stress of the COVID-19 pandemic, followed patterns seen in previous PPI and DMHI research. For example, the PARK sample primarily included non-Hispanic, white, well-educated women, mirroring socio-demographic characteristics seen in related trials (Hendriks, Warren, et al., 2019; Moskowitz et al., 2021). Like other studies describing psychological well-being during the pandemic (Daly & Robinson, 2021; VanderWeele et al., 2020), PARK participants reported clinically significant elevations in anxiety and depression and decrements in positive affect at baseline. With only 42% retained in assessments and 30% of participants never accessing the PARK website, retention and usage rates closely mirrored pre-pandemic trends of RCTs and publicly available apps (Baumel et al., 2019; Christensen et al., 2009; Fleming et al., 2018) — including another one specifically designed for coping with the pandemic, COVID Coach (Jaworski et al., 2021). Moreover, compared to participants who accessed the PARK website at least once, those who never logged in to PARK reported higher depression, anxiety, and impact of the pandemic on their resources, as well as lower meaning and purpose, reiterating earlier findings that psychological symptoms can deter DMHI usage (Borghouts et al., 2021).

Thus, even during a time that necessitated web-based delivery of most activities, this format can still fail to reach key segments of the population — namely, people of color, adults with lower socioeconomic status and limited access to resources, and those with a heavier psychological burden. Admittedly, our team did not strive to recruit a representative sample or engage in targeted recruitment and retention efforts. Psychological research and dissemination, including healthcare and the digitial health marketplace, should more actively heed guidelines for inclusive practices (Buchanan et al., 2021; Patalay & MacDonald, 2022).

Within the select sample who did access the PARK website, several predictors of usage ran counter to prior research. Across the variety of usage metrics we examined, we found greater usage of PARK among older participants. Whereas previous DMHI studies showed higher engagement among younger participants (Onyeaka et al., 2021), this contrary finding in PARK may stem from pandemic-related restrictions on in-person activities; for example, a web-based tool such as PARK might have been particularly accessible to middle-aged adults who were increasingly at home with children and older adults closely adhering to distancing precautions given their increased risk of COVID and its sequelae. Although we did not link individuals’ dates of participation with their local public health measures, pandemic-related reductions in the availability of in-person activities might also explain why usage of PARK home practice exercises was more common among participants with higher self-reported social isolation.

Participants who qualified as completers (i.e., they accessed all 10 skills or all 9 home practice exercises) also differed from trends in prior research. PARK completion was more likely among older adults, people of color, and participants with higher baseline levels of anxiety. These findings underscore the need to remove barriers to enrollment in clinical research and access to DMHI. They also parallel findings from COVID Coach, a publicly available app for coping with the pandemic, which demonstrated a positive association between psychological symptoms at baseline and days of app usage (Jaworski et al., 2021). Together, these findings indicate that, as the pandemic drove increased reliance on phone- and web-based mental health resources (Sorkin et al., 2021), it might also have shifted trends in who uses DMHI.

In simple longitudinal tests of PROMIS scores, all well-being metrics statistically significantly improved over time, with meaningful improvements seen in anxiety, depression, social isolation, and positive affect (i.e., all but meaning and purpose). Because PARK was not tested in a randomized controlled trial, these changes could have resulted from factors unrelated to the intervention. For example, studies have reported co-occurrence of declines in mental health and increases in pandemic-related stressors such as regional elevations in COVID prevalence and restricted mobility (Daly & Robinson, 2021; Santomauro et al., 2021). Psychological well-being in PARK participants could have fluctuated with local changes in the pandemic, which we did not measure.

Despite the single-arm study methodology, capitalizing on naturally occurring differences in PARK usage allowed us to approproximate a test of efficacy by comparing changes in well-being over time in participants who used at least some of the PARK skills vs. those who did not. These results provide preliminary evidence that using at least some of the PARK skills facilitated greater decreases in anxiety and social isolation. In particular, this was clinically significant for anxiety, where only participants who accessed PARK skills showed drops in anxiety from mildly elevated at baseline to within normal limits at follow-up.

In addition to the limitations of the single-arm design, this study did not account for all factors that could have influenced retention, usage, or participants’ well-being. For example, we did not measure factors previously demonstrated to predict DMHI usage, such as experience with technology and mental health literacy (Borghouts et al., 2021). Additionally, given that use of DMHI has been positively associated with COVID case rates (Sorkin et al., 2021), some attrition could have occurred during lulls in the pandemic. We did not thoroughly assess pandemic-related changes such as increased caregiving responsibilities, difficulty transitioning to remote work, and interpersonal conflict within the home, which contributed to global increases in depression and anxiety (Alzueta et al., 2021). However, in addition to PROMIS well-being scores, we included measures of COVID impact, which allow us to account for pandemic-specific perceived stress. In previous US samples, these individual-level responses to and appraisals of the pandemic were predictive of mental health outcomes, whereas objective threat of COVID was not (Nikolaidis et al., 2022).

Caveats about the select sample and relatively low engagement with PARK notwithstanding, we can draw several conclusions from this study. First, the stress of the pandemic led to a need for programs such as PARK, and many people have relied on DMHI to be nimble and responsive to this significant, global event (Park et al., 2021; Sorkin et al., 2021). Our team made PARK widely available by request, and among those who accessed it, multiple well-being metrics improved. Factors that previously were barriers to DMHI usage, such as older age and psychological symptoms (Borghouts et al., 2021; Onyeaka et al., 2021), in this case predicated greater engagement. Still, these findings from PARK alone are not proof that PPIs work and should be widely distributed. Instead the current results provide more information about how well-being may change among individuals enrolled in a PPI, along with individual (e.g., age) and contextual (e.g., social isolation) factors that may influence who is most likely to engage in programs like PARK. They also underline the importance of continuing to adapt our research practices and our interventions to ensure that science and health include everyone (Buchanan et al., 2021; Patalay & MacDonald, 2022).