Comparing efficacy of telehealth to in-person mental health care in intensive-treatment-seeking adults.

The heightened acuity in anxiety and depressive symptoms catalyzed by the COVID-19 pandemic presents an urgent need for effective, feasible alternatives to in-person mental health treatment. While tele-mental healthcare has been investigated for practicability and accessibility, its efficacy as a successful mode for delivering high-quality, high-intensity treatment remains unclear. This study compares the clinical outcomes of a matched sample of patients in a private, nation-wide behavioral health treatment system who received in-person, intensive psychological treatment prior to the COVID-19 pandemic (N = 1,192) to the outcomes of a distinctive group of patients who received telehealth treatment during the pandemic (N = 1,192). Outcomes are measured with respect to depressive symptoms (Quick Inventory of Depressive Symptomatology-Self-Report; QIDS-SR) and quality of life (Quality of Life Enjoyment and Satisfaction Questionnaire; Q-LES-Q). There were no significant differences in admission score on either assessment comparing in-person and telehealth groups. Patients in the partial hospitalization level of care stayed longer when treatment was remote. Results suggest telehealth as a viable care alternative with no significant differences between in-person and telehealth groups in depressive symptom reduction, and significant increases in self-reported quality of life across both groups. Future research is needed to replicate these findings in other healthcare organizations in other geographical locations and diverse patient populations.

Introduction

In response to the coronavirus disease 2019 (COVID-19) pandemic, healthcare organizations around the world adapted to remote behavioral health treatment to meet an increased need for high-quality mental health care. With lockdowns and stay-at-home orders, the pandemic presented an acute need for mental health treatment to be virtually accessible—regardless of patient or provider setting—as well as near-equivalent in quality to what a person would have received had they attended in person. This paper examines the current landscape of remote mental health care as a viable treatment delivery method and uses clinical outcomes data from patients undergoing in-person and telehealth mental health treatment at a large, multistate behavioral health system to illustrate telehealth's comparability to traditional, in-person care.

The US Centers for Disease Control and Prevention (CDC) reported that symptoms of anxiety and depressive disorders among adults “increased considerably” when comparing reports from April through June of 2019 to the same time period in 2020 (Czeisler et al., 2020). In the survey with 5,412 adult respondents, the CDC measured anxiety and depressive symptoms related to the pandemic using the four-item patient health questionnaire (PHQ-4) and COVID-19 trauma-related symptoms using the six-item Impact of Event Scale. Overall, 40.9% of survey respondents reported at least one adverse mental health condition, including symptoms of anxiety disorder or depressive disorder (30.9%), and 26.3% reported symptoms of a trauma- and stressor-related disorder because of the pandemic. Suicidal ideation in the last 30 days was reported by 10.7% of respondents. In 2021, the CDC published results of the Household Pulse Survey, a self-report online survey, which showed that the percentage of adults who reported experiencing symptoms related to depression and anxiety disorders increased from 36.4% to 41.5% between August 2020 and February 2021 (Vahratian et al., 2021). Additionally, respondents who reported unmet mental health needs increased from 9.2% to 11.7% in the same time period. Historically there is evidence that mental health needs increase during a pandemic (Soklaridis et al., 2020). Past outbreaks, such as severe acute respiratory syndrome (SARS) in 2003, directly resulted in heightened anxiety related to the virus (Cheng and Cheung 2005; Chong et al., 2004), as well as increased stigmatization and discrimination of minority groups (Ren et al., 2020). Feelings of uncertainty, isolation, and confusion over rapidly changing health guidelines have exacerbated feelings of anxiety and mental distress (Rajkumar, 2020). What sets the COVID-19 pandemic apart is the already widespread use of telecommunication platforms for remote work, learning, and basic healthcare. Although prior studies have investigated the impact of telehealth compared to traditional in-person treatment, there has been limited research into these differences in treatment settings beyond the outpatient level (Hilty et al., 2013). This is a novel offering of the current analysis.

Mental healthcare providers have been exploring remote treatment options for over six decades (Hilty et al., 2013). In this domain, providers use videoconferencing for therapy, evaluations, and medication management. In some cases, clinicians use telehealth to supplement in-person treatment. Previous research has shown that telehealth reduces patient-incurred costs and time associated with travel (Rabinowitz et al., 2010) and improves accessibility of mental healthcare for those living in rural areas (Manfredi et al., 2005; Morland et al., 2010; Weiner et al., 2011). Some even argue that remote treatment is superior to in-person consultations for some patients (Pakyurek et al., 2010; Storch et al., 2011). In the context of public health, others have highlighted telehealth as a way for patients to maintain human connection during social distancing and lockdowns (Whaibeh et al., 2020). Findings are mixed with respect to patient satisfaction of psychiatric telehealth services (Frueh et al., 2005; Jacob et al., 2012; Nelson et al., 2003; Ruskin et al., 2004; Zimmerman et al., 2021a), yet remote care has been shown to improve access to mental health treatment in some cases, especially for individuals experiencing linguistic or financial obstacles (Chong and Moreno, 2012; Moreno et al., 2012; Mucic 2010; Ye et al., 2012). For example, Mucic (2010) highlights that the availability of bilingual clinicians through remote care can help overcome travel requirements for both patients and providers, as well as lead to higher patient satisfaction. In the case of the pandemic, telehealth treatment may also help minority groups navigate greater exposure risk to COVID-19 by allowing them to remain remote and socially distanced distanced during care (Price-Haywood et al., 2020). Prior to the pandemic, telehealth was largely reported as comparable to in-person treatment in terms of its feasibility in obtaining favorable treatment outcomes in a variety of symptom profiles (De Las Cuevas et al., 2006; Lopez et al., 2004; Nelson et al., 2003; Richardson et al., 2009; Rojas et al., 2020; Ruskin et al., 2004; Shore et al., 2008; Urness et al., 2006; Yellowlees et al., 2010). Zimmerman and colleagues describe how patients in a partial hospitalization program reported significant reduction in symptoms whether they received in-person or telehealth treatment (Zimmerman et al., 2021b). However, to date, there are few other complete comparisons of patient outcomes prior to and following the pandemic, specifically as they relate to the efficacy of remote versus in-person treatment (although see Hom et al., 2020 for preliminary data). Further, the efficacy of telehealth as a treatment alternative has not previously been examined in a comparison of partial hospitalization and intensive outpatient programming, namely how level of care and acuity might affect outcomes.

The present study analyzes the clinical outcomes of patients who underwent mental health treatment in this telehealth format during the COVID-19 pandemic and compares their outcomes to patients from the same healthcare system who completed in-person care prior to the pandemic. The research questions were twofold: (1) How did treatment delivery (in-person vs. telehealth) affect patient outcomes across Rogers as a nation-wide provider; and (2) does treatment delivery modulate therapeutic response for patients in different levels of care (PHP or IOP)? Considering some service lines utilize therapies traditionally conducted in-person—for example, exposure therapies or behavioral activation exercises—it was hypothesized that any between-group differences may be due to the difficulty associated with implementing these methods virtually in such a short timeframe. If this is the case, patients who received in-person treatment are expected to show greater symptom reduction than patients who received telehealth treatment. However, consistent with the Zimmerman et al. (2021b) study, if the therapies employed translate as planned to the virtual format, we do not expect significant differences in scores at discharge comparing in-person and telehealth groups.

Methods

Participants and procedure

The Rogers Behavioral Health (Rogers) Institutional Review Board is committed to the ethical principles for the protection of human subjects in research which guide the IRB's deliberations and decision-making, including respect for persons, beneficence, and justice. This study was approved with exempt determination by the Rogers Institutional Review Board (IRB). Because this study contained retrospective data analysis only and no prospective data collection, there was no consent process. Only de-identified data was used for analyses. All adult patients between 18 and 65 were eligible for inclusion. If patients were missing either an admission or discharge score for either of the two measures, they were excluded from analyses.

In March of 2020, following the onset of the COVID-19 pandemic, Rogers converted partial hospitalization (PHP) and intensive outpatient (IOP) treatment programs to telehealth. Prior to the pandemic, telehealth services were occasionally provided on an as-needed basis. The organization-wide transition to telehealth leveraged existing technological solutions for videoconferencing and scheduling sessions. If patients did not have a device to access their therapy sessions, laptops were provided. To ensure high-quality treatment, fidelity checks of treatment delivery were conducted by clinical supervisors and directors of clinical services. As pre-existing in-person treatment was protocolized and manualized, an existing audit process was leveraged to ensure compliance with treatment groups and to identify gaps in telehealth treatment delivery as compared to in-person care. Specific aspects of treatment varied by services line (e.g., OCD PHP versus Depression Recovery PHP), however there was overlap in main treatment functions. Specifically, individual meeting invites were sent to each patient for every treatment element daily. Individual sessions included those with their assigned therapist and with their psychiatrists. Group sessions may have included cognitive behavioral therapy skills or possibly dialectal behavior therapy skills depending on the program the patient was enrolled in. Couples or family sessions were also delivered via telehealth. All treatment manuals and materials were sent electronically for patients to use during telehealth programming. Completed forms were emailed back to treatment team members to review in future sessions. Patients completed self-reported outcome measures in a protocolized fashion, at home, and all scores received from these assessments were integrated into the patient's treatment. Close attention was paid to patient attendance for each treatment element and local police phone numbers and emergency contact information was collected and made accessible for all treatment team members in case someone did not appear for treatment in a timely manner. In addition to emergency contact information, safety protocols were developed to ensure response to safety concerns was timely and consistent.

The research study included in this submission aligns with the IRB-approved protocol. Participants include 2,384 adult patients who completed either PHP or IOP programs at Rogers, 1,192 patients in each of the in-person and telehealth groups. Patients in the telehealth group received treatment between July 2020 and March 2021. The group of patients who received telehealth was matched to an equally sized subsample of 1,192 patients who received in-person care between May 2013 and December 2019. Groups were matched using optimal pairwise matching computed with the MatchIt package in R (Ho et al., 2011), where covariates unrelated to treatment were balanced between in-person and telehealth groups. Covariates that were balanced in this way include age, sex, race, diagnosis, and level of care, ensuring that comparable demographic groups could be compared in addition to having an equal proportion of PHP and IOP patients included in each of the groups. Patients whose sex and/or race were unknown were excluded from the matching process and were not included in the analyzed dataset (Table 1 ). There were no significant differences with respect to diagnosis across the dataset either comparing in-person to telehealth groups or with level of care (Table 2 ).

Table 1

Comparison of age, sex assigned at birth, and race of in-person and telehealth groups.

Partial Hospitalization (PHP)
	In-Person n = 950	Telehealth n = 950	Comparison
	Mean ± SD	Mean ± SD
Age	31.53 ± 11.81	31.69 ± 11.84	t(3798) = −0.41	p = .68
	% (n)	% (n)
Sex			Χ2 (df = 1, 0.02)	p = .88
Female	66.0% (627)	66.4% (631)
Male	34.0% (323)	33.6% (319)
Race			Χ2 (df = 4, 0.08)	p = 1
Asian	1.6% (15)	1.7% (16)
Black or African American	2.9% (28)	2.8% (27)
Native Hawaiian or Pacific Islander	0.4% (4)	0.4% (4)
White	93.5% (888)	93.4% (887)
Multiple	1.6% (15)	1.7% (16)
Intensive Outpatient (IOP)
	In-Person n = 242	Telehealth n = 242	Comparison
	Mean ± SD	Mean ± SD
Age	34.31 ± 13.29	33.88 ± 13.19	t(966) = 0.50	p = .62
	% (n)	% (n)
Sex			Χ2 (df = 1, 0.32)	p = .57
Female	61.6% (149)	64.5% (156)
Male	38.4% (93)	35.5% (86)
Race			Χ2 (df = 4, 1.95)	p = .58
Asian	3.7% (9)	4.5% (11)
Black or African American	7.4% (18)	4.5% (11)
Native Hawaiian or Pacific Islander	0.0% (0)	0.0% (0)
White	86.4% (209)	88.4% (214)
Multiple	2.5% (6)	2.5% (6)

Note. Age is denoted as mean ± standard deviation. Demographic percentages are denoted with counts in parentheses. T-tests were used to test for significant differences when age was the dependent variable. Chi-square tests were used to detect differences in sex and race.

Table 2

Comparison of diagnosis category across in-person and telehealth groups.

Partial Hospitalization (PHP)
	In-Person n = 950	Telehealth n = 950	Comparison
	% (n)	% (n)	Χ2	p
Anxiety	50.7% (482)	55.3% (525)	(df = 1, 3.73)	.06
Feeding/eating	8.9% (85)	7.9% (75)	(df = 1, 0.55)	.46
Mood	77.9% (740)	74.8% (711)	(df = 1, 2.29)	.13
Neurodevelopmental	11.2% (106)	12.4% (118)	(df = 1, 0.61)	.43
OCD	23.9% (227)	25.2% (239)	(df = 1, 0.34)	.56
Social	9.5% (90)	8.7% (83)	(df = 1, 0.23)	.63
Substance abuse/addictions	20.3% (193)	22.9% (218)	(df = 1, 1.79)	.18
Trauma	20.3% (193)	23.9% (227)	(df = 1, 3.33)	.07
Intensive Outpatient (IOP)
	In-Person n = 242	Telehealth n = 242	Comparison
	% (n)	% (n)	Χ2	p
Anxiety	59.5% (144)	55.4% (134)	(df = 1, 0.68)	.41
Feeding/eating	2.5% (6)	5.0% (12)	(df = 1, 1.44)	.23
Mood	68.6% (166)	60.7% (147)	(df = 1, 2.93)	.09
Neurodevelopmental	10.3% (25)	11.6% (28)	(df = 1, 0.08)	.77
OCD	22.7% (55)	18.6% (45)	(df = 1, 1.02)	.31
Social	9.9% (24)	8.3% (20)	(df = 1, 0.23)	.64
Substance abuse/addictions	28.5% (69)	21.5% (52)	(df = 1, 2.82)	.09
Trauma	14.0% (34)	12.8% (31)	(df = 1, 0.07)	.79

Note. Data are shown as proportions with counts in parentheses. Chi-square test results and p-values are reported.

Regardless of treatment delivery mode, prior to admitting to treatment, patients' psychological symptoms are assessed over the phone. Licensed psychiatrists and psychologists with expertise in the patient's behavioral health area of concern review the phone screen interviews to determine treatment program appropriateness and to recommend level of care. PHP and IOP are multidisciplinary programs that involve individual, group, and family therapies along with medication management. Patients in PHP attend treatment 6 h per day, five days a week, and patients in IOP complete 3 h of treatment five days a week. After they are admitted, patients work with a psychologist to complete a diagnostic assessment where diagnoses are confirmed using the Diagnostic and Statistical Manual-5th Edition (DSM-5; American Psychological Association, 2013). Patient-reported outcome measures (PROMs) are collected from each patient at the beginning of treatment, either weekly or biweekly during treatment, and at the end of treatment. Patients received the same treatment content and frequency of treatment sessions regardless of when they completed treatment. In-person and telehealth providers participated in the same internal training program, ensuring a high level of similarity in treatment received by in-person and telehealth patient groups. Due to staff turnover, some providers were consistent across in-person and telehealth timeframes, whereas others were unique to one timeframe.

Two PROMs were used to index treatment efficacy: the Quick Inventory of Depressive Symptomology-Self Report (QIDS-SR; Rush et al., 2003) and the Quality of Life Enjoyment and Satisfaction Questionnaire - Short Form (Q-LES-Q; Schechter et al., 2007). Both assessments are administered upon admission, discharge, and either weekly or biweekly across all adult programs, service lines, and levels of treatment at Rogers. The Quick Inventory of Depressive Symptomatology Self-Report (QIDS-SR; Rush et al., 2003) is a 16-item self-report measure that asks respondents to indicate on a scale of 0–3 (0 = low, 3 = high) how well each item describes them for the past week. The QIDS-SR assesses nine domains of depression symptomatology: sleep, mood, weight, concentration, guilt, suicidal ideation, interest, fatigue, and psychomotor changes. Responses are summed to provide a single outcome metric, which can range from 0 to 27, where a higher score indicates a more severe level of depression symptomatology. The Quality-of-Life Enjoyment and Satisfaction Questionnaire (Q-LES-Q; Schechter et al., 2007) is a 16-item self-report measure that asks respondents to indicate on a scale of 1–5 (“very poor” = 1, “very good” = 5) how satisfied they have been in the past week with aspects of their general health, well-being, and feelings about their life. Total scores are reported as percentages ranging from 0 to 100%, with higher percentages indicating greater enjoyment and satisfaction.

Statistical analyses were performed using R (R Core Team, 2021). To measure differences between in-person and telehealth groups, we conducted 2 × 2 between-subjects analyses of variance (ANOVAs) with treatment modality (in-person, telehealth) and level of care (PHP, IOP) as independent variables. One ANOVA was conducted for each of total QIDS-SR score at admission, score at discharge, and percent change over time between admission and discharge. The same three tests were conducted for Q-LES-Q scores. Planned pairwise comparisons within each significant factor were conducted using t-tests (e.g. PHP compared to IOP). For all analyses, a two-tailed level of significance of 5% was adopted.

Results

There were no significant differences in admission score on either the QIDS-SR (F(2380) = 1.06, p = .30) or the Q-LES-Q (t(2380) = 0.26, p = .61) between in-person and telehealth groups (Table 3 ). For QIDS-SR scores, there was a significant main effect of level of care only at both admission (F(1, 2380) = 88.00, p < .001) and at discharge (F(1, 2380) = 48.07, p < .001). Planned pairwise comparisons showed that PHP scores at both time points, on average, were higher than IOP scores, which is clinically expected. The main effect of treatment delivery was trending but not significant at discharge only (F(1, 2380) = 3.60, p = .06) such that patients who received telehealth treatment had a higher average QIDS discharge score compared to in-person. The interaction between treatment delivery and level of care was not significant.

Table 3

Comparison of clinical assessment outcomes at admission and discharge for in-person and telehealth groups. Data are shown as mean ± standard deviation with analysis of variance results for the group factor.

Clinical Assessment	Total Score	In-Person	Telehealth	Cohen's D	F(1, 2380)	p	η2	Post-hoc comparisons
QIDS
ADMISSION
Treatment Delivery					1.06	.30	0.00
In-person	14.15 ± 5.07
Telehealth	14.36 ± 5.27
Level of Care					88.00	<.001***	0.04	t(746) = 9.35, p < .001***
PHPa		14.54 ± 5.04	14.95 ± 5.10
IOP		12.60 ± 4.87	12.04 ± 5.30
Treatment Delivery  × Level of Care					3.55	.06	0.00
DISCHARGE
Treatment Delivery					3.60	.06	0.00
In-person	8.42 ± 5.12			1.13
Telehealtha	8.82 ± 5.27			1.05
Level of Care					48.07	<.001***	0.02	t(798) = 7.28, p < .001***
PHPa		8.82 ± 5.14	9.15 ± 5.31	1.12
IOP		6.85 ± 4.71	7.49 ± 4.91	1.04
Treatment Delivery  × Level of Care					0.34	.56	0.00
In-person PHP				1.12
In-person IOP				1.20
Telehealth PHP				1.11
Telehealth IOP				0.89
% CHANGEb
Treatment Delivery					0.01	.90	0.00
In-person	−37.34% (37.38%)
Telehealth	−37.06% (37.47%)
Level of Care					0.00	.96	0.00
PHP		−36.50% (37.43%)	−37.04% (34.73%)
IOP		−37.34% (37.38%)	−37.06% (37.47%)
Treatment Delivery  × Level of Care					2.67	.10	0.00
Q-LES-Q
ADMISSION
Treatment Delivery					0.26	.61	0.00
In-person	45.89 ± 16.07
Telehealth	46.22 ± 16.33
Level of Care					78.00	<.001***	0.03	t(742) = −8.77, p < .001***
PHP		44.64 ± 15.87	44.55 ± 15.94
IOPa		50.78 ± 15.94	52.75 ± 16.22
Treatment Delivery  × Level of Care					1.59	.21	0.00
DISCHARGE
Treatment Delivery					0.73	.39	0.00
In-person	61.05 ± 17.62			0.90
Telehealth	61.67 ± 17.48			0.91
Level of Care					39.14	<.001***	0.02	t(780) = −6.46, p < .001***
PHP		59.73 ± 17.47	60.73 ± 17.71	0.93
IOPa		66.23 ± 17.27	65.31 ± 16.07	0.86
Treatment Delivery  × Level of Care					1.17	.28	0.00
In-person PHP				0.90
In-person IOP				0.93
Telehealth PHP				0.96
Telehealth IOP				0.78
% CHANGEb
Treatment Delivery					1.62	.20	0.00
In-person	21.46% (35.94%)
Telehealth	21.06% (34.81%)
Level of Care					2.69	.10	0.00
PHP		20.21% (42.93%)	21.65% (36.79%)
IOP		21.46% (35.94%)	21.06% (34.81%)
Treatment Delivery  × Level of Care					2.33	.13	0.00

Note: Significance threshold used is 0.05. F-statistics and partial eta-squared (η2) reflect the ANOVA results at the group level. T-tests reflect significant pairwise comparisons. Cohen's D effect size reflects the change between admission and discharge score and is presented alongside discharge data only.

Significantly greater.

% Change denotes percent change in total score (e.g. admission, discharge) over time.

The same pattern of results seen for QIDS scores was also seen for Q-LES-Q scores, namely only a significant main effect of level of care at both admission (F(2380) = 78.00, p < .001) and at discharge (F(2380) = 39.14, p < .001). There was no main effect of treatment delivery or interaction between treatment delivery and level of care. Planned pairwise comparisons showed that Q-LES-Q admission and discharge scores were higher for patients in IOP relative to PHP, which is an expected result given the difference in degree of symptom severity at these different levels of care.

Pairwise Cohen's d effect sizes were calculated, as well as descriptive statistics for the change in QIDS-SR and Q-LES-Q scores from admission to discharge for all. Pairwise comparisons show the degree of change between admission and discharge scores comparing in-person and telehealth groups was not significant at either IOP or PHP levels of care for either QIDS-SR or Q-LES-Q. Effect sizes, overall, were moderate to high at all levels of care for both assessments. Length of stay was not significantly different comparing in-person and telehealth IOP groups (t(481) = −1.66, p = .10), but it was significantly different comparing PHP groups, such that patients receiving telehealth stayed 2.8 days longer in treatment than patients seen in person (t(1877) = −6.62, p < .001).

Discussion

To demonstrate the efficacy of telehealth treatment relative to in-person treatment, we compared clinical outcomes scores at admission and discharge for adult patients (1,192 in-person and 1,192 telehealth) across PHP and IOP programs. Patients who received PHP telehealth stayed an average of 2.8 days longer in treatment than the PHP in-person group; there was no difference in length of stay in IOP programs. This significant finding of increased length of stay in the PHP telehealth group is consistent with prior work (Zimmerman et al., 2021b). There were no significant differences in QIDS or Q-LES-Q discharge scores for either IOP or PHP groups, and there were no significant differences in the degree of change between admission and discharge scores for any group. Effect sizes were moderate to large across treatment modalities and levels of care. These data support remote treatment as a viable alternative to in-person mental health services, specifically as both in-person and remote patients experienced symptom reduction, and both populations reported improvements in quality of life.

There are recognized limitations of this study. Patients were not randomized into in-person and telehealth groups; rather, the external variable of the COVID-19 pandemic forced group membership in our sample. However, there were no between-group differences in QIDS-SR and Q-LES-Q admission scores at either the holistic level or by level of care, which suggests that no one group was more acute than the other at onset of treatment as indexed by these assessments. Further, matching in-person and telehealth groups for pre-treatment covariates—age, sex, race, and level of care—allowed us to examine the dependent variable of interest more closely, namely treatment outcomes, independent of these other factors. Additionally, findings are limited by the lack of a structured clinician-administered diagnostic assessment at admission which would strengthen the diagnoses established. However, licensed psychiatrists with years of experience diagnosing mental health disorders conducted the initial diagnostic assessments, and patient charts would be updated to reflect any changes to diagnoses made later in treatment.

Results have implications for both patients considering remote treatment options as well as organizations looking to implement alternatives to in-person care. Despite the difference in treatment delivery, our large sample size from a system-wide perspective supports telehealth as being as effective as in-person care for intensive-treatment-seeking individuals. For patients living far away from treatment centers or for patients who cannot leave home to attend in-person treatment, these results demonstrate telehealth treatment as a viable alternative to in-person treatment.

Author statement

Nyssa Z. Bulkes, Ph.D.: Methodology, Formal Analysis, Writing – Original Draft.

Kaley Davis, M.S.: Writing – Reviewing & Editing, Project Administration.

Brian Kay, Ph.D.: Resources, Writing – Reviewing & Editing, Conceptualization.

Bradley C. Riemann, Ph.D.: Resources, Writing – Reviewing & Editing, Conceptualization.

Declaration of competing interest

None.