Patient satisfaction with partial hospital telehealth treatment during the COVID-19 pandemic: Comparison to in-person treatment.

Most research evaluating telehealth psychiatric treatment has been conducted in outpatient settings. There is a lack of research assessing the efficacy of telehealth treatment in more acute, intensive treatment settings such as a partial hospital. In the face of the COVID-19 pandemic, much of ambulatory behavioral health treatment has transitioned to a telehealth, or virtual, format. In the present report from the Rhode Island Methods to Improve Diagnostic Assessment and Services (MIDAS) project, we compared patient satisfaction of partial hospital services delivered via telehealth to in-person treatment provided to patients treated prior to the COVID-19 outbreak. The sample included 240 patients who were treated virtually from May, 2020 to October, 2020, and a comparison group of 240 patients who were treated in the in-person partial program a year earlier. Patients completed self-administered measures of patient satisfaction after the initial evaluation and at the end of treatment. For both the in-person and telehealth methods of delivering partial hospital level of care, patients were highly satisfied with the initial diagnostic evaluation and were optimistic at admission that treatment would be helpful. At the completion of treatment, both groups were highly satisfied with all components of the treatment program and almost all would recommend treatment to a friend or family member. Thus, patient satisfaction was as high with telehealth partial hospital treatment as with in-person treatment.

Introduction

In the face of the COVID-19 pandemic, most ambulatory behavioral health treatment has been required to transition to a telehealth format due to public health recommendations and legal guidelines for social distancing (Lewnard and Lo, 2020; Wright and Cardill, 2020). With continued advances in technology, telehealth services for mental health treatment have gained increasing attention as a cost-effective way to increase accessibility to evidence-based treatments (Gros et al., 2013; Ralston et al., 2019). Telehealth interventions are generally acceptable, feasible, and comparable in efficacy to in-person mental health services (Drago et al., 2016; Shigekawa et al., 2018).

The empirical literature on telehealth for mental disorders has been limited to outpatient settings and has focused on patients with single disorders. The few studies intentionally treating patients with co-occurring disorders are largely preliminary in nature and use small samples (Khatri et al., 2014; Smith et al., 2021; Kennedy et al., 2020). Data is lacking in evaluating telehealth treatment delivery in partial hospital and other intensive treatment settings. In these higher level of care settings, patient severity and the risk of self-harm and suicidal behavior is greater than in outpatient practice thereby raising concerns as to whether telehealth treatment could be provided while maintaining patient safety and patient satisfaction. Studies of telehealth in outpatient settings often exclude patients with suicidal ideation (Fortney et al., 2007; Fortney et al., 2013; Mohr et al., 2012), though some preliminary reports have addressed the issue of treating suicidal patients (Rojas et al., 2020).

Partial hospital programs (PHPs) and intensive outpatient treatment programs face unique concerns and challenges in adopting a telehealth platform (Inchausti et al., 2020; Horn et al., 2020). In working virtually with acute psychiatric patients who require a higher level of care, many of whom may be transitioning from inpatient programs, appropriate risk management is essential. This can be perceived as intrusive and burdensome and reduce patient satisfaction. Furthermore, for group therapy-based programs, additional considerations regarding privacy and technology constraints are needed, again potentially impacting satisfaction. Despite these concerns, Hom and colleagues (2020) described the changes made in adapting their PHP to a telehealth platform and reported high rates of acceptability amongst the first 10 patients discharged from the program.

Patient satisfaction is recognized as an important construct to assess in evaluating the quality of medical care. While there are questions as to how accurate satisfaction with treatment is as a marker of quality, measures of satisfaction have been found to be significantly associated with external, independent assessments of the quality of care, thereby validating their utility as an indicator of quality (Holcomb et al., 1998; Shipley et al., 2000).

Studies of outpatient samples have found telehealth treatment to be both feasible and acceptable (see Bashshur et al., 2016; Hubley et al., 2016 for reviews). However, few studies directly compared patient satisfaction based on in-person and telehealth formats and the results have been mixed (Morland et al., 2010; Holden and Dew, 2008; Urness et al., 2006).

As part of ongoing efforts for the Rhode Island Methods to Improve Diagnostic Assessment and Services (MIDAS) project, we have routinely assessed patient satisfaction with treatment. In our transition to a telehealth-based program as a response to the COVID-19 pandemic, we continued to evaluate patient satisfaction in our PHP. In this paper, we compare satisfaction of PHP services delivered via telehealth to in-person treatment provided to patients treated prior to the COVID-19 outbreak.

Methods

Setting

The study was completed in the Rhode Island Hospital Adult PHP—an acute care setting serving a range of presenting concerns referred from various clinical settings. A multidisciplinary team of psychiatrists, psychologists, clinical social workers, postdoctoral fellows, and doctoral level graduate student therapists delivered the treatment. All intake assessments, individual therapy, psychiatry visits, and group therapy sessions were conducted virtually using real-time audio and visual computer-based communication using the Zoom tele-health platform, business account version.

Intervention and adaptation to telehealth

Upon referral to the program, administrative staff and research assistants contacted patients to describe the program and assess inclusion criteria. Inclusion criteria were broad to maximize access to care and encompassed most diagnoses and presenting concerns. To participate in the telehealth program, patients needed a computer, tablet, or smartphone with access to the internet. Additional inclusion criteria included availability of a private location where patients could participate confidentially in individual and group therapy sessions. Patients were excluded from the PHP if they had a primary substance use disorder or imminent suicidal or homicidal ideation with plan and intent. Patients with active suicidal ideation were routinely included in the program, except when their suicidal ideation was accompanied by both plan and intent such that the team could not be certain they could maintain safety. Inclusion and exclusion criteria were the same for the telehealth and in-person programs.

Additional safety procedures were implemented to address the unique challenges of delivering treatment via telehealth in an acute care setting in which patients frequently present with safety concerns, including suicidal ideation, self-injurious behavior, and aggressive ideation and behavior. First, to address the challenge of tracking patient attendance and location in a telehealth program, a check-in procedure was implemented in which patients participate each morning in a brief Zoom check-in group in which administrative staff record their attendance. Following the meeting, attendance is relayed to the treatment team along with daily updated email and physical address information for each patient. As patients are not physically present in the program, they could not be accompanied if transfer to inpatient or emergency room care was needed. For this reason, this information was necessary to send emergency support to patients’ residences in instances of worsening suicidal ideation or safety concerns. Patients were also required to identify an emergency contact support person and to submit release of information paperwork upon beginning the program.

Second, one of the therapists from the PHP functioned as a clinical and technical support resource who was available during program hours to respond by phone or Zoom visit to urgent patient needs. This included needs for urgent clinical support outside of individual sessions, assistance with troubleshooting technical problems, and assistance connecting patients to their individual providers. Third, the team developed and administered guidelines for patients upon intake for effective participation in the virtual format. These included participating from a private location with the camera on at all times to ensure confidentiality, guidelines on procedures for leaving groups for breaks and individual sessions, recommendations for engaged, effective participation in virtual groups (e.g., participating in group seated and minimizing distractions), as well as contact information for the clinical/technical support person. Finally, recommendations to staff were made to ensure patient confidentiality and safety when conducting group and individual sessions, including recommended settings to select when setting up sessions in the Zoom telehealth platform (e.g., enabling the waiting room feature to allow group leaders to verify that each member they admit to a group session is a patient in the program who is presenting to the correct group).

Similar to the in-person PHP, admissions to the virtual program are on a rolling basis with patients completing varying lengths of treatment depending on the symptoms and problems being addressed. Patients attend daily individual therapy sessions of approximately 30 to 45 min, as well as regular psychiatry sessions. Patients also attend 3 daily therapy groups plus an optional mindful meditation group, including two didactic groups focused on delivering skills, and one process-oriented group in which a group leader assists patients to apply the skills to day-to-day concerns they raised in the group. The focus, content and structure of these groups was consistent with the in-person PHP (see Morgan et al., 2021 for details).

Measures

At admission, the patients completed the Clinically Useful Patient Satisfaction Scale (CUPPS; Zimmerman et al., 2017a, 2017b). The partial hospital version of the CUPSS is a 16-item scale covering 3 areas: clinician attitude and behavior (12 items), office environment and staff (2 items), global satisfaction and expectation of improvement (2 items). We deleted the office environment and staff items when administering the scale to the patients receiving telehealth treatment.

The initial validation study of the CUPSS included 500 PHP patients (Zimmerman et al., 2017a). The internal consistency of the CUPSS was high (Cronbach's alpha = .93), and all item-scale correlations were significant. All items were significantly correlated with each of the indicators of global satisfaction. A second study of the reliability and validity of the CUPSS, in more than 1,300 PHP patients, replicated the results of the first study (Zimmerman et al., 2017b).

On the day of discharge from the program, patients completed a satisfaction scale on which they rated their satisfaction with each element of the program as well as their overall satisfaction with treatment. The patients also indicated if they would recommend the program to a friend or family member.

During the in-person program, the admission and discharge scales were handed to the patients by their treating clinicians and returned to the support staff or placed in a covered box in the check-in area near the support staff. In the virtual program, patients were sent links by both support staff and their clinicians to complete the scales online. The Rhode Island Hospital institutional review committee approved the research protocol, and all patients provided informed consent. Consent in the in-person program was obtained on a paper consent form, whereas in the virtual program it was obtained on an electronically signed form.

Data analysis

T-tests were used to compare the telehealth and in-person groups on continuously distributed variables and chi-square statistics were used to compare categorical variables.

Results

Patient characteristics

During the 5 months from May, 2020 to October, 2020 499 patients were admitted to the telehealth program of whom 445 (89.2%) have ended their treatment and have been discharged. Because of delays in getting approval from the institutional review board and clinician oversight in requesting consent for participating in the study, 63 (12.6%) patients were not consented. An additional 77 (15.4%) patients refused consent. This left a sample of 305 consenting patients–58 of whom were previously been treated in the program in-person, and 7 of whom were treated twice within the telehealth program. Of the 240 consenting patients for whom it was their first admission, three quarters completed their treatment in the program (75.0%, n = 180) and one-quarter dropped out during the course of treatment (25.0%, n = 60). We compared the 240 consenting patients to the 140 patients who were not consented or refused consent on demographic and diagnostic variables. There were no significant differences between the groups on these variables.

The in-person treatment comparison group was selected from 584 consented patients admitted and discharged from the program during the 5 months from May, 2019 to October, 2019. Of those patients, 416 were first admissions. To match the number of patients who attended the telehealth program for the first time during the same time frame in 2020, every other alternate patient was removed from the database, leaving an equivalent sample size of 240 patients.

The in-person and telehealth treated groups were similar in demographic characteristics (Table 1 ). There were more patients with a principal diagnosis of generalized anxiety disorder (GAD) in the in-person group, and more patients had a principal diagnosis of panic disorder with or without agoraphobia in the telehealth group. There were no other differences between the treatment groups in their principal psychiatric diagnosis (Table 2 ).

Table 1

Demographic characteristics of partial hospital patients treated in-person or in a telehealth format.

	In-Person		Telehealth		χ2	p level
	(n = 240)		(n = 240)
Gender, % (n):					1.44	.49
Male	31.4	(74)	29.2	(70)
Female	66.5	(162)	67.5	(162)
Transgender or non-binary	1.7	(4)	3.3	(8)
Race, % (n):					4.99	n.s.
White	72.4	(174)	74.2	(178)
Hispanic	14.3	(34)	13.3	(32)
Black	4.6	(11)	7.1	(17)
Asian	1.7	(4)	1.3	(3)
Other	5.9	(14)	4.2	(10)
Education, % (n):					6.23	n.s.
Less Than High School Graduate	6.3	(15)	2.6	(6)
High School Diploma, GED, or some college	57.5	(138)	58.9	(136)
4-Year College Degree	25.0	(60)	25.0	(60)
Graduate Degree	11.3	(27)	12.6	(29)
Marital Status, % (n):					7.21	n.s.
Married	31.7	(76)	27.9	(67)
Living Together	12.1	(29)	14.2	(34)
Widowed	3.3	(8)	0.4	(1)
Separated	4.2	(10)	3.8	(9)
Divorced	11.3	(27)	11.3	(27)
Never Married	37.5	(90)	42.5	(102)
Age*, M (SD):	37.88	(14.00)	36.23	(13.94)	t = 1.30	.20

n.s. indicates not significant.

*Age was compared by t-test.

Table 2

Principal diagnoses of partial hospital patients treated in-person or in a telehealth format.

	In-person		Telehealth
	(n = 240)		(n = 240)		χ2	p level
Mood Disorders, % (n):
Major Depressive Disorder	49.6	(117)	44.0	(102)	1.48	0.22
Persistent Depressive Disorder	2.5	(6)	3.0	(7)	0.10	0.76
Bipolar I Disorder	4.2	(10)	4.3	(10)	0.05	0.97
Bipolar II Disorder	3.4	(8)	2.2	(5)	0.71	0.42
Other Mood Disorder	1.7	(4)	3.0	(7)	0.89	0.35
Anxiety Disorders, % (n):
Panic Disorder	1.3	(3)	4.3	(10)	4.00	0.05
Panic Disorder with Agoraphobia	0.4	(1)	3.0	(7)	4.68	0.03
Agoraphobia without Panic	0.0	(0)	0.0	(0)
Social Anxiety Disorder	0.0	(0)	0.4	(1)	1.02	0.31
Specific Phobia	0.0	(0)	0.0	(0)
Posttraumatic Stress Disorder	10.2	(24)	7.3	(17)	1.18	0.28
Generalized Anxiety Disorder	11.9	(28)	6.5	(15)	4.09	0.04
Obsessive-Compulsive Disorder	0.0	(0)	1.3	(3)	3.07	0.08
Other Anxiety Disorder	0.4	(1)	1.7	(4)	1.87	0.17
Substance Use Disorders, % (n):
Alcohol Abuse/Dependence	0.0	(0)	0.4	(1)	1.02	0.31
Drug Abuse/Dependence	0.4	(1)	0.0	(0)	1.00	0.32
Any Eating Disorder, % (n):	0.4	(1)	0.4	(1)	0.00	1.00
Any Psychotic Disorder, % (n):	2.5.	(6)	2.2	(5)	0.08	0.78
Any Somatoform Disorder, % (n):	0.4	(1)	0.9	(2)	0.35	0.55
Any Impulse Control Disorder, % (n)	0.4	(1)	0.0	(0)	0.99	0.32
Any Adjustment Disorder, % (n):	5.1	(12)	9.1	(21)	2.81	0.09
Borderline personality disorder, % (n)	2.1	(10)	3.4	(8)	0.20	0.66
Other Diagnoses	0.0	(0)	0.4	(1)	1.02	0.31

Satisfaction with the initial evaluation

The rate of completion of the CUPSS after the initial evaluation by the psychiatrist was significantly lower in the telehealth cohort (52.7% vs. 91.3%, X 2 = 88.53, p < .001). In the telehealth cohort, we compared the demographic and diagnostic characteristics of the patients who did and did not complete the CUPSS and found that patients who completed the CUPSS were more likely to be diagnosed with GAD (11.2% vs. 3.8%, X 2= 6.16, p < .05). There were no other significant differences between the groups.

The data in Table 3 shows that the mean scores on the CUPSS items were nearly identical in the patients who were treated in the telehealth and in-person programs. At most, the mean scores on the items differed by one-tenth of a point on the 5-point scale. Approximately 90% of the patients in the virtual and in-person groups indicated that they were very or extremely satisfied with the initial evaluation (89.3% vs. 90.6%, X 2 = .98, NS). Likewise, the majority of patients in both the virtual and in-person groups were hopeful that they would get better (79.4% vs. 77.1%, X 2 = 1.82, NS).

Table 3

Clinically Useful Patient Satisfaction Scale (CUPSS) scores in patients treated in the partial hospital in-person or via telehealth.

CUPSS Item	In person (n = 219)		Telehealth (n = 131)		t value	p level
	Mean	SD	Mean	SD
The evaluation was thorough and complete.	3.6	0.7	3.5	0.7	1.46	.15
My diagnosis was explained in clear way.	3.2	1.2	3.3	1.0	-0.37	.71
My questions were answered to my satisfaction.	3.6	0.8	3.6	0.7	-0.15	.88
My treatment was discussed in a clear and understandable way.	3.4	1.1	3.4	0.8	0.01	.99
I was asked for my opinion about treatment.	3.4	1.1	3.4	0.9	0.10	.92
I was told what to do if my symptoms got worse.	3.2	1.3	3.1	1.2	0.18	.86
My doctor seemed genuinely interested in me.	3.6	0.8	3.7	0.6	-0.76	.45
My doctor seemed to understand my problems.	3.6	0.8	3.6	0.6	-0.57	.57
My doctor treated me with respect.	3.8	0.5	3.8	0.4	0.18	.86
My doctor seemed to know what he/she was doing.	3.8	0.5	3.8	0.5	0.45	.65
I felt I could trust my doctor.	3.7	0.7	3.7	0.6	-0.01	1.00
My doctor asked if I had any questions.	3.8	0.7	3.8	0.4	-0.80	.43
Overall satisfaction with initial evaluation	3.5	0.7	3.4	0.7	0.29	.77
Expectation of improvement	3.2	0.8	3.2	0.8	-0.30	.76

Satisfaction with treatment

Satisfaction with treatment was evaluated on the day of discharge; patients who dropped out of the program before their discharge date were not sent the program satisfaction scale. There was no difference in the percentage of patients who completed treatment between the telehealth and the in-person formats (73.0% vs. 71.7%, X 2 = 0.11, NS). There was no difference in the percentage of patients who discharged from the in-person program due to nonattendance (6.2% vs. 10.8%, X 2 = 3.28, NS), were transferred to inpatient care (0.8% vs. 1.3%, X 2 = 0.21, NS), or withdrew from treatment due to dissatisfaction with the program (3.7% vs. 2.1%, X 2 = 1.16, NS).

Amongst the patients who completed the program, there was no difference in the completion rate of the discharge satisfaction scale between the in-person and the telehealth groups (54.2% vs. 58.5%, X 2 = .92, NS). More than 90% of the patients in the in-person and virtual groups indicated that they were very or extremely satisfied with their treatment (92.2% vs. 93.2%, X 2 = 1.40, NS). Similarly, more than 95% of the patients treated in both formats indicated that they would recommend the treatment program to a friend or family member (96.1% vs. 95.6%, X 2 = 1.20, NS). Satisfaction was high for each element of the program (Table 4 ). The only significant difference was greater satisfaction with the interpersonal process group in the in-person group.

Table 4

Satisfaction with treatment at the end of treatment in the partial hospital delivered in-person or via telehealth.

Item	In-person (n = 130)			Telehealth (n = 140)
	Mean	SD	Mean	SD	t value	p level
ACT group	3.1	1.0	3.1	0.9	0.24	.81
Interpersonal group	3.6	0.7	3.2	1.1	3.16	.01
Skills group	3.1	1.0	3.1	0.9	-0.46	.65
Individual sessions with psychiatrist	3.6	0.7	3.7	0.6	-0.76	.45
Number of sessions with psychiatrist	3.7	0.7	3.7	0.7	0.19	.85
Individual sessions with therapist	3.7	0.8	3.7	0.6	-0.31	.76
Length of sessions with therapist	3.6	0.8	3.6	0.7	0.07	.95
Overall satisfaction with program	3.5	0.8	3.5	0.8	-0.17	.86
Recommend program to friend or family	3.8	0.6	3.7	0.7	0.74	.46
Perception of improvement	3.0	1.0	2.9	1.0	0.71	.48

Satisfaction with telehealth treatment in patients who previously were treated in-person

Twenty-five patients treated in the telehealth program had previously been treated in-person and completed the discharge satisfaction scale for both admissions. There were no significant differences in the ratings for the telehealth and in-person programs.

Discussion

In an intensive acute care setting consisting of daily group and individual therapy sessions as well psychopharmacological management, for both telehealth and in-person methods of delivering treatment, patients were highly satisfied with the initial diagnostic evaluation and were optimistic at admission that treatment would be helpful. Likewise, at the completion of treatment, both groups were highly satisfied with the treatment that they had received and would recommend treatment to a friend or family member.

While there are many studies demonstrating the efficacy of telehealth, and satisfaction with telehealth, little research has focused on patients treated in an intensive PHP that includes both individual and group therapy. Patients in PHPs are generally more severely ill than patients treated in outpatient settings. To be sure, as described in the Methods section, added precautions were taken to ensure that emergencies could be addressed in the telehealth treated patients. Because a PHP is essentially an outpatient treatment, albeit more intensive in terms of the frequency of visits (5 days per week) and the duration of each visit (6 hours per day), it is routine to assess risk and conduct safety planning interventions. We did not refuse admission of suicidal patients to our PHP, whether conducted virtually or in-person, unless a high level of intent was judged to be present whereupon the patient was referred for inpatient care. In fact, a small percentage of patients in both treatment formats were referred for inpatient admission though there was no significant difference between the formats in this regard.

While levels of satisfaction were high overall in both treatment formats, there was one difference—patients were more satisfied with the interpersonal group when it was conducted in person. This group is more of a process-oriented group than a didactic group. To be sure, the majority of patients were nonetheless very or extremely satisfied with the interpersonal group when conducted virtually; however, the level of satisfaction was significantly lower than in-person delivery (81.3% vs. 92.9%, X 2 = 9.38, p < .05). While satisfaction was lower with this group, this did not impact the overall level of satisfaction with the telehealth program. Nonetheless, this finding raises the question of whether process-oriented outpatient group therapy will be as well received in telehealth settings as in person.

The present study was not designed to compare the level of satisfaction of in-person and telehealth treatment formats. We adopted the virtual platform because of the COVID-19 pandemic. Thus, we did not randomize patients to the treatment formats. Rather we examined the satisfaction with treatment in sequentially recruited cohorts and only controlled for the months of year the patients were admitted to the PHP. Fortunately, there were no differences between the cohorts in their demographic characteristics and few differences in psychiatric diagnosis. While a randomized, controlled trial would be desirable, such a study in a partial hospital setting is unlikely to ever be conducted because of the need to double the clinical staff needed to run parallel PHPs at the same time.

It was easier to collect data when the patients were treated in-person. When our program was conducted in-person, almost all patients completed the satisfaction survey at admission. In the virtual program the response rate was approximately 50%. Direct person-to-person contact, in which the forms are handed to the patients by their treating clinicians, likely enhanced completion rates when compared to sending patients electronic links to surveys to be completed online at home. However, there were no differences in the demographic and clinical characteristics of the patients who did and did not complete the various measures.

Finally, a few words about the future. We anticipate that we will deliver telehealth PHP level of care for the foreseeable future during the COVID-19 pandemic to enhance the safety of our patients and staff. Moreover, several patients whom we have treated using a telehealth format have commented that they never would have presented for in-person treatment even if there was no pandemic. Some of these patients had medical illnesses that made in-person treatment attendance more difficult to manage. For other patients, transportation limitations made in-person treatment more difficult. Thus, telehealth partial hospital treatment may continue either as a stand-alone program or perhaps included in a combined in-person and virtual program. Of course, decisions about how care is delivered in the future will be significantly influenced by insurance company reimbursement. Hopefully, regulations will be passed to ensure that telehealth treatment is reimbursed at the same level as in-person treatment. If not, this will likely undermine efforts to make telehealth treatment more widely available to patients.

CRediT authorship contribution statement

Mark Zimmerman: Conceptualization, Methodology, Writing - original draft, Supervision, Project administration, Funding acquisition. Isabel Benjamin: Investigation, Data curation, Writing - review & editing. Julianne Wilner Tirpak: Investigation, Data curation, Writing - review & editing. Catherine D'Avanzato: Investigation, Data curation, Writing - review & editing.

Declaration of Competing Interest

None.