Psychotherapists' perception of their clinical skills and in-session feelings in live therapy versus online therapy during the COVID-19 pandemic: a pilot study.

During the coronavirus disease 2019 (COVID-19) pandemic, many psychotherapists who were used to seeing their patients in face-toface setting adapted to providing therapies online. In the present pilot study, we investigated therapist current experiences of online therapy compared to live therapy. Twenty-nine therapists completed Clinical Skills, Difficulties in Practice, and in-sessions feelings of Flow, Boredom and Anxiety of the Trainee Current Progress Report, giving a score for each item in two different conditions: Live Therapy and Online Therapy. Compared to Live Therapy, in Online Therapy therapists reported significantly less Clinical Skills, whereas Difficulties in Practice did not differ in the considered conditions. With regard to in-sessions feeling, therapists reported significantly lower scores of Boring and higher scores of Flow in Live Therapy compared to Online Therapy, whereas Anxiety did not differed in the considered conditions. ©Copyright: the Author(s).

Introduction

Online therapy can be defined as ‘any type of professional therapeutic interaction that makes use of the Internet to connect qualified mental health professionals and their clients’ (Rochlen et al., 2004). The efficacy of online therapy is supported by clinical trials in which it has been compared to waiting-list control groups (Klein & Richards, 2001; Lange et al., 2000; Lange, van de Ven, Schrieken, & Emmelkamp, 2001) and to traditional faceto- face therapies (Cohen & Kerr, 1999; Day & Schneider; 2002), and it has been documented for a variety of clinical conditions including post-traumatic stress disorder (Poletti et al., 2020; Backhaus et al., 2012), anxiety disorders (Poletti et al., 2020), depression (de Bitencourt Machado et al., 2016), obsessive-compulsive disorder (Simpson, 2009), panic attacks and agoraphobia (Poletti et al., 2020), bulimia nervosa (Simpson, 2009, de Bitencourt Machado et al., 2016) and mixed diagnoses (Backhaus et al., 2012). Moreover, a number of studies have investigated various aspects of the therapeutic relationship in online therapy, such as therapeutic alliance (Cook & Doyle, 2002; Reynolds et al., 2013) and trust (Fletcher-Tomenius & Vossler, 2009), and have provided qualitative description of online therapy processes (Cipolletta et al., 2018; William et al., 2009). This early evidence accounts for similar features of the therapeutic relationship in online therapy and face-to-face therapy (Sucala et al., 2012; Cook & Doyle, 2002). In the recent past of these studies, in Italy online therapy was confined to very specific situations, such as individuals living in isolated areas, ‘internet- friendly’ patients and therapists or therapies realized as part of clinical trials specifically proposed to test online therapy efficacy. During the coronavirus disease 2019 (COVID-19) pandemic lockdowns, instead, the use of online therapy has become usual in the Italian context. Most psychotherapists, who were used to seeing their patients in face-to-face setting, adapted to provide therapies online, via videoconferencing or via telephone. Thus, the COVID-19 pandemic is offering the opportunity to investigate features of online therapy in large scale.

Among different interesting aspect of online therapy experience, we have notified that, despite available evidence accounts for a general efficacy of online therapies, most therapists continue to prefer traditional live therapy (van der Vaart et al., 2014) or they have relevant reservations about online therapy (Cipolletta & Mocellin, 2018). These preferences suggest the existence of differences in their experience of online compared to live therapy. However, to the best of our knowledge, no study has investigated features of online therapy experience from the therapist point of view. This became the aim of the present pilot study.

We started from previous investigations of psychotherapists’ point of view and experiences in doing therapy that have been object of attention by the Interest Section on Therapist Training and Development of the Society for Psychotherapy Research (SPRISTAD) (Orlinsky et al., 2015; Messina et al., 2019) and its ancestor of the SPR Collaborative Research Network (CRN) (Orlinsky et al., 1999; Orlinsky et al., 2001). In this context, core aspects of the therapists’ experiences from different countries, professions, and theoretical orientations were identified (Orlinsky et al., 2005). Among others contributions, the SPR/CRN has collected data concerning the quality of therapist involvement in doing psychotherapy, such as their perception of clinical skills and difficulties in doing therapy, as well as in-session feelings (Orlinsky, Ronnestad & Willutzki, 2010; Orlinsky et al., 1991; Orlinsky et al., 1999a), observing their variation by therapist and across time (Orlinsky et al., 1999; Orlinsky & Ronnestad, 2005). Moreover, variation of these dimensions have been confirmed to be important features of psychotherapists development during their training (Messina et al., 2017).

The aim of the present study is to extend these previous investigations considering how therapist experience may vary in association to live (traditional) versus online settings. Following SPRISTAD tradition, we focused in three main dimensions of healing involvement and stressful involvement in therapist experience (Orlinsky & Ronnestad, 2005). First, we considered therapist perception of their abilities (healing involvement) and difficulties in doing therapy (stressful involvement) in online compared to live psychotherapy sessions. Second, we were interested in therapists’ in-sessions feelings. We took into consideration the experience of flow as a component of healing involvement, defined as a deep interest and engagement felt during therapy sessions, and considered an important component of healing involvement in doing therapy, and feelings of anxiety and boredom, considered components of stressful involvement in doing therapy (Orlinsky & Ronnestad, 2005). In our hypothesis, differences in therapists experiences of healing involvement (perceived abilities and in-session feelings of flow) and stressful involvement (perceived difficulties and in-session negative feelings) may differ in live therapy compared to online therapy.

Methods

Therapists

A convenience sample of 29 volunteer therapists with different theoretical orientations and with experiences in doing therapy in online settings were recruited through online social networks. The general features of therapists sample are summarized in Table 1. As shown in Table 1, therapists’ experience in online sessions became more frequent during the COVID-19 lockdown and the use of online sessions was partially maintained post lockdown period.

Table 1.

General information about therapists sample.

Age	M=39.14; SD=9.76
Gender	Female 68.96% (n=20) Male 31.03% (n=9)
Theoretical orientation
Psychodynamic	34.48% (n=10)
Cognitive-behavioral	27.59% (n=8)
Systemic-familiar	20.69% (n=6)
Humanistic-existential	10.34% (n=3)
Other	6.89% (n=2)
Years of experience as therapist	M=12.69; SD=10.79
Current sessions per week	M=12.93; SD=10.11
Current live sessions per week	M=7.86; SD=8.74
Online sessions per week
Pre-COVID-19	M=1.41; SD=0.56
Lockdown	M=4.37; SD=1.19
Current (post-lockdown)	M=3.00; SD=1.35

M, mean; SD, standard deviation.

Instrument: Trainee Current Progress Report

Parts of the Trainee Current Progress Report (TCPR) were used to collect the data of the present study. The TCPR is the core longitudinal instrument of the SPRISTAD project (Orlinsky et al., 2015) and it has been shown to be suitable to measure changes in therapists perceptions of their experiences, with sufficient sensitivity in measuring change in therapists over the course of the time (Messina et al., 2017). Among the various areas relevant to therapists’ description and evaluation of their therapeutic practice, for the aims of the present study we considered the subscales Clinical Skills, Difficulties in Practice, in-sessions feelings of Flow, Anxiety and Boredom. A previous study has showed good to acceptable internal reliability in the TCPR subscales Clinical Skills (α=0.97), Difficulties in Practice (α=0.82), Anxiety (α=0.73), Boredom (α=0.73), Flow (α=0.72) (Messina et al., 2017).

Clinical skills

The multiple item subscale for Clinical Skills includes 12 items (rated on an 11-point scale from 0=‘not at all’ to 10=‘very much’) that ask respondents how they perceive themselves on various aspects of technical skill (e.g., ‘How much mastery do you have of the techniques and strategies involved in practicing therapy?’) and relational skill (e.g., ‘How effective are you at engaging patients/clients in a working alliance?’). In the present study, the internal reliability of the subscales Clinical Skills was excellent (α=0.96).

Difficulties in practice

The subscale Difficulties in Practice included 12 items (rated on an 11-point scale from 0=‘never’ to 10=‘very often’) relating to how often the therapist feel self-doubt regarding one’s own ability as a therapist (e.g., ‘Lacking in confidence that you can have a beneficial effect on a client’), being unable to move therapy with a client in a constructive direction (e.g., ‘Bogged down with a client in a relationship that seems to go nowhere’). In the present study, the internal reliability of the subscales Difficulties in practice was excellent (α=0.92).

In-session feelings (flow, anxiety and boredom)

In-session feelings were evaluated asking therapists the rating of a list of possible in-session feeling on a 6- point scale (from 0=‘never’ to 5=‘very often’). The subscale Flow included the items inspired, stimulated, engrossed, challenged; the subscales Anxiety included the items anxious, overwhelmed, pressured, trapped; and the sub-scale Boredom included the items absent, bored, drowsy. In the present study, the internal reliability was good for subscales Boredom (α=0.85) and Anxiety (α=0.78), whereas it was problematic in the case of subscale Flow (α=0.49)

Procedures

Data were collected in the post-lockdown period. After giving their informed consent, therapists completed an online form with the selected parts of the TCPR. In the same moment, therapists evaluated their current experiences in online therapy and live therapy. They reported their ratings of perceived Clinical Skills, Difficulties in Practice, and in-session feelings of Flow, Anxiety and Boredom, referred to two different conditions: Live Therapy and Online Therapy. Thus, we presented each subscale 2 times, one time referred to current experience in online therapy (for example ‘Currently, doing online therapy…’) and one time referred to current experience in live therapy (for example ‘Currently, doing live therapy…’).

Statistical analyses

Due to absence of normality in data distribution, we used permutation t-test for statistical analyses. In permutation method P-value are obtained from the sample-specific permutation distribution of that statistic, rather than from the theoretical distribution as in common parametric methods (Pesarin & Salmaso, 2010; LaFleur & Greevy, 2009). We compared mean scores of all dependent variables (Clinical Skills, Difficulties in Practice, in-session feelings of Flow, Boredom and Anxiety) in Live Therapy and Online Therapy conditions in the same group of therapists, with variable therapist considered as random factor. In these analyses, we also included age, gender, theoretical orientation, experience and experience in online therapy as covariates. Moreover, only for explorative purposes, we also evaluated difference between Live Therapy and Online Therapy conditions in single items of Clinical Skills and Difficulties in Practice subscales.

Results

Therapists perceive differently their performance in doing therapy in live and online settings Significantly higher scores of Clinical Skills were observed in Live Therapy compared to Online Therapy (t=3.34, P<0.001). Moreover, this difference was observed in most single items of the subscale (see Table 2 for single items results). Instead, no significant results were reported in the subscale Difficulties in Practice (t=0.69, P<0.519). Exploring results related to single-items, we observed a statistically significant result for the item ‘Unable to generate sufficient momentum to move therapy with a client in a constructive direction’ (t= –2.04, P=0.035) and a statistical trend for the item ‘Unable to have much real empathy for a client’s experiences’ (t= –1.83, P=0.078). In both cases, scores were lower for Live Therapy compared to Online Therapy (more difficulties in Online Therapy).

Table 2.

Perceived clinical skills in live therapy and online therapy.

	Live therapy	Online therapy	Live therapy versus online therapy
	M (SD)	M (SD)	t	P
Clinical skills	7.87 (0.72)	6.67 (1.84)	3.34	<0.001**
How effective are you at engaging patients/clients in a working alliance?	8.20 (0.94)	6.86 (2.15)	3.40	<0.001**
How ‘natural’ (authentically personal) do you feel?	8.62 (1.08)	7.08 (2.12)	3.98	<0.001**
How much mastery do you have of the techniques and strategies involved in practicing therapy?	7.83 (1.14)	6.72 (1.98)	3.42	0.001**
How well do you understand the moment-by-moment process in sessions?	7.34 (1.20)	6.59 (2.08)	2.14	0.045*
How effectively do you communicate your understanding and concern to clients?	8.66 (0.97)	7.21 (2.06)	4.89	<0.001**
How well are you able to detect and deal with the emotional reactions that your patients/clients have towards you?	8.31 (1.14)	6.76 (2.01)	4.16	<0.001**
How good are you at making constructive use of your own personal reactions to patients/clients?	8.07 (1.22)	6.79 (1.84)	4.07	<0.001**
How much precision, subtlety and finesse have you attained in your therapeutic work?	7.45 (1.30)	6.62 (2.18)	2.48	0.018*
How effective are you in helping your patients/clients?	7.62 (0.56)	7.07 (1.91)	1.74	0.108
Do you find yourself thinking of clients at times between sessions?	6.62 (1.76)	5.89 (2.30)	1.63	0.129

M, mean; SD, standard deviation. **P<0.001; *P<0.01.

With regards to in-session feelings, significant differences have been reported in the case of Flow, with more experience of flow-related feeling in Live Therapy compared to Online Therapy (t=2.83, P=0.012), and Boredom, with less experience of boredom-related feelings in Live Therapy compared to Online Therapy (t= –2.76, P=0.011), whereas no significant differences were found in the case of Anxiety (see Tables 3 and 4 for in-session feelings results).

Table 3.

Perceived difficulties in practice of live therapy and online therapy.

	Live therapy	Online therapy	Live therapy versus online therapy
	M (SD)	M (SD)	t	M (SD)
Difficulties in practice	2.76 (0.88)	2.68 (1.02)	0.69	0.519
Lacking in confidence that you can have a beneficial effect on a client	3.10 (1.72)	3.24 (2.32)	–0.33	0.755
Unsure how best to deal effectively with a client	3.24 (1.98)	2.83 (2.05)	1.03	0.311
In danger of losing control of the therapeutic situation to a client	1.21 (1.52)	1.41 (1.82)	–0.67	0.504
Unable to have much real empathy for a client’s experiences	1.28 (1.58)	1.83 (2.09)	–1.83	0.078°
Uneasy that your personal values make it difficult to maintain an appropriate attitude towards a client	2.03 (1.55)	1.52 (1.30)	1.91	0.065
Unable to generate sufficient momentum to move therapy with a client in a constructive direction	1.93 (1.33)	2.69 (2.16)	–2.04	0.035*
Demoralized by your inability to find ways to help a client	2.31 (1.54)	2.59 (1.92)	–0.89	0.407
Guilty about having mishandled a critical situation with a client	2.62 (2.34)	2.03 (1.88)	1.73	0.105
Unable to comprehend the essence of a client’s problems	1.83 (1.49)	1.90 (1.68)	–0.24	0.805
Unable to withstand a client’s emotional neediness	2.28 (1.85)	2.31 (2.30)	–0.17	0.896
Unable to find something to like or respect in a patient	1.07 (1.58)	0.93 (1.33)	0.72	0.482
Bogged down with a client in a relationship that seems to go nowhere	2.48 (1.70)	2.72 (2.25) only	–0.99	0.319

M, mean; SD, standard deviation. °P<0.09; *P<0.01.

We did not find significant effects of age, gender, theoretical orientation, experience and experience in online therapy.

Discussion and Conclusions

In the present pilot study, we investigated psychotherapists’ perception of online therapy compared to live therapy (or traditional therapy), with a focus on variables that in the context of psychotherapists’ development literature have been considered relevant dimension of therapist experience: perception of clinical skills, perception of difficulties in psychotherapy practice, and in-session positive and negative feelings.

According to our results, the quality of therapists’ experience differed between traditional therapy and online therapy. This differences clearly emerges in therapists’ perception of reduced clinical skills in online therapy compared to live therapy. For example, therapists reported different perception of effectiveness in engaging the patients in therapeutic alliance, in their feeling of being ‘natural’, in communication and in the management of transfer and controtransfert issues and others. Instead, in case of therapist perceptions of difficulties in psychotherapy practice differences did not reach statistical significance. Taken together, these results suggest that even if therapists did not perceived clear difficulties or errors in online sessions, at the same time they have a general impression of not practice therapy as the best they can.

May this differences in therapists’ perceived performance influence psychotherapy effectiveness? On the basis of our data, it is not possible to draw conclusion regarding how much the observed difference in therapists’ perception of their therapeutic skills may influence psychotherapy outcome. From psychotherapy effectiveness research, however, it is clear that psychotherapy outcome is importantly predicted by the person of the therapist (from 5% in controlled studies and to 17% in health care studies) (Baldwin & Imel, 2013). The effect sizes for therapist effects are 0.13-0.74, the relapse rates of severe psychiatric illnesses are between 25 and 100% depending on the therapist and the frequency of therapy discontinuations is related to the therapist variable (Baldwin & Imel, 2013). According to this evidence, therapists’ performance may have an influence.

In the present study, we also observed that the quality of psychotherapists’ experience in online and live therapy differed reported in-session feelings, with decreased flow and increased boredom in online therapy. In this case, the focus was on therapists’ feelings, which may appear less directly connected to psychotherapy outcomes. Nevertheless, previous studies have clearly related therapists’ insession feelings with psychotherapy outcomes, with faster symptom reduction in presence of high flow and low boredom (Heinonen et al., 2014; Heinonen et al., 2012). Thus, we can hypothesize that also differences between in-session feelings in online and live therapy may influence therapy outcomes producing differences in term of effectiveness (note, however, that the reliability of Flow subscale was not satisfactory in the present study).

In sum, the results of the present study, together with therapists’ variable studies, bring into question the effectiveness of online therapy compared to more traditional therapies. On the other hand, early comparisons of traditional face-to-face and online therapies did not report differences in terms of efficacy (Cohen & Kerr, 1999; Day & Schneider; 2002). This inconsistence can be explained with several hypothesis. A hypothesis is that therapists’ perception may reflect a real decline in therapists performance in online therapy, but this decline is not significant enough to influence psychotherapy outcome (at least as measured in clinical trials). An alternative hypothesis is that therapists’ performance decline has significant effects on psychotherapy outcomes but the experimental design of comparisons studies is not suitable for the investigation such effects. For example, existing comparison studies have not considered the number of drop-outs as outcome variable, and therapists’ skills seem to have a relevant impact on dropout rates (Roors and Werbart, 2013). Second, clinical studies involved specific groups of therapists, such as therapists with specific trainings in online therapy or scientific interest for online therapies, elements that may influence their experience of online therapy. With this regard, it is important to mention that a specific training in online therapy may help therapists’ in cope against the difficulties that online setting implies, with a possible reduction of differences with traditional therapy.

The issues touched in the present article may have relevant implications for the understanding of online therapy, enlarging the scientific debate from efficacy as reported in clinical trials to clinical effectiveness evaluated extending the investigation to other important factors, including the therapists’ point of view and experiences. However, important limitation of the present study should be considered. First, the generalizability of our findings remains to be tested because of the small convenience sample of therapists. Second, we did not consider patients-related variables, such as their diagnosis, psychological impairment severity or preferences for online versus live therapy. Third, we did not control if therapists’ perceptions regarded therapies that switched from live to online settings or therapies that have maintained the same modality throughout the time. Thus, the continuation of data collection is required for a stronger empirical anchoring of the hypotheses emerged in the present pilot study.

Table 4.

In-session feelings in live therapy and online therapy.

	Live therapy	Online therapy	Live therapy versus online therapy
	M (SD)	M (SD)	t	M (SD)
Flow	6.12 (1.14)	5.44 (1.63)	2.83	0.012*
Boredom	1.59 (1.24)	2.38 (1.96)	–2.76	0.011*
Anxiety	1.59 (0.98)	1.92 (1.79)	–1.33	0.198

M, mean; SD, standard deviation. *P<0.01.