Development of the Internet-Delivered Cognitive Behaviour Therapy Undesirable Therapist Behaviours Scale (ICBT-UTBS).

Internet-delivered cognitive behaviour therapy (ICBT) is often provided with therapist assistance via asynchronous secure emails, but there is limited research on undesirable behaviours exhibited by therapists in their correspondence with patients. In this study, an ICBT-Undesirable Therapist Behaviour Scale (ICBT-UTBS) was developed and used to assess the nature, frequency, and correlates of undesirable therapist behaviours in routine practice. Thematic analysis was used to identify undesirable therapist behaviours in 720 emails sent to 91 randomly selected patients in the context of a previous clinical trial of transdiagnostic ICBT for depression and anxiety. The following undesirable behaviours were identified, albeit infrequently, in therapist emails: inadequate detail (6.4%), unaddressed content (4.0%), unsupportive tone (0.6%), missed correspondence (0.6%), inappropriate self-disclosure (0.6%), and unmanaged risk (0.3%). At least one undesirable behaviour was found in 10.7% of all emails coded. Moreover, 37.4% of patients received at least one email containing an undesirable therapist behaviour. Number of undesirable therapist behaviours was not correlated with patient engagement, working alliance, treatment satisfaction, or patient outcome variables. However, undesirable therapist behaviours were negatively correlated with patient gender and therapist characteristics (e.g., clinical setting, therapist profession). The results of the present study provide preliminary psychometric support for the ICBT-UTBS, a measure of ICBT treatment integrity. In the future, the ICBT-UTBS should be used in combination with the ICBT-Therapist Rating Scale (ICBT-TRS), a measure of desirable or recommended therapist behaviours, for training purposes and to monitor ICBT therapists in routine practice.

Introduction

There is growing interest in the use of therapist-assisted Internet-delivered cognitive behaviour therapy (ICBT) in routine practice in order to improve patient access to treatment (Titov et al., 2018). This form of treatment typically involves patients reviewing online treatment materials weekly over the course of several months. As patients progress through treatment, they are provided with therapist assistance, most commonly using asynchronous secure emails (Andersson, 2016). The body of research on this form of treatment is expanding rapidly, with one recent review reporting over 300 controlled trials have assessed ICBT for various conditions (Andersson et al., 2019). Reviews largely conclude that ICBT shows comparable effects to face-to-face cognitive behaviour therapy (Andersson et al., 2014).

With a rising number of routine care clinics offering ICBT comes the need to develop tools that can assist with efficiently and effectively monitoring ICBT implementation in clinical practice (Andersson et al., 2019; Proctor et al., 2011; Titov et al., 2018). One implementation outcome that should be addressed is treatment integrity (also known as treatment fidelity) – the extent to which an intervention is implemented as intended (Perepletchikova et al., 2007; Proctor et al., 2011). At the provider level, an essential component of treatment integrity is the quality of program delivery, which can be assessed by monitoring adherence to recommended therapist behaviours (Hermes et al., 2019; Plumb and Vilardaga, 2010; Proctor et al., 2011). In ICBT, this would involve examining the extent to which therapists demonstrate both desirable and undesirable behaviours in their email communication with patients.

In an effort to learn more about quality of program delivery from a provider perspective, Hadjistavropoulos et al. (2018) developed and evaluated the ICBT-Therapist Rating Scale (ICBT-TRS) to measure the presence and quality of various therapist behaviours that are recommended in ICBT. The scale is based on nine recommended therapist behaviours: builds rapport, seeks feedback, provides symptom feedback, provides psychoeducation, facilitates understanding, praises effort, encourages practice, clarifies administrative procedures, and communicates effectively. Three psychology graduate students rated therapist emails on a binary scale for adherence (absent/present) and quality (inadequate/competent). Therapists were from Saskatchewan, Canada and were registered professionals or graduate students under supervision in psychology, social work, and nursing. Promisingly, among 39 therapists, most recommended behaviours were identified as being present in emails and were rated as adequate quality, providing evidence of therapist fidelity. A preliminary examination of the ICBT-TRS revealed that higher quality ratings were associated with higher patient engagement, therapists who specialized in ICBT, and therapists who had a psychology background. ICBT-TRS ratings were not related to patient treatment satisfaction or symptom reductions.

Similarly, Mol et al. (2018) created a checklist to assess Dutch therapists' adherence to online feedback instructions in the context of ICBT blended with face-to-face therapy for depression. The checklist consisted of six categories and 19 subcategories that were assessed on a dichotomous scale (present/not present): 1) greeting/ending (e.g., include an appropriate greeting and ending); 2) communication skills (e.g., begin the message with a compliment, formulate sentences as hypotheses); 3) structure (e.g., give feedback on maximum two subjects, respond within 3 working days); 4) referring (e.g., refer to diary or next session); 5) readability (e.g., short, clear sentences and paragraphs); and 6) writing style (e.g., limit abbreviations and misspellings, correct emoticons). The professional backgrounds of the 19 therapists were not described by the authors, however it was noted that they all completed a 4-hour group training workshop and received monthly group supervision. The authors reported good therapist adherence to the majority of feedback instructions (greeting/ending = 95.9%; writing style = 93.6%; structure = 87.7%; communication skills = 69.4%; readability = 68.0%). ‘Referring’ was the only category with poor adherence (34.0%). Structure and readability ratings were negatively correlated with session completion. No associations were reported between depressive symptom change and adherence to feedback instructions.

A notable feature of existing ICBT fidelity measures is their exclusive focus on the presence of recommended or desirable therapist behaviours. This has also been the case with previous studies that have attempted to classify therapist communication in ICBT (Holländare et al., 2016; Paxling et al., 2013; Sanchez-Ortiz et al., 2011; Schneider et al., 2016). It seems likely that alongside recommended behaviours, therapists in ICBT have the potential to exhibit undesirable behaviours that could negatively affect treatment quality, patient satisfaction, engagement, and outcomes. To our knowledge, no existing research has assessed for the presence of undesirable behaviours in therapist emails sent to patients during ICBT. The face-to-face psychotherapy literature has explored several ineffective or harmful therapist behaviours, including confrontation, self-disclosure, rigidity, cultural arrogance, ignoring alliance ruptures, and unaddressed countertransference (Norcross and Lambert, 2018). However, it is unclear if those behaviours are also relevant to ICBT.

The purpose of the current study was to extend the literature on therapist behaviours in ICBT by developing the ICBT-Undesirable Therapist Behaviour Scale (ICBT-UTBS). This was accomplished by reviewing therapist emails with the goal of identifying behaviours that could potentially threaten ICBT treatment integrity. Subsequent to development of the ICBT-UTBS, we then examined: 1) the frequency of undesirable behaviours in therapist emails; and 2) correlates of undesirable therapist behaviours, including relationship of the ICBT-UTBS domains with patient variables (e.g., treatment engagement, satisfaction, and outcomes) and therapist variables (e.g., therapist professional background, clinical setting). We also explored the relationship between the ICBT-UTBS and recommended therapist behaviours assessed by the ICBT-TRS. While we expected undesirable therapist behaviours might be present in therapist emails and we had some prior knowledge of undesirable therapist behaviours in face-to-face psychotherapy (e.g., confrontation, rigidity, unaddressed countertransference), we made no formal hypotheses about the frequency or nature of the undesirable behaviours given the lack of previous ICBT research on this topic.

Methods

Patients

The current study made use of the same data as a previous study in which the ICBT-TRS was developed and evaluated (Hadjistavropoulos et al., 2018). The sample consisted of 91 patients (75 of whom completed post-treatment measures) randomly selected from an open trial of transdiagnostic therapist-assisted ICBT for depression or anxiety (ISRCTN42729166; Hadjistavropoulos et al., 2016). The original trial was approved by the institutional research ethics boards involved.

All patients completed an online screening and subsequent telephone interview to assess inclusion/exclusion criteria including: 1) being at least 18 years of age and residing in Saskatchewan; 2) having access to a computer with Internet connection; 3) consenting to participate and for their physician to be notified about their participation in ICBT; 4) exhibiting symptoms of depression or anxiety; 5) the absence of severe or unmanaged mental health symptoms (e.g., suicidality, schizophrenia); and 6) not receiving regular psychotherapy at the time of enrollment. More information about the inclusion and exclusion criteria can be found in Hadjistavropoulos et al. (2016). The average age of patients in the current study was 39 (SD = 13.42; range = 19–74), 65% were female, 83% were Caucasian, 67% were employed, and 47% lived in a city of over 200,000 people. On average, patients' pre-treatment depression score on the Patient Health Questionnaire 9 (PHQ-9; Kroenke et al., 2001) was 11.79 (SD = 5.08), with 67% reporting clinically significant depression. On the Generalized Anxiety Disorder 7 (GAD-7; Spitzer et al., 2006), patients' average score was 11.31 (SD = 4.97), with 63% reporting clinically significant anxiety. Twenty one percent of patients reported subclinical depression or anxiety scores.

ICBT course

Patients participated in an 8-week transdiagnostic ICBT program for anxiety and depression that consists of 5 lessons: 1) psychoeducation; 2) cognitive restructuring; 3) managing physical symptoms; 4) graduated exposure; and 5) relapse prevention (Dear et al., 2015; Titov et al., 2015). Each lesson consists of text and images, patient stories, lesson summaries, and homework assignments. Patients in this study logged in to the program an average of 23.20 times over the course of 8 weeks (SD = 12.03; range 3–60).

Therapist assistance

Patients were assigned to the first available therapist who worked in a specialized ICBT clinic (n = 14 therapists treating 45 patients) or a community clinic (n = 25 therapists treating 46 patients). Therapists were registered mental health professionals (n = 26) or graduate students under supervision (n = 13) with professional backgrounds in psychology (9 registered; 10 students), social work (16 registered; 3 students), or social work/nursing (1 registered). In total, 63 patients were treated by a therapist with a background in psychology and 28 patients were treated by a therapist with a background in social work or social work/nursing. The majority of therapists were female (92.3%). All therapists completed a one-day training workshop prior to providing ICBT (Hadjistavropoulos et al., 2012) followed by a period of at least two months of supervision. In addition, registered therapists were able to consult with a supervisor as needed and student therapists engaged in ongoing supervision.

Patients were encouraged to email their therapist as many times as needed each week. Therapists were instructed to spend 15 to 20 min emailing their patients once per week on a predetermined day, regardless of whether or not they received an email from the patient. Therapists reviewed patient emails, weekly symptom scores, and progress through the online content before composing their email each week. In their emails, therapists were instructed to: 1) be unconditionally supportive and warm; 2) ask about treatment progress; 3) provide symptom feedback; 4) respond to questions; 5) assist with skill use; 6) reinforce progress and skill practice; 7) encourage lesson completion and skill use; and 8) clarify administrative procedures (Hadjistavropoulos et al., 2012). Therapists were discouraged from being critical of the patient, writing emails that were too lengthy or extremely short, using jargon/slang, or engaging in inappropriate self-disclosure. In the current study, therapists sent an average of 7.76 emails to each patient (SD = 1.62; range 1–10) and received 4.46 emails in return (SD = 3.03; range 0–16). On average, patients waited 3.72 days (SD = 3.12; range 0–19) for an email response from their therapist.

In addition to therapists' email communication, patients were sent brief automated emails and occasionally received phone calls from their therapist. The brief automated emails described the availability and content of lessons and reminded patients to complete questionnaires. Therapists phoned patients if they had not logged in to the course in at least 7 days or reported a significant increase in symptoms during the week (e.g., anxiety, depression, suicidality). Therapists were trained to add a contact note to a patient's file whenever telephone contact was initiated, including the data and time of attempted contact, the general topic of discussion if the patient was reached, and any follow-up action that was required.

Measures

In addition to providing demographic information during the online screening, patients completed online symptom measures at pre-treatment, the beginning of each lesson, post-treatment, and three-month follow-up. Depressive symptoms were assessed using the PHQ-9 (range 0–27; ≥ 10 = clinical depression; Kroenke et al., 2001) and anxiety symptoms were measured using the GAD-7 (range 0–21; ≥10 = clinical anxiety; Spitzer et al., 2006). Both measures have strong psychometric properties (Kroenke et al., 2001; Spitzer et al., 2006) and demonstrated strong internal consistency in the current sample (Cronbach's α = 0.86 and 0.88 respectively). At post-treatment, patients completed the Working Alliance Inventory-Short Revised (WAI-SR; α = 0.90; Tracey and Kokotovic, 1989) and responded to questions about treatment satisfaction (e.g., would you recommend the treatment to a friend (yes/no)? Was the course worthwhile (yes/no)?). All therapist emails were also rated using the ICBT-TRS, which demonstrated high inter-rater reliability (ICC = 0.91) and internal consistency in the current sample (Cronbach's α = 0.91; Hadjistavropoulos et al., 2018).

Analyses

Undesirable therapist behaviour scale development

Therapist emails were the primary source of data for the present study. To facilitate analysis, emails sent within the same week were coded as one email and brief administrative emails (e.g., informing the patient a therapist was unable to send their usual email due to illness) were not coded. In total, 720 emails were examined. In addition, the researchers reviewed: 1) the time stamps on each email – to determine whether therapists were sending emails to their patients on at least a weekly basis; 2) patient emails – to determine whether therapists were responding to patient comments/questions; and 3) telephone contact notes – to determine if telephone contact was made in lieu of an email.

Thematic analysis was used to inductively identify potential undesirable behaviours without the use of a pre-existing coding guide (Braun and Clarke, 2006). An inductive approach was chosen because there has been no previous research on undesirable therapist behaviours in ICBT. Three research assistants with varying educational backgrounds (i.e., master's degree in clinical psychology, master's degree in counseling psychology, completing undergraduate social work degree) were responsible for the first round of data analysis. All three coders attended an ICBT training workshop (Hadjistavropoulos et al., 2012), had experience delivering ICBT under supervision, and had knowledge of the recommended therapist behaviours described by Hadjistavropoulos et al. (2018).

Data analysis was an iterative process that began with the three coders reading therapist e-mails to familiarize themselves with the data and establish initial codes. Behaviours were not coded as undesirable if they were considered to be the absence of a recommended therapist behaviour, because such behaviours are captured by the ICBT-TRS (Hadjistavropoulos et al., 2018). For example, if the therapist failed to build rapport, provide psychoeducation, or encourage practice it was not counted as an undesirable behaviour, but rather the absence of a recommended behaviour. All undesirable behaviours that were not captured by the ICBT-TRS were coded, no matter how infrequently they occurred. Next, the coders came together to discuss the initial codes and create the coding guide. Related or overlapping codes were combined into themes to represent each undesirable therapist behaviour. For instance, separate codes were originally created for failing to respond to a patient question and failing to respond to a comment made by the patient, but those codes were considered overlapping and were thus combined into one theme. The coders then used the coding guide to comprehensively analyze all the therapist emails. Each undesirable therapist behaviour was rated as 0 (absent) or 1 (present). Finally, to ensure the accuracy of the coding, a fourth coder (who was completing a doctoral degree in clinical psychology) reviewed all coded material to ensure it fit with the definitions provided in the coding guide. In the rare event of a coding discrepancy between the first and second set of coders, the first author, who was not involved with the initial coding, was responsible for making the final coding decision using the coding guide as well as her knowledge of and experience with ICBT.

This coding process resulted in the identification of six undesirable therapist behaviours that that form the basis of the ICBT-UTBS: 1) inadequate detail; 2) unaddressed content; 3) unsupportive tone; 4) missed correspondence; 5) inappropriate self-disclosure; and 6) unmanaged risk. Table 1 provides descriptions and examples of the undesirable therapist behaviours. Appendix 1 includes a series of questions that helped researchers assess undesirable therapist behaviours. For every patient, the following equation was used to calculate domain scores for each of the six undesirable therapist behaviours:

Table 1

Undesirable therapist behaviour descriptions, examples, and frequencies.

Behaviour	Description	Example	Emailsn/720 (%)	Patientsn/91 (%)
Inadequate detail	Email is extremely short and is missing a large portion of the qualities that are expected in a therapist emaila.	Therapist writes a brief email (i.e., one short paragraph) that includes no symptom feedback or indication when the next lesson will become available.	46 (6.4%)	19 (20.9%)
Unaddressed content	Therapist ignores or does not address comment or question from previous patient email.	Therapist fails to answer a question about cognitive restructuring asked in the patient email.Therapist makes no reference to any content from the patient's most recent email.	29 (4.0%)	22 (24.2%)
Unsupportive tone	Tone of email is unsupportive or critical.	Therapist bluntly states that patient has not logged in for many weeks without providing a statement of support (e.g., “I see that you have not logged on in 17 days”).	4 (0.6%)	3 (3.3%)
Missed correspondence	Therapist misses a scheduled communication without providing advanced warning.	Therapist is unable to send their scheduled email due to a sudden illness, but no email is sent to the patient informing them of the therapist's absence.	4 (0.6%)	3 (3.3%)
Inappropriate self-disclosure	Therapist self-disclosure that has no clear therapeutic purpose or detracts from the patient's situation.	Therapist makes reference to their own anxiety or depressive symptoms.	4 (0.6%)	2 (2.2%)
Unmanaged risk	Therapist does not appropriately address an increase in patient symptom severity or suicidality.	Therapist mentions an increase in suicidal ideation in their email, but does not phone patient to follow up.Therapist does not initiate telephone contact when a patient reports a symptom score increase of >5 on the depression symptom measure.	2 (0.3%)	2 (2.2%)

See Hadjistavropoulos et al. (2018) for more about recommended therapist behaviours.

The ICBT-UTBS total score was calculated for each patient using the following equation:

Statistical analyses

Internal consistency was measured using Cronbach's alpha coefficient, with ≥0.70 as the standard for reliability (Tavakol and Dennick, 2011). Internal consistency for the ICBT-UTBS total was excellent (Cronbach's α = 0.93). Descriptive statistics were used to calculate the overall frequency of undesirable therapist behaviours, as well as the percentage of emails that contained each undesirable therapist behaviour. In addition, correlational analyses were used to examine the relationship of the average ICBT-UTBS total score and domain scores to: 1) patient background variables; 2) patient engagement variables; 3) patient satisfaction and therapeutic alliance; 4) patient outcomes; 5) therapist variables; and 6) ICBT-TRS total and domain scores. Spearman's rank-order correlation coefficient (Spearman's Rho) was used because it is a robust test that guards against violations of statistical assumptions (e.g., normality) and is appropriate with mixed variable types (e.g., ordinal and continuous). Alpha was set at p < .01 as a partial control for the number of correlations examined.

Results

ICBT-UTBS frequencies

The frequencies of the undesirable therapist behaviours are presented in Table 1. The most commonly occurring undesirable behaviour was inadequate detail (6.4% of emails), followed by unaddressed content (4.0% of emails). The remaining four undesirable behaviours were rarely observed (<1.0% of emails). At least one undesirable behaviour was present in 77 out of 720 therapist emails (10.7%). When examined per patient, 34 out of 91 randomly selected patients (37.4%) received at least one email with an undesirable behaviour. Twenty patients were exposed to only one undesirable behaviour (22.0%), 12 patients experienced two distinct undesirable behaviours (13.2%), and two patients experienced three or four distinct undesirable behaviours (2.2%). No patient was exposed to more than four undesirable therapist behaviours. The average number of undesirable behaviours per patient was 0.98 (SD = 1.76; range = 0–8).

Twenty-three out of 39 therapists demonstrated at least one undesirable therapist behaviour (59.0%), with 18 therapists practicing in a community clinic setting (13 social work/nursing; 5 psychology) and 5 therapists practicing in a specialized ICBT clinic setting (4 psychology; 1 social work). Of the 34 patients who received an email with at least one undesirable therapist behaviour, 26 were treated by a therapist from a community clinic setting (18 social work/nursing; 8 psychology) and 8 were treated by a therapist from a specialized ICBT clinic (1 social work/nursing; 7 psychology).

Inter-correlations

Correlations between the ICBT-UTBS total score and domain scores are presented in Table 2. Strong to weak positive correlations were found between the ICBT-UTBS total score and four of the domains: inadequate detail (r = 0.78; p < .001), unaddressed content (r = 0.73; p < .001), unsupportive tone (r = 0.32; p < .01), and missed correspondence (r = 0.32; p < .01). The vast majority of the undesirable therapist behaviours were not correlated with each other, with a few exceptions. Inadequate detail demonstrated a weak positive correlation with unaddressed content (r = 0.37; p < .001) and unsupportive tone (r = 0.38; p < .001), while a strong positive correlation was observed between unsupportive tone and missed correspondence (r = 0.67; p < .001).

Table 2

Correlations between ICBT-UTBS total score and domains.

	1	2	3	4	5	6	7
1. ICBT-UTBS Total	–
2. Inadequate Detail	0.78⁎⁎	–
3. Unaddressed Content	0.73⁎⁎	0.37⁎⁎	–
4. Unsupportive Tone	0.32⁎	0.38⁎⁎	0.08	–
5. Missed correspondence	0.32⁎	0.25	0.20	0.67⁎⁎	–
6. Inappropriate Self-Disclosure	0.16	−0.08	−0.08	−0.03	−0.03	–
7. Unmanaged Risk	0.16	0.10	−0.08	−0.03	−0.03	−0.02	–

Note: ICBT-UTBS = Internet-Delivered Cognitive Behaviour Therapy – Undesirable Therapist Behaviours Scale.

p < .01.

p ≤ .001.

Total score correlations

The ICBT-UTBS total score was not significantly correlated with most patient background variables, patient engagement, working alliance, treatment satisfaction, or patient outcomes (Table 3). There was a weak negative correlation between ICBT-UTBS total score and patient gender (r = −0.30; p < .01), such that a greater number of undesirable therapist behaviours were present in emails sent to female patients. With regard to therapist characteristics, the ICBT-UTBS total score was moderately negatively correlated with therapist clinical setting (r = −0.42; p < .001) and profession (r = −0.42; p < .001). Namely, a greater number of undesirable therapist behaviours was associated with therapists who worked in a community clinic setting and had a background in social work/nursing compared to those who worked in a specialized ICBT clinic and had a psychology background.

Table 3

ICBT-UTBS total and domain score correlations with background, engagement, working alliance, treatment satisfaction, patient outcomes, and therapist characteristics.

	ICBT-UTBS total	Inadequate detail	Unaddressed content
Background
Age	−0.09	−0.03	−0.04
Female (0)/Male (1)	−0.30⁎	−0.20	−0.17
Not married (0)/Married (1)	0.09	−0.08	0.07
No university (1)/University (1)	−0.02	−0.06	−0.06
Unmedicated (0)/Medicated (1)	0.08	−0.06	0.13
GAD-7 pre-treatment	0.02	−0.01	0.07
PHQ-9 pre-treatment	0.04	0.11	0.01
Engagement
Log-ins	0.01	−0.19	0.14
Lessons started	−0.15	−0.32⁎	−0.02
Emails to therapist	0.02	−0.20	0.16
Emails from therapist	0.04	−0.05	−0.13
Phone-calls from therapist	−0.11	−0.10	−0.11
Working alliance
WAI-SR-Total	−0.10	−0.27	0.11
WAI-SR-Bond	−0.09	−0.26	0.10
WAI-SR-Task	−0.06	−0.26	0.11
WAI-SR-Goal	−0.14	−0.23	0.05
Treatment satisfaction
Not recommend (0)/Recommend (1)	0.12	0.07	0.09
Not worthwhile (0)/Worthwhile (1)	−0.09	−0.05	−0.07
Patient outcomes
GAD-7 change score	0.21	0.11	0.18
PHQ-9 change score	0.01	−0.05	0.05
Therapist characteristics
Community clinic (0)/Specialized clinic (1)	−0.42⁎⁎	−0.35⁎⁎	−0.26
Social work + Nursing (0)/Psychology (1)	−0.42⁎⁎	−0.34⁎⁎	−0.28⁎
Graduate student (0)/Registered provider (1)	0.16	0.18	0.07

Note: Correlations for unsupportive tone, missed correspondence, inappropriate self-disclosure, and unmanaged risk are not reported due to their low incidence. ICBT-UTBS = Internet-Delivered Cognitive Behaviour Therapy – Undesirable Therapist Behaviours Scale; PHQ-9 = Patient Health Questionnaire-9; GAD-7 = Generalized Anxiety Disorder-7; WAI-SR = Working Alliance Inventory-Short Revised.

p < .01.

p ≤ .001.

Domain correlations

At the domain level, correlations are not reported for unsupportive tone, missed correspondence, inappropriate self-disclosure, and unmanaged risk because these behaviours were observed so infrequently in the present study. With regard to the remaining domains, inadequate detail and unaddressed content, only a few significant correlations were observed for background variables, engagement, working alliance, treatment satisfaction, and patient outcomes (Table 3). Specifically, inadequate detail demonstrated a weak negative correlation with number of lessons started (r = −0.32; p < .01), such that short emails that lacked detail were associated with patients who started fewer lessons. With regard to therapist characteristics, inadequate detail was weakly negatively correlated with therapist setting (r = −0.35; p < .001), suggesting that therapists who worked in a community clinic setting were more likely to send emails that were short and lacking adequate detail. Moreover, weak negative correlations were observed between therapist profession and inadequate detail (r = −0.34; p < .001) and unaddressed content (r = −0.28; p < .01), such that each behaviour was associated with therapists from a social work or nursing background.

Correlations with the ICBT-TRS

Moderate negative correlations were observed between the ICBT-UTBS and ICBT-TRS total scores (r = −0.48; p < .001), indicating that a greater number of undesirable therapist behaviours was associated with fewer recommended therapist behaviours (see Table 4). The ICBT-UTBS total score was also negatively correlated with all ICBT-TRS domains, with the exception of seeks feedback, provides feedback, and praises effort. Weak to moderate negative correlations were also identified between several of the ICBT-UTBS and ICBT-TRS domains. Inappropriate self-disclosure and unmanaged risk were not correlated with any recommended therapist behaviours. The recommended therapist behaviour provides feedback was not significantly correlated with the ICBT-UTBS total or domains scores.

Table 4

Correlations between ICBT-TRS and ICBT-UTBS total and domain scores.

	ICBT-UTBS total	Inadequate detail	Unaddressed content	Unsupportive tone	Missed correspondence	Inappropriate self-disclosure	Unmanaged risk
ICBT-TRS total	−0.48⁎⁎	−0.46⁎⁎	−0.33⁎	−0.27⁎	−0.24	−0.02	0.05
Builds rapport	−0.35⁎⁎	−0.22	−0.35⁎⁎	−0.30⁎	−0.30⁎	0.06	0.06
Seeks feedback	−0.26	−0.11	−0.27⁎	0.00	0.00	−0.09	0.06
Provides feedback	−0.14	−0.24	−0.06	−0.09	−0.01	0.01	−0.06
Provides psychoeducation	−0.55⁎⁎	−0.48⁎⁎	−0.38⁎⁎	−0.29⁎	−0.29⁎	−0.08	−0.02
Facilitates understanding	−0.39⁎⁎	−0.43⁎⁎	−0.23	−0.29⁎	−0.22	0.06	0.05
Praises effort	−0.15	−0.22	−0.09	−0.27⁎	−0.27⁎	0.14	0.02
Encourages practice	−0.36⁎⁎	−0.42⁎⁎	−0.22	−0.17	−0.20	−0.03	0.04
Clarifies administrative procedures	−0.40⁎⁎	−0.47⁎⁎	−0.30⁎	−0.29⁎	−0.29⁎	0.07	0.07
Communicates effectively	−0.46⁎⁎	−0.38⁎⁎	−0.31⁎	−0.20	−0.31⁎	−0.23	0.09

Note: ICBT-TRS = Internet-Delivered Cognitive Behaviour Therapy – Therapist Rating Scale; ICBT-UTBS = Internet-Delivered Cognitive Behaviour Therapy – Undesirable Therapist Behaviours Scale.

p ≤ .01.

p ≤ .001.

Discussion

To our knowledge, this is the first study to explore undesirable therapist behaviours in email correspondence with ICBT patients. The purpose of the present study was to create the ICBT-UTBS and explore factors related to undesirable therapist behaviours with the goal of identifying behaviours that have the potential to compromise ICBT treatment integrity. Qualitative analysis of therapist emails, in the context of patient emails and telephone contact notes, led to the identification of six undesirable behaviours or domains that form the basis of the ICBT-UTBS: 1) inadequate detail; 2) unaddressed content; 3) unsupportive tone; 4) missed correspondence; 5) inappropriate self-disclosure; and 6) unmanaged risk.

In general, the incidence of undesirable therapist behaviours was low (10.7% of emails), suggesting that trained therapists are infrequently demonstrating undesirable therapist behaviours in routine practice. This provides evidence of treatment quality, an important facet of treatment integrity (Hermes et al., 2019; Proctor et al., 2011). When examined per patient, however, 37.4% received at least one email with an undesirable behaviour. Together, these findings indicate that although a very small proportion of all therapist emails contain undesirable behaviours, it is not uncommon for patients to be exposed to at least one undesirable behaviour over the course of treatment. Importantly, a positive correlation was found between the ICBT-UTBS and ICBT-TRS, with a greater number of recommended behaviours being associated with fewer undesirable behaviours. Based on these findings, it may be useful to consider both desirable and undesirable therapist behaviours in concert.

The most prevalent undesirable therapist behaviour, found in 6.4% of all emails, was inadequate detail, which refers to emails that are obviously short, lacking detail, and missing several qualities that would be expected in a weekly email (e.g., friendly greeting, symptom feedback, praise for effort, encouragement to practice, reference to the content of the next lesson). Approximately 20% of patients received at least one email of this kind during the course of treatment, likely as a result of time pressure. That is, therapists only have a specified amount of time each week to correspond with each patient, so there may be occasions when they are forced to send a briefer-than-usual email.

The second most common undesirable therapist behaviour was unaddressed content (4.0% of all emails), which refers to the therapist failing to address a comment or question from the patient's most recent email. Approximately one quarter of patients received an email with this type of behaviour. In some cases, unaddressed content may be an unfortunate oversight by the therapist, but it is also possible that the therapist intended to leave the question/comment unacknowledged. For example, a therapist could make the intentional decision to leave a patient's question or comment unaddressed if the therapist deemed it not helpful to the therapeutic process. Unfortunately, when post-hoc coding, it is not possible to evaluate the intention of the therapist, only what is observable (Plumb and Vilardaga, 2010).

The four remaining undesirable therapist behaviours were present in <1% of all emails. Unsupportive tone was found in 0.6% of therapist emails to 3.3% of patients. Similar to face-to-face therapists, ICBT therapists are trained to be warm, unconditionally supportive, and encouraging in their correspondence with patients, therefore, the low incidence of unsupportive statements is not surprising (Hadjistavropoulos et al., 2012; Norcross and Lambert, 2018). In this study, most incidents of unsupportive tone related to the therapist calling attention to the patient's disengagement with the program (e.g., patient had not logged in for over 7 days) without including a subsequent statement of support. These types of statements, which could be interpreted by the patient as critical or judgmental, are examples of poor quality support that has the potential to threaten treatment integrity.

Missed correspondence, which involves an unexplained lack of correspondence with the patient on the scheduled day, was also found in 0.6% of therapist emails to 3.3% of patients. In routine practice, it is reasonable to expect that, on occasion, a therapist might be unavailable on their usual check-in day due to illness or other unforeseen circumstances. We suggest that so long as someone associated with the ICBT program (e.g., supervisor, administrator) informs the patient that their therapist will be unable to check in on their usual day, the behaviour should not be considered undesirable. Only unexplained absences that jeopardize the patient's sense of consistent support should be considered an undesirable behaviour. Notably, coding of this undesirable therapist behaviour assumes the therapists maintained accurate records (i.e., made note of telephone contact that was made in lieu of an email).

Inappropriate self-disclosure was found in only 0.6% of emails to 2.2% of patients. Although self-disclosure in and of itself is not prohibited in therapist emails, any self-disclosure that does not serve a clear therapeutic purpose (e.g., self-disclosure that is excessive or inappropriate) should be considered an undesirable behaviour (Henretty and Levitt, 2010). It seems important that ICBT therapists should be explicitly trained to distinguish between appropriate and inappropriate self-disclosure given that one study found a weak positive correlation between appropriate self-disclosure and depression symptom reduction (Holländare et al., 2016).

Finally, unmanaged risk was found in only 0.3% of emails to 2.2% of patients. In this study, all examples of unmanaged risk related to therapists not taking appropriate steps when a patient reported an increase in suicidal ideation (i.e., phoning the patient to conduct a risk assessment). Although not observed in the present study, unmanaged risk might also be demonstrated if a therapist failed to respond according to policy when a patient's weekly symptom measures indicated a significant deterioration in symptoms. Incidents of unmanaged risk may be extremely rare, but they are of paramount importance to monitor, as it relates to risk of patient harm.

The undesirable therapist behaviours identified in the current study have some commonalities with ineffective or harmful therapist behaviours described in the face-to-face psychotherapy literature (Norcross and Lambert, 2018). Inappropriate self-disclosure and unmanaged risk are behaviours that seem likely to occur in both ICBT and face-to-face therapy. Similarly, unsupportive tone corresponds with therapist criticism in face-to-face therapy and missed correspondence could be considered equivalent to an in-person therapist missing an appointment. Inadequate detail and unaddressed content are slightly more difficult to compare to face-to-face therapy, because they are so strongly tied to the format of email communication. It seems plausible, however, that variations of these themes could be demonstrated by therapists in face-to-face therapy. Overall, our findings suggest there is value in further exploring similarities and differences in undesirable therapist behaviours between ICBT and face-to-face therapy in the future.

The ICBT-UTBS demonstrated excellent internal consistency in the present study, which supports the overall reliability of the scale. Examination of the inter-correlation between the ICBT-UTBS total and domain scores revealed several significant relationships. These findings indicate that inadequate detail and unaddressed content, and to a lesser extent unsupportive tone and missed correspondence, are the greatest contributors to the ICBT-UTBS total score. Moreover, the significant correlations across domains suggest that certain undesirable behaviours are more likely to occur together (e.g., missed correspondence and unsupportive tone). Nevertheless, the fact that the correlations between domains were weak to moderate in strength provide evidence that the domains are related but distinct.

When correlations between the ICBT-UTBS and treatment outcomes were explored, a few significant findings were observed. We found higher ICBT-UTBS total scores were associated with female patients and inadequate detail was associated with starting fewer lessons, but more research is needed to explore the directionality of these relationships and to determine whether mediating variables are at play. Therapists from all practice settings and professional backgrounds demonstrated undesirable therapist behaviours, suggesting that this issue is relevant to all ICBT therapists, yet undesirable therapist behaviours were significantly correlated with therapist practice setting and professional background. Similar to what was found for recommended therapist behaviours (Hadjistavropoulos et al., 2018), ICBT-UTBS scores were higher among therapists working in a community clinic setting, which might be reflective of those therapists having competing demands for their time (e.g., provision of face-to-face and internet intervention). Moreover, therapists in the community may have received less supervision given that they were not practicing in the same physical space as their ICBT supervisor. Lower ICBT-UTBS scores among therapists with a psychology background could reflect differences in training, as therapists with social work/nursing backgrounds had mostly undergraduate training and therapists with psychology backgrounds had graduate training. However, it seems likely that professional setting is a more important factor than professional background, given that the vast majority of therapists with social work/nursing backgrounds who demonstrated undesirable therapist behaviours worked in a community clinic setting. This has important implications for clinical practice. Namely, it may be necessary for therapists working in community setting to receive supervision on a more regular basis (e.g., have their emails reviewed by their supervisor) to ensure that undesirable therapist behaviours are not being perpetuated.

Perhaps most importantly, ICBT-UTBS total and domain scores were not significantly correlated with working alliance, treatment satisfaction, or patient outcomes. These findings were not surprising, given that past research exploring the relationship between recommended therapist behaviours and treatment outcomes is characterized by inconsistent findings (Hadjistavropoulos et al., 2018; Holländare et al., 2016; Mol et al., 2018; Paxling et al., 2013; Schneider et al., 2016). Although it is possible the lack of significant correlations was due to low power for identifying small effects, lack of variability in ICBT-UTBS scores, or the dichotomous nature of some variables, it seems more likely that patient outcomes are simply robust to infrequent undesirable therapist behaviours. That is, the presence of one or two undesirable therapist behaviour during the course of treatment may go unnoticed or be overlooked by the patient. It may only become problematic to outcomes if therapists with little to no training demonstrate serious and/or frequent undesirable behaviours.

Strengths and limitations

The current study has several notable strengths. Previous research has not yet attended to undesirable behaviours even though this is an important component of treatment integrity. We observed six distinct behaviours by examining the written content of therapist emails in the context of the content of patient emails, the timing of the therapist emails, and telephone contact notes. This resulted in a better understanding of undesirable therapist behaviours that was not previously gleaned from studies of therapist behaviours in ICBT (Holländare et al., 2016; Mol et al., 2018; Paxling et al., 2013; Sanchez-Ortiz et al., 2011; Schneider et al., 2016). It also allowed for the creation of the ICBT-UTBS, a reliable measure of undesirable therapist behaviours that can be used in routine practice or training settings to monitor quality of therapist support. Another strength is that the present study included therapist emails that were sent in the context of an open trial of transdiagnostic ICBT program in a routine care setting. Other studies on therapist behaviours have been conducted with disorder-specific ICBT programs and in the context of strictly controlled randomized controlled trials (Holländare et al., 2016; Paxling et al., 2013). Furthermore, the inclusion of 39 therapists with different professional backgrounds (e.g., psychology, social work), varying levels of experience (e.g., registered, students), and from two distinct practice settings (e.g., specialized ICBT clinic, community clinic) enhances the generalizability of the findings and allowed us to identify factors that were distinctly associated with increased prevalence of undesirable therapist behaviours (e.g., professional background, practice setting). Finally, our decision to adjust the p-value to 0.01 reduces the risk of type I errors (i.e., findings of false “significance”) when conducting exploratory analyses.

Despite its strengths, the present study is not without limitations. First, therapists involved with the open trial received intensive training on effective communication with patients in the context of ICBT, which may have lessened the overall frequency of undesirable behaviours observed. Moreover, it is possible there are additional undesirable therapists behaviours not captured in the current study that may emerge if this research was replicated in another setting. For example, unmanaged risk could include instances where therapists fail to respond appropriately to significant symptom deterioration, not only an increase in suicidality. Second, we were only able to code observable therapist behaviours without insight into the therapist's intentions. This important contextual information may explain the presence of certain undesirable behaviours. Third, findings from the current study are purely correlational and thus it is not possible to infer causality. For example, the negative correlation observed between inadequate detail and number of lessons started could be interpreted in one of two ways: (a) therapists were more likely to send short emails that lacked detail when the patient started fewer lessons (i.e., therapists had less to say in these cases), or (b) patients tended to start fewer lessons when the emails sent by their therapist were short and lacking detail. Subsequent research is necessary to explore these correlational relationships. Lastly, identifying undesirable therapist behaviours requires clinical judgement. While we have provided as much information as possible to assist other researchers/clinicians when using the ICBT-UTBS, there will always be occasions where the coder will have to make a subjective judgement about whether to code an undesirable therapist behaviour as present or absent.

Future directions

Additional research is needed to replicate the current study's findings in diverse practice settings where therapists have received different training or the ICBT program includes alternative components (e.g., mandatory homework, exclusively email communication). Such studies should confirm the ICBT-UTBS domains, the measure's psychometric properties and factor structure, and its relationship (or lack thereof) to treatment outcomes. It will be especially important to confirm that undesirable therapist behaviours are unrelated to poorer patient engagement, working alliance, or outcomes even if the behaviours were more prevalent than in the current study. It is possible that patients are robust to infrequent undesirable behaviours, but may be more effected if undesirable behaviours occur repeatedly. If necessary, the ICBT-UTBS should be tailored to the practice setting and ICBT program it will be used in. For example, the criteria for unmanaged risk might need to be modified if initiating telephone contact is not the procedure employed by the program when patient's report increases in symptoms/suicidality. Future research should also investigate the clinical utility of using the ICBT-UTBS and ICBT-TRS in concert as training tools and to monitor quality of therapist support provided during ICBT. Experimental studies may also use both measures when exploring the effect of removing or manipulating selected therapist behaviours (e.g., systematically increasing or decreasing the length/detail of therapist emails).

Conclusions

Six distinct undesirable therapist behaviours were identified, albeit infrequently, in therapist emails to patients taking part in an 8-week transdiagnostic ICBT program. Those behaviours form the basis of the ICBT-UTBS, a reliable tool that can be used to in training and to monitor therapist behaviours in research and routine clinical practice. Promisingly, ICBT-UTBS total scores were not related to treatment outcomes, suggesting that patient outcomes are robust to infrequent undesirable therapist behaviours. In combination with the ICBT-TRS, this new measure will help ensure the quality of support provided by ICBT therapists and thus treatment integrity.

Funding

Funding for this study was provided by the Canadian Institutes of Health Research, Canada (grant # 293379 & grant # 152917), Saskatchewan Health Research Foundation, and Rx & D Health Research Foundation, Canada. Funders had no involvement in the study design, or data collection, analysis, or interpretation.

Declaration of Competing Interest

None.

Behaviour	Questions to help with decision makinga
Inadequate detail	Is the email a reasonable length (e.g., at least 2–3 paragraphs)?Does the email contain most of the features expected in a therapist email (e.g., friendly greeting, symptom feedback, praise for effort, encouragement to practice, reference to the content of the next lesson)?Note: Inadequate detail should only be coded if the email is obviously shorter than normal and lacking numerous features expected in an email.
Unaddressed content	Does the therapist make reference to the content of the patient's most recent email (e.g., respond to a question or a comment made by the patient)?Is there any indication the question/comment was addressed in a telephone contact or the therapist intended to leave the question/comment unaddressed?Note: Unaddressed content should not be coded if there is no patient email from the preceding week.
Unsupportive tone	Does the email come across as warm, friendly, and supportive (e.g., with no sense of criticism, judgement, or curtness)?Does the therapist include one or more supportive statements in their email (e.g., praise, encouragement)?Note: Unsupportive tone should only be coded if the email is obviously negative or critical.
Missed correspondence	Does the date stamp on the therapist email suggest that the email was sent on the scheduled day each week?If there is no email, was there are telephone contact note instead?Note: Missed correspondence should not be coded if the therapist informed the client ahead of time that a check-in would be missed or another staff member informed the client their therapist wouldn't be available on the scheduled day.
Inappropriate self-disclosure	Is there an obvious reason why the therapist engaged in self-disclosure (e.g., to build rapport, to normalize an experience)?Is the self-disclosure appropriate (e.g., not too personal, professional) and relevant (e.g., related to something the client said or is experiencing)?Note: Inappropriate self-disclosure should only be coded if it is obvious the self-disclosure was inappropriate or not relevant.
Unmanaged risk	In their email to the patient, does the therapist reference an increase in symptoms/suicidality and inform the patient a telephone contact will be initiated?Was a telephone contact attempted?Note: Unmanaged risk should only be coded if the patient's weekly symptom measures indicate significant symptom deterioration or an increase in suicidality. What exactly constitutes significant symptom deterioration or an increase in suicidality depends on the measures being used.

If the answer to one or more of the questions in each section is ‘no’, this may be indicative of an undesirable therapist behaviour; however, judgement on the part of the coder may be required.