Scales for assessing therapeutic adherence and competence in dialectical behaviour therapy for PTSD: development and analysis of psychometric properties.

Background: The assessment of therapeutic adherence and competence is essential to understand mechanisms that contribute to treatment outcome. Nevertheless, their assessment is often neglected in psychotherapy research. Aims/Objective: To develop an adherence and a treatment-specific competence rating scale for Dialectical Behaviour Therapy for Posttraumatic Stress Disorder (DBT-PTSD), and to examine their psychometric properties. Global cognitive behavioural therapeutic competence and disorder-specific therapeutic competence were assessed using already existing scales to confirm their psychometric properties in our sample of patients with PTSD and emotion regulation difficulties. Method: Two rating scales were developed using an inductive procedure. 155 videotaped therapy sessions from a multicenter randomised controlled trial were rated by trained raters using these scales, 40 randomly chosen videotapes involving eleven therapists and fourteen patients were doubly rated by two raters.
Results: Both the adherence scale (Patient-level ICC = .98; αs = .65; α p  = .75) and the treatment-specific competence scale (Patient-level ICC = .98; αs = .78; α p  = .82) for DBT-PTSD showed excellent interrater - and good reliability on the patient level. Content validity, including relevance and appropriateness of all items, was confirmed by experts in DBT-PTSD for the new treatment-specific competence scale.
Conclusion: Our results indicate that both scales are reliable instruments. They will be useful to examine possible effects of adherence and treatment-specific competence on DBT-PTSD treatment outcome.

Introduction

The experience of childhood abuse (CA) may have many far-reaching consequences for a person’s mental health. Not only do a great number of CA-survivors suffer from Post Traumatic Stress Disorder (PTSD), but also a comorbid Borderline Personality Disorder (BPD) is common (McLean & Gallop, 2003; Scheiderer, Wood, & Trull, 2015).

A treatment approach specifically developed for patients with PTSD and additional severe difficulties in emotion regulation is Dialectical Behaviour Therapy for PTSD (DBT-PTSD; Bohus et al., 2013; Steil, Dyer, Priebe, Kleindienst, & Bohus, 2011). Originally developed as an inpatient treatment specifically tailored to patients with PTSD after CA and difficulties in emotion regulation, DBT-PTSD demonstrated its excellent efficacy in several trials both in an inpatient as well as in an outpatient setting (Bohus et al., 2013; Bohus et al., 2020; Steil et al., 2011; Steil et al., 2018). DBT-PTSD combines principles of standard DBT (Linehan, 1996) with trauma-focused cognitive and exposure-based elements as well as innovative interventions (for a detailed description, refer to Steil et al., 2018). The outpatient treatment administers a training in emotion regulation skills within individual treatment sessions before using cognitive restructuring and imaginal exposure with trauma-related memories. Additionally, patients can receive therapist telephone coaching in crisis situations. The therapists receive regular case consultation. DBT-PTSD is structured in five phases: 1.) commitment phase; 2.) planning and motivation phase; 3) identification and addressing of trauma-related escape strategies; 4) application of exposure; and 5) ‘regain a life worth living’.

To ensure that a manualized treatment such as DBT-PTSD is implemented as intended and to accurately interpret results of outcome studies, the assessment of treatment integrity in psychotherapy research is essential (Waltz, Addis, Koerner, & Jacobson, 1993). Treatment integrity is an important construct to ensure internal validity of outcome trials, and it can furthermore be used to examine the relationship between treatment integrity and treatment outcome (Weck, Bohn, Ginzburg, & Stangier, 2011). Thereby it brings elucidation relating to possible mechanisms of treatment success and can help to identify central therapeutic skills and interventions (Perepletchikova & Kazdin, 2005), which is especially important when studying newly developed treatments such as the DBT-PTSD. Treatment integrity is considered to comprise therapeutic adherence, therapeutic competence, as well as treatment differentiation. While treatment adherence is defined as the extent to which a therapist uses interventions as described in the treatment manual, therapeutic competence refers to the level of therapeutic skills used by the therapist when delivering the treatment (Waltz et al., 1993). Hence, in addition to the treatment manual, a competent therapist will also respond flexibly to issues arising in the respective treatment context, such as the stage of therapy or symptom severity.

According to Barber, Sharpless, Klostermann, and McCarthy (2007), therapeutic competence can be further subdivided into different domains: global cognitive behavioural therapeutic (CBT) competence, defined as general clinical skills and knowledge of a therapist, and limited-domain competence, divided in the skilful application of treatment- or disorder-specific interventions.

The third component of treatment fidelity, treatment differentiation, describes the extent to which treatments can be distinguished from other treatments (Perepletchikova & Kazdin, 2005).

Despite the importance of treatment integrity, many treatment studies lack its assessment. One reason for this could be that assuring and assessing therapeutic adherence and competence is a challenging process: In addition to the most basic elements such as providing a treatment manual as well as therapist training and supervision, the ‘silver bullet’ in the assessment of adherence and competence is the rating of videotaped therapy sessions (Bellg et al., 2004). Not only is this method time consuming, it is expensive as well (Perepletchikova, Hilt, Chereji, & Kazdin, 2009). Furthermore, such ratings require highly qualified raters as well as valid and reliable rating scales to assess therapeutic competence and adherence. Often, adequate rating instruments are lacking or their psychometric properties are not reported (Perepletchikova, Treat, & Kazdin, 2007).

This is also true for research on psychological treatments for PTSD. Only a few randomised controlled trials in the field of PTSD assess and analyze therapeutic adherence and competence. In most of these studies the rating scales that were used are not reported and even less their psychometric properties (Barber, Triffleman, & Marmar, 2007; e.g. Blanchard et al., 2003; Holder, Holliday, Williams, Mullen, & Surís, 2018; Kubany et al., 2004; Schnurr et al., 2003).

When separately regarding the constructs of adherence and competence in the field of PTSD research, we find studies assessing therapeutic adherence to Cognitive Processing Therapy (CPT, Resick et al., 2008), which report psychometric properties for the rating scales (e.g. Farmer, Mitchell, Parker-Guilbert, & Galovski, 2017; Marques et al., 2019; Resick, Nishith, Weaver, Astin, & Feuer, 2002), and from our workgroup (Dittmann et al., 2017; Gutermann et al., 2015). These adherence rating instruments are understandably only applicable when the CPT manual is used, other adherence rating scales addressing PTSD treatments are missing.

Also, for the assessment of therapeutic competence in the treatment of PTSD, only a very limited amount of adequate assessment instruments is available: In one study by Paivio, Holowaty, and Hall (2004), a rating scale measuring competence to administer imaginal confrontation with CA memories and its psychometric properties are reported, however, the respective scales are not published. Furthermore, our workgroup has developed and published PTSD-related competence rating scales and their psychometric properties (Dittmann et al., 2017; Gutermann et al., 2015).

All in all, there is a lack of reliable and valid rating instruments concerning therapeutic adherence and competence in the field of PTSD treatment research.

Regarding adherence measurement in DBT, there is only a very limited amount of adequate adherence assessment instruments. The scale most often cited in the literature is the University of Washington Adherence Scale (DBT-ACS: Linehan & Korslund, 2003), which is an advancement of the DBT expert rating scale (Linehan, Lockard, Wagner, & Tutek, 1996). It comprises 66 items within 12 subscales and a global adherence scale to assess adherence by trained coders (Miga, LoTempio, Michonski, & Hunter, 2020). Reported ICCs of 0.93 suggest a good interrater reliability of the scale (Linehan et al., 2015). However, these scales are only available as unpublished manuscripts.

Other checklists for treatment integrity assessment of DBT exist (e.g. McCay et al., 2016), yet their psychometric properties are not reported. Another checklist was developed by DiGiorgio, Glass, and Arnkoff (2010), comprising 31 items summarising core DBT techniques that can be answered by therapists after each session. Again, psychometric properties of the scale are not available.

There is a lack of availability of empirically validated adherence measurements for DBT. This, as well as the fact that DBT-PTSD is a modular treatment that, additionally to DBT interventions, also comprises elements of trauma focused treatments and innovative interventions, calls for the development of an adequate and psychometrically sound measurement instrument of treatment adherence to DBT-PTSD.

The availability of such measurement instruments is the prerequisite for an investigation of the effect of therapist adherence and competence on treatment outcome. Thus far, only a few studies investigated the relationship between therapeutic adherence/ competence and treatment outcome. One study by Ginzburg et al. (2012) indicated that the treatment- and disorder-specific competences are more likely to predict treatment outcome than adherence. In a study by Marques et al. (2019) the authors found different effects of adherence and competence on different treatment outcomes: higher levels of adherence were associated with greater reduction in depressive symptoms, whereas higher levels of competence were related to greater reduction in PTSD symptoms.

Other studies have found an influence of the therapists’ adherence but not of competence on the treatment progress (e.g. Hogue et al., 2008).

In summary, in order to estimate the role of therapeutic adherence and competence on a scientific basis, further investigation is required. Inconsistent results may be due to the lack of reliable assessment procedures. Therefore, our present study aims to provide methodologically sound measures for therapeutic adherence and competence for the DBT-PTSD, where presently no such scales exist.

In the present study we therefore report the development of a short and time economic adherence rating scale for DBT-PTSD allowing to assess therapeutic adherence during treatment session by rating randomly drawn videotaped therapy sessions. Additionally, we developed a treatment-specific competence rating scale according to the DBT-PTSD manual. The aim of this study was to analyze the psychometric properties (interrater reliability and internal consistency) of these newly developed scales and to analyze their associations with global CBT competence (measured by an already existing scale measuring global CBT competence called Cognitive Therapy Scale (CTS; Young & Beck, 1980)) and disorder specific competence (measured by the Competence Rating Scale for PTSD which was developed by our workgroup (CRS-PTSD; Dittmann et al., 2017)). Our further aim is to confirm the psychometric properties of the CTS and the CRS-PTSD. Thus we do not put forward directional hypotheses.

Method

Study design

The data for this study was collected from a clinical multicenter randomised controlled trial that examined the efficacy of Dialectical Behaviour Therapy for PTSD (DBT-PTSD; Bohus et al., 2013; Steil et al., 2011) and CPT (Müller-Engelmann, Dittmann, Weßlau, & Steil, 2016; Resick et al., 2008) in patients with PTSD and borderline personality symptoms following childhood sexual/physical abuse (CSA/CPA) (the RELEASE study; Treating Psychosocial and Neural Consequences of Childhood Interpersonal Violence in Adults; German Clinical Trials Registration ID: DRKS00006095). The research on treatment competence and adherence with regard to the RELEASE study as well as the study itself was funded by the German Ministry of Science and Education. Further descriptions of the RELEASE study design can be found in Bohus et al. (2019) and the results are reported in Bohus et al. (2020).

In the present study, 155 videotaped therapy sessions of 71 female patients from the DBT-PTSD group were rated by rater 1. Furthermore, 14 patients were randomly chosen and the respective therapy sessions were additionally rated by rater 2, to analyze reliability and validity of the scales.

Treatment

DBT-PTSD is a modular treatment programme that comprises up to 45 sessions in one year and integrates trauma-focused cognitive and exposure-based interventions with the modes and principles of standard Dialectical Behaviour Therapy (DBT; Linehan et al., 1993). For a detailed description of DBT-PTSD, see (Steil et al., 2018).

Raters

The two raters, who were clinical psychologists with 3 and 8.5 years of clinical experience, had received intensive training in DBT-PTSD by visiting various workshops in DBT as well as DBT-PTSD. Both had treated patients with DBT-PTSD under supervision. Video tapes of three pilot cases were evaluated and discrepancies discussed before starting the assessments for the present study. During the study the ratings of every fifth video were compared between the two raters and differences discussed by both raters to avoid drifting apart. The ratings were assessed independently by each rater and not altered after comparing and discussing them. This is consistent with previous research studies on similar rating scales (Gutermann et al., 2015; Weck, Hilling, Schermelleh-Engel, Rudari, & Stangier, 2011b).

Participants and therapists

Altogether, 155 videotapes of 23 therapists and 71 patients were rated by rater 1. For each patient, 2–4 out of 41 videotaped treatment sessions (with the exception of the first four sessions) were randomly selected, including one of the first half of the therapy course and one of the second half. All patients were females with a mean age of 38.22 years (SD = 11.25 years; range = 19-62 years).

For the assessment of reliability and validity of the scales, 40 videotapes of 11 therapists and 14 patients were additionally rated by the second rater: again, for each patient, 2–4 videotaped treatment sessions from the first and second half of therapy were randomly selected. All patients were females with a mean age of 33.71 years (SD = 6.2 years; range = 20-43 years).

In addition to PTSD criteria, all patients met at least three criteria for borderline personality disorder and fulfilled two to five comorbid mental disorder diagnoses, mostly major depression and/or anxiety disorders assessed with the SCID I Interview (Wittchen, Zaudig, & Fydrich, 1997). All therapists were psychologists who were trained and supervised in DBT-PTSD.

Adherence rating scale for DBT-PTSD (release version)

The Adherence Rating Scale for DBT-PTSD (Release Version) (ARS-DBT-PTSD-R) was developed based on the respective DBT-PTSD treatment manual which was used in the trial. The items of the scale are developed to evaluate if specific interventions described in the manual are applied during the treatment session. The scale was developed in German.

For the structure, e.g. number of anchors of our new adherence rating scale, we took into account the structures of already existing adherence rating scales such as the Therapeutic Adherence Scale for Developmentally Adapted Cognitive Processing Therapy (TAS D-CPT) by Gutermann et al. (2015) or the Adherence Rating Scale for CPT (ARS-CPT) (Dittmann et al., 2017).

The ARS-DBT-PTSD-R consists of three parts. Part one includes general elements of the DBT-PTSD manual which should be present in every session. These include, for example, ‘Reviewing the diary card’ or ‘Guiding the mindfulness exercise’. Part two includes special interventions for the respective therapy phase such as ‘Psychoeducation on PTSD’ or ‘Carrying out exposure exercise’. Part three includes an item to assess whether other techniques as described by the manual were used.

Items are rated using a three-point Likert scale with 0 = not adherent, 1 = partly implemented/adherent to some extent and 2 = adherent with descriptions of adherent and non-adherent behaviour based on the manual for DBT-PTSD. Part 2 additionally includes the possibility for rating ‘not applicable’ if this intervention is not part of the respective session.

In the end, the mean for the session specific intervention items (part two, item 16–23) is calculated, then a sum score is built, containing this mean and all 23 other items.

Competence rating scale for DBT-PTSD (release version)

For assessing treatment-specific therapeutic competence (Waltz et al., 1993), we newly developed six items together with a group of experts in DBT-PTSD treatment. These items represent the therapeutic skills considered most important in DBT-PTSD such as ‘Optimal activation of the trauma network’. We called this scale the Competence Rating Scale for DBT-PTSD (Release Version) (CRS-DBT-PTSD-R). The same 7-point Likert scale as applied in the CRS-PTSD is used. A sum score of these six items is built to form a treatment-specific competence score.

For measuring global CBT competence, we referred to the already existing Cognitive Therapy Scale (CTS; Young & Beck, 1980), which is a rating scale originally designed to assess therapeutic competence in Cognitive Behaviour Therapy for depression (Young & Beck, 1980). In the present study we used an adaption of the German version (Weck, Hautzinger, Heidenreich, & Stangier, 2010), which contains 14 items rated on a 7-point Likert scale ranging from 0 = poor competence to 6 = excellent competence and has shown good psychometric properties (ICC = .85–93; Weck et al., 2010). Additionally, we included one more item (Item 11: ‘Focus on the behavioral model’) to account for the use of this scale in more behaviour-oriented treatments.

According to the recommendation of Waltz et al. (1993), we also measured disorder-specific therapeutic competence using the PTSD-specific Competence Rating Scale developed by our workgroup (Competence Rating Scale for PTSD, CRS-PTSD; Dittmann et al., 2017). The seven items of this scale represent PTSD-specific therapeutic competences such as for example ‘Identification and Modification of avoidance’. For further information concerning the development of the CRS-PTSD see Dittmann et al. (2017). The seven items are rated on a 7-point Likert scale ranging from 0 = poor competence to 6 = excellent competence and have shown good psychometric properties (ICC = .97; Cronbach’s α = .92; Dittmann et al., 2017).

Content validation of the competence rating scale for DBT-PTSD

Three experts and developers of DBT-PTSD (RS, KP and Anne Dyer) were asked to evaluate the relevance (the extent to which the item’s content is important for the assessment of therapeutic competence in PTSD treatment) and appropriateness (whether the wording and examples we used in the item description were appropriate for assessing the content) of the newly developed six items of the CRS-DBT-PTSD-R. They used a four-point Likert scale from 0 = not at all relevant/appropriate to 3 = extremely relevant/appropriate. The experts of DBT-PTSD were not involved in the video ratings however two of them (RS & KP) worked as supervisors in the treatment study.

Statistical analysis

Calculations to determine interrater reliability were based on the judgements of the 40 videotapes rated by both raters. To determine interrater reliability, we calculated intraclass correlation coefficients (ICC, Shrout & Fleiss, 1979) for each item as well as for the sum scores of each scale. To select the appropriate ICC form, we followed recommendations by Koo and Li (2016) and chose the two-way random effects model with single rater type and absolute agreement (Koo & Li, 2016). To account for the multilevel structure of our data (sessions (Level 1) nested within patients (Level 2), rated by both raters), we calculated session-level and patient-level ICCs based on design 1 recommended by Ten Hove, Jorgensen, and van der Ark (2021), using Markov-Chain Monte Carlo Estimation (MCMC, Ten Hove et al., 2021). The session-level ICC gives information on the degree to which the ordering of sessions within patients is independent across raters, whereas the patient-level ICC gives information on the degree to which the ordering of patients is independent of raters and therefore about the consistency in the assessment of patients (Ten Hove et al., 2021). The session-level ICC was calculated based on only 2–4 sessions per patient, a cluster size that can be considered very small (see Ten Hove et al., 2021) and hence unlikely to provide stable results. Therefore session-level ICCs must be interpreted with caution.

ICCs were interpreted as follows: ICC ≤ .50 is considered poor; .50 < ICC ≤ .75 is considered as moderate, .75 < ICC ≤ .90 is considered as good and ICC > .90 is considered as excellent (Koo & Li, 2016).

The reliability of the adherence scale and the three competence scales was calculated based on the procedure by Bonito, Ruppel, and Keyton (2012) on reliability estimates for multilevel designs. We calculated session-level reliability (αs) and patient-level reliability (αp) using the following formulas: where σ2 is the variance, p is the number of items in the respective scale and n is the average number of sessions per patient (Bonito et al., 2012). Scores exceeding α = .70 are considered acceptable (DeVellis, 1991).

To assess the relationship between therapeutic adherence and competence, ratings of 155 videotapes from 71 patients were used. Sum scores of the adherence scale and the three competence measures were calculated for each videotape. For the 40 sessions rated by both raters, the mean sum score of both raters per videotape was built. As 2–4 sessions per patient were rated (hence each patient provided more than one data point) the assumption of independence of error between observations was violated. Therefore, we calculated repeated measures correlation using the rmcorr package (Bakdash & Marusich, 2017).

Effects were considered small when r > .10, moderate when r > .30 and large when r > .50 (Cohen, 1988). All calculations were carried out with R.

Results

The patient-level and session-level ICCs for all adherence items and the sum score, as well as the mean, standard deviation values and the range of the minimum/maximum ratings for each item are presented in Table 1.

Table 1.

Patient-level and Session-level Intraclass Correlation Coefficients, Mean, Standard Deviation and Range of the Adherence Rating Scale for Dialectical behaviour Therapy for PTSD (Release Version).

Item	ICC(A,1)	M (SD)	Min/Max
1. Guiding the mindfulness exercise		1.06 (0.96)	0/2
Session-level	.92
Patient-level	.99
2. Reviewing the diary card		1.35 (0.86)	0/2
Session-level	.58
Patient-level	.97
3. Reviewing homework		1.06 (0.90)	0/2
Session-level	.93
Patient-level	.98
4. Setting the agenda		0.53 (0.73)	0/2
Session-level	.51
Patient-level	.94
5. Dealing with difficult situations in everyday life, e.g. interpersonal conflicts, homelessness, pregnancy		1.23 (0.77)	0/2
Session-level	.37
Patient-level	.95
6. Considering the dynamic treatment hierarchy		1.60 (0.61)	0/2
Session-level	.55
Patient-level	.96
7. Dealing with problematic behaviour		1.60 (0.64)	0/2
Session-level	.65
Patient-level	.97
8. Identification and modification of escape and avoidance		1.48 (0.66)	0/2
Session-level	.40
Patient-level	.95
9. Application of cognitive techniques to explore /identify/modify trauma-related dysfunctional beliefs		1.40 (0.55)	0/2
Session-level	.15
Patient-level	.92
10. Conclusion of session		0.89 (0.76)	0/2
Session-level	.76
Patient-level	.88
11. Assigning homework		1.11 (0.94)	0/2
Session-level	.49
Patient-level	.92
12. Applying principles of reinforcement		1.49 (0.51)	0/2
Session-level	.02
Patient-level	.91
13. Time management		1.11 (0.81)	0/2
Session-level	.56
Patient-level	.95
14. Overall strategy		1.64 (0.57)	0/2
Session-level	.56
Patient-level	.90
15. Implementation of DBT-PTSD interventions in general		1.62 (0.57)	0/2
Session-level	.74
Patient-level	.94
16. Session specific intervention components (Items16-23)		1.39 (0.61)	0/2
Session-level	.60
Patient-level	.92
24. Interventions from treatments other than DBT-PTSD		1.78 (0.53)	0/2
Session-level	.43
Patient-level	.94

Note. ICC(A,1) = Intraclass correlation coefficients for both raters; Min = lowest score of the judges’ ratings on a scale from 0 to 2 and Max = highest score of the judges’ ratings on a scale from 0 to 2.

The patient-level ICC of the adherence sum score showed high rater concordance with ICC = .98, also for all items, the patient-level ICCs exceeded .80 indicating excellent interrater reliability on the patient level. On the session level, the ICC of the adherence sum score was moderate (ICC = .67). Concerning the session-level ICC of the items, a poor session-level ICC was found for items 5, 8, 9, 11, 12 and 24, an excellent session-level ICC was found for items 1 and 3. One item (item 10) had a good session-level ICC, all other items had a moderate session-level ICC.

For the ARS-DBT-PTSD-R, the reliability on the session-level was moderate (αs = .65) the patient-level reliability was good (α = .75).

The patient-level and session-level ICCs for all competence items and for the sum score of each competence scale, as well as the mean, standard deviation values and the range of the minimum/maximum ratings for each item are presented in Table 2. For the CRS-DBT-PTSD-R, also means and standard deviations of the experts’ ratings of relevance and appropriateness are included in Table 2. The interrater-reliability for the sum score of the CRS-DBT-PTSD-R was excellent on the patient-level with an ICC = .98 and moderate on the session-level with an ICC = .68.

Table 2.

Patient-level and session-level Intraclass Correlation Coefficient, Mean, Standard Deviation and Range of all Items for the Cognitive Therapy Scale (CTS), the Competence Rating Scale for PTSD (CRS-PTSD) and the Competence Rating Scale for Dialectical Behaviour Therapy for PTSD (Release Version) (CRS-DBT-PTSD-R).

				Relevanceb		Appropriatenessb
Item	ICC(A,1)	M (SD)	Min/Maxa	M	SD	M	SD
Therapeutic Competence Rating Scale for DBT-PTSD (Release Version) (CRS-DBT-PTSD-R)
1. Adequate development of skills with respect to the regulation of distress and emotionsSession-levelPatient-level	.54.94	3.32 (1.16)	0/5	2.50	0.71	1.50	0.71
2. Promotion of a mindful and benevolently supportive attitude of the patient towards him-/herself; promotion of a meta-cognitive level during treatmentSession-levelPatient-level	.60.97	3.48 (1.34)	1/6	2.50	0.71	2.00	0.00
3. Appropriate promotion of corrective emotional experiences in relation to the trauma and its consequencesSession-levelPatient-level	.06.89	3.75 (1.00)	1/5	3.00	0.00	3.00	0.00
4. Optimal activation of the trauma networkSession-levelPatient-level	.49.94	2.98 (1.08)	0/6	3.00	0.00	2.50	0.71
5. Focus on validation strategiesSession-levelPatient-level	.25.82	4.40 (0.99)	2/6	2.50	0.71	2.00	1.41
6. Adequate motivation of the patientSession-levelPatient-level	.42.95	3.38 (1.18)	1/6	3.00	0.00	2.00	1.41
Total CRS-DBT-PTSD-R scoreSession-levelPatient-level	.68.99	21.81 (5.51)	5/31	-	-	-	-
Cognitive Therapy Scale (CTS)
1. AgendaSession-levelPatient-level	.69.96	1.30 (1.09)	0/5	-	-	-	-
2. Dealing with problems/ questions/objectionsSession-levelPatient-level	.08.95	3.65 (1.00)	2/5	-	-	-	-
3. Clarity of communicationSession-levelPatient-level	.25.84	4.17 (1.03)	1/6	-	-	-	-
4. Pace and efficient use of timeSession-levelPatient-level	.26.95	2.72 (1.26)	0/5	-	-	-	-
5. Interpersonal effectiveness: empathySession-levelPatient-level	.37.94	4.43 (0.89)	3/6	-	-	-	-
6. Resource orientationSession-levelPatient-level	.03.95	3.92 (1.19)	2/6	-	-	-	-
7. Reviewing previously assigned homeworkSession-levelPatient-level	.92.97	2.12 (1.90)	0/5	-	-	-	-
8. Use of feedback and summariesSession-levelPatient-level	.07.84	3.22 (1.17)	0/6	-	-	-	-
9. Guided discoverySession-levelPatient-level	.14.93	3.27 (1.14)	0/5	-	-	-	-
10. Focus on the cognitive modelSession-levelPatient-level	.05.85	3.10 (1.12)	0/6	-	-	-	-
11. Focus on the behavioural modelSession-levelPatient-level	.59.92	3.68 (1.20)	0/6	-	-	-	-
12. RationaleSession-levelPatient-level	.31.92	3.58 (1.28)	0/6	-	-	-	-
13. Selection of appropriate strategiesSession-levelPatient-level	.60.94	3.65 (1.37)	0/5	-	-	-	-
14. Implementation of techniquesSession-levelPatient-level	.42.86	3.37 (1.03)	1/5	-	-	-	-
15. Homework settingSession-levelPatient-level	.36.91	2.00 (1.82)	0/5	-	-	-	-
Total CTS scoreSession-levelPatient-level	.72.99	49.64 (14.42)	6/74
Therapeutic Competence Rating Scale for PTSD (CRS-PTSD)
1. Managing patient emotionsSession-levelPatient-level	.05.86	3.53 (0.95)	1/5	-	-	-	-
2. Balance between change- and acceptance-oriented interventionsSession-levelPatient-level	.38.99	3.40 (1.23)	1/5	-	-	-	-
3. Interpersonal effectiveness: confidenceSession-levelPatient-level	.03.96	4.05 (0.93)	1/6	-	-	-	-
4. Identification and modification of avoidance behaviourSession-levelPatient-level	.47.96	3.30 (1.26)	0/6	-	-	-	-
5. Dealing with and using the therapist´s own emotionsSession-levelPatient-level	.06.96	4.05 (1.09)	2/6	-	-	-	-
6. Managing PTSD-specific symptomsSession-levelPatient-level	.30.93	3.47 (1.09)	1/5	-	-	-	-
7. Contingency managementSession-levelPatient-level	.52.94	3.27 (1.23)	0/6	-	-	-	-
Total CRS-PTSD scoreSession-levelPatient-level	.58.99	25.76 (6.74)	3/37	-	-	-	-

Note. ICC(A,1) = Intraclass correlation coefficients for both raters; Min = lowest score of the judges’ ratings on a scale from 0 to 6 and Max = highest score of the judges’ ratings on a scale from 0 to 6; Relevance and appropriateness were assessed on a scale ranging from 0 to 3.

On the patient level, rater concordance was also high for the individual items (All patient-level ICCs > .80). Session-level ICCs were moderate for items 1 and 2, the other items showed poor session-level interrater reliability.

The reliability of the CRS-DBT-PTSD-R was good on the session-level (αs = .78) as well as on the patient-level (α = .82).

The experts confirmed the relevance and appropriateness of all items of the CRS-DBT-PTSD-R (see Table 2). No missing items were named.

Psychometric properties of the CTS and CRS-PTSD

For the CTS, interrater-reliability of the total score was excellent on the patient-level with an ICC = .98 and moderate on the session-level with an ICC = .72. Reliability was good on session- and patient-level with αs = .71 and α = .75. For the CRS-PTSD total score, we found an excellent patient-level ICC = .99 and a moderate session-level ICC = .58, as well as good session-level (αs = .86) and patient-level (α = .86) reliability.

Associations between therapeutic adherence and competence

The repeated measures correlation analysis revealed large positive correlations between the sum score of the adherence scale and the three competence measures (see Table 3). Treatment-specific competence, measured by the CRS-DBT-PTSD-R, was also highly positively correlated with global CBT-competence and disorder-specific competence.

Table 3.

Repeated Measures Correlations between Therapeutic Competence and Therapeutic Adherence.

	Therapeutic Adherence	Therapeutic Competence
	ARS-DBT-PTSD-R	CTS	CRS-PTSD	CRS-DBT-PTSD-R
ARS-DBT-PTSD-R	–	.77***	.61***	.65***
CTS		–	.77***	.72***
CRS-PTSD			–	.82***
CRS-DBT-PTSD-R				–

Note. *** p < .001; ARS-DBT-PTSD-R = Adherence Rating Scale for Dialectical Behaviour Therapy for PTSD (Release Version); CTS = Cognitive Therapy Scale; CRS-PTSD = Competence Rating Scale for PTSD; CRS-DBT-PTSD-R = Competence Rating Scale for Dialectical Behaviour Therapy for PTSD (Release Version).

Discussion

The aim of the present study was to assess the psychometric properties of two newly developed rating scales for therapeutic adherence and competence with regard to DBT-PTSD, a treatment for survivors of CA suffering from both PTSD as well as difficulties in emotion regulation. Additional aims were to assess global CBT competence and treatment-specific therapeutic competence with already existing scales and to confirm their psychometric properties. At last, the association between the different adherence and competence measures was examined.

Our results indicate very high interrater-reliability on the patient level and good patient-level reliability for both newly developed scales ARS-DBT-PTSD-R and CRS-DBT-PTSD-R. On the session-level, interrater-reliability of the sum scores of both scales was moderate, as well as the reliability of the ARS-DBT-PTSD-R (< .70). One reason for this could be that only 2–4 sessions per patient were rated. According to Bonito et al. (2012), group sizes can at least in part influence the calculation of alpha. The CRS-DBT-PTSD-R showed good reliability on the session-level, as well as good content validity, indicating that all items are relevant and appropriate. DBT-PTSD-specific competence and adherence can therefore be reliably assessed in future studies using our scales.

Concerning the ratings of the individual items, the rater concordance on the patient level was also high. This indicates that also specific aspects of competence in and adherence to DBT-PTSD and their possible relation to treatment effect can be examined, as shown in previous studies for other treatments (Farmer et al., 2017; Ginzburg et al., 2012).

ICCs on the session level showed a greater range from poor to excellent interrater reliability, which is again, most likely due to the very small sample size (N = 2 to N = 4) on the session level. In future studies in which interrater reliability of adherence/competence scales is assessed with multilevel ICCs, more videotaped sessions per patient should be rated in order to produce more stable session level ICCs.

The assessment of the psychometric properties of already existing competence rating scales also showed high interrater reliability. Our results for the CTS were comparable with those of previous studies (e.g. Dittmann et al., 2017 reported an ICC of .97; Weck et al., 2010 an ICC of .90).

Furthermore, the results for the CRS-PTSD were similar to previous findings (Dittmann et al., 2017). In this previous study, the CRS-PTSD was also validated as part of the RELEASE study but for PTSD patients treated with CPT. The authors found an ICC of .97. Our comparable results indicate that the CRS-PTSD can reliably be used to assess PTSD-specific competence in different types of PTSD treatments. It should be noted that the raters in our study were well-trained in DBT-PTSD as they participated in workshops and had treated patients with DBT-PTSD before. Hence, we recommend that clinically inexperienced raters receive training before rating competence and adherence in PTSD treatment studies to become familiar with the treatment manuals.

As all scales were developed in German, only the German version is validated in this study. We provide an English translation in the supplement, that has not yet been validated, it cannot be ruled out that the translation may have an impact on the psychometric properties.

We furthermore analyzed the association between the different scales and found high correlations between the adherence to DBT-PTSD and all three competence measures. Also, global CBT competence, disorder-specific competence, and treatment-specific competence were highly interrelated in our study. Concerning the high correlations between competence and adherence measures, our results are in line with findings from previous studies (for example Gutermann et al. (2015) reported r = .65 and Ginzburg et al. (2012) r = .69). Thus, consistent with previous research, our results suggest a close connection of both constructs.

However, it is important to consider that adherence and competence cannot be seen as identical constructs (Barber et al., 2007). This becomes especially evident in studies examining the relationship between competence, adherence and treatment outcome, which suggest that adherence and competence function differently in explaining different treatment outcomes: For example, Hogue et al. (2008) found that adherence but not competence predicted treatment outcome. Other studies indicate that adherence and competence predict different symptom reductions (Marques et al., 2019). Importantly, findings concerning the association between adherence, competence and treatment outcome are very heterogenous and the relationship between them remains unclear: A meta-analyses assessing the effect of adherence and competence on symptom change in 36 studies found no effect neither in adherence nor in competence on treatment outcome (Webb, DeRubeis, & Barber, 2010). The authors point out that more research on the topic is necessary. Hence, our newly developed scales can be helpful instruments for further research concerning the effect of adherence and competence and PTSD symptom reduction.

A possible explanation for the high intercorrelations between adherence and competence in our study could be the result of the simultaneous rating of adherence and competence by the raters who therefore perceive them as highly related (Barber et al., 2007).

Our results also suggest a close connection between the different domains of competences (treatment-specific, PTSD-specific, and global CBT competence). Again, these findings are comparable to those of Dittmann et al. (2017), who found strong associations between adherence to CPT, PTSD-specific and treatment-specific competence to CPT (r = .62–.85). These high correlations suggest multi-collinearity between the scales and therefore raise the question if the competence scales measure different constructs at all. In part, the high correlations could be explained again with the simultaneous rating of all three competence types, but it can still not be ruled out that treatment-, disorder-, and general CBT competence are so closely interconnected that they are essentially the same construct. Further research on the differentiation of the different types of competence is necessary. For example, if the competence types would have differential effects on treatment outcome, as suggested by first empirical findings on the effect of treatment specific competence (Ginzburg et al., 2012), this could be a support for the differentiation of the competence types. Furthermore, it should be analyzed if the DBT-PTSD specific competence scale can differentiate between DBT-PTSD and other types of trauma treatment, as for example CPT, to further support the validity of the scale. For this kind of research, and to further extend literature in this field, the development and hence availability of new rating scales for specific competences is necessary (see also Barber et al., 2007). Our newly developed CRS-DBT-PTSD-R provides such a new instrument and hence the basis for further research.

Nevertheless, along with the CRS-PTSD and the CTS, our newly developed scale allows for a very fine-grained evaluation of different aspects of therapeutic competence as recommended by Barber et al. (2007). The analyses of the impact of different types of competences on PTSD treatment outcome might provide important information to understand therapeutic skills that are essential in the treatment of PTSD patients in general and of DBT-PTSD more specifically. Information gained through these analyses can be used to enhance the training of therapists working with PTSD patients.

Limitations

There are a few limitations to our study. Regarding the scales, it is important to consider that they only assess adherence and competence during treatment sessions and not during phone coaching and consultation team, which are both also parts of DBT-PTSD. However, we believe that the most important interventions are administered during sessions.

Additional limitations pertain to the study design. First, our study comprised a sample of only female patients with PTSD after CA. The generalizability of our results is therefore limited.

Second, all ratings included in the calculations of the psychometric properties of the scales were performed by the same two raters who each rated all videotapes of each patient with respect to all four measures of adherence and competence. During this procedure, the former ratings of the same patient e.g. the adherence ratings that were done before the competence ratings may have influenced the later ones. This procedure may have led to a shared variance and therefore contribute to a high intercorrelation between the different measures. A solution to overcome this limitation could be that different raters assess different scales in future studies. Related to this, the possible multi-collinearity of the three competence scales raises questions about the differentiation between the three competence types. As priorly discussed, further investigation on the differentiation between the different competence types should be carried out in future studies.

The third limitation is the independence of our raters (and experts). Both raters (C.D. and M.M.-E.) were involved in the treatment study and received similar training in DBT-PTSD. It is possible that this led to an increase in interrater agreement. Furthermore, raters regularly met to discuss their ratings in order to avoid systematic discrepancies in their ratings. Even though ratings were not altered afterwards, this approach could potentially inflate interrater agreement. As the DBT-PTSD is a newly developed treatment approach for PTSD, only a few experts in the field exist. It was therefore not possible to find independent raters with a high expertise, who were not involved in our study, to be able to rate treatment-specific competence in a satisfactory way. Also, the discussion of rated videotapes is an approach that has been used previously in other studies examining the reliability of adherence and competence measures (Weck et al., 2011b). In future studies it should be examined if raters with less involvement in DBT-PTSD and without regular meetings would achieve comparably high levels of interrater agreement.

Apart from these limitations, the results of this study indicate that the newly developed scales ARS-DBT-PTSD-R and CRS-DBT-PTSD show good psychometric properties and are therefore appropriate measures to assess treatment specific competence and adherence to DBT-PTSD.

Click here for additional data file.

Click here for additional data file.

Click here for additional data file.

Click here for additional data file.