Development of a patient-report measure of psychotherapy for depression.

Despite clear indications of need to improve depression treatment, practical tools that efficiently measure psychotherapy are not available. We developed a patient-report measure of psychotherapy for depression that assesses Cognitive Behavioral (CBT), Interpersonal (IPT), and Psychodynamic therapies. 420 patients with depression from a large managed behavioral health care organization completed the measure. The three subscales measuring CBT, IPT, and Psychodynamic Therapy showed good internal consistency, appropriate item-total correlations, and were supported by a 3-factor structure. Our results suggest that a patient questionnaire is a promising approach for assessing psychotherapy in quality improvement interventions.

Mental health care provided in most practice settings uses procedures that are different from the treatments found to be effective in research settings (Institute of Medicine 2001). Quality improvement interventions are effective at increasing rates of evidence-based depression care (Katon et al. 1995, 1996; Wells et al. 2000). Unfortunately, managed behavioral health care settings are unlikely to be able to engage in large scale quality improvement interventions for psychotherapy because no efficient, easily administered measures of the extent to which psychotherapy procedures provided are evidence-based are available. In this study, we present initial findings on the development of a patient-report measure of the extent to which therapists use procedures of Cognitive Behavioral Therapy (CBT), Interpersonal Therapy (IPT), or Psychodynamic Therapy.

Psychotherapy for depression is an appropriate area to begin development of a patient-report measure of psychotherapy procedures. Depression has a high prevalence and early age of onset, making depressive disorders a leading cause of disability worldwide (Murray and Lopez 1996; Lopez et al. 2006). Two forms of well-defined psychotherapy for depression, CBT and IPT, have been well-established as effective interventions for depression (USDHHS 1999). Yet many depressed patients do not receive evidence-based psychotherapy (Wang et al. 2000; Young et al. 2001). Quality improvements efforts greatly improve the outcomes for depression (Gilbody et al. 2003), yet are not feasible for psychotherapy without ways to measure quality care. An efficient measure of the extent to which psychotherapy procedures consistent with CBT or IPT are used could enable managed behavioral health organization to document the impact of quality improvement interventions for psychotherapy for depression.

Current methods to determine if psychotherapy is evidence based are not feasible for quality improvement efforts. Trained observers code audio or videotapes of sessions to determine adherence to a particular psychotherapy. Although not a measure of psychotherapy, caregiver reported adherence to principles of MST have proven to be reliable and valid predictors of child outcome in randomized trials, as well as usual care settings (Schoenwald 2008). In fact, caregiver reports of adherence to principles of MST have proved better predictors than therapist reports of adherence (Schoenwald 2008). In this study, we reviewed existing rating scales and psychotherapy literature to develop a patient self-report measure of Cognitive Behavioral, Interpersonal, and Psychodynamic Therapies that could easily be administered as part of a quality improvement effort. We include Cognitive Behavioral and Interpersonal Therapies because they are well validated forms of care. We include Psychodynamic Therapy because 25% of therapists in practice use this approach (Norcross et al. 2002). The goals of our study were to present preliminary evidence of the psychometric characteristics of our patient-report measure of psychotherapy for depression.

Methods

We sought to develop a patient-report instrument to describe three psychotherapy approaches used in the treatment of depression. The goals of the analyses presented here were to evaluate the (a) psychometric properties of the measure, (b) preliminary data on validity of the scale, and (c) propose a revision of the measure based on these initial evaluations.

Measures

Psychotherapy Techniques

The instrument, the Psychotherapy Practice Scale—Patient Depression Care Version (PPS Patient), was designed to provide a tool to describe the psychotherapeutic techniques used in the treatment of depression from the patient perspective. The instrument focuses on three therapeutic approaches: CBT, IPT, and Psychodynamic Therapy. CBT and IPT techniques were included because these approaches have strong empirical support for the treatment of major depression (Butler et al. 2006; Chambless et al. 1998; Chambless and Ollendick 2001; Frank and Spanier 2006). Psychodynamic Therapy has limited empirical support but was included given the prominence of this orientation in clinical training and practice.

Content and item development was guided by review of clinical literature on efficacy and essential therapy components and existing observation coding tools of adherence and competence (including rating scales used in the NIMH Treatment of Depression Collaborative Research Program). Generated items were reviewed in consultation with at least two clinical experts in each of the three therapeutic approaches to ensure inclusion of key therapeutic techniques essential to each type of therapy. Items were further refined by conducting cognitive interviews with 12 current psychotherapy patients who completed the measure.

The questionnaire includes 30 items to assess techniques that are key components of CBT (11 items), IPT (9 items), and Psychodynamic Therapy (11 items). Item content is provided in Table 1. The instrument asked the patient to rate the frequency that their therapist used a particular technique in the course of treatment on a 7-point scale ranging from ‘never’ (1) to ‘always’ (7). Items were presented in random order.

Table 1

Item—subscale correlations

	Items	CBT	IPT	Psycho-dynamic	% Missing	Variability
Cognitive behavioral therapy items
C_Q11	My therapist helped me to understand the beliefs behind my thinking. (For example, “I am a worthless person” or “If people get to know me, they won’t like me.”)	0.74a	0.70	0.64	3	1.42
C_Q13	My therapist helped me to create statements that I could use to respond to my negative thoughts	0.75a	0.68	0.62	2	1.65
C_Q21	My therapist asked me to do things that I enjoy doing between sessions	0.63a	0.58	0.49	4	1.56
C_Q22	My therapist helped me to develop a schedule of activities	0.63a	0.55	0.48	7	1.56
C_Q26	My therapist worked with me to come up with a plan for each session	0.71a	0.60	0.50	4	1.63
C_Q28	My therapist asked me to try specific things to see if my thoughts were true	0.70a	0.60	0.58	5	1.63
C_Q31	My therapist helped me to use evidence or facts to question my thoughts	0.74a	0.71	0.66	2	1.59
C_Q33	My therapist helped me to see mistakes in my thinking. (For example, all-or-nothing thinking or always fearing the worst will happen.)	0.73a	0.72	0.67	0	1.54
C_Q34	My therapist helped me to understand how my thoughts and feelings are related	0.75a	0.76	0.72	6	1.48
C_Q37	My therapist asked me to do “homework” between sessions. (“Homework” can include activities like writing something down or practicing something you learned during your session.)	0.61a	0.49	0.48	5	1.60
C_Q39	My therapist asked me about the kinds of thoughts I have when I am upset. (For example, “My spouse divorced me because I am a terrible person.”)	0.69a	0.68	0.65	5	1.58
Interpersonal psychotherapy items
I_Q10	My therapist and I discussed how good my relationships with other people are	0.57	0.63a	0.57	2	1.30
I_Q14	My therapist helped me to explore ways I could improve relationships with people in my life	0.67	0.73a	0.62	2	1.51
I_Q16	My therapist and I discussed how I might get along with other people in the future	0.72	0.81a	0.71	0	1.51
I_Q18	My therapist focused more on how I get along with people today than how I got along with people in the past	0.63	0.70a	0.56	2	1.53
I_Q19	My therapist and I discussed conversations that I had with other people	0.57	0.70a	0.63	1	1.54
Psychodynamic psychotherapy items
D_Q12	My therapist had me talk about issues as they came to my mind	0.59	0.57	.61a	2	1.52
D_Q15	My therapist helped me to talk about things in my childhood that make it difficult for me to talk about issues today	0.55	0.63	0.71a	1	1.64
D_Q17	My therapist and I explored the deeper meaning of my thoughts and expectations	0.73	0.74	0.72a	5	1.54
D_Q23	My therapist and I discussed how I felt about him/her	0.50	0.49	0.42a	10	1.48
D_Q27	My therapist allowed me enough time to explore topics or ideas that I was interested in	0.47	0.48	0.56a	0	1.33
D_Q29	My therapist helped me to talk about things in my childhood that were very painful	0.56	0.63	0.70a	0	1.63
D_Q35	My therapist and I explored my fears of being rejected by other people	0.66	0.71	0.63a	8	1.63
D_Q36	I chose the issues we discussed in my therapy	0.31	0.35	0.45a	8	1.22
I_Q20	My therapist helped me to explore different ways to make a decision	0.78	0.76a	0.71	0	1.59
I_Q24	My therapist helped me to understand how my problems might be related to my relationships with others	0.67	0.77a	0.70	4	1.52
I_Q25	My therapist and I explored how I could change my social activities	0.68	0.67a	0.55	1	1.45
I_Q30	My therapist and I discussed my ideas about how others should act	0.53	0.64a	0.61	2	1.54
I_Q32	My therapist asked me about the feelings I had when I was with other people	0.70	0.77a	0.65	1	1.48
I_Q38	My therapist and I discussed the good and bad parts of how I got along with others in the past	0.69	0.76a	0.71	6	1.49

aItems are associated with the original theoretical scale

Patient Demographic Variables

We used administrative data to determine age and gender for patients. Survey data measured patient education.

Clinician Demographic Variables

We used administrative data to determine age, gender, education level and years of clinical experience. Survey data measured ethnic/cultural identity.

Clinician Therapeutic Primary Orientation

Clinicians were asked to select their “main theoretical orientation” when treating depression. Options included Cognitive Behavioral, Interpersonal, psychoanalytic, psychodynamic, supportive, and other. Because many of our clinicians were eclectic, we developed a measure of the extent to which they practiced each of our three major orientations. Using clinician responses on the frequency with which they delivered CBT, IPT and Psychodynamic Therapy, we classified clinicians into low (<25% quartile), medium (25–75% quartile) and high (>75% quartile) categories on each of the three therapies.

Recruitment and Participants

We selected patients from OptumHealth Behavioral Solutions (formerly United Behavioral Health) who were recently diagnosed with major depression, who had at least three outpatient psychotherapy visits within the prior 6 weeks, and who had been treated by a high-volume network provider. A high-volume provider was defined as a clinician (MD, PhD, or MSW) who had treated 10 or more adult patients in the past year. We surveyed 2,417 eligible patients. Patients were sent a cover letter and consent form, patient survey instrument (described below), and an additional consent to allow us to contact their psychotherapist by mail. All patient participants received a $10 gift card. With patient permission, we then mailed survey packets (cover letter, consent form, signed copy of patient consent, and clinician survey instrument) to their treating clinician. All study procedures were approved by the RAND and UCLA Human Subjects Protection Committees.

Analyses

Examination of Psychometric Properties

We examined the internal consistency reliability (Cronbach’s alpha; Cronbach and Warrington,1951) with the goal of achieving at least an alpha of 0.7 for each of the three hypothesized scales. We also examined the correlation between hypothesized composites. We expected that the IPT and Psychodynamic Therapy composites would be most highly correlated given that IPT is derived from Psychodynamic Therapy. We expected the composites to have low to moderate intercorrelations. We examined item total scale correlations which allowed us to determine how highly an item correlated with its hypothesized scale and how that correlation related to the item’s correlation with other scales. To further determine the psychometric properties of each item, we examined the percent missing for the item and the variability of each item.

Examination of Factor Structure

Although we designed our patient report measure to have a three-factor structure, we were uncertain whether the measure would demonstrate this structure in a sample of patients seen by clinicians in practice who were predominantly eclectic in orientation. Thus, the use of exploratory factor analysis seemed the most appropriate analytic technique for this stage of measurement development research (Floyd and Widaman 1995; Reise et al. 2000; Weersing et al. 2002). Exploratory factor analyses using oblique (promax) rotation with varimax prerotation and squared multiple correlations as communality estimates priors were used to examine the initial composites and determine if the three therapeutic approaches would form three correlated subscales. The number of factors was determined using eigenvalues, adequate loadings (<0.40) on one factor with other loadings 0.20 lower on other factors, and the interpretability of the rotated factor pattern matrix.

Preliminary Validity Analyses

We examined whether clinician ratings of their own techniques predicted patient self-rated subscale scores for similar therapeutic orientations. For patient reported scales, we use subscores designed to measure CBT, IPT, and Psychodynamic Therapy. Individual items were standardized (z-scores with a mean of 0 and standard deviation of 1), to ensure that an item’s variability did not impact its weight in the subscore. Scores were transformed into T-scores, with a mean of 50 and standard deviation of 10 for ease of interpretation. We compared scores on each of the three patient-reported subscales with clinician-reported theoretical orientation (low, medium, high) using t-tests. To determine the effect of the three types of clinician reported therapy orientation on patient reported behavior, we used a multivariate analysis of variance (MANOVA) with clinician rated behavior (CBT, IPT, Psychodynamic Therapy) as independent variables and patient reported clinician behavior (CBT, IPT, Psychodynamic Therapy) as dependent variable.

Development of a Short-Form Measure

To determine whether it is possible to shorten the measure, we examined each for poor discrimination, negative impact on internal consistency, percentage of missing, and loading on the wrong factor.

Results

Participants

A total of 420 patients (17.4% response rate) and 159 high volume clinicians responded to the survey. Patients who responded were mostly adults between the age 35 and 54 (53.8%), female (74.2%), and received only psychotherapy (versus psychotherapy and medication management, 71.1% vs. 28.9%). Their clinicians were mostly master’s level therapists (62.4%), with largest percentage falling below age 30 (66.9%).

Evaluation of 30-Item Instrument

Cronbach’s alphas for each subscale were high, CBT = 0.93, IPT = 0.93 and Psychodynamic Therapy = 0.86 (Nunnally and Bernstein 1994). Scales were highly correlated with one another (r = 0.78–0.85). As predicted, the measures of the two dynamic therapies (IPT and Psychodynamic Therapy) were more highly correlated (r = 0.85) as compared with the correlation of CBT to IPT and Psychodynamic Therapy (r = 0.82 and 0.78, respectively).

Psychometric properties evaluated included item-subscale correlations, percentage of missing for each item, and variability of the item (Table 1). Results showed appropriate primary correlations (e.g., CBT technique items correlated most highly with the CBT subscale, etc.), as well as moderate secondary correlations between the subscales. Although, correlations were fairly high among all subscales, the lowest correlation was observed between CBT and Psychodynamic Therapy items. However, the highest correlation was observed for CBT and IPT items, rather than IPT and Psychodynamic Therapy items, as originally hypothesized. The Psychodynamic Therapy subscale had larger percentages of missing items than did the CBT and IPT subscales. Variability of items was relatively similar across scales.

Factor Structure

Table 2 presents the factor loadings, eigenvalues, and percentage of variance for each factor. There were two eigenvalues over one (14.74, 1.22, 0.91, 0.65, 0.54 and so on) with the third eigenvalue approaching one (0.91). When items were assigned to a factor that had standardized regression coefficients of at least 0.40, the three factor solution was interpretable. The three rotated factors accounted for 61% of the total variance. The first factor accounted for 81% of the proportion of total variance and the seven items that loaded on that factor appeared to tap a “structured, behavioral orientation,” most consistent with CBT techniques. However, it did include one item about encouraging social activities, which was originally developed as an IPT item. The second factor accounted for 7% of the proportion of total variance and seven items loaded on this factor. These items were originally Psychodynamic Therapy and IPT items and appeared to reflect a focus on past childhood experiences as well as an examination of relationships and problems, which we called “insight about relationships”. The third factor accounted for 5% of the proportion of total variance and 5 items that reflect a “general talking” factor, where patients had control over their session. Most of these items were from the Psychodynamic Therapy scale, but an item from each of the other two scales associated with “talking” loaded on this factor. There were also items that loaded on more than one factor. Items that loaded on to more than one factor were considered nonspecific with low discrimination, as listed in Table 2. Thus, three factor solution was not necessarily consistent with the original hypothesized orientations, instead they reflected (1) structured specific behavioral factor (most similar to CBT), (2) insight oriented factor (consistent with both Psyhodynamic Therapy and IPT), and (3) unstructured talking factor.

Table 2

Rotated Factor Loadings from a Principal Axis Factor Analysis of Items from the PPS (N = 420)

		Factors
		1	2	3
Items
C_Q26	Come up with a plan for each session	0.75	0.25	0.12
C_Q22	Develop a schedule of activities	0.70	0.25	0.08
C_Q28	Try specific things to see if my thoughts were true	0.62	0.32	0.23
C_Q13	Respond to my negative thoughts	0.61	0.18	0. 50
I_Q25	Explored how I could change my social activities	0.61	0.35	0.23
C_Q21	Do things that I enjoy doing between sessions	0.60	0.11	0.36
C_Q37	Do “homework” between sessions	0.57	0.17	0.24
D_Q29	Talk about things in my childhood that were very painful	0.17	0.70	0.31
I_Q30	Discussed my ideas about how others should act	0.23	0.67	0.22
I_Q38	Discussed the good and bad parts of how I got along with others	0.39	0.63	0.33
D_Q15	Talk about things in my childhood that make it difficult for me to talk about issues today	0.16	0.62	0.39
I_Q32	Feelings I had when I was with other people	0.44	0.60	0.30
I_Q24	Understand how my problems might be related to my relationships with others	0.40	0.57	0.37
D_Q35	Explored my fears of being rejected by other people	0.34	0.55	0.38
I_Q16	Get along with other people in the future	0.45	0.53	0.44
D_P12	Talk about issues as they came to my mind	0.28	0.21	0.62
D_Q27	Explore topics or ideas that I was interested in	0.12	0.23	0.60
C_Q34	Understand how my thoughts and feelings are related	0.43	0.38	0.60
I_Q10	Discussed relationships with other people	0.32	0.30	0.51
D_Q36	Chose the issues we discussed in my therapy	0.01	0.21	0.50
Nonspecific items
C_Pll	Understand the beliefs behind my thinking	0.49	0.21	0.62
I_Q18	Get along with other people today	0.44	0.40	0.35
C_Q31	Use evidence or facts to question my thoughts	0.51	0.52	0.30
D_Q23	Discussed how I felt about the therapist	0.40	0.41	0.07
D_Q17	Explored the deeper meaning of my thoughts	0.38	0.43	0.58
I_Q14	Explore ways I could improve relationships	0.44	0.34	0.51
I_Q19	Conversations that I had with other people	0.24	0.48	0.49
I_Q20	Explore different ways to make a decision	0.53	0.36	0.54
C_Q33	See mistakes in my thinking	0.45	0.42	0.48
C_Q39	Kinds of thoughts I have when I am upset	0.43	0.40	0.44
Eigen value		14.74	1.22	0.91
% of Variance		81	7	5

Bold values indicate eigen values above .50

Preliminary Evaluation of Validity

A multivariate analysis of variance (MANOVA) was conducted to determine the effect of the three types of clinician reported therapy orientation (CBT, IPT, and Psychodynamic Therapy) on three dependent variables (patient reported CBT, IPT or Psychodynamic Therapy). Clinician reported therapy orientation was categorized into low, medium, and high for each orientation. Significant differences were found among the clinician reported therapy subscale on the patient reported dynamic subscale (Wilks’ λ = 0.91, F (24,159) = 2.12, P < 0.05), suggesting the clinicians who self rated their therapy as higher on the Psychodynamic subscale were significantly more likely to have patients rate them high on the Psychodynamic subscale as well. Whereas clinician rated subscales of IPT and CBT were not as predictive of patient ratings on the therapy subscales.

Univariate analyses of variance (ANOVAs) for each dependent variable were conducted as follow-up tests to the MANOVA. The ANOVAs of the patient reported Psychodynamic Therapy (F(2,159) = 4.87, P < 0.01) and CBT (F(2,159) = 3.86, P = 0.023) were significant, whereas the Patient Reported IPT was nonsignificant (F(2,159) = 0.87, P = 0.41). Although main effect was not found for IPT, the level of IPT techniques endorsed by clinicians corresponds to the mean patient rated IPT scores, suggesting a similar trend. Table 3 presents the means and standard deviations of the patient reported therapy orientation for the three levels of each clinician reported therapy orientation (CBT, IPT, Psychodynamic Therapy) groups.

Table 3

Patient mean scores (30 Items) by clinician (Low–Medium–High) behavioral orientation

	Clinician—CBT behavior				Clinician—IPT behavior				Clinician—psychodynamic behavior
	Low N = 46	Medium N = 73	High N = 40	P-value	Low N = 47	Medium N = 67	High N = 45	P-value	Low N = 47	Medium N = 74	High N = 38	P-value
PT Mean CBT	3.51(1.11)	3.77(1.04)	4.13(0.93)	0.02*	3.61(1.12)	3.87(1.08)	3.83(0.95)	0.38	3.63(1.03)	3.82(1.10)	3.90(1.00)	0.48
PT Mean IPT	3.82(1.16)	3.85(1.01)	4.22(1.18)	0.17	3.74(1.13)	3.94(1.11)	4.12(1.04)	0.27	3.76(1.04)	3.98(1.05)	4.06(1.26)	0.42
PT Mean DYN	3.91(1.11)	3.91(0.97)	4.17(1.00)	0.37	3.74(1.14)	4.07(1.00)	4.08(0.87)	0.17	3.67(1.03)	3.98(0.96)	4.35(1.03)	0.01**

* P < .05; ** P < .01

Shortened Version

Based on the psychometric properties of the full scale (see Table 4), we considered a shortened version of this patient self-report measure. We retained items that had best psychometric properties for its hypothesized scale for the shortened version, which resulted in 15 items assessing CBT (6 items), IPT (6 items), and Psychodynamic Therapy (3 items). These items are in bold in Table 4.

Table 4

Evaluation of 30-item version

	Item text	Poorer discriminationa	Negative impact on internal consistencyb	Higher % missing	Loaded on wrong scale
	In thinking of your sessions with your therapist, HOW OFTEN:
CBT
C_Q11	Understand the beliefs behind my thinking	✓			✓
C_Q13	Respond to my negative thoughts
C_Q21	Do things that I enjoy doing between sessions
C_Q22	Develop a schedule of activities
C_Q26	Come up with a plan for each session
CC_Q28	Try specific things to see if my thoughts were true
C_Q31	Use evidence or facts to question my thoughts	✓
C_Q33	See mistakes in my thinking	✓
C_Q34	Understand how my thoughts and feelings are related		✓		✓
C_Q37	Do “homework” between sessions
C_Q39	Kinds of thoughts I have when I am upset	✓	✓
IPT
I_Q10	Discussed relationships with other people				✓
I_Q14	Explore ways I could improve relationships	✓			✓
I_Q16	Get along with other people in the future
I_Q18	Get along with other people today	✓
I_Q19	Conversations that I had with other people	✓
I_Q20	Explore different ways to make a decision	✓			✓
I_Q24	Understand how my problems might be related to my relationships with others
I_Q25	Explored how I could change my social activities
I_Q30	Discussed my ideas about how others should act
I_Q32	Feelings I had when I was with other people
I_Q38	Discussed the good and bad parts of how I got along with others
DYN
D_Q12	Talk about issues as they came to my mind
D_Q15	Talk about things in my childhood that make it difficult for me to talk about issues today				✓
D_Q17	Explored the deeper meaning of my thoughts	✓			✓
D_Q23	Discussed how I felt about the therapist	✓		✓
D_Q27	Explore topics or ideas that I was interested in
D_Q29	Talk about things in my childhood that were very painful				✓
D_Q35	Explored my fears of being rejected by other people
D_Q36	Chose the issues we discussed in my therapy			✓

Stronger items are bolded

aCorrelation with its own scale was not at least two standard errors higher than its correlation with other scales in the 30-item matrix

bCronbach’s alpha would increase if item were removed in 30-item model

The shortened instrument demonstrated good psychometric properties and an increased differentiation between factors. We conducted exploratory factor analyses using oblique (promax) rotation with varimax prerotation and squared multiple correlations as communality estimates priors to determine the initial composites (see Table 5). The number of factors was determined using eigenvalues and the interpretability of the rotated factor pattern matrix. There was only one eigenvalue over one (6.85, 0.93, 0.63, 0.19 so on) with the second eigenvalue approaching one (0.93); the average eigenvalue was 0.52. When items were assigned to a factor that had standardized regression coefficients of at least 0.30, the three factor solution was interpretable and all items loaded highly on the corresponding three therapy approaches.

Table 5

Rotated factor loadings from a principal axis factor analysis of 15 item version from the PPS (N = 540)

	Items	Factors
		1	2	3
C_Q26	Come up with a plan for each session	0.75	0.29	0.05
C_Q22	Develop a schedule of activities	0.67	0.27	0.08
C_Q28	Try specific things to see if my thoughts were true	0.63	0.30	0.32
I_Q13	Respond to my negative thoughts	0.62	0.18	0.18
C_Q25	Explored how I could change my social activities	0.61	0.33	0.21
C_Q21	Do things that I enjoy doing between sessions	0.60	0.24	0.23
I_Q32	Feelings I had when I was with other people	0.37	0.69	0.19
I_Q38	Discussed the good and bad parts of how I got along with others	0.40	0.67	0.23
I_Q30	Discussed my ideas about how others should act	0.19	0.67	0.20
I_Q16	Get along with other people in the future	0.44	0.64	0.28
I_Q24	Understand how my problems might be related to my relationships with others	0.40	0.63	0.25
I_Q19	Conversations that I had with other people	0.23	0.60	0.38
D_Q27	Explore topics or ideas that I was interested in	0.20	0.24	0.67
D_Q36	Chose the issues we discussed in my therapy	0.06	0.18	0.64
D_Q12	Talk about issues as they came to my mind	0.37	0.26	0.57
Eigen value		6.85	0.93	0.63
% of Variance		87	12	8

Bold values indicate eigen values above .50

Discussion

The psychometric and validity analyses of our Psychotherapy Practice Scale—Patient Depression Care Version offer evidence that patients are able to identify psychotherapeutic techniques and that developing a short patient measure of evidence-based care is promising. With further development and testing, this scale measuring the extent to which psychotherapy is evidence-based could be a vital tool in improving quality of psychotherapy in managed mental health care settings. This tool could be a cost effective method for identifying quality care and measuring improvements in providing evidence-based psychotherapy for depression.

This initial study is a solid first step towards developing a brief measure of the extent to which depression care is evidence-based. To develop this measure further, it should be used within ongoing studies of CBT, IPT and Psychodynamic Therapy where coded videotape validation is available to validate the measure. Alternative report formats might also be considered, including forced choose options, which might improve the reliability and validity of the measure. With clear validation, the measure could then be used to better measure care as usual in practice settings, as well as serve as a measure of evidence-based care in quality improvement interventions.

Although this first study of a patient measure of evidence-based care for depression provides useful information, several limitations should be noted. First, only 17% of our patient sample participated in this study. This number is quite typical of survey response rates in managed health care studies, but nonetheless raises questions about the representative nature of our sample. Second, providers in managed behavioral health care centers tend to be eclectic in their approach to psychotherapy. This was true of our clinicians. This prohibited us from being confident that our findings regarding the shortened scale would hold in settings where more rigorous adherence to a particular therapy occurs. Third, while our measure should evaluate if treatment is more consistent with evidence-based care, without including assessment of the intensity and sequencing of intervention approaches, the measure would not be an indication of the fidelity of care to a particular model. Finally, the scales in this study were highly related. The extent to which this represents an accurate representation of eclectic therapy as opposed to a measurement problem within the scale cannot be determined.

Despite limitations, this is the first study to our knowledge to propose a short patient-administered measured of evidence-based care for depression. A short patient-administered measure is particularly appealing because unlike providers, patients would not inherently have a bias towards reporting about evidence-based care as would clinicians who may feel they “should” be providing care according to guidelines. With further validation, this patient measure of evidence-based depression care could be a very promising tool for measuring and improving quality of psychotherapy.