Development of a clinician report measure to assess psychotherapy for depression in usual care settings.

Although mental health policy initiatives have called for quality improvement in depression care, practical tools to describe the quality of psychotherapy for depression are not available. We developed a clinician-report measure of adherence to three types of psychotherapy for depression-cognitive behavioral therapy, interpersonal therapy, and psychodynamic therapy. A total of 727 clinicians from a large, national managed behavioral health care organization responded to a mail survey. The measure demonstrated good psychometric properties, including appropriate item-scale correlations, internal consistency reliability, and a three-factor structure. Our results suggest that this questionnaire may be a promising approach to describing psychotherapy for depression in usual care.

Introduction

Recent mental health policy initiatives have called for quality improvement interventions to ensure that evidence-based care is practiced (New Freedom Commission on Mental Health 2003). Quality improvement efforts in psychotherapy have been hampered because there are no easily obtained measures of the extent to which psychotherapy is evidence-based. In this paper, we present preliminary findings regarding a short, self-report measure assessing therapist adherence to three types of psychotherapies for depression.

We focus on depression because it is a prevalent and debilitating disorder, with evidence-based psychotherapy treatments. Lifetime prevalence of major depression is 16.6%, making it the single most prevalent psychological disorder (Kessler et al. 2005). According to World Health Organization estimates, by the year 2020, major depression will become the second leading cause of disability worldwide (World Health Organization 2001). Depression costs employers about $44 billion a year, primarily due to loss of productivity at work and lack of high quality depression care (Stewart et al. 2003). Further, patients with affective disorders have a high degree of morbidity and low health utility compared to most chronic health conditions (Wells and Sherbourne 1999). While antidepressant treatment is an efficacious option, most patients prefer psychotherapy (Alvidrez and Azocar 1999; Dwight-Johnson et al. 2000). Given the prevalence and burden of depression, it is likely the most common condition seen by psychotherapists, thus making depression an important target.

Substantial empirical support exists for the efficacy of two forms of psychotherapy for treating depression in adults: cognitive behavioral therapy (CBT) and interpersonal therapy (IPT; Chambless et al. 1998). Previous works suggests that there has been a lag between discovering efficacious forms of treatment and incorporating them into routine patient care (Institute of Medicine 2001). Efforts to improve the quality of psychotherapy for depression rely on the ability to measure the extent to which either CBT or IPT are practiced.

One indication of how a therapist might treat depression that could readily be available to potential patients or a behavioral health plan hoping to improve the quality of depression care is the therapist’s therapeutic or theoretical orientation. Unfortunately, endorsing a therapeutic orientation (e.g., cognitive-behavioral) may not translate into implementing key components of the model when practicing psychotherapy. While roughly a quarter of psychologists identify themselves as cognitive-behavioral and another quarter as psychoanalytic/psychodynamic, a full one third of therapists identify themselves as eclectic/integrative—indicating their use of techniques from more than 1 orientation (Norcross et al. 2002). National data on therapeutic orientation across disciplines are not available. There is also evidence of striking discrepancies between theoretical belief and clinical practice (Buckley et al. 1979). Further, therapists may endorse a particular therapeutic orientation (e.g., cognitive-behavioral) to increase their appeal when applying to join managed care panels who want therapists practicing evidence-based care. The availability of information about a therapist’s actual therapeutic practice would enable quality improvement efforts to identify the extent to which a particular psychotherapy is practiced.

Current clinical research measures of the extent to which a therapy follows an evidence-based practice are not feasible for quality improvement efforts in mental health settings. These approaches have relied on trained observers to code audio or videotapes of sessions to determine adherence to a particular therapy model. Several process rating scales have been developed to assess adherence to a treatment model. For example, the Cognitive Therapy Scale (Dobson et al. 1985) is a checklist that evaluates the extent to which therapists adhere to CBT and requires administration by trained professionals listening to tapes of sessions. In this study, we reviewed existing ratings scales and psychotherapy literature to develop a clinician self-report measure that could easily be administered as part of a quality assessment and improvement effort. This study represents a first step toward a clinician-reported instrument that could prove useful in describing the extent to which therapists adhere to CBT, IPT or psychodynamic therapy. We developed a clinician-report psychotherapy questionnaire that assesses the frequency that the clinician used cognitive-behavioral (CBT), interpersonal (IPT), and psychodynamic (DT) techniques in treating a particular patient with depression. The goals of the following analyses were to reduce the length of the pilot questionnaire and begin to understand the validity of the hypothesized subscales.

Methods

We sought to develop a clinician-report instrument to describe three psychotherapy approaches used in the treatment of depression. To develop the item content for the pilot questionnaire, we reviewed relevant literature on the essential components of three therapy approaches: cognitive behavioral, psychodynamic, and interpersonal. To evaluate the questionnaire, we analyzed data collected from clinicians associated with a large managed behavioral health care organization. The goals of the analyses presented here were to refine the questionnaire and to begin to understand the validity of the items.

Measure

The instrument, the Psychotherapy Practice Scale—Clinician Depression Care Version (PPS Clinician), was designed to provide a tool to describe the psychotherapeutic techniques used in the treatment of depression. The instrument focuses on three therapeutic approaches: CBT, IPT, and DT. CBT and IPT techniques were included because these approaches have strong empirical support for the treatment of major depression (Butler et al. 2006; Chambless et al. 1998; Chambless and Ollendick 2001; Weissman et al. 2000). While DT is inherently difficult to subject to randomized controlled trials and thus has limited empirical support (Tanenbaum 2005), a recent survey indicated that roughly a quarter of psychologists identify themselves as psychoanalytic/psychodynamic (Norcross et al. 2002), providing support for including dynamic therapy as well.. Content coverage and item development was guided by review of clinical literature on efficacy and essential therapy components (Barlow 2007; Beck 1995; Jacobson et al. 1996; Klerman et al. 1984), existing observation coding tools of adherence and competence (Barber et al. 2003; Dobson et al. 1985; Hill et al. 1992; Weersing et al. 2002), and consultation with clinical experts to determine the key therapeutic techniques essential to each type of therapy when treating depression. Items often include clinical terminology that would be easily recognizable by clinicians who use a particular technique. We also developed a companion patient report instrument (not reported here) that removed clinical terminology and focused on concrete, observable clinician behaviors (Miranda et al. 2010).

The instrument assesses how frequently a therapist uses specific techniques in the course of treating a patient. The full version of the questionnaire included 28 items to assess techniques that are key components of CBT (8 items), DT (11 items), and IPT (9 items). Items are listed in Table 2, while the full formatted instrument with instructions is available from the first author. The instrument asked the clinician to select a single, recently treated adult patient diagnosed with depression. Clinicians were asked to rate the frequency that they used each technique with that patient on a 7-point scale ranging from ‘never’ (1) to ‘always’ (7). In an effort to minimize patterned responding or endorsing only items from a particular orientation, technique items are not labeled or grouped by therapy approach. An additional question asked whether the clinician’s approach in treating this patient was typical of the approach used with other depressed patients.

Table 2

Evaluation of 28 item version

	Item texta In thinking of your sessions with this patient, HOW OFTEN:	Item-total correlationb	Poorer discriminationc	Negative impact on internal consistencyd	Higher % missing	Lower variability (SD)
CBT
Q2	Were you directive or instructive?	0.39			✓
Q6	Did you encourage the patient to notice connection between their thoughts and feelings?	0.49
Q7	Did you help this patient understand the beliefs and assumptions behind their thinking (e.g., core beliefs, cognitive schemas)?	0.59
Q11	Did you help this patient understand which thought are helpful and which thoughts are not (e.g., explain the cognitive triad, identify negative thinking)?	0.70
Q21	Did you assign “homework” between sessions (e.g., asked patient to complete Mood Rating Scale or record of thoughts, feelings, or activities)?	0.63
Q23	Did you ask this patient to do things they enjoyed doing between sessions (e.g., behavioral activation, increasing pleasurable activities, use of pleasure ratings)	0.58
Q26	Did you help this patient with activity monitoring and scheduling (e.g., completing an activity monitoring chart, recording activity pleasure ratings, developing an activity schedule)?	0.59
Q27	Did you help this patient create statements they could use to respond to negative thoughts (e.g., practicing rational responses, using reattribution or alternative reasoning)?	0.70
DT
Q1	Did you allow time for this patient to explore topics or ideas that were of interest to them?	0.35		✓		✓
Q3	Did you encourage this patient to choose the issues discussed in therapy?	0.33		✓		✓
Q5	Did you discuss this patient’s feelings toward you (e.g., transference)?	0.53
Q8	Did you encourage this patient to talk about things in their childhood that made it difficult to discuss present-day issues?	0.70
Q10	Did you explore this patient’s fears of being rejected by other people?	0.51	✓
Q14	Did you help this patient become aware of reactions that are defense mechanisms (e.g., repression, projection)?	0.61
Q17	Did you incorporate interpretation of dreams and fantasies?	0.58
Q18	Did you encourage this patient to talk about issues as they came to mind?	0.53
Q19	Did you consider countertransferential issues in developing your understanding of this patient (e.g., your feelings toward this patient that might be relevant to your sessions?	0.60
Q20	Did you explore the deeper emotional meaning of this patient’s concerns or behaviors (e.g., subconscious motives)?	0.67
Q28	Did you discuss developmental issues with the patient (e.g., relevant positive or negative childhood events)?	0.66
IPT
Q4	Did you explore possible changes that could be made in interpersonal relationships and social activities?	0.58				✓
Q9	Did you place more emphasis on getting along with people today than how this patient got along with people in the past?	0.31	✓	✓
Q12	Did you discuss specific conversations that this patient had with other people?	0.40
Q13	Did you encourage this patient to adopt the “sick role” (i.e., excuse patient from usual social roles and instead focus on working to recover their health)?	0.05	✓	✓	✓	✓
Q15	Did you relate depressive symptoms to grief, role disputes, role transitions, or interpersonal deficits?	0.45	✓		✓
Q16	Did you help the patient to understand that addressing interpersonal situations may help to improve their depression?	0.64	✓
Q22	Did you assess the positive and negative aspects of how this patient got along with others in the past (i.e., a prior social role, dysfunctional patterns, depth of intimacy in previous relationships)?	0.54
Q24	Did you discuss the current quality of patient’s relationships with other people?	0.57			✓
Q25	Did you examine the emotional response this patient had during interpersonal interactions?	0.58

aQuestion numbers are associated with the original fielded survey. Stronger items are bolded

bPearson product-moment correlations between the item and the scale score corrected for overlap

cCorrelation with its own scale was not at least two standard errors higher than its correlation with other scales in the 28-item matrix

dCronbach’s alpha would increase if item were removed in 28-item model

Clinicians were also asked to select their “main theoretical orientation” when treating depression. Options included cognitive behavioral, interpersonal, psychoanalytic, psychodynamic, supportive, and other. Some clinicians selected multiple orientations, so we created a “primary orientation” variable where clinicians could fall into only one category. We created “Multiple Orientations/Eclectic” to reflect clinicians who selected more than one option. Due to low frequencies, we also combined the psychoanalytic and psychodynamic orientation into a single dynamic orientation and combined supportive with other. Clinicians also reported the number of patients with depression they had treated with at least five sessions of psychotherapy in the past 6 months.

Recruitment and Respondents

A total of 2,200 “high volume” clinicians (600 Psychiatrists, 600 Psychologists and 1,000 Masters-level therapists) that had treated at least ten adult patients in the past year were randomly selected from the clinical panel of United Behavioral Health (UBH), a large, national managed behavioral health organization (since renamed OptumHealth Behavioral Solutions). Clinicians received a packet containing an introductory letter, 1-page study description, informed consent form, and questionnaire via standard mail. Clinicians were offered an educational CD-ROM allowing them to earn 1 h of continuing education credit for returning the questionnaire. Non-respondents received an identical follow-up mailing approximately 2 weeks after the initial mailing. The study was approved by the RAND Institutional Review Board.

A total of 687 clinicians returned the questionnaire by mail for a raw response rate of 31%. To determine an adjusted response rate, we removed 21 sampled clinicians who were ineligible because they no longer practiced, did not treat adult clients, or had died; therefore, the adjusted response rate was 32% (687/2,179). In addition to the mailing to the sampled clinicians, electronic copies of all materials were posted on UBH’s clinician website, allowing any clinician in their network to participate. An additional 40 clinicians completed the questionnaire by obtaining materials through the website. While online respondents reported fewer years of clinical practice (12.6 vs. 16.1, P < 0.05) and a higher proportion of master’s degrees (100 vs. 42%, P < 0.001) compared to mail respondents, there were no demographic differences in gender, age, or ethnicity (Caucasian). Based on these analyses, we determined that the online sample was sufficiently comparable to be combined with the mail sample. The combined sample included a total of 727 respondents.

Comparisons of demographic characteristics for responders (both mail and online, n = 727) and non responders (n = 1,492) yielded some differences (Table 1). Responders were more likely to be older, Caucasian, female, and have a master’s degree. There was no difference in number of years of clinical practice. The pattern of these results was the same when the online responders were excluded.

Table 1

Characteristics of responders vs. non-responders

	Responders (n = 727)	Non-responders (n = 1,492)
Female	58.5% (425)*	50.2% (749)
Caucasian	80.2% (583)*	66.3% (989)
Education
Masters	56.1% (408)*	37.2% (556)
MD	7.0% (51)	35.0% (522)
PhD	36.9% (268)	27.8% (414)
Age
Mean (SD)	56.0 (8.4)*	54.0 (9.3)
Years practice
Mean (SD)	16.1 (8.3)	16.1 (9.4)

Respondents include both mail and online respondents

* P < .001

Responding clinicians reported seeing an average of 24 patients (SD = 15.4) for at least five psychotherapy sessions in the prior 6 months. Clinicians reported a range of primary theoretical orientations. Over half endorsed cognitive-behavioral (51%), followed by eclectic/multiple orientations (21%), psychodynamic/psychoanalytic (12%), other (10%), and interpersonal (7%). The majority of clinicians (95%) reported that the approach used with the selected patient was the approach that they typically used with their depressed patients.

Analyses

Approach to Refining the Instrument

We conducted analyses to determine the psychometric properties of the instrument and inform instrument revisions. We believed that items assessing the three therapeutic approaches would form three subscales (or multi-item composites). We conducted exploratory factor analyses using oblique (promax) rotation with varimax prerotation and squared multiple correlations as communality estimates priors to determine the initial composites. The number of factors was determined using eigenvalues and the interpretability of the rotated factor pattern matrix. There were three eigenvalues over one (7.25, 2.98, 1.13, 0.78, 0.57, 0.46, and so on) with an average eigenvalue of 0.44. When items were assigned to a factor that had standardized regression coefficients of at least 0.30, the three factor solution was interpretable and represented the three therapy approaches. There were some notable secondary loadings, but we proceeded with the hypothesized subscales.

We incorporated several pieces of information to determine recommendations for item deletion including internal consistency reliability, corrected item total correlations, item discrimination, subscale intercorrelations, item missing, and limited observed variability. Specifically, we evaluated internal consistency reliability using Cronbach’s alpha, with the goal of achieving at least an alpha of .7 for each of the 3 hypothesized scales (Cronbach and Warrington 1951). Each item’s impact on internal consistency estimates was also noted. We also examined the relationships among the items using a multi-trait, multi-item correlation matrix, which allowed us to determine how highly an item correlated with its hypothesized scale and how that correlation related to the item’s correlation with other scales. We also examined the correlation between hypothesized composites. We expected that the IPT and DT composites would be most highly correlated given that IPT was developed using some concepts from DT. We also expected that the CBT and DT composites would have the lowest correlation given that these two therapies have quite distinct therapeutic approaches. Although we expected the composites to have low to moderate intercorrelations, we believed that each of the composites would provide unique information. We also noted whether an item had higher rate of missing or lower standard deviation than other items.

Approach to Evaluating the Shortened Instrument

The shortened instrument was re-evaluated using the analytic approach describe above, with 1 addition. We also conducted confirmatory factor analyses to evaluate the factor structure of the instrument. We anticipated that the items represented three distinct, but correlated factors of CBT, IPT, and DT. In addition to testing a 3-factor model, we evaluated competing 1- and 2-factor models to determine whether the 3-factor model provided a significant increase in model fit over the competing, more parsimonious models. A well-fitting 1-factor model would suggest that the items do not assess distinct types of therapy, but instead reflect a general approach to treating depression. The 2-factor model was specified so that the IPT and DT items loaded on the first factor (given the conceptual relationships between these therapies), and the CBT items loaded on the second factor. If this model provided a superior fit, it would suggest that the CBT items were distinct, but the IPT and DT items demonstrated a great degree of overlap. A priori, we favored the 3-factor model as conceptually superior and supporting the clinical literature that indicates these to be distinct therapies. To judge model fit, we used CFI (comparative fix index) and RMSEA (root-mean-square error of approximation) (Loehlin 1998). These analyses were conducted in SAS and Mplus (Muthen and Muthen 1998–2004).

In summary, to determine the items retained in the shortened instrument, we evaluated internal consistency reliability, corrected item total correlations, item discrimination, subscale intercorrelations, item missing, observed variability and factor models to select a subset of items that best represented each of the three therapy approaches.

Preliminary Validity Analyses

We created subscores designed to reflect CBT, DT, and IPT. Individual items were standardized (z-scores with a mean of 0 and standard deviation of 1), to ensure that an item’s variability did not impact its weight in the subscore, and then averaged. Scores were transformed into T-scores, with a mean of 50 and standard deviation of 10 for ease of interpretation. We compared scores on each of the three subscales by clinician-reported primary theoretical orientation using t-tests. In addition, we evaluated the correlation of each subscore with the number of years the clinician had been practicing.

Results

Evaluation of Fielded 28-Item Instrument

Table 2 provides a summary of key quantitative and qualitative results for the fielded 28-item instrument. This evidence guided the selection of a subset of 16 better performing items.

Confirmation of the Shortened Composite Structure

Based on the findings from the pilot test, we revised and shortened the PPS-Clinician. The shortened version presented here included 16 items assessing CBT (6 items), DT (6 items), and IPT (4 items). These items are in bold in Table 2.

The shortened instrument demonstrated good psychometric properties. The multi-trait, multi-item correlation matrix showed appropriate primary correlations (e.g., CBT technique items correlated most highly with the CBT subscale), although some moderate secondary correlations between IPT and DT items were observed (Table 3). Internal consistency reliability was acceptable (above 0.7) for each subscale (Table 4). Further, the correlations among the subscales supported the a priori hypotheses about the relationship among the different therapeutic approaches (Table 4). That is, the highest correlation was observed between DT and IPT, while the lowest correlation was observed between CBT and DT.

Table 3

Multi-trait multi-item correlation matrix

		CBT	DT	IPT
Q7	Behind thinking	0.51	0.23	0.34
Q11	Helpful thoughts	0.67	0.10	0.27
Q21	Homework	0.65	0.06	0.28
Q23	Enjoyable activities	0.58	0.12	0.40
Q26	Activity monitoring	0.61	0.16	0.25
Q27	Rational statements	0.71	0.05	0.26
Q5	Transference	0.12	0.54	0.27
Q8	Talk @ childhood	0.22	0.59	0.46
Q17	Dream interpretation	0.05	0.61	0.32
Q18	Came to mind	0.06	0.50	0.37
Q19	Countertransference	0.11	0.62	0.41
Q20	Deeper meaning	0.12	0.68	0.41
Q16	Interpersonal to improve	0.34	0.38	0.54
Q22	Pos/neg past interpersonal	0.31	0.49	0.56
Q24	Current quality	0.31	0.31	0.64
Q25	Emotional response	0.28	0.42	0.66

Table 4

Properties of shortened composites

	Cronbach’s Alpha	Correlation among composites
		CBT	DT
CBT	0.84	1.00
DT	0.82	0.15	1.00
IPT	0.79	0.39	0.52

Confirmatory factor analyses supported a 3-factor solution representing CBT, DT, and IPT. These analyses provided adequate fit for a 3-factor model (CFI = 0.87, RMSEA = 0.089) and improved fit over the 1-factor model (CFI = 0.53, RMSEA = 0.167) and the 2-factor model that specified DT and IPT on the same factor (CFI = 0.78, RMSEA = 0.114).

Preliminary Evaluation of Validity

There is also preliminary support for the validity of the instrument. We compared scores on each of the three subscales by the primary theoretical orientation as reported by the clinician. Figure 1 presents mean t-scores for each of subscores by clinician self-reported primary orientation. Clinicians who indicated their orientation (using the primary orientation variable) to be cognitive-behavioral scored significantly higher on the CBT subscale (P < .0001) and significantly lower on the DT subscale (P < .0001) than non-cognitive-behavioral therapists. Dynamic therapists scored significantly higher on the DT subscale (P < .0001) and significantly lower on the CBT subscale (P < .0001) than non-Dynamic therapists. IPT therapists had significantly lower scores on the CBT subscore (P < .0001), but did not differ on the IPT subscore compared to non-IPT therapists. However, when therapists were allowed to endorse multiple orientations, instead of only a single primary orientation, those who endorsed IPT did score significantly higher on the IPT subscore (P < .01). These subscore-by-orientation analyses provide preliminary support for the validity of the CBT and DT subscales, but they also suggest that the relationship between self-reported orientation and scores on the questionnaire may be more tenuous for IPT therapists than for therapists who endorse either CBT or DT.

Fig. 1

Mean T-scores by primary orientation

Additional analyses suggested that a higher frequency of using CBT techniques was significantly associated with fewer years of practice (r = −0.20, P < .0001), while use of IPT and DT techniques were not associated with years of practice.

Discussion

This study provides preliminary support for the reliability and validity of the three subscales assessed in the Psychotherapy Practice Questionnaire—Clinician Depression Care Version. The revised clinician instrument includes 16 items that assess the use of CBT (6 items), IPT (4 items), and DT (6 items) techniques with a single adult patient with depression. These 16 items were selected from a larger set of 28 items based on results from the pilot test. The revised PPS-Clinician demonstrates good psychometric properties. The factor structure of the instrument corresponds to the intended domains of CBT, IPT, and DT, and, in general, subscores relate to therapist orientation as expected. We believe that this 16-item version of the questionnaire would likely take 10–15 min for a clinician to complete. Our results suggest that a clinician-report questionnaire may be a promising, feasible approach to describing psychotherapy for depression in usual care and an efficient way to measure the extent to which clinicians practice evidence-based depression care.

Our results also provide some information about usual care psychotherapy for depression. While it may not be surprising that about half of responding clinicians endorsed cognitive-behavioral as their primary orientation, a substantial portion (21%) endorsed multiple orientations, suggesting that it is important to understand how clinicians implement eclectic approaches that blend techniques from multiple orientations and how these approaches relate to outcomes. Our results also suggest an inverse relationship between years of practice and CBT delivery. These findings may reflect efforts to disseminate CBT by emphasizing this approach in graduate programs for training new therapists.

While we believe that our approach is a promising direction in the development of a tool to describe psychotherapy for depression in usual care, there are limitations to our work. A chief concern is the unknown validity of clinician report of use of various psychotherapy techniques. It is unknown whether clinicians can reflect back on their treatment of a particular patient and accurately report on how frequently they used different techniques. Clinicians may also be susceptible to social desirability biases, such as overreporting using techniques from an approach endorsed by a managed care organization. Further, our use of self-reported therapeutic orientation as a preliminary evaluation of validity has limitations because it is unclear how this variable relates to actual practice. Therefore, an essential step in the validation of this approach would be to compare responses to these questionnaire items with ratings of audiotapes by trained raters. Future research will address the relationship between this clinician-report questionnaire, the patient-report version, and expert ratings of audiotapes.

The response rate we obtained (32%) is not uncommon in surveys of managed care network clinicians and highlights the challenges in conducting survey-based quality monitoring. Nevertheless, our response rate may limit our ability to generalize our findings to non-responding clinicians.

We also believe, however, that development work within a large national managed behavioral health care organization (MBHO) is a strength because it provides an opportunity to understand the practice of therapy for depression from the perspective of the many and diverse clinicians delivering care to the general population. Approximately 95% of American workers are covered by employer-sponsored managed care plans (Kaiser Family Foundation and Health Research and Educational Trust 2003). In addition by 2002, 58% of the Medicaid population were enrolled in managed care (Centers for Medicare and Medicaid Services 2002). MBHO’s are also likely to be interested in tools to facilitate quality assessment and quality improvement efforts. For example, recent research has shown that an MBHO-based depression program that identifies and promotes evidence-based treatment for depression identified in employees can significantly improve clinical and workplace outcomes (Wang et al. 2007). MBHOs, with their large provider panels, are in the position to monitor the quality of care provided to their members, and they have the organizational infrastructure to provide valuable information and feedback to a large number of geographically dispersed clinicians, including information on clinical outcomes, and the relationship of outcomes to treatment processes such as type of psychotherapy.

We believe that this clinician reported questionnaire has the potential to fill an important gap in understanding usual care psychotherapy. Questionnaire-based methods such as the one we describe here could provide a more practical, less costly alternative to expert rater methods. A practical tool would be valuable in describing usual care psychotherapy and in evaluating how use of these techniques predicts patient outcomes. As MBHOs have become more accountable to members and customers for quality of care, such a tool would allow for greater monitoring and could inform the development of quality improvement interventions.