The Relationship Between Homework Compliance and Therapy Outcomes: An Updated Meta-Analysis.

The current study was an updated meta-analysis of manuscripts since the year 2000 examining the effects of homework compliance on treatment outcome. A total of 23 studies encompassing 2,183 subjects were included. Results indicated a significant relationship between homework compliance and treatment outcome suggesting a small to medium effect (r = .26; 95% CI = .19-.33). Moderator analyses were conducted to determine the differential effect size of homework on treatment outcome by target symptoms (e.g., depression; anxiety), source of homework rating (e.g., client; therapist), timing of homework rating (e.g., retroactive vs. contemporaneous), and type of homework rating (e.g., Likert; total homeworks completed). Results indicated that effect sizes were robust across target symptoms, but differed by source of homework rating, timing of homework rating, and type of homework rating. Specifically, studies utilizing combined client and therapist ratings of compliance had significantly higher mean effect size relative to those using therapist only assessments and those using objective assessments. Further, studies that rated the percentage of homeworks completed had a significantly lower mean effect size compared to studies using Likert ratings, and retroactive assessments had higher effect size than contemporaneous assessments.

Introduction

Cognitive and behavior therapies are often considered “first-line” treatments for a number of psychiatric disorders, with various meta-analyses demonstrating the efficacy of these therapies for conditions such as anxiety disorders (Hofmann and Smits 2008; Otto et al. 2004), depression (Dobson 1989; Spek et al. 2007), and substance-use disorders (Duttra et al. 2008). While cognitive and behavior therapies have been established on theoretical foundations, the efficacy of these interventions may lie in their strong history of utilizing homework assignments as a mechanism toward producing beneficial treatment outcomes. That is, practice of skills outside of therapy (i.e., homework) allows clients to master the skills believed necessary to affect symptoms, generalize these skills to their natural settings, and promote prolonged symptom improvement through extending therapeutic aspects of treatment beyond the completion of therapy (Kazantzis and Lampropoulos 2002).

Indeed, the importance of homework for producing positive therapy outcome was demonstrated in a previous meta-analysis (Kazantzis et al. 2000). In their analysis, a Pearson r effect size of .22 was reported for the relationship between homework compliance and therapy outcome in a sample of 1,327 subjects across 27 studies. These results suggest that greater compliance with homework is associated with beneficial treatment outcome, with the strength of the association falling between Cohen’s small and medium effect size cutoffs (Cohen 1988; Kraemer et al. 2003).

Kazantzis et al. (2000) analysis was the first study to examine the type of homework activity and the nature of the client’s presenting problem as moderating variables of homework effectiveness. The presenting problems were categorized as depression, anxiety-related disorders, and other outpatient. The results of this meta-analysis showed the following mean effect sizes for problem type: depression (.22), anxiety (.24), and other outpatient (.17), with homework effects being significantly greater for the treatment of depression than the “other outpatient” sample. Additionally, results indicated that effect sizes were robust across the type of homework completed (no single type, relaxation, or social skills) and time of homework compliance assessment (regular intervals or posttreatment), but differed by the source of homework compliance assessment. Specifically, studies that utilized client and therapist ratings had a significantly lower mean effect size relative to those using objective measures of homework compliance.

In the 8 years since Kazantzis, Deane, and Ronan’s meta-analysis on the effects of homework assignments on treatment outcome, homework has continued to remain “both a traditional and integral component of contemporary manual-based cognitive-behavioral therapy (CBT) approaches” (Coon and Thompson 2003, p. 53). Further, there continues to be support for the effectiveness of cognitive-behavioral interventions to prevent the onset, relapse, and recurrence of a number of psychological disorders (Hollon 2003). The meta-analysis conducted by Kazantzis et al. (2000) included homework-related studies spanning from 1980, 1 year following Beck’s emphasis on regularly using homework in cognitive-behavioral therapy for depression (Beck et al. 1979), through 1998, a time when homework in therapy had been incorporated into a more diverse range of clinical conditions (Kazantzis et al. 2000). Therefore, a significant amount of variance as a function of time may exist within this analysis.

The present study is an updated meta-analysis of the relationship between homework compliance and treatment outcome. We hypothesized that greater homework compliance would be significantly associated with improved treatment outcome. Given that the previous meta-analysis found some evidence that targeted symptoms and source of homework ratings may moderate the effect of homework compliance, we further examined whether treatment target (e.g., symptoms of anxiety, depression, etc.) and source of rating (e.g., therapist, objective) moderated the relationship between homework compliance and therapy outcome. A novel aspect of this meta-analysis is that we examine the moderating effect of rating type (e.g., Likert rating, percentage of homeworks completed).

Methods

Sample

To identify candidate studies for inclusion in our review, the following inclusion/exclusion criteria were used: (a) studies must have been published between January, 2000 and September, 2008, (b) the study must have been published in English, and (c) the study must have been a treatment study examining pre- and post-treatment outcome and measured some aspect of homework compliance. Guided by these criteria, we searched PsychArticles, PsychInfo, and Medline databases for journal manuscripts published between January 2000 and September 2008 using the key terms homework and compliance and (therapy or psychotherapy or psychosocial intervention or intervention). From this search 87 articles were found. We read the abstracts from these articles to identify potential studies for inclusion as well as manuscript citations to identify further manuscripts that may have initially been missed in our initial search. Articles that were eliminated dealt with methods for improving homework compliance rather than the impact of homework compliance on treatment outcome. Additionally, articles that were book chapters or dissertations were excluded. Twenty-three studies encompassing 2,183 subjects met the inclusion criteria for the meta-analysis and were therefore included in the present study.

Classification and Coding Systems

Only studies looking at the relationship between homework compliance and the therapeutic outcome were included in the present study. In addition to the relations between homework compliance and outcome, the following elements were considered as moderator variables:

Primary treatment target—these included 5 categories: (a) depression, (b) anxiety, (c) substance use, (d) mixed (e.g., both anxiety and depression), and (e) other (e.g., functioning);

Source of homework rating—Four categories were included in this rating: (a) therapist (Likert rating), (b) client (Likert rating), (c) objective (e.g., number of assignments turned in), and (d) client and therapist (e.g., both client and therapist rated homework compliance and average ratings were used).

Type of homework rating—Three categories of homework rating were coded: (a) Likert scale, (b) number of assignments completed, and (c) percentage of homework completed.

Timing of homework rating—Two categories of timing were coded: (a) retroactive ratings of homework compliance (e.g., a single rating at the end of treatment), and (b) contemporaneous ratings of homework compliance (e.g., assessment of homework at each therapy session).

Year of study—In this analysis, we used weighted regression to determine if the linear variable “year of publication” moderated the effect size of homework on outcome.

Calculation of Effect Sizes

Effect size r was used to characterize the relationship between homework compliance and therapy outcome for each of the 20 studies. For studies that did not report correlation coefficients (r), available study statistics were converted to r according to standard formulas (Hunter and Schmidt 1990). As mentioned above, effect sizes were determined by two independent reviewers and for the majority of studies agreement was reached. In three cases, discrepancies were determined by discussion between the two reviewers and a third reviewer. For those studies where available statistics were not readily converted to r, we used the standardized regression coefficient (β; n = 7) or semi-partial correlation coefficient (n = 3) as a proxy for r (Peterson and Brown 2005). Once study-level correlation coefficients were calculated they were weighted, aggregated, and their heterogeneity was assessed with the Q statistic (Hedges and Olkin 1985) using a random effects model.

Results

Characteristics of the Sample

Characteristics of the 23 studies included in this meta-analysis are presented in Table 1. Overall, the number of participants in these studies ranged from 10 to 641, with a mean of approximately 95 participants (median n = 46). Eight studies targeted symptoms of anxiety, 5 targeted symptoms of depression, 3 targeted substance use, and 1 targeted a mix of symptoms. The remaining 6 studies targeted a variety of symptoms including psychosis, body image, and everyday functioning; these were coded as “other”. As for the source of homework ratings, 11 used therapist ratings, 2 used client ratings, 8 used an objective rating, and 2 used both client and therapist ratings. A total of 9 studies used a Likert rating of homework compliance, 7 used the number of homework assignments completed, and 7 used the percentage of homeworks completed.

Table 1

Description of studies included in the analysis

Study	N	Target symptoms	Homework rating	Rating type	Timing of assessment
Abramowitz et al. (2002)	28	Anxiety	Therapist	Likert	Retroactive
Bogalo and Moss-Morris (2006)	24	Other (Global relief)	Objective	# Completed	Contemporaneous
Burns and Spangler (2000)	521	Depression	Client + Therapist	Likert	Retroactive
Carroll et al. (2008)	34	Substance Use	Objective	# Completed	Contemporaneous
Carroll et al. (2005)	48	Substance Use	Therapist	Categorical	Contemporaneous
Cash and Hrabosky (2003)	25	Other (Body image distress)	Objective	# Completed	Contemporaneous
Coon and Thompson (2003)	58	Depression	Therapist	% Completed	Contemporaneous
Cowan et al. (2008)	641	Depression	Objective	% Completed	Contemporaneous
Dunn et al. (2006)	29	Other (Psychotic Symptoms)	Therapist	Likert	Retroactive
Gonzalez et al. (2006)	123	Substance Use	Therapist	% Completed	Contemporaneous
Granholm et al. (2006)	32	Other (Everyday Functioning)	Objective	# Completed	Contemporaneous
Hughes and Kendall (2007)	132	Anxiety	Therapist	Likert	Contemporaneous
Rees et al. (2005)	94	Mixed	Client	# Completed	Contemporaneous
Schmidt and Woolaway-Bickel (2000)	48	Anxiety	Therapist	% Completed	Contemporaneous
Tolin et al. (2007)	10	Other (Hoarding)	Therapist	Likert	Contemporaneous
Westra and Dozois (2006)	40	Anxiety	Client + Therapist	Likert	Contemporaneous
Westra et al. (2007)	25	Anxiety	Client	Likert	Contemporaneous
Wetherell et al. (2005)	65	Anxiety	Objective	% Completed	Contemporaneous
Woods et al. (2002)	82	Anxiety	Objective	# Completed	Contemporaneous
Woody and Adessky (2002)	53	Anxiety	Therapist	Likert	Contemporaneous

Effects of Homework Compliance on Therapy Outcome

The overall effect size r between homework compliance and treatment outcome was .26 (95% CI = .19–.33; P < .001), indicating that across treatment targets, sources of homework ratings, and type of homework ratings, greater homework compliance was associated with improved treatment outcome. The overall effect fell within the small-to-medium range (Cohen 1988). This result supported our first hypothesis. Effect sizes ranged from .08 to .93, and the homogeneity analysis indicated significant heterogeneity in results (Q = 39.38, df = 19, P = .004). The fail-safe n (Rosenthal 1979) was computed to be 618.

Moderator Analyses

Results of our 3 moderator analyses are presented in Table 2, and information on study details (e.g., duration, modality, outcome measures) are found in Table 3. Our first moderator analysis examined the effect of homework on treatment outcome by treatment target (e.g., symptoms of anxiety or depression). Overall, treatment target did not significantly moderate the relationship between homework compliance and treatment outcome (Q = .39, df = 4, P = .983). As seen in Table 2, the effect sizes were remarkably robust, ranging from .22 for anxiety to .27 for substance use outcomes.

Table 2

Mean effect sizes moderator analyses

	N	Mean ES	95% Low	95% High
Treatment target
Anxiety	8	.22	.13	.32
Depression	3	.24	.18	.29
Substance use	3	.27	.13	.41
Mixed	1	.25	.05	.46
Other	5	.26	.06	.45
Homework rater
Therapist	9	.24	.15	.33
Client	2	.32	.13	.50
Objective	7	.16	.09	.22
Client and therapist	2	.35	.27	.44
Homework rating type
Likert	8	.31	.25	.38
# Homeworks completed	6	.22	.10	.34
% Homeworks completed	5	.17	.10	.23
Timing of homework
Contemporaneous	20	.19	.14	.24
Retroactive	3	.36	.28	.44

Table 3

Study details

Study	Intervention	Duration	Modality	Mean age	Randomized clinical trial?	Outcome measure(s)
Abramowitz et al. (2002)	Exposure and ritual prevention (EX/RP)	18 sessions	Individual	35.2	No	Y-BOCS scores
Bogalo and Moss-Morris (2006)	Self-help manual or control	3 months	Individual	39	Yes	Irritable bowel symptom severity
Burns and Spangler (2000)	CBT alone or CBT with medication	12 weeks	Individual	37.2	Yes	BDI and HSCL-90 depression score
Carroll et al. (2008)	CBT or treatment as usual	8 weeks	Individual	41.6	Yes	Most consecutive days of abstinence; % days of abstinence during treatment; % of urine specimens negative for all drugs
Carroll et al. (2005)	CBT or IPT	12 weeks	Individual	34.8 (tx group only)	Yes	% of cocaine positive urines, self-report daily frequency of cocaine use
Cash and Hrabosky (2003)	CBT	3 weeks	Individual	24.7	No	SIBID scores
Coon and Thompson (2003)	Medication alone, CBT alone or CBT with medication	3–4 months	Individual	66.6	Yes	BDI, HAM-D

Our second moderator analysis examined the source of homework ratings (e.g., therapist, client). Results of this analysis indicated a significant moderating effect of homework source (Q = 13.83, df = 3, P = .003). Studies that utilized combined client and therapist ratings had a significantly larger mean effect size than those that utilized objective ratings (P < .001). No significant differences were observed between the other sources of homework ratings.

Our third moderator analysis was for the type of homework compliance rating (e.g., Likert scale). Results of this analysis indicated that type of homework rating significantly moderated the relationship between homework compliance and therapy outcome (Q = 9.51, df = 2, P = .009). Post-hoc analyses indicated that studies utilizing Likert ratings of homework compliance had a significantly higher mean effect size compared to studies using a percentage rating (i.e., percentage of homeworks completed) of homework compliance (P = .002). No significant differences were observed between Likert and total number of homeworks completed or between total number completed and percent completed (P-values > .05).

Our fourth analysis was for timing of homework compliance (e.g., retroactive vs. contemporaneous). Results of this moderator analysis indicated that retroactive ratings of homework compliance (e.g., a single rating of compliance provided at the end of treatment) demonstrated a significantly higher effect size than contemporaneous ratings (e.g., ratings made after each therapy session; Q = 11.90, df = 1, P < .001). Specifically, the mean correlation between homework compliance and outcome was .36 for retroactive ratings and .19 for contemporaneous ratings.

A final analysis examined the moderating effect of publication year. Results of this analysis indicated that year of publication did not moderate the effect of homework on treatment outcome (P = .264).

Discussion

This meta-analysis examined the relationship between homework compliance and treatment outcome across 23 studies and over 2,000 participants. Similar to results found by Kazantzis et al. (2000), greater homework compliance was associated with improved treatment outcome (r = .27). These results were consistent across a variety of target symptoms including symptoms of anxiety (r = .22), depression (r = .24), and substance use (r = .27), suggesting that compliance with homework is an important component of psychotherapy regardless of the target symptoms. Indeed, this finding is consistent with cognitive and behavioral theories, which suggest that mastery of skills learned in therapy via practice of such skills is important for producing positive treatment outcomes (i.e., improving symptoms).

In the present study, the two most common sources of homework ratings were therapists and objective ratings (e.g., counting the number or percentage of homework turned in), and we found that the source of homework ratings moderated the relationship between homework compliance and treatment outcome. Specifically, when both clients and their therapists provided homework ratings, effect sizes were significantly higher (r = .35) than when objective ratings were used (r = .16). However, because only two studies utilized both client and therapist ratings, these results should be interpreted with caution. Indeed, the two studies that utilized therapist and patient ratings of compliance used quite different methods for assessing homework compliance and had quite different sample sizes. Moreover, our analysis averaged the therapist and patient rating of homework compliance, despite the fact that these ratings may not always be strongly correlated. Indeed, the study by Westra and Dozois (2006) reported only a modest correlation between therapist and client compliance ratings. Again, given the small number of studies utilizing this method and the limitations mentioned here, readers should take caution about interpreting these findings as particularly meaningful.

These findings might be interpreted in a number of different ways. First, they may suggest that future studies of this relationship should utilize both types of ratings, at least on the assumption that this effect size discrepancy is real. Alternatively, this discrepancy in findings might highlight the inherent limitations of using “subjective” ratings as a means of assessing homework compliance. For example, therapists who provide homework ratings may give better scores to those who are doing better in therapy (i.e., “he’s doing better, so he must be doing his homework”).

There were no significant differences between groups when comparing other sources of homework ratings. However, although objective ratings did not differ from client alone or therapist alone ratings, it is interesting to note that our findings differ from those of Kazantzis et al. (2000), who found that objective ratings had a higher overall correlation with treatment outcome. This may be due to the difference in defining “objective” assessment between the two meta-analyses. Specifically, whereas Kazantzis defined “objective” as an electronic marker of homework compliance, our analysis considered “objective” to mean studies that counted the number of homeworks turned into therapists.

Studies that used Likert scales to rate homework compliance had a significantly higher mean effect size (r = .31) than those rating the percentage of homeworks completed (r = .17). Further, studies using Likert scales were higher, but not significantly so, than studies using the number of homeworks completed. While this finding is difficult to explain, it may be due to the fact that Likert ratings might inadvertently reflect quality and quantity ratings, whereas a summary variable such as percent or total homeworks completed reflect quantity only. For example, during the course of therapy, clients may be asked to regularly (e.g., once each day) practice homework. However, they may present at the next therapy session and describe one excellent (and extremely beneficial) example of how he/she practiced homework over the past week. Therapists who rated client homework from 0 (poor) to 6 (outstanding) might rate this compliance relatively high on the scale. In contrast, clients who report doing homework every day but who had difficulty with the assignment or who described it as unhelpful might be rated relatively lower in terms of compliance. Further, Likert scales provide the therapist and the client with a range to rank homework completion. This can be opposed to percentage of homeworks completed and number of homeworks completed, which are often scored on a dichotomous (completed or did not complete) scale. If a client completes part of a homework assignment, the client is given some credit for compliance, even if the effort is minimal.

Further, a “timing effect” was found for contemporaneous versus retrospective ratings of homework completion in that retrospective ratings were a significantly better predictor of outcome than contemporaneous ratings. This may have been due to a bias effect for retroactive ratings. For example, it is possible that patients who have appeared to have done well in therapy could have been rated by their therapist or themselves as more compliant with homework assignments. These results may provide insight into differences in objective versus subjective ratings (i.e., higher effect size for subjective ratings than objective assessments), in that objective ratings are most typically contemporaneous by nature (e.g., paperwork that was turned into and/or discussed with the therapist), and therefore appear more reliable in assessing compliance than retroactive or subjective ratings of compliance.

These issues (objective vs. subjective; Likert vs. non-Likert) highlight the important issue of how we define homework compliance. Specifically, they highlight the important issue of the purpose of conducting a homework analyses, which is to discover the “true nature” of the relationship between homework compliance and treatment outcome, not findings ways of manipulating methods to demonstrate larger effects. Determining the true effect indeed involves finding increasingly “objective”, or bias-free methods of assessing homework compliance. To this end, Kazantzis et al. (2004) has described novel methods of assessing homework in therapy research (e.g., the Homework Rating Scale), which include the assessment of homework quality. However, there has yet to be any consistent use of these methods. We strongly recommend new research incorporate these new methods of assessing homework compliance, as well as develop more objective and accurate means of assessing homework quantity and quality in treatment research and outcome.

There are several limitations to the current review. As previously mentioned, there have been problems with the objective assessment of homework compliance. Additionally, the current review did not examine demographic moderators (i.e., age, gender, ethnicity, education) or the severity of psychopathology (e.g., Major Depressive Disorder vs. Dysthymia; Substance Abuse vs. Substance Dependence) that could contribute to homework compliance. These variables were not included in the current study’s moderator analysis as they were not examined in the results of the studies reviewed. Research has found that clients comply less with homework directives if they have greater and/or more long-lasting symptomology (Worthington 1986). In addition to demographic moderators and severity of psychopathology, other things to keep in mind when considering the relationship between a client’s homework compliance and therapeutic gain are pharmacotherapy (e.g., is the client on antidepressant medications?), if the client is involved in another form of treatment (e.g., social skills training), and use of coping mechanisms for dealing with stress (e.g., does the client take action in response to stress or become less productive? Addis and Jacobson 2000). The results, however, demonstrate a more generalized view of the effects of homework compliance on therapy outcome across a span of different psychological diagnoses and diverse demographic characteristics.

A further limitation of the current review is that it did not take into account the client-therapist relationship. Research has found that a positive and trusting client-therapist relationship may aid recovery in mental illness (Green et al. 2008) regardless of homework. Additionally, the strength of the relationship between the client and the therapist could contribute to homework compliance, with a stronger working relationship leading to increased homework compliance. Without looking at the client-therapist relationship as a moderator between homework compliance and treatment outcome, there is a possibility that the relationship alone contributed to the improvements seen in the clients. However, as mentioned by Kazantzis et al. (2000), there exists an abundance of research that demonstrates the positive effects of the use of homework in therapy on treatment outcome.

Finally, the current review did not examine the client’s attitude towards homework. A negative attitude towards homework, even if the homework is completed, could potentially limit the likelihood that the client will continue to practice the skills learned once therapy is completed. Motivation, lack of effort, and readiness to change are other variables that were not explored in the current study, which are factors that have been found to be correlated with homework compliance (Neimeyer et al. 2008; Yovel and Safren 2007). Addis and Jacobson (2000) examined the relationship between clients acceptance of the treatment rationale and the degree to which clients completed homework, and concluded that the ability to provide a convincing treatment rationale may be one of the crucial skills which determines the success of CBT in real-world clinical settings. Further studies would benefit from exploring these areas in regard to homework compliance.

In sum, the results of this meta-analysis suggest that on the whole, greater compliance with homework is related to improved treatment outcome, and this relationship is robust across a variety of treatment targets (e.g., depression, anxiety, and substance use). However, this study also highlights discrepancies in effect sizes surrounding the method of assessing homework compliance (e.g., objective vs. subjective). Specifically, higher effect sizes were found when therapists and clients both evaluate homework compliance. On one hand, clinicians may desire making homework compliance a collaborative part of treatment (e.g., to structure therapy whereby review of homework is an integral part of sessions). On the other hand, these discrepancies may highlight the inherent limitations in using subjective assessments of homework compliance. To this regard, it may be increasingly important for more standardized and objective methods of assessing homework compliance that are less prone to bias and that capture the true nature of the relationship between homework compliance and treatment outcome. In this vein, suggestions on incorporating homework into therapy and improving compliance are available in the literature (Beck 1995; Tompkins 2004), as are forms for measuring multiple aspects of homework compliance (Kazantzis et al. 2004).