Clinicians' Personal Theories of Developmental Disorders Explain Their Judgments of Effectiveness of Interventions.

Evidence-based psychotherapy requires clinicians to consider theories of psychopathology and evidence about effectiveness, and their experience when choosing interventions. Research on clinical decision making indicates that clinicians' theories of disorders might be personal and inform judgments and choices beyond current scientific theory and evidence. We asked 20 child therapists to draw models of how they believed that biological, psychological, environmental, and behavioral factors interact to cause and maintain four common developmental disorders. They were also asked to judge the effectiveness of interventions recommended in the literature. Therapists showed only fair agreement about the factors and a slight to fair agreement about the causal relations between these, and just fair agreement about interventions' effectiveness. Despite these disagreements, we could predict effectiveness judgments from therapists' personal theories, which indicates that clinicians use personal theories in decision making. We discuss the implications of these findings for evidence-based practice.

According to guidelines for evidence-based practice (EBP; e.g., APA Presidential Task Force on Evidence-Based Practice, 2006), clinicians are expected to consider scientific evidence in their professional practice. Scientific evidence includes insights into the complex processes by which psychological, biological, and environmental factors interact to cause and maintain psychological and psychiatric problems (Carr, 2006; De Los Reyes & Kazdin, 2006; Kendall, Holmbeck, & Verduin, 2004; Kendler, 2005; Kiesler, 1999; Morton, 2004). Theoretical models of specific disorders and theories of psychopathology summarize and integrate these insights. For most psychological disorders several theoretical models have been proposed, which are not necessarily overlapping. For example, there are several empirically supported theories on ADHD (cf. Barkley, 2006), some of which focus on the underlying neurobiological processes, whereas others focus on how learning and experience maintain and aggravate patient’s problems. EBP also prescribes clinicians to use research evidence about effectiveness and efficacy of interventions (e.g., APA Presidential Task Force on Evidence-Based Practice, 2006). Clinicians are supposed to know the respective evidence and choose the treatment that presumable works the best for their patient. Finally, EBP asks clinicians to take into account their professional experience. Clinicians should monitor and assess their performance and use these insights when planning a treatment. Thus, clinicians are expected to consider empirically supported scientific theories about disorders, empirically proven effective interventions, and their personal experience when judging the effectiveness of interventions and deciding on a treatment. Moreover, EBP suggests to use the current best evidence in a “conscientious, explicit, and judicious” way (Sackett et al., 1996).

From the perspective of cognitive psychology, EBP is very challenging. It requires clinicians to integrate declarative knowledge acquired from textbooks, guidelines, and research papers with instrumental knowledge acquired through experiential learning. It also asks clinicians to engage in deliberate decision making, carefully thinking about the positive and negative consequences that may result from the treatment while considering recommendations from guidelines and the wishes of the patient. Cognitive psychology provides theoretical models explaining how clinicians may solve the tasks posed by EBP.

The first task is to integrate the different types of knowledge. Several theoretical accounts propose that clinicians develop personal theories of specific disorders on the basis of their experience and knowledge (e.g., Custers, Boshuizen, & Schmidt, 1998; Garb, 2005). For example, Charlin, Boshuizen, Custers, and Feltovich (2007) assume that clinicians develop so-called illness scripts, which include signs and symptoms of a disease or disorder, its typical presentation, and the underlying causal mechanism that leads to and maintains the problems of patients suffering from this disease. According to these psychological theories, clinicians’ theories are dependent on their individual knowledge and experience. Personal theories are then used to make diagnoses, to explain a client’s problems, and to decide on a treatment.

There is quite a bit of evidence that clinicians have personal theories that influence their judgments and decisions. Ahn, Proctor, and Flanagan (2009) asked mental health clinicians about their beliefs about the causes of psychiatric disorders as defined in the Diagnostic and Statistical Manual of Mental Disorders (DSM–IV–TR; American Psychiatric Association [APA], 2000). The researchers inquired to which extent the disorders were caused by biological, psychological, or environmental causes. Their first finding was that clinicians had assumptions about the causes of the various disorder, although the DSM defines the disorder without any reference to cause. They also found that if clinicians believed that a disorder had mainly a biological basis, they believed in no or few psychological or environmental causes, and vice versa. Furthermore, clinicians’ causal beliefs were related to their treatment preferences. Clinicians preferred medical treatments more often if they believed that a disorder had a biological origin, and they preferred psychotherapeutic treatments more often if they believed that it had a psychological or environmental cause (Ahn et al., 2009).

Further studies indicate that clinicians not only have beliefs about the causes leading to a disorder, but also have beliefs about the causal relations among the symptoms of mental disorders (Kim & Ahn, 2002). This is especially noteworthy, as the DSM makes no respective assumptions. Kim and Ahn (2002) showed that clinicians assume some symptoms to causally affect many other symptoms (i.e., causally central symptoms), some symptoms to be mere effects of other symptoms (i.e., noncentral symptoms), and some symptoms to be causally disconnected from others. It turned out that clinicians’ beliefs about the causal centrality of symptoms (i.e., their causal impact on other symptoms) affected clinicians’ diagnostic classifications, memories of case information, and inferences (Kim & Ahn, 2002; Proctor & Ahn, 2007). Clinicians weighted causally central symptoms more in classification, remembered them better, and based inferences about other features on them.

The second task posed by EBP to clinicians is to make a deliberate, well-thought-out decision on the best course of treatment. In cognitive psychology, a number of theories have been proposed that explain how personal theories provide a basis for deciding on a course of action (e.g., Hastie & Pennington, 2000; Pliske & Klein, 2003; Sloman & Hagmayer, 2006). Causal model theories of decision making (Hagmayer, Meder, Osman, Mangold, & Lagnado, 2010; Sloman & Hagmayer, 2006) assume that clinicians acquire a causal model of a disorder (i.e., a personal theory) through experiential, deliberate, and social learning. This causal model encompasses the factors that cause and maintain symptoms, as well as the causal mechanisms connecting the factors and symptoms. Interventions (i.e., treatments in mental health) may target causes, maintaining factors, or symptoms, and can be part of the model. For example in cognitive-behavioral therapy (CBT), cognitive restructuring is assumed to target dysfunctional beliefs and attributions, whereas techniques of emotion regulation may directly target emotional instability as a symptom.

A generic causal model of a disorder can be used in several ways in judgment and decision making (for more details on how causal models may support clinical decision making, see Hagmayer & Fernbach, in press; Hagmayer & Witteman, in press). First, it allows a clinician to identify causally central factors, which are important to assess in an explanatory diagnosis and important to consider in treatment planning. Second, a causal model allows a decision maker to infer which treatment will be more effective. Interventions that target causally central factors, affecting many other factors and symptoms, are likely to be more effective to improve the condition of the patient than treatments targeting only a single symptom. Third, a causal model enables the decision maker to predict the consequences that should result from different interventions. These predictions can then be used to evaluate the progress of a treatment. If predicted changes happen, the client is probably on track.

The aim of the present study was to investigate clinicians’ personal theories of disorders and whether these theories are related to their expectations about the effectiveness of interventions. Our main question was to what extent we could predict clinicians’ judgments about the effectiveness of interventions for a disorder from their personal theories for the disorders (i.e., their causal model). We also investigated whether different clinicians have similar theories for a disorder, and whether clinicians agree about the effectiveness of possible interventions for each disorder. If clinicians base their personal theories on scientific theories, supported by evidence from systematic research, one may expect to find a rather high agreement. However, as pointed out before, often several theories can be found in the literature. This situation may lead to a rather low agreement. The same might be true for the agreement about the effectiveness of interventions. Systematic research provided us with good evidence about which therapies and treatment programs are effective and which are not (or substantially less effective). Rather little, however, is known, about the differential effectiveness of individual intervention techniques or the causal mechanisms by which they work (De Los Reyes & Kazdin, 2006; Kazdin, 2008).

The research presented here goes beyond prior research. Previous studies investigated certain parts of clinicians’ personal theories. Kim and Ahn (2002) explored only clinicians’ assumptions about the causal relations among DSM–IV symptoms (APA, 1994). As the DSM–IV hardly contains references to deeper causes, clinicians’ personal theories consisted mostly of factors on the behavioral or cognitive level, but neglected biological or environmental factors. Ahn et al. (2009) asked clinicians only to indicate the degree to which disorders are due to biological, social, or environmental factors. They did not inquire about the causal relations connecting causes, symptoms, and other relevant variables within each disorder. Hence, it is still an open question of which factors and causal relations are included in clinicians’ personal theories when they are requested to come up with a comprehensive account.

Previous studies also indicated that clinicians’ causal assumptions are related to treatment preferences (e.g., Ahn et al., 2009; Flores, Cobos, López, Godoy, & Ganzález-Martín, 2014), but no study investigated how personal theories of disorders are linked to inferences about the relative effectiveness of treatments. De Kwaadsteniet, Hagmayer, Krol, and Witteman (2010) studied clinicians’ causal models of individual clients (i.e., their clinical case conceptions) and found that these models predicted clinicians’ judgments of the relative effectiveness of different interventions for the specific client. But they did not explore clinicians’ personal theories of the disorder.

Outline of Experiment and Hypotheses

We asked experienced child therapists to draw their personal causal theories of four prevalent developmental disorders—attention-deficit/hyperactivity disorder (ADHD), autism, conduct disorder, and reactive attachment disorder—and to rate the effectiveness of 10 common and recommended types of interventions. For each of these four disorders several theories have been proposed in the literature, which are supported by empirical evidence (for respective overviews, see Carr, 2009; Mash & Barkley, 2014). In addition, meta-analytic reviews and national guidelines make suggestions for evidence-based treatments (e.g., National Institute for Health and Care Excellence, 2008, 2011, 2013, 2015). In the Netherlands, the country in which the present study was conducted, there are guidelines for each of the four disorders, for example the Dutch Multidisciplinary Guideline ADHD (Trimbos-instituut, 2007, 2011) and guidelines for effective interventions from the Dutch Institute for Youth (NJi, 2013). Clinicians engaging in EBP should be aware of these scientific theories and the guidelines (APA Presidential Task Force on Evidence-Based Practice, 2006).

We expected that clinicians would have rather elaborate causal theories with biological, psychological, and environmental factors explaining the presence of the problems and symptoms characterizing each disorder. This expectation is based on the theory view of categorization (Murphy & Medin, 1985), which assumes that categories like disorders have a theory-like structure, and on the fact that current scientific theories of the four disorders are often complex, especially when an integrative account is proposed. For example, the dual-pathway model of ADHD (Sonuga-Barke, 2004) contains 19 different variables despite omitting genetic factors, which have been proven to be causally relevant as well.

We expected to find rather low agreement among clinicians with respect to their causal theories for a disorder. Developmental disorders are explained in divergent ways by different scientific theories (see, e.g., Morton, 2004). There are no universally agreed theories capturing all relevant aspects of the four disorders investigated in our study (cf. Carr, 2009). So, even if clinicians would base their personal theories on scientific theories published in the literature and ignore their professional experience, we would not expect a high degree of consensus. Our expectation is also supported by previous studies investigating clinicians’ causal beliefs about disorders. Ahn and colleagues (2009) found only moderate agreement among clinicians about the degree to which biological, psychological, and environmental factors contribute to the disorders defined in the DSM–IV (interrater correlations were .48, .53, and .32, respectively). Kim and Ahn’s (2002) studies indicate that the agreement about causal mechanisms connecting the symptoms characterizing a disorder is probably even lower (Kendall’s Ws for interrater agreement for 10 familiar disorders were between .09 and .46). In addition, several studies have shown that clinicians’ causal inferences about individual clients have poor concordance. For example, De Kwaadsteniet et al. (2010) found kappas of .20 and .29 for causal factors, and kappas of .26 and .29 for causal relations between factors. Kuyken, Fothergill, Musa, and Chadwick (2005) found percentages of agreement of 7% to 73%, with less agreement for more theory driven inferences. Persons and Bertagnolli (1999) found intraclass correlation coefficients for agreement about inferences of the schemas underlying overt problems between .13 and .66.

We also expected no high agreement among clinicians’ ratings of the effectiveness of interventions. Conclusive evidence for the effectiveness of many interventions is still lacking, and existing evidence is often not conclusive for all client populations and contexts (APA Presidential Task Force on Evidence-Based Practice, 2006; Carr, 2009; Ollendick & King, 2000). For example, a recent Cochrane Review (Otasowie, Castells, Ehimare, & Smith, 2014) found the evidence for the usage of tricyclic antidepressant medication for children and adolescents with ADHD to be of low quality. Moreover, there is often very little evidence about which of two evidence-based interventions is more effective. The cited Cochrane review on antidepressant medication, for example, identified only one trial in which an antidepressant was compared to standard methylphenidate treatment. There is also still much unknown about the causal mechanisms that lead to and maintain disorders, and about the mechanisms by which interventions cause change (Bieling & Kuyken, 2003; Borsboom & Cramer, 2013; De Los Reyes & Kazdin, 2006; Kazdin, 2008). Available guidelines on the assessment and treatment of developmental disorders in the Netherlands (NJi, 2013; Trimbos-instituut, 2007) should raise agreement among clinicians, but doubts about the evidence base may lower the agreement. Therefore, we expected no high agreement.

Finally, we did expect that clinicians’ individual effectiveness judgments for interventions would match their personal causal theories of the disorders, despite our expectation of low consensus among clinicians about theories for disorders and about effective interventions. Based on the causal model theory of decision making (Sloman & Hagmayer, 2006) and the existing evidence, we thus expected that effectiveness judgments could be predicted from clinicians’ personal theories. Note that such a predictive relation is possible even when therapists do not agree about the theories of the disorders and the effectiveness of different interventions.

Method

Participants

Out of 24 child therapists working at three intramural institutions, 20 agreed to participate. The therapists treated boys and girls, aged 12 to 18 years, who had serious behavioral problems. Four therapists declined, because of the large time investment. Participants’ average experience was 5.0 years, ranging from 2 to 11 years (SD = 3.2). The median numbers of clients that participants had seen with each of the four disorders in the previous year were 8 for ADHD, 8 for autism, 10 for conduct disorder, and 15 for reactive attachment disorder (RAD). Seventeen clinicians were female. All participants had a clinical master’s degree in child psychology or special education; five participants also had a post-master’s degree, which implies that they have had an additional 2 years of training with courses and supervision. In the Netherlands, youth care professionals are often female (about 75%), and 43% are younger than 35 (Hollander, Van Klaveren, Faun, & Spijkerman, 2013). Our sample is relatively young, with a mean age of 28.4 (SD = 3.6). The level of education is highly representative of the level of education of Dutch youth care therapists, although increasingly often post-master’s degrees are being required and obtained (NIP, 2016; NJi, 2015).

Participants received no compensation for their time investment (2 to 3 hours per participant). When the study was finished, results were presented and discussed with clinicians during an intervision conference.

Materials and procedure

We investigated child therapists’ causal theories about ADHD, autism, conduct disorder (CD), and RAD. These four disorders were the most prevalent disorders in the institutions our participants worked at. All participants engaged in a theory drawing task and an intervention rating task, with a time lag of 2 weeks in between. The time lag was necessary to prepare individualized materials for each participant (see Interventions rating task for details).

Theory drawing task

The experimenter made an appointment with each participant to make the drawing for the first disorder at participants’ offices. The order of disorders was counterbalanced across participants. Materials for the four disorders were the same, except for the names of the respective disorders written on top of all pages.

We asked participants to depict their personal theories in the causal modeling framework of Morton (2004). This framework can be used as a tool to analyze and compare personal as well as scientific theories, to scrutinize their causal claims, and to see whether they provide a coherent explanation (e.g., Fava & Morton, 2009; Krol, Morton, & De Bruyn, 2004). In the causal modeling framework, symptoms and causal factors are assigned to a biological, cognitive, and behavioral level. Environmental factors are represented next to these three levels (see Fig. 1). Genetic factors, conditions of the brain, and other physical factors are assigned to the biological level. Factors on the cognitive level include, for example, executive functions, information processing, and emotion regulation. On the behavioral level all observable behaviors are placed. Environmental factors are, among others, family factors, school factors, or societal/cultural factors. Causal relations between elements within the same level and between elements on different levels are represented with arrows. The framework imposes one restriction on the possible relations: A direct relation between a factor on the biological and a factor on the behavioral level is impossible; it should always be mediated by one or more factors on the cognitive level.

Fig. 1.

Exemplary causal model for autism. The model is representative with respect to the number of variables and its overall complexity. Variables that were presented on the symptom list are in italics. The participant had drawn bidirectional relations between the two environmental variables and all behavioral variables. We indicate these with asterisks to enhance the clarity of the figure.

Participants received stepwise instructions to draw their personal theory for the first disorder. All instructions were typed, although participants could always ask for clarification. First, participants received a list of symptoms that were extracted from the descriptions of the four disorders in the DSM–IV-TR (APA, 2000). For all four disorders, participants received the same list (symptoms in italics in Table 1). The order of symptoms was randomized for each participant. We asked participants to indicate which symptoms they believed to be typical for the first disorder, that is, symptoms that often occurred with the disorder. Participants could add symptoms that they believed were typical but were not on the list. Second, participants received an empty causal modeling framework on an A4-sized sheet of paper, indicating the biological, cognitive, environmental, and behavioral levels (see Fig. 1). They were asked to place the typical symptoms for the disorder in the framework, on the respective levels.

Table 1.

Variables Included in Participants’ Theories

Level	Variable	ADHD	Autism	Conduct disorder	Reactive attachment disorder
Environmental	Parent-child interactions	0.15	0.15	0.40	0.20
	Pathogenic care	0.15	0.25	0.30	0.70
	Family problems	0	0.05	0.35	0.15
	Traumatic experience	0.05	0.05	0.20	0.10
	Peers’ influence	0.10	0	0.25	0
	School problems	0.05	0.05	0.10	0
	Lack of structure	0.40	0.25	0.10	0.05
	Social reactions	0.15	0.20	0	0.15
	Social isolation	0	0.15	0.10	0
Biological	Birth complications	0	0.05	0	0.05
	Genetic disposition	0.30	0.40	0.25	0
	Neurological structure	0.15	0.55	0.15	0.15
	Neurotransmitter dysregulation	0.30	0	0.10	0.05
	Hypo-arousal	0	0	0.15	0.05
	Food/medication	0.05	0.05	0.05	0
	Temperament	0	0	0.20	0.05
Cognitive	Intelligence/cognitive skills	0.35	0.50	0.20	0.15
	Executive functions	0.10	0.25	0.05	0
	Language delay	0	0.10	0	0
	Information processing	0.40	0.65	0.10	0.25
	Social information processing	0	0.20	0	0.05
	Rigidity	0.05	0.25	0.15	0
	Emotional/affect regulation problems	0.10	0.15	0.05	0.40
	Coping mechanisms	0	0	0.10	0.10
	Impulsivity	0.52	0	0.20	0
	Hyperactivity	0.55	0	0.05	0
	Inattention	0.75	0.05	0.05	0
	Lack of guilt, remorse	0.05	0.40	0.95	0.65
	Negative interpretations	0.10	0.20	0.75	0.60
	Low frustration tolerance	0.60	0.30	0.85	0.50
	Self-image	0.45	0.20	0.25	0.75
	Lack of empathy	0	0.90	0.60	0.65
Behavioral	Aggression	0.14	0.20	1.0	0.50
	Poor school performance	0.70	0.15	0.50	0.20
	Restricted, repetitive, stereotypic behavior	0	1.0	0	0.05
	Destruction of property	0.30	0	0.85	0.20
	Temper outbursts	0.20	0.40	0.50	0.25
	Violations of rules	0.30	0	0.90	0.25
	Stubbornness	0.30	0.35	0.45	0.35
	Blames others	0.20	0.05	0.85	0.50
	Hyperactive behavior	0.55	0	0.05	0
	Impulsive behavior	0.75	0	0.30	0.10
	Deceitfulness/theft	0	0	0.85	0.20
	Problems in communication	0.20	1.0	0.15	0.15
	Risk seeking behavior	0.75	0	0.80	0.45
	Ambivalent relations	0	0	0	0.20
	Distrust	0	0	0	0.20
	Inattention	0.40	0	0	0
	Odd reactions to stimuli	0	0.35	0.05	0
	Attachment	0.05	0.20	0.20	0.90
	Conflicts with family/authorities	0.52	0.25	0.90	0.75
	Problems in social interactions	0.65	1.0	0.50	0.75

Note: ADHD = attention-deficit/hyperactivity disorder. Values are proportions of the 20 participants. Variables that were presented on the symptom list are italicized.

Third, we asked participants to think about the biological, cognitive, and environmental factors they believed to cause and maintain the disorder, and to insert these at the appropriate levels in the framework. Fourth, participants were asked to indicate with arrows the causal relations they believed existed between the factors, between factors and symptoms, and between symptoms. We explained (a) that factors at one level can have relations with factors or symptoms at other levels, and with factors within the same level, (b) that factors may affect other factors directly, or indirectly, that is, via other factors, and (c) that mutual effects can be depicted with double arrows. Visualizations of these relations were shown in a separate figure, without naming any factors or symptoms.

Finally, participants received a questionnaire asking them to rate how representative their drawing was for their thinking about the disorder. Completing all tasks for the first disorder took about half an hour. Figure 1 shows a participant’s causal model for autism.

Participants were thanked and received a package of materials to make the drawings for the remaining three disorders at their own convenience. The package consisted of the instructions (the same as for the first disorder), symptom lists, empty frameworks, and questionnaires. The order of disorders was counterbalanced across participants. We asked them to keep the indicated order. Participants could send an email in case they had questions.

Participants added very few new symptoms. Only two symptoms were mentioned more than once. These were “ambivalent relations” and “distrust,” mentioned by four participants for RAD. We included these two in our analyses.

Interventions rating task

After participants returned their drawings, questionnaires inquiring about interventions’ effectiveness and effects were sent 2 weeks later by email. In between individualized materials for each participant were prepared. The first part of the questionnaire was identical for all participants. For all four disorders the same 10 interventions were listed in a table: (a) medication, (b) neuro-feedback, (c) psychoeducation, (d) parent support, (e) family therapy, (f) teacher support, (g) CBT, (h) aggression regulation therapy, (i) social skills training, and (j) creative therapy. These interventions were selected because they are recommended in the Dutch national guidelines for effective interventions (NJi, 2013; cf. Carr, 2009) and frequently used for at least one of the four disorders. Participants were asked to indicate the effectiveness of interventions for each disorder, with 1 indicating that the intervention would be the most effective, 2 that it would be the second most effective, and 3 that it would be the third most effective intervention. Also, they could indicate 0 for the intervention they believed to be least effective. It was explained that they should judge the effectiveness of each intervention as if it were the only intervention and that they should not consider other aspects like time or money. We decided to ask for the most and least effective interventions to get a broader range of rankings that could be compared to predictions derived from participants’ personal theories.

The next part of the questionnaire was individualized for each participant. For each disorder the participant received a separate table with column headings presenting the 10 interventions, and rows presenting the factors and symptoms included in the participant’s personal theory of the disorder. The participant received no information about the causal relations she or he had included in the models. The participant was asked to indicate which factors or symptoms were directly affected by each intervention (if any), by placing an X in the respective cells. We explained that interventions’ effects were indirect if these were achieved via other factors or symptoms. Questionnaires were returned by email.

Assignment of factors to categories

Participants devised a large number of causal factors affecting the symptoms. We assigned these factors to categories. Categories were based on Carr’s (2006) summary of different theories for ADHD, behavior disorders, and autism. A total of 24 different categories resulted. Two coders independently assigned all factors from participants’ models to categories. They agreed in 93.8% of their assignments (ranging from 89.5% to 97.8% for the four disorders). Disagreements were resolved by discussion. Participants sometimes placed the same variable (symptom or factor) on more than one level within the same model. Participants disagreed most often about the assignment of attachment, conflicts with family or authorities, and problems in social interactions. Some participants assigned these variables to the environmental level, others to the behavioral level, and some to both levels. Therefore we decided to add the interpersonal level, which has been proposed by Carr (2006), and to assign these three variables to this new level. A second major point of disagreement was the assignment of impulsivity, inattention and hyperactivity in ADHD. Some therapists assigned these variables to the cognitive, others to the behavioral, and some to both levels. It is interesting that Carr (2006) distinguishes for all three between a behavioral and a cognitive feature. As participants seem to endorse this distinction, we decided to recode these three variables into cognitive or behavioral impulsivity, inattention, or hyperactivity. Table 1 shows the resulting list of variables on which all further analyses were based.

Results

Personal theories

First, we assessed the symptoms, factors, and causal relations in participants’ personal theories. Participants’ models contained on average 12.7 variables and 25.2 causal relations (11.3 to 15.9 variables and 18.7 to 34.8 causal relations, for the different disorders). They judged their models to be reasonably adequate representations of their personal theories (M = 4.4, SD = 0.7 on a scale of 1 [not at all representative] to 7 [perfect]). Table 1 shows the proportion of participants who had included each variable in their model (with symptoms provided on the symptom list in italics).

As Table 1 and the average causal maps presented in Figure 2 indicate, participants created disorder-specific models. We calculated agreement among participants with respect to the variables included in their theories using Fleiss’s (1971) kappa, a generalization of Cohen’s kappa. It quantifies the agreement of the judgments of more than two people on a scale from −1 to 1, with 0 indicating no agreement. The resulting values were κ = .29 for ADHD, κ = .40 for autism, κ = .43 for CD, and κ = .33 for RAD. These kappa values indicate fair agreement for ADHD and RAD, and moderate agreement for autism and CD (cf. Landis & Koch, 1977).

Fig. 2.

Average causal models generated by clinicians. Shown variables were included by at least 50% of participants, variables with a thick border by 80%. Solid arrows indicate causal links included by at least 50% of participants, dashed arrows links that were included by at least 50% of participants who included the two connected variables in their model. Variables not connected by causal links in the graphs were generally connected in the causal map generated by the individual clinician.

We also calculated clinicians’ agreement about causal relations. A causal relation between two variables could be included only if both variables were present. To capture agreements that were representative for at least a considerable subgroup of participants, we included only pairs of variables into the analysis that were present in the models of at least 25% of participants (n = 5). Otherwise agreements would reflect only the fact that participants did not include the variables. For ADHD 148 pairs of variables fulfilled the criterion, 158 for autism, 346 for CD, and 199 for RAD. If participants agreed about causal relations, they were to either include or exclude causal relations for these pairs of variables. We again calculated agreements using kappa. The results showed only fair agreement among participants: κ = .29 for ADHD, κ = .26 for autism, and κ = .23 for RAD. Almost no agreement was found for CD, κ = .13. Taken together, these results show that participants’ theories of the disorders did not agree with each other very much.

The rather low agreement is also apparent in the causal models presented in Figure 2. For each disorder there were only a few variables that were included by a majority of participants and even fewer causal relations endorsed by at least 50% of participating clinicians.

Judgments of interventions

Table 2 shows how often each type of intervention was chosen as one of the three most effective. It also shows the mean rank assigned to each intervention with lower numbers indicating higher effectiveness. The most effective intervention received rank 1, the second most effective rank 2, the third most effective rank 3, and the least effective intervention rank 10. Whenever participants assigned the same number to several interventions, ranks were reassigned so that the rank sum remained constant across participants (a rank sum of 55, which equals the sum of ranks 1–10). This procedure was necessary for subsequent statistical analyses.

Table 2.

Judgments of Interventions for Effectiveness: Percentage of Participants Who Considered Each Type of Intervention as One of the Three Most Effective, and Mean Ranks (and SD) Assigned to Them

Intervention	ADHD	Autism	Conduct disorder	Reactive attachment disorder
Medication	901.72(0.29)	305.85(0.60)	56.98(0.44)	08.32(0.38)
Neuro-feedback	156.20(0.48)	07.10(0.27)	07.80(0.36)	07.45(0.33)
Psychoeducation	624.00(0.48)	852.48(0.44)	156.08(0.53)	454.43(0.55)
Parent support	554.08(0.52)	553.83(0.57)	404.68(0.52)	205.65(0.46)
Family therapy	116.83(0.46)	106.50(0.40)	454.50(0.55)	802.78(0.58)
Teacher support	296.20(0.51)	156.13(0.39)	106.35(0.36)	56.53(0.32)
Cognitive-behavioral therapy	265.55(0.40)	156.48(0.48)	703.63(0.49)	852.78(0.43)
Aggression regulation therapy	06.80(0.20)	56.73(0.30)	902.15(0.32)	106.18(0.26)
Social skills training	146.33(0.40)	753.28(0.46)	156.20(0.39)	56.35(0.20)
Creative therapy	07.50(0.35)	106.65(0.45)	106.65(0.36)	504.55(0.47)

As Table 2 shows, participants considered different interventions as most effective for each disorder. We calculated participants’ agreement about the effectiveness of interventions for each disorder. First, we calculated kappa for whether an intervention was chosen as one of the three most effective. The resulting values were κ = .32 for ADHD, κ = .38 for autism, κ = .36 for CD, and finally κ = .43 for RAD. These kappa values indicate fair (ADHD, autism, CD) and moderate agreement (RAD; cf. Landis & Koch, 1977). Second, we calculated multiple rank order correlations among participants for each disorder using Cohen’s w. The resulting values were w = .44 for ADHD, w = .39 for autism, w = .42 for CD, and w = .53 for RAD, which again indicate moderate agreements.

Prediction of judgments of interventions from personal theories

To predict the effectiveness of the different interventions, we included the interventions as additional factors in participants’ theoretical models. We calculated the causal centrality of the interventions as a measure of their causal influence on all other variables within the model. Highly central interventions directly or indirectly affect all or most other variables, whereas peripheral interventions target only variables not affecting other variables. Calculations were based on the computational model devised by Sloman, Love, and Ahn (1998; see the appendix for details). Centralities of all 10 interventions were calculated and turned into a rank order with the most causally influential intervention receiving rank 1. This was done for every participant and every disorder. Next, these rankings were correlated with the actual ranking of the interventions made by each participant. Table S3 in the Supplemental Material available online shows the resulting correlations for each participant and each disorder. The analysis showed that individual causal maps corresponded to participants’ estimates of efficacy often quite well, 34 out of 80 correlations were ρ ≥ .50, indicating an at least moderate correlation (cf. Cohen, 1988; Ferguson, 2009). For more than half of the clinicians, at least two of the four correlations were ρ ≥ .50. Effectiveness judgments of two participants corresponded almost perfectly with the predictions derived from their models for all four disorders. There were three participants, though, whose maps did not correlate with their estimates for neither disorder.

To analyze correlations statistically, we averaged them across participants for each disorder using a Fisher’s z transformation. Mean correlations were ρ = .55 for ADHD, ρ = .47 for autism, ρ = .56 for CD, and ρ = .55 for RAD. All four correlations were significantly larger than zero with p < .05. To use a more robust test of significance, we also did a bootstrap analysis. It turned out that all four mean correlations were significant with p < .01.

Discussion

We investigated clinicians’ personal theories for ADHD, autism, CD, and RAD to see (a) whether clinicians had similar theories for these disorders, (b) whether clinicians had similar expectations for the effectiveness of common interventions for each disorder, and crucially (c) whether clinicians’ judgments of interventions could be predicted from their personal theories of how biological, cognitive, environmental, and behavioral factors interact to cause the symptoms of a disorder. Participating clinicians worked in similar intramural institutions, and had ample experience with children with the four disorders. Still, clinicians’ personal theories varied considerably, resulting in no more than moderate agreements. Also, we found only fair to moderate agreement among participants’ judgments about the relative effectiveness of 10 common and recommended interventions for the disorders. Despite the low level of agreement we could predict participants’ judgments of the effectiveness of interventions from their personal theories. More precisely, clinicians’ causal models predicted judgments of efficacy on the group level, and they often predicted the judgments of individual clinicians for specific disorders. Thus, our findings indicate that clinicians’ personal theories about disorders are likely to be important for clinicians’ reasoning about interventions. They at least sometimes seem to use their personal theories to assess effectiveness.

These results are in line with theories assuming that decision makers use causal models to infer the effects of available interventions (Hagmayer & Witteman, in press; Sloman & Hagmayer, 2006). They also extend previous research on clinical judgments. Specifically, they add to the findings of Ahn et al. (2009), who showed that clinicians hold beliefs about the biological, psychological, and social nature of DSM–IV–TR disorders and that these beliefs are related to preferences for medication or psychotherapy. Going beyond this study, we analyzed clinicians’ theories about the causal relations among biological, environmental, cognitive, and behavioral variables, and showed that these assumptions are related to the judgments of specific types of interventions that are recommended in the literature for these disorders. Finally, it was found earlier that clinicians construct different causal models to explain the problems of the same client, and that from these individual models intervention judgments could be predicted (De Kwaadsteniet et al., 2010). The present results indicate that differences in client models and intervention judgments could at least partly be due to different personal theories of disorders.

Strategies to assess effectiveness

Personal theories explained participants’ judgments of effectiveness quite well in many, but not all cases, and almost perfectly for only two participants. They failed to make any accurate prediction for three participants. These findings indicate that participants probably used a variety of different strategies to rate the effectiveness of interventions.

Inferences about effectiveness can be drawn from causal models in several ways. One strategy would be to activate the causal model and run a mental simulation to predict the consequences of each intervention (Pliske & Klein, 2003; Sloman & Hagmayer, 2006). A simpler causal heuristic would be to activate a causal model and rate interventions that target root or causally central causes as more effective than interventions targeting mere symptoms (cf. Ahn et al., 2009). Both strategies may underlie the significant correlations we found.

Another judgment strategy that some participants may have used would be to rely on the results of effectiveness studies or on recommendations from evidence-based guidelines. The Dutch guideline for children and adolescents with ADHD advises medication, psychoeducation, and parent support (NJi, 2013; Trimbos-instituut, 2007, 2011). Participants’ ratings (as shown in Table 2) tended to be in line with these recommendations. For CD behavioral training for parents is advised for children younger than 12 years old, CBT and family therapy for children 12 and older (NJi, 2013; cf. Carr, 2009; Dretzke et al., 2005; Ollendick & King, 2000). Participants who worked mostly with teenagers showed a strong preference for behavioral interventions, but fewer than half ranked family therapy as one of the three most effective interventions. For the treatment of autism and RAD, good evidence is scarce (cf. Carr, 2009; Ollendick & King, 2000). NJi (2013) advises the use of psychoeducation and parent or family support for autism first. Our participants seem to follow this advice. For RAD, parent support and family therapy is recommended (NJi, 2013). Participating clinicians only partially followed this advice. CBT was chosen more often as most effective.

Some clinicians may have relied on their clinical experience. If they have often experienced success with one type of intervention they might generally expect this intervention to be effective for a particular disorder. This strategy may explain judgments that were unrelated to clinicians’ personal theories. Clinical experience, however, may also support personal theories and therefore strengthen the belief in them.

To find out which strategy or combination of strategies individual clinicians use, carefully controlled experimental studies involving many cases would be needed (for potential methodologies, see Glöckner & Witteman, 2010; Schulte-Mecklenbeck, Kühberger, & Raynard, 2011).

Personal theories versus scientific theories

We used Morton’s (2004) causal modeling framework to analyze clinicians’ personal theories of four common developmental disorders. The same framework has been used before to analyze scientific theories of three of these disorders. Krol et al. (2004) analyzed scientific theories for CD, and Morton (2004) summarizes scientific theories of autism and ADHD. Although we did not intend to compare participants’ theories with scientific theories, we noticed a few interesting points. First, it seems that participants’ personal theories tended to include less specific factors than scientific theories. Although some participants did use specific factors (e.g., “abnormality in gray mass”), such specific factors were rare. In addition, several participants had used the same variable on different levels in one model. For example, “intelligence” was placed at both the cognitive and biological levels, and “communication problems” was placed at the biological, cognitive, environmental, and behavioral levels. Ahn and colleagues (2009) reported a similar observation. They found that clinicians sometimes rated a specific causal factor to be biological, psychological, and environmental at the same time.

Second, participants presented quite elaborate personal theories with respect to the causal relations among the various cognitive and behavioral factors and symptoms. Feedback loops occurred frequently in participants’ drawings. Scientific theories tend to be simpler at this point, focusing on major causal mechanisms. They are also less complex on the behavioral level, often just mentioning a few disorder-defining symptoms. This finding may be due to the fact that our participants started the task with the identification of typical symptoms from a list. This procedure might have cued participants to focus on the behavioral and cognitive level. In addition, dynamic networks of symptoms may be important in clinicians’ personal theories, either from recent growing scientific interest in the analysis of disorders as dynamic networks (e.g., Borsboom & Cramer, 2013), or from clinical experience. According to dynamic network approaches, disorders are constituted by an interaction of symptoms (emotions, cognitions, behaviors), and negative vicious circles, rather than by symptoms, which are effects of underlying root causes.

Finally, participants seemed to have integrated elements of different scientific theories in their personal theories. It is interesting that one of the conclusions that Krol et al. (2004) arrived at was that different scientific theories for CD complement each other. Thus, not only would it have made sense for participants to combine different theories in their personal theories, this would also be in line with EBP. However, only a few clinicians mentioned specific scientific theories when queried about them in the questionnaires. For autism, the theory of mind was mentioned five times. With RAD, attachment theory was mentioned three times, twice with the specification that it was Bowlby’s. Apparently, participants were often not aware of the origin of the scientific theories they may have used.

Limitations

Our study focused on four developmental disorders and 10 respective interventions and involved 20 experienced child psychotherapists. We cannot be sure how results generalize to other disorders, interventions, and professionals providing mental health care. However, the level of disagreement that we found was striking, given that our participants worked with similar clients in three similar institutions and had plenty of experience with the four disorders. Hence we would expect even less agreement across clinicians having a different therapeutic or occupational background and working with other groups of clients. We would still expect to find coherence between personal theories and effectiveness ratings because a high sensitivity to causal beliefs has been found for a broad variety of disorders, across different disciplines, and different levels of expertise (Ahn et al., 2009; De Kwaadsteniet, Kim, & Yopchick, 2011; Kim & Ahn, 2002; Proctor & Ahn, 2007).

Another limitation is that participants were asked to rate only effectiveness. They did not have to make actual choices for individual clients. Intervention choices in practice also depend on other factors like budget, time, personal characteristics of clients (e.g., motivation) and clinicians (e.g., experience), and institutions (e.g., protocols; cf. Clarkin & Levy, 2004; Nelson & Steele, 2008).

Implications

Our results indicate that many clinicians rely on their personal causal theories, which differ to a rather large extend across clinicians, when judging the effectiveness of interventions. This finding raises several important implications for research in clinical decision making, professional training, and evidence-based clinical practice. We will discuss these in turn.

One implication is that we need to know more about how clinicians acquire their personal theories, which later guide their decisions. At present we know rather little about how clinicians integrate their knowledge about scientific theories with knowledge derived from clinical practice. Bayesian theories of learning (e.g., Tenenbaum, Kemp, Griffiths, & Goodman, 2011) provide a conceptual and formal framework that would allow us to describe and explain how experience and knowledge interact in learning. Respective studies will help us to understand why personal theories of clinicians diverge to so such a large degree. A second important empirical question is when clinicians use personal causal theories to judge treatment options and to make decisions on care. Theoretical analyses indicate that it may not always be wise to rely on personal theories (Hagmayer & Witteman, in press). For straightforward diagnoses of disorders for which the most effective treatments are known, rule-based or script-based reasoning might be appropriate. By contrast, if a client presents with multiple disorders, or an earlier treatment was ineffective, causal reasoning may help the clinician to identify the most effective intervention. Whether clinicians use their personal theories in a way that improves their treatment choices is not known yet.

Our findings also entail that current professional training—at least in the Netherlands—seems insufficient for clinicians to acquire highly valid and reliable theories of developmental disorders. Therefore, an important question is how training programs should be redesigned for clinicians (practicing and in training) to learn about empirically well-validated scientific theories and integrate these theories into their personal theories, such that personal theories become more similar and evidence-based. The same question arises for knowledge about effective interventions, and their mechanisms of change. If clinicians are supported in acquiring evidence-based theories of disorders that include evidence-based treatments, they may become more likely to apply scientific evidence in their practice. Clinicians may also need training and support in the explicit formulation and testing of causal hypotheses for specific clients (cf. De Kwaadsteniet et al., 2010; Ghaderi, 2007).

Finally, there are important implications of our findings for EBP. The first is that clinicians need evidence that would allow them to form valid, evidence-based personal theories. Unfortunately, for many interventions and disorders there is still an incomplete evidence base (cf. Carr, 2009; Ollendick & King, 2000). In particular, mechanisms that lead to and maintain disorders and mechanisms by which interventions cause change are still largely unknown (Bieling & Kuyken, 2003; Borsboom & Cramer, 2013; De Los Reyes & Kazdin, 2006; Kazdin, 2008). As much is still unknown, clinicians’ personal theories may vary considerably and therefore afford different judgments and decisions. This explains why clinicians’ personal theories at present can be a source of bias (Garb, 2005) rather than a pillar of EBP. Therefore, clinicians should be aware that their causal beliefs may only partially conform to existing evidence, but may nevertheless shape their judgments and decisions. That is why they should be advised to consider other sources of evidence whenever possible.

The finding that clinicians consider causal mechanisms when judging the effectiveness of treatment options also has implications for the communication of scientific evidence about the efficacy of interventions. To inform and change clinical practice toward EBP, it may be important to communicate information about the mechanisms by which interventions cause their effects. This information would enable clinicians to incorporate scientific information about the effectiveness of interventions into their personal theories more easily. In consequence, they may sooner apply new insights. Once we have and communicate a more complete understanding of the mechanisms causing and maintaining disorders and of the mechanisms making interventions effective, clinicians may have more valid personal theories, and make more reliable and evidence-based decisions (cf. Grove, 2005; Shrout, 2011).