Problem-solving strategies in psychiatry: differences between experts and novices in diagnostic accuracy and reasoning.

BACKGROUND: The purpose of this study was to examine and compare diagnostic success and its relationship with the diagnostic reasoning process between novices and experts in psychiatry.
METHODS: Nine volunteers, comprising five expert psychiatrists and four clinical clerks, completed a think-aloud protocol while attempting to make a DSM-IV (Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition) diagnosis of a selected case with both Axis I and Axis III diagnoses.
RESULTS: Expert psychiatrists made significantly more successful diagnoses for both the primary psychiatric and medical diagnoses than clinical clerks. Expert psychiatrists also gave fewer differential options. Analyzing the think-aloud protocols, expert psychiatrists were much more organized, made fewer mistakes, and utilized significantly less time to access their knowledge than clinical clerks. Both novices and experts seemed to use the hypothetic-deductive and scheme-inductive approaches to diagnosis. However, experts utilized hypothetic-deductive approaches significantly more often than novices.
CONCLUSION: The hypothetic-deductive diagnostic strategy was utilized more than the scheme-inductive approach by both expert psychiatrists and clinical clerks. However, a specific relationship between diagnostic reasoning and diagnostic success could not be identified in this small pilot study. The author recommends a larger study that would include a detailed analysis of the think-aloud protocols.

Introduction

Reviewing the literature reveals no specific studies examining or exploring the nature of the structure of psychiatric knowledge or its development processes, and there are no studies which have investigated the differences between psychiatric experts and psychiatric novices in these domains. However, there is an indication from research that categorizations of physical illness by doctors may play a central role in the reasoning processes used by them in arriving at diagnostic and treatment decisions.1

Experts have a more complex and organized knowledge structure, which allows them faster access to their knowledge and enables them to be more efficient and faster at problem-solving than novices. It has also been demonstrated that clinical reasoning is idiosyncratic and varies from clinician to clinician. This may be due to many factors, depending on the nature of the knowledge, its depth, how the knowledge has been structured, on perception of the problem to be solved, and the clinician’s age. Expert performance is predominantly mediated by complex acquired skills and physiological adaptations, and deliberate practice can lead to anatomical changes resulting from adaptations to intense physical activity. 2 We commonly use semantic anchors as reminders, and Bordage has emphasized semantic competencies among strong diagnosticians who make diagnoses on more complex semantic connections as opposed to weak diagnosticians.

Elstein et al initially described hypothetic-deductive reasoning as being the only reasoning method used by novices and experts to reach a diagnosis.3 However, many subsequent authors have viewed this reasoning strategy as a weak method of problem-solving. For example, Schmidt et al described a four-stage theory with gradual development of knowledge network into scripts that eventually solidify in the mind as representations of the original learned problem, but developed by experience and practice. Experts then consider related diagnostic hypotheses by means of a succession of limited comparisons. The problem-solving strategy utilized is dependent on the knowledge structure available to the problem-solver and the knowledge structure available depends on the domain in which the problem to be solved resides. This forms the basis for the scheme-inductive reasoning utilized by experts to access their knowledge. Before becoming more expert in problem-solving, learners progress through several transitional stages, characterized by different knowledge structures, ie, elaborated casual networks, abridged networks, illness scripts, and instance scripts.4

The purpose of this research was to examine and compare diagnostic success in psychiatry and its relationship with the diagnostic reasoning process among clinical clerks and experts in psychiatry.

Materials and methods

This was a pilot study utilizing a convenience sample of volunteers from the faculty and clinical clerks.

Participants

Two sets of volunteers participated in the experiment. One sample was formed of five expert psychiatrists, with at least 10 years experience as independent consultants in the field of psychiatry, as clinicians or as faculty members at University of Calgary. The other sample was composed of four clinical clerks, who completed their didactic courses of instruction as medical students and had at least 2 weeks of exposure to clinical psychiatry in the same faculty.

Materials

A case scenario with multi-axial diagnoses confirmed by the DSM IV (Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition) was selected and described (Table 1), so that it covered a wide range of expertise at different levels of the psychiatric and medical domains used in psychiatric practice. This included a detailed and organized description of the relevant history and mental state of a real patient (Appendix A). Information both relevant and irrelevant to the diagnoses were included. On a separate page, the participants were instructed to write the diagnosis along the five DSM-IV axes. The primary diagnosis on Axis I in this case was major depression. The diagnosis was confirmed by the treating psychiatrist using the Mini- International Neuropsychiatric Interview schedule.5

Table 1

DSM-IV diagnosis scoring guide for Axis I and Axis II

Diagnostic DSM-IV Axes	Diagnosis (dependent variables including items in the differential)	Scoring (minimum–maximum)
Axis I	Major depression, recurrent, generalized anxiety and obsessive-compulsive disorder (or symptoms), history of bereavement, alcohol dependence, and cocaine abuse	(0–6)
Axis III	Severe hyponatremia, likely due to abnormal antidiuretic hormone excretion syndrome	(0–1)

Abbreviation: DSM-IV, Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition.

Procedure and data collection

Participants were asked to read the case presented carefully and to think aloud about how to make a detailed DSM-IV diagnosis along Axis I (diagnosis of formal psychiatric disorders) and Axis III (diagnosis of medical conditions). While thinking aloud, they were asked to describe how the diagnosis was reached on each of these two axes, and to record their thoughts clearly on a tape recorder.

Each participant was tested individually, was provided with a small tape-recorder to use while thinking aloud during diagnosis of the case, and was asked to keep track of and report the time taken to complete the task. Two independent experts, ie, a psychiatrist and a psychologist, reviewed the tapes, which were collected from all expert psychiatrists and clinical clerk participants, and from the verbal discourse, the most predominant diagnostic reasoning strategy used was analyzed and determined, ie, hypothetic-deductive or scheme-inductive.

The diagnoses provided by each participant were reviewed to estimating the likelihood of successful primary psychiatric and medical diagnoses on Axes I and III. The number of correct differential diagnoses on Axis I and the time taken to complete the experiment was estimated for each participant (Table 1).

The frequency of diagnostic success on each axis of the DSM-IV was compared between clinical clerks and expert psychiatrists. The odds of correct diagnoses on the different DSM-IV axes, number of differentials on Axis I, and time taken to complete the experiment were used as the dependent variables. The relationship between the diagnostic strategy used and frequency of diagnostic success was examined and compared between clinical clerks and expert psychiatrists.

The study was approved by the conjoint ethics board of the University of Calgary, and all participants consented to the anonymous use of their data for the purpose of this study. The Statistical Package for Social Sciences (SPSS Inc, Chicago, IL) was used to analyze the data, and a P value < 0.05 was considered to be statistically significant.

Results

Participants in this study were five faculty members (mean age 48 years), with a mean 14 years of experience as independent consultants, and four clinical clerks (mean age 26 years) who had completed medical school and at least 2 months of psychiatry training.

Reliability of case diagnosis and diagnostic reasoning strategies

There was an interrater reliability coefficient of 0.8 among the experts for the DSM-IV Axes I and III diagnoses when the tapes were reviewed independently, and there was a 100% agreement between them with regard to the diagnostic reasoning strategies used by participants.

Relationship between diagnostic reasoning and diagnostic success

Table 2 summarizes the differences between the expert psychiatrists and clinical clerks with regard to the likelihood of a correct primary diagnosis of the case along the two axes of the DSM-IV, the number of differential diagnoses, and time taken to complete the experiment.

Table 2

Results of the Mann–Whitney test comparing diagnostic accuracy between experts and novices

Mann–Whitney test U value for P	CCs (n = 4)		EPs (n = 5)
	MR	SR	MR	SR
Odds of correct primary diagnosis (Axis I) 2.5 0.025	3.13	12.50	6.5	32.5
Odds of correct medical diagnosis (Axis II) 0.0001 0.005	2.5	10.0	7.0	35.0
Number of diagnoses on Axis I differential 1.5 0.029	7.13	28.5	3.3	16.5
Time to complete experiment 00 0.014	7.5	30.0	3.0	15.0

Abbreviations: CCs, clinical clerks; EPs, expert psychiatrists; MR, mean rank; SR, sum of ranks.

The independent-samples Mann–Whitney U test was used to examine differences between expert psychiatrists and clinical clerks on the following tasks: first, the odds of the correct primary psychiatric diagnosis on Axis I and a primary medical diagnosis on Axis III; second, the number of differentials made on Axis I; and third, the time taken to complete the experiment. Close inspection of Table 2 demonstrates that expert psychiatrists had a significantly greater likelihood than clinical clerks of diagnostic success, and took significantly less time to discuss the case and access their knowledge. The most commonly used diagnostic reasoning strategy by both novices and experts was hypothetic-deductive. Expert psychiatrists gave significantly fewer correct differentials on axis I, and were more likely than clerks to reach the correct primary psychiatric diagnosis. All clinical clerks failed to make the correct medical diagnosis on Axis III for a patient with severe hyponatremia. In contrast with the clerks, all expert psychiatrists gave the correct medical diagnosis and two experts out of five were also able to discuss the causes for hyponatremia in more depth.

Diagnostic reasoning

From the analysis it was determined that both expert psychiatrists and clinical clerks utilized hypothetic-deductive more than scheme-inductive approaches. Two of the experts (40%), and one clerk (25%) seemed to adopt a predominantly scheme-inductive approach. These participants seemed to stay focused on the most relevant primary diagnosis and to work through fewer differentials.

Discussion

Our pilot study, although the earliest of its kind in psychiatry, sheds light on how diagnostic success in clinical psychiatry relates to expertise, and our results may assist in improving clinical psychiatric teaching and supervision methods for clinical clerks in psychiatry. Diagnostic reasoning in psychiatry is complex. Publication of the DSM-III in 1980 represented an attempt to distinguish between the mentally well and the mentally ill, and the essential focus of psychiatric knowledge shifted from the clinically-based biopsychosocial model to a research-based medical model.6 Agreement on a diagnostic classification only partially resolves the problem, because the concrete value of the classification is dependent on the reliability and validity of the primary data available from the patient.7 However, development of the epidemiological approach in mental health is still hampered by a number of methodological difficulties. First, there are problems of case definition, ie, the subjectivity of ratings, and interrater issues may make it difficult to draw a sharp line between cases and noncases. Second, there are problems of classification of mental disorders, taking into account the complex network of comorbidity and dual diagnoses, which are commonly found with psychiatric disorders among the psychiatric population. Clinicians at times may find a less stigmatizing diagnosis more palatable to make, despite it being a less convincing one.

In the present pilot study, the majority of participants (75% clinical clerks and 60% experts) used the hypothetic-deductive approach. This could be due to the fact that there is always missing information when making a psychiatric diagnosis. Corroborating the history given by patients and seeking further information from other sources appears to be crucial to psychiatric diagnosis. Elstein et al explained that even experts use the hypothetic-deductive approach because not all information relevant to the diagnosis is available to them in the initial interview.3 It has also been demonstrated in the psychiatric literature that combining a structured interview with a review of the patient’s medical records appears to produce a more accurate primary diagnosis and to identify more secondary diagnoses than routine clinical methods.8 It is understandable why expert psychiatrists performed better on reaching more correct diagnoses, and were faster at accessing their knowledge than clerks. Performance is dependent on both the time spent practicing the relevant skills and on the quality of actual clinical practice.9

Notwithstanding the small sample size and the test situation being limited to one case scenario, the present pilot study is the first of its kind, to the author’s knowledge, to examine differences in clinical diagnostic accuracy between novices and experts in psychiatry.

Future research should include a heterogeneous group of case scenarios from different psychiatric diagnoses on different DSM axes and a larger sample of expert and novice volunteers in order to examine the relationship between diagnostic reasoning and diagnostic success in more depth. The main limitation of this research is that it was a pilot study with few participants and only included one case scenario of a real patient.

Conclusion

Novices may be less likely than experts to make successful diagnoses of psychiatric conditions and important comorbid medical conditions. Experts have faster access to their knowledge structure than novices. However, a specific relationship between diagnostic reasoning and diagnostic success could not be identified in this small pilot study. Close supervision during training with an emphasis on intentional good quality clinical practice should be considered during clerkship rotations in psychiatry. The author recommends that future research should include more case scenarios and a larger sample of participants in a more elaborate controlled study to examine differences in knowledge structure between novices and experts and their relationship with diagnostic reasoning.

Axis I: Psychiatric diagnosis…………………………………………………………
Axis Ill: Medical diagnosis…..………………………………………………………
Demographics
Age:		Gender: M/F		Faculty/Student (circle)
Duration in minutes:………….. Years of experience…………. Signature………….