Diagnostic Reasoning Assessment Toolkit: Guided Reflection and Standardized Cases for At-Risk Final-Year Medical Students.

INTRODUCTION: A failing diagnostic reasoning performance may represent student deficiency in a number of potential areas. However, many standard clinical skills assessments do not offer detailed assessments of diagnostic reasoning ability. This toolkit was designed to identify specific learner deficiencies with respect to diagnostic reasoning by focusing on individual student remedial work and by standardizing faculty evaluation.
METHODS: Educational objectives were derived from institutional patient care competency learning objectives at the Indiana University School of Medicine. Review of existing clinical skills remediation literature yielded a design that combined two learning methods: guided reflection and standardized patient cases.
RESULTS: Over the 2014-2015 academic year, 12 final-year medical students used this resource to help develop an individual remedial learning plan prior to retaking a failed standardized assessment. Students were generally satisfied with the combined guided reflection and standardized case learning methods. DISCUSSION: Unique final-year medical student scheduling pressures, combined with a reporting time line for both institutional high-stakes OSCE remediation exams and the USMLE Step 2 Clinical Skills exam, incentivized failing students to schedule a retest on a short time line, often leaving little time for critical preparation. This resource offered an opportunity to efficiently spend limited preparation time to individualize exam preparation using a variety of faculty facilitators. The simplistic design was readily deployable to multiple faculty remediation mentors. Our institution can now provide a standardized diagnostic reasoning remedial evaluation using numerous clinical faculty based at any of our nine campuses.

Educational Objectives

By the end of this activity, learners will be able to:

Demonstrate prioritization of key data collection.

Correctly apply history and physical exam elements to different clinical contexts.

Evaluate and prioritize diagnostic possibilities.

Formulate diagnostic plans for different clinical contexts.

Interpret clinical data and apply foundational knowledge.

Create supporting documentation of diagnostic reasoning.

Introduction

Diagnostic reasoning is a critical element of physician clinical practice. Clinical diagnostic reasoning represents a complex, metacognitive process that is challenging to assess and to remediate due to its being a multidimensional construct. Given the broad range of patient care skills implicit in diagnostic reasoning assessment, students are often not provided specific performance data sufficient to guide further clinical skills development. The United States Medical Licensing Exam (USMLE) Step 2 Clinical Skills score report defines failing student performance in three broad categories: communication and interpersonal skills, spoken English proficiency, and integrated clinical encounter (ICE).[1] The ICE component focuses on diagnostic reasoning assessment. At the Indiana University School of Medicine, a standardized patient objective structured clinical exam (OSCE) is used to assess the diagnostic reasoning capacity of final-year medical students. Score reporting of this institutional OSCE assessment is modeled after that of the USMLE Step 2 Clinical Skills, and the diagnostic reasoning subcomponent of student performance is reported separately. Failures of diagnostic reasoning assessments potentially represent student deficiency in a variety of clinical elements, including, but not limited to, technical skill performance, clinical skill application, time management, and diagnostic reasoning.

In order to identify specific diagnostic reasoning elements that contribute to learner deficiency, educational objectives were derived from both behaviors and domains associated with the Indiana University School of Medicine institutional patient care competency learning objectives. Educational objectives are summarized in Table 1.

Table 1.

Design of Diagnostic Reasoning Learning Objectives

Domain	Institutional Learning Objective-Derived Behavior	Toolkit Educational Objective
Patient Care Skills	Efficiently and accurately obtaining key clinical data from patient encounters	—Demonstrate prioritization of key data collection—Correctly apply history and physical exam elements to different clinical contexts
Clinical Reasoning	Thoughtfully applying interview questions, physical exam maneuvers, and diagnostic testing both to create and to refine differential diagnoses	—Evaluate and prioritize diagnostic possibilities—Formulate diagnostic plans for different clinical contexts—Interpret clinical data and apply foundational knowledge
	Justifying differential diagnoses with organized documentation of supporting clinical data	—Create supporting documentation of diagnostic reasoning

Guidelines for development of competency-based remediation programs have suggested that programs should generally incorporate four basic elements: early assessment via multiple tools; identification of specific behavior deficiencies with individual learning plan development; instruction that involves practice, feedback, and reflection; and reassessment to certify competence.[2] Educators have generally agreed that individualization of remediation efforts strengthens clinical remediation programs and that faculty are a limiting resource for successful implementation of such programs.[3] Guided reflection with feedback has been used to help to engage learners in identifying learning needs and in developing an individual learning plan.[4] While medical schools often provide structured remediation resources for students who fail to meet minimum clinical competency standards, clinical skills remediation programs have been reported as often lacking in standardization or validation, and much is unknown about the best design of successful clinical remediation resources.[5,6]

Best-practices literature recommendations were reviewed to develop a standardized resource that focuses on individual student preparation for repeat diagnostic reasoning assessment. Scheduling challenges unique to the institutional high-stakes OSCE assessment and to USMLE Step 2 Clinical Skills testing necessitated the creation of a resource that is uniform, efficient, and portable. In the experience of students reported here, individualized remediation efforts generally required urgent instruction following notification of a failing diagnostic reasoning performance in order to facilitate timely retesting. Geographic disparity of the student body over a nine-campus system posed a challenge to delivering a consistent remediation pathway with many different faculty members. Thus, efficient remediation resources specifically designed for this population were needed for swift deployment on demand.

This resource targets final-year medical students who are deficient in diagnostic reasoning. Prerequisite experience with clinical practice allows faculty deploying this resource to provide quality feedback specific to educational objectives. Additionally, prior faculty experience with standardized patient or role-playing learning methods (e.g., oral boards preparation) is advantageous.

Methods

This resource was designed to assist medical students who are deficient in diagnostic reasoning skills in developing an individualized remedial learning plan. Selection of learning methods and overall design were based on existing literature recommendations, as outlined in the Introduction.

The institutionally developed guided reflection exercise (Appendix A) was designed to engage the learner in the development of an individual learning plan following a failing diagnostic reasoning performance but prior to standardized patient case activities. When first discussing goals for the session with the learner, faculty should invite the learner to discuss the initial draft of an individualized learning plan. Student preparation, along with faculty knowledge of a student's failing performance, can yield a valuable gap analysis to help drive individualized learning plan creation.

The original standardized patient cases (Appendices B–E) were developed by me and designed for timed tabletop exercises. While standardized patients represent a generally preferred method for case-based assessments, the use of standardized patients for individual learner needs is often limited by personnel, physical place, and time resources. Faculty role-playing may be used as a surrogate for available standardized patient resources. Each case should be timed, and faculty should consistently use a reliable timing device with an alarm to strictly adhere to the allotted time for each portion of each case. One timing model, described by the USMLE Step 2 Clinical Skills instructions, is to allow students 15 minutes to complete the patient encounter, followed by a subsequent 10 minutes to document a patient note, associated diagnostic reasoning, and desired diagnostic studies.[1]

For each standardized case, students should be provided the first three pages of the case. Page one contains door note information about each patient scenario. Pages two and three are provided for the learner to document a patient note and diagnostic reasoning associated with the case. This format is consistent with USMLE Step 2 Clinical Skills instructions.[1] Subsequent pages of each case provide patient information such as history and physical examination information, some diagnostic possibilities with supporting clinical data, and suggested case debriefing questions.

History information, physical exam findings, and potential diagnoses can all shape discussions regarding diagnostic reasoning performance. Diagnoses and diagnostic studies suggested by this resource are not exhaustive, and other reasonable approaches should be considered. After each case, faculty should guide the student through a critical debriefing of his or her performance using suggested case debriefing questions provided at the conclusion of each case.

Appendix F is a collection of original clinical vignettes that I developed. These vignettes range from low to high complexity. Faculty should use these vignettes for additional exercises as guided by learner need.

Results

The toolkit resources were implemented as a diagnostic reasoning remediation pathway for final-year students at the Indiana University School of Medicine. During the 2014–2015 academic year, 15 students were identified as deficient in diagnostic reasoning via the ICE score component of either the USMLE Step 2 Clinical Skills or the institutional high-stakes final-year OSCE remediation exam. Twelve of these students used the toolkit resources to generate an individual learning plan for remediation. Results are summarized in Table 2.

Table 2.

At-Risk Student Performance on Next Attempt at Diagnostic Reasoning Assessment

Student Cohort	Assessment Method	Passed Next Attempt	Failed Next Attempt	Total
ICE failure	USMLE Step 2 Clinical Skills	2	1a	3
	Institutional OSCE	0	0
ICE failure + toolkit	USMLE Step 2 Clinical Skills	10	0	12
	Institutional OSCE	2	1b

Abbreviations: ICE, integrated clinical encounter; OSCE, objective structured clinical exam; USMLE, United States Medical Licensing Exam.

Subsequently remediated following toolkit assessment, passing on third attempt.

Subsequently required individual clinical remediation course.

Medical students failing the USMLE Step 2 Clinical Skills ICE component on first attempt represented 2% of student examinees in 2014–2015.[7] Indiana University School of Medicine students were selected as a convenience sample in an attempt to offer toolkit resources to each student with a failing diagnostic reasoning performance. This student sample represented a USMLE Step 2 Clinical Skills exam ICE component first-attempt failure rate of approximately 2% for the school in 2014–2015. The two students who were identified by an institutional high-stakes final-year OSCE remediation exam represented approximately 1% of the school's 2014–2015 final-year students.

Formal evaluation of the resource was not performed by learners or by faculty. Students were generally satisfied with the combined guided reflection and standardized case learning methods. This design allowed learners and instructors to focus on individual learner needs within the stated objectives. All sessions involved individual instruction for a single learner by a single instructor. I worked with each student individually, and two chief residents served as additional instructors for some of these students. Both chief residents relayed positive feedback regarding the efficacy of the materials and the methods.

Discussion

Best-practices recommendations were applied to create a simple, standardized, transferrable remediation resource that reliably promotes individual remediation planning for our target learner population. Results of using this resource at our institution were both consistent and positive. Combining detailed objectives with a consistent methodology yielded detailed performance feedback about diagnostic reasoning. The guided reflection tool was effective at engaging learners in development of an individual learning plan. Individual faculty instruction provided a large volume of specific and immediate feedback for the learner.

Unique final-year medical student scheduling pressures, combined with a reporting time line for both institutional high-stakes OSCE remediation exams and the USMLE Step 2 Clinical Skills exam, incentivized failing students to schedule a retest on a short time line, often leaving little time for critical preparation. This resource offered an opportunity to efficiently spend limited preparation time to individualize exam preparation using a variety of faculty facilitators. The simplistic design was readily deployable to multiple faculty remediation mentors. Our institution can now provide a standardized diagnostic reasoning remedial evaluation using numerous clinical faculty based at any of our nine campuses.

The results were limited by small intervention and control samples of remediation students, as well as by a small faculty sample. All data were collected via convenience sampling. In addition to these toolkit resources, numerous variables may have contributed to students successfully passing the next examination attempt including, but not limited to, practice effect, extracurricular study materials, and interval educational experiences within the programmed curriculum.

Our institution continues to employ this resource in final-year diagnostic reasoning remediation pathways. The school plans to revise cases and debriefing questions to better focus tools on specific objectives, as well as to add more standardized cases and debriefing items as needed. One future direction may be to measure skill retention following remedial work employing this resource, as this may reflect deep learning promoted by these tools.

A. Guided Reflection Exercise.docx

B. Practice Case 1.docx

C. Practice Case 2.docx

D. Practice Case 3.docx

E. Practice Case 4.docx

F. Clinical Vignettes.docx

All appendices are peer reviewed as integral parts of the Original Publication.