A Psychiatric-Specific Entrustable Professional Activity for the Evaluation of Prospective Psychiatric Residents: Towards a National Standard.

INTRODUCTION: Entrustable professional activities (EPAs) were developed as a way to ensure adequate skills of the medical school graduate. While the 12 EPAs apply to all medical specialties, EPA 1, "Gather a history and perform a physical examination," applies most explicitly to psychiatry through the performance of a mental status exam. Although proficiency in performing a psychiatric interview and mental status exam evolves throughout a psychiatrist's professional life, basic proficiency is essential in order to function as a psychiatry intern. We developed a tool for assessing the mental status exams conducted by future psychiatry residents.
METHODS: Our tool contains both a video of a psychiatrist interviewing a patient and a mental status exam rating sheet that can be used when students present a mental status exam orally or in writing. We incorporated feedback from psychiatry educators at an annual meeting of the Association for Medical Student Educators in Psychiatry, followed by the reiteration of the video and the rubric. Subsequently, the rubric was verified on the performance of a cohort of 13 third- and fourth-year medical students from three institutions.
RESULTS: In their mental status exam presentations, students covered all the items measured by the rubric. There was a significant difference between the third- and fourth-year medical students in describing the cognitive exam. DISCUSSION: Overall, our tool offers an opportunity to standardize mental status presentations by senior medical students who wish to specialize in psychiatry.

Educational Objectives

After using this resource, faculty will be able to evaluate medical students' performance on the mental status exam in a standardized manner.

After using this resource, medical students will be able to:

Present a mental status exam.

Recognize gaps in their abilities to conduct the mental status exam after being offered feedback from the mental status exam rating sheet.

Introduction

A competency-based curriculum in contemporary medical education organizes the educational experience around skill proficiency, underscores performance outcomes, and ultimately promises greater accountability to patients and society.[1] Graduate medical education accreditation agencies historically have required educational programs to implement competency-based evaluations. The Accreditation Council for Graduate Medical Education has developed a matrix of milestones corresponding to each developmental stage in specialty training. In 2014, the Association of American Medical Colleges followed suit with the development of entrustable professional activities (EPAs) as a way to ensure that the medical school graduate possesses an adequate set of competencies on the first day of residency.[2]

Although proficiency in performing a psychiatric interview and mental status exam evolves throughout a psychiatrist's entire career, basic proficiency is essential for a new medical graduate entering into a psychiatry residency program. Though medical students are often assessed on their ability to perform a mental status exam both clinically and on observed structured clinical examinations during their psychiatric clerkship, this skill often takes dozens of repetitions to perform well. As such, medical students expecting to pursue a career in psychiatry are expected to continue to refine this skill throughout the rest of their clinical years before being adjudicated as having a full EPA and therefore being prepared for residency.

Here, we depict the development of an assessment tool for a psychiatry-specific EPA to be used with senior medical students pursuing psychiatric residencies, particularly regarding their ability to identify, describe, and document abnormal physical exam findings as part of EPA 1 (i.e., gathering a history and performing a physical examination).[2] In the case of psychiatry, this may refer to the mental status exam. The mental status exam is the primary (and often only) portion of the physical examination that all psychiatrists use with each patient encounter.[3] The mental status exam is often abbreviated in other disciplines, whereas in psychiatry, it is used in its whole form.

Often, the last assessment of a student's skill occurs at the end of his or her clerkship. This does not allow for the opportunity to see further growth of this skill throughout the remainder of the student's education. Our tool can be used in a capstone assessment. Whereas the clerkship-level education and evaluation should be sufficient for the graduating student who is not going into psychiatry, they may not constitute an EPA for a future psychiatric resident. Therefore, this tool may be used to assure that the future psychiatrist meets EPA 1 for psychiatry.

The ability to perform and report a complete mental status examination is a unique EPA for future psychiatrists. Although resources that teach the psychiatric interview with the mental status component are available on MedEdPORTAL,[4] our resource is unique in that it specifically addresses the mental status exam and the systematic evaluation of its presentation. It prepares the trainee for higher-stakes performance evaluations such as the United States Medical Licensing Examination Step 2 Clinical Skills exam and ultimately for the basic psychiatric residency milestone of Psychiatric Formulation and Differential Diagnosis, which states that a competent psychiatric resident “organizes, summarizes, reports, and presents to colleagues the information obtained from the patient evaluations.”[5] This tool allows for a standardization of the interview, which leads to a specific expectation on the mental status exam, and reduces the time burden on evaluator who would otherwise have to find a patient, observe the learner's evaluation, and then review the mental status exam.

Methods

This mental status exam rating tool comprises a video recording of a psychiatric interview and a scoring rubric with anchors describing levels of competency in presenting a mental status exam. The video (Appendix A) was filmed by one the authors (Benjamin T. Griffeth), utilizing a script adapted from Klapheke.[6] With Dr. Klapheke's endorsement, his script was expanded to focus on the cognitive examination.

The scoring rubric, referred to as the mental status exam rating sheet (Appendix B), was developed by one of the authors (Adriana Foster) and utilized for 4 years in the psychiatry clerkship at a large medical school (Georgia Regents/Medical College of Georgia) with positive feedback from students and faculty. The rubric includes an evaluation on the following items: appearance, behavior, psychomotor activity, speech, affect, mood, thought process, thought content, perception, insight, judgment, Montreal Cognitive Assessment (MoCA) global score, and clarity of presentation. Each item is measured as either a pass, minimal pass, or fail. An overall score is also assigned as a pass or fail. An item failure is assigned when a student fails to address a section or provides a response that is abjectly incorrect (e.g., identifying a patient's smiling and open affect as depressed). A passing score is awarded when the answer is consistent with the expectation of a PGY1 resident (e.g., the correct identification of pressured speech in a patient presenting with rapid speech that is difficult to understand or difficult to interrupt). A minimal pass score is assigned for close-to-correct descriptions (e.g., if the student describes a patient with pressured speech as having fast speech). An overall failure score may stem from incorrect presentation of data (e.g., a report of normal speech in a patient with pressured speech) or failure to address the criteria at all (e.g., no report of the speech of the patient).

This tool was presented at the 2015 and 2016 Association of Directors of Medical Student Education in Psychiatry (ADMSEP) Annual Meetings, and feedback was solicited from meeting attendees during workshop and poster presentations.[7,8] Each category of the mental status examination as depicted in the video was validated by the psychiatric educators present at the meeting as being complete and correct, with the exceptions being the patient's insight and judgment. The rubric was considered correct as presented. The video was then amended, incorporated in the final version of the tool as presented here, and submitted to a majority of the clerkship directors who had been at the original presentation. The consensus of this second panel, which offered insights throughout the 2015–2016 academic year and during the 2016 ADMSEP meeting, was that the video and rubric represented a full and correct tool for assessing and reporting a mental status examination.

Finally, at the Florida International University Herbert Wertheim College of Medicine, the University of South Alabama College of Medicine, and the University of South Carolina School of Medicine Greenville, three fourth-year students who were choosing psychiatry as their intended residency and nine third-year students who had yet to make a choice of intended residency but who had completed the psychiatric clerkship watched the standardized video of a psychiatrist performing a limited examination on a patient and were allowed to take notes during the viewing. Each student wrote out a complete mental status examination with no time limit. The students were not aware of the rubric content and described the mental status exam exclusively based on the teaching they had received in the clerkship. The clerkship faculty at each of the three sites then graded the students' work using the peer-validated rubric.

Results

All 12 students passed the rubric as a whole. From 169 grading points (12 students each graded on a 13-item rubric), students received full passing grades for 153 out of 169 (90.5%). Of the 12 students, only five had more than one minimal pass, and only one posted a failure of any component. The cognitive exam component received the most minimal pass responses as well as the only item failure (missing the MoCA score by more than 2 points and not addressing the areas of deficit). Comparison of the nine third-year students to the three fourth-year students revealed that only the ability to document the cognitive exam was different between third-year and fourth-year students (i.e., one out of nine third-year students passing vs. all three fourth-year students passing). This difference was found to be significant using the Fisher exact test. Due to the small number of participants in each group, we used this test to compare categorical data as a more appropriate statistical method. A comparison of item scores across the two student groups may be seen in the Table.

Table.

Percentage of Correct Answers Across Medical Student Cohorts as Measured With the Mental Status Exam Rating Sheet

Mental Status Exam Item	Third Year (n = 9)	Fourth Year (n = 3)	Total Sample (N = 12)
Appearance	100	100	100
Behavior	100	100	100
Psychomotor activity	100	100	100
Speech	88.9	100	91.7
Affect	88.9	100	91.7
Mood	88.9	100	91.7
Thought process	100	100	100
Thought contenta	66.7	100	75.0
Perception	100	100	100
Insight	88.9	100	91.7
Judgment	88.9	100	91.7
MoCAb	11.1	100	33.3
Clarity	100	100	100

All comparisons across cohorts made using the Fisher exact test. Unless otherwise indicated, p = 1. Abbreviation: MoCA, Montreal Cognitive Assessment.

p = .509.

p = .018.

Discussion

As medical education moves toward increasingly competency-based assessments, this module offers the opportunity to have a nationally available, validated assessment tool for EPA 1, particularly to “identify, describe, and document abnormal mental status exam findings.”[2] Though clerkship is the only formal education in psychiatry at most schools, ongoing exposure to psychiatric patients and psychiatric tools throughout the clinical years gives graduating students a more robust education than what is received in the psychiatry clerkship alone. Although our initial sample of students is small, there is evidence of content validity given the fact that we were able to score all 13 items on the rubric based on the responses provided by students.

Explanations as to why the fourth-year medical students performed better on the rubric than the third-year students validate our expectation that broader clinical exposure is additive to the clerkship's teaching and includes such benefits as greater patient contact and greater exposure to the mental status exam. Additional sources of variance between the third-year and fourth-year students may center on the greater interest of fourth-year students intending to go to psychiatry residency versus third-year students who are as yet undecided or have opted to not go into psychiatry. The ability to show variance between the groups through this evaluation tool allows clerkship directors to validate that fourth-year medical students seeking psychiatric residency are able to demonstrate EPA 1 whereas those who have completed only the third year of medical school psychiatric rotation are not able to demonstrate EPA 1.

There are a small number of limitations with this resource. The number of students in the pilot was small, although their robust performance indicates the strength of the tool and their preparation. Additionally, student performance could not be measured against another assessment of a mental status examination as the comparison test would not have been similar between the three testing sites. However, having a standard video to present to students, as well as a standardized rubric, brings objectivity to the examination and will allow comparisons to be made between different sites and campuses at the same medical schools listed previously. Additionally, with a vast majority of students successfully completing most of the sections appropriately, the current tool may be limited in its ability to discriminate between those fully developed in the EPA and those who are progressing.

Another limitation is that the rubric gives subheading and examples for potential responses of the student to the various portions of the mental status exam but does not indicate correct answers or differential responses that would be acceptable. Future rubrics would benefit from removal of examples that are not germane, such as hypnogogic hallucinations and jealous delusions, as well as inclusion of stronger descriptions of the expected definitions of insight and judgment.

Faculty will benefit from this tool by being able to identify specific areas of remediation that may need to be developed before the future resident can be said to have a complete EPA. In the future, we plan to engage medical schools from other regions in the United States for comparisons of student performance.

A. EPA MSE Video.wmv

B. Mental Status Exam Rating Sheet.docx

All appendices are peer reviewed as integral parts of the Original Publication.