Evaluation of the use of script concordance test in a multicampus psychiatric pharmacy elective course.

INTRODUCTION: Evaluating a student's ability to accept complexity, uncertainty, and ambiguity as part of clinical practice is difficult in a classroom setting using written tests. This study was conducted to explore the feasibility and validation of using a script concordance test (SCT) to evaluate pharmacy student knowledge and clinical competence in a psychiatry elective course.
METHODS: This study involved prospective validation of psychiatry-focused SCT questions using a panel of practicing psychiatric pharmacists and retrospective review of student performance on the same SCT questions. The reliability of the SCT was also evaluated using Cronbach alpha coefficient.
RESULTS: A total of 13 practicing psychiatric pharmacists participated in the validation phase of the study of 75 questions. Pharmacy student scores (n = 17) averaged 39.79 (±5.02) points, and psychiatric pharmacist scores averaged 50.11 (±4.51) points, representing mean percentages of 61.2% and 77.1%, respectively, on the adjusted exam. The Cronbach alpha was 0.94. DISCUSSION: The development of a valid and reliable SCT to test student psychiatric pharmacy knowledge and clinical competence after taking a psychiatry elective course was feasible.

Introduction

Cases in psychiatry was a multicampus (four campuses of a college of pharmacy in the southeast region) elective course that allowed the pharmacy student to become knowledgeable on psychiatric illnesses via didactic coursework, application to and discussion of patient cases, and critical evaluation of the literature. Students applied knowledge obtained to current events and clinical controversies related to psychiatry. The course was taught via an electronic platform (Blackboard Collaborate [BBC], Blackboard Inc, Washington, DC) used to deliver multiple active learning activities to reinforce the learning of psychiatric illnesses and key concepts. The BBC contained tools, such as polls and whiteboards, used to help students become more familiar with script concordance test (SCT) questions and for case discussions. It also easily facilitated the use of debates (with whiteboards and breakout rooms), which students perceived was effective at granting them a better understanding of both sides of the controversy or treatment option. Further details regarding a description of the class, as well as example SCT questions, have been published previously.1 Based on anonymous survey, students at all campuses felt the learning platform, methods, and strategies employed were effective and conducive to their learning and confidence in being able to manage psychiatric illnesses.

Students completing this course should gain an overall proficiency with evaluating medical literature, deeper understanding and appreciation for patients with psychiatric illness, and ability to critically evaluate issues relating to mental illness with special emphasis on psychotropic medications. The course had several objectives; however, the specific primary learning objective that was the focus of this research was to evaluate the student's ability to accept complexity, uncertainty, and ambiguity as part of clinical practice. Anecdotally, when multiple-choice questions are used, wherein there is only one correct answer choice and the remainder are incorrect, both students and faculty express frustration that, in clinical practice, there is rarely one correct answer. The ability of a multiple-choice exam to test clinical competence (clinical skills, knowledge, and problem-solving ability) may be unclear although it is possible with “well-constructed” questions.2,3 The advantage of a multiple-choice exam is the ease and speed of grading, especially compared to written-answer responses, which could also be subject to interpretation by the grader. To address these concerns, the SCT was identified and used as the tool to assess students' ability to meet this objective by the end of the course. The SCT, which is case based and can be validated, has been documented in the literature4,5 to assess clinical reasoning competence related to the ability to interpret information under uncertain or incomplete conditions that may simulate the ambiguity of true clinical practice. The student must use not only relevant factual knowledge, but also appropriate reasoning skills; there is no one correct answer in SCT.4,5 Students first read the case description and then consider a treatment (or modification) or monitoring. Given a new piece of information, the student must then formulate a decision about how the new information would influence the student's actions or thinking; this would be rated on a 5-point Likert scale.4,6,7 Because the assessment uses a Likert scale, grading of exams could be done quickly and more efficiently, especially in a larger class size, than if short answers or essay exams had been used to evaluate the student's clinical decision-making skills given a patient case. The recommended number of questions per case is 3 questions.5 According to Fournier et al,5 the desired number of experts on the panel to validate the SCT should be between 10 and 20, wherein fewer than 10 experts involved greater error in estimating reliability, 15 experts were required for acceptable reliability in high-stakes exams, and greater than 20 experts yielded minimal improvements in reliability estimates. To validate the SCT and create the rubric, at least 10 panel members (in this case, pharmacists practicing in psychiatry with overall clinical experience in the field) were needed to reduce error in reliability.5

The long-term research goal of this study was to be able to identify accurate assessments of student understanding of psychiatry topics taught in a classroom setting. The purpose of this current study was to reevaluate student performance at the end of the elective course using the same SCT that would be validated by expert panelists.

Methods

This was a retrospective study using a prospectively validated SCT. The SCT has always been the primary assessment method of student knowledge and learning since course inception. When the SCT was initially created by the course coordinator, the rubric was also created by the same person, who was a practicing clinical pharmacist in an acute inpatient psychiatric setting. Students took the exam and were graded based on this rubric prior to the validation of the test. Students were also allowed to use any resource deemed helpful in answering case-based questions. These resources would also be available to the practicing pharmacist, hence mimicking practice. In the prospective component, psychiatric pharmacists practicing in the clinical setting were recruited via an email sent to the College of Psychiatric and Neurologic Pharmacists (Lincoln, NE) listserv. A total of 10 to 15 current and actively practicing (engaging in direct patient care) psychiatric pharmacists were required to validate the SCT. Because this was not considered a high-stakes exam for which failing the exam would lead to course failure and/or failure to progress in the overall pharmacy curriculum, a goal of 10 to 15 pharmacists was set. These pharmacists, considered as panel members, took the SCT (via Qualtrics; Provo, UT), and their responses were used to create the rubric. Questions were assigned points based on panel member responses. According to Lubarsky et al,4 “ideal SCT questions are those that produce a range of expert responses clustered around a modal answer.”(p190) To this end, questions that yielded a broad distribution, including answers selected from all 5 categories of the Likert scale, were removed for questionable reliability and/or validity. The exam metrics were then reanalyzed. Alternatively, questions on which experts agreed on one single answer are considered to perform similarly to multiple-choice questions, wherein only one correct or best answer exists and the remaining are considered incorrect but do not necessarily represent poor question quality. Questions performing this way were kept in the adjusted scoring. Grant funding was obtained from the University of Georgia Faculty Achievement in Classroom Teaching Grant award committee, the Instructional Design and Technology Office, and the College of Pharmacy to aid in recruitment of psychiatric pharmacists working in various settings to answer the SCT questions in order to validate the overall test for students to take at the end of the semester. Compensation was offered to participating pharmacists with a $25 Amazon (Seattle, WA) gift card upon completion of the SCT. Informed consent was obtained from all pharmacists participating in the validation phase.

The expert key was created using responses from the validation phase. All prior students who completed the exam (fall 2015, fall 2016, and fall 2017 semesters) would have their exams regraded based on the validated scoring. The variability of pharmacist answers to each SCT question were evaluated as well as student performance on SCT after “poorly performing” test questions were removed. To assign points per question, the number of times an answer choice was chosen was divided by the modal response. See Table 1 for example scoring for a specific item. Demographic information of the practicing pharmacists was obtained. Descriptive statistics were used to characterize demographic information as well as SCT responses. Similar to other SCT studies, the Cronbach alpha coefficient was used to measure the internal validity to determine scale reliability of the SCT. This study was approved by the University of Georgia Institutional Review Board.

TABLE 1

Example scoring of question item for rubric determination

Likert Scale Anchor	Answer	Responders Selecting Answer Choice, %	No. Responders Selecting Answer Choice, n	Raw Calculation	Points Assigned
2	Absolutely indicated	53.85	7	7/7	1.00
1	Somewhat indicated	38.46	5	5/7	0.71
0	Neither indicated or contraindicated	7.69	1	1/7	0.14
–1	Somewhat contraindicated	0.00	0	0/7	0
–2	Absolutely contraindicated	0.00	0	0/7	0
	Total	100	13

Results

A total of 13 practicing pharmacists completed the SCT during the validation phase. All pharmacists who responded to the recruitment email and completed the exam were included. More than half of the pharmacists practiced in the adult and/or acute inpatient settings. Although variability was present in years of practice, all pharmacists spent at least 15 hours a week in clinical practice with the majority indicating 30 to 40 hours of practice. Variability also existed across psychiatric illness although all pharmacists encountered schizophrenia spectrum and other psychotic disorders, bipolar and related disorders, depressive disorders, and anxiety disorders at least weekly. See Table 2 for additional demographic information. Upon completion of the exams by the practicing psychiatric pharmacists, the exam rubric was created using the scoring system according to the method described by Lubarsky et al.4

TABLE 2

Pharmacist demographic information for validation of script concordance test

Parameter	Value (N = 13)
Psychiatric practice setting (may be multiple)a
Academia	5
Adult	9
Child and adolescent	2
Geriatric	5
Acute inpatient	8
Outpatient	6
Long-term inpatient	3
Nursing home	1
Teaching hospital	6
Specialty outpatient clinic	5
Community hospital	1
Otherb	1
Years in clinical practice in psychiatric setting
Less than 1	1
1 to <3	2
3 to <5	5
5 to <7	0
7 to <10	2
10+	3
Hours per week spent in clinical practice
Under 5	0
5 to <10	0
10 to <15	0
15 to <20	3
20 to <30	1
30 to 40	9
Disease states encountered routinely (at least weekly)
Anxiety disorders	13
Bipolar and related disorders	13
Depressive disorders	13
Schizophrenia spectrum and other psychotic disorders	13
Personality disorders	9
Substance-related and addictive disorders	9
Trauma- and stressor-related disorders	9
Sexual dysfunctions	7
Sleep-wake disorders	6
Neurocognitive disorders	5
Disruptive, impulse-control, and conduct disorders	3
Gender dysphoria	3
Obsessive-compulsive and related disorders	3
Somatic symptom and related disorders	2
Dissociative disorders	0
Elimination disorders	0
Feeding and eating disorders	0
Neurodevelopmental disorders	0
Paraphilic disorders	0
Currently Board Certified in Psychiatric Pharmacy	9

Values are not mutually exclusive; a practicing psychiatric pharmacist may work in multiple areas.

Description provided by pharmacist was correctional healthcare, managed care.

Students' exams were then rescored according to the rubric created during the validation phase. Of 17 students who completed the exam over various semesters, the mean score was 45.91 points out of a possible 75 points (61.2%) using the original exam with no questions discarded. The median score was 45.0 (60.0%). The lowest and highest scores were 38.92 (51.9%) and 56.34 (75.1%), respectively. When scoring the psychiatric pharmacist exams against the rubric, the mean score was 57.26 points (76.3%), the median score was 58.51 (78.0%), and the range of scores was 48.49 (64.7%) to 61.58 (82.1%).

A total of 10 questions displaying broad variability (pharmacists selected all answer choices on the scale) were removed, and the exam results of the remaining 65 questions were then evaluated. The Cronbach alpha coefficient of the expert panel responses was 0.94. The mean score (SD) of students' exams was then 39.79 (±5.02) points out of 65 points or 61.2%. When scoring the psychiatric pharmacist exams against the rubric, the mean score (SD) was 50.11 (±4.51) points, which yielded a mean percentage of 77.1%.

Discussion

From discussion with faculty at other schools of pharmacy, it appears faculty who are teaching the content are then also responsible for creating test questions or developing other assessment methods. Problems with this include inaccurate assessment of student knowledge and competency if the faculty is a poor question writer, tests trivial facts, or creates overly easy or difficult questions (eg, leading to questions being thrown out or the overall score being curved). Using the SCT could be a feasible means by which to assess pharmacy student and trainee knowledge in an accurate, valid, and reliable way. A possible barrier may be identifying practicing pharmacists or specialists who would be available or have the time to assist in test validation. It is also a new testing method with which students would need to become familiar. In this course, the BBC polling method was used to teach students how to answer SCT questions. Once a question was shown and all students selected an answer, responses were posted anonymously, and the course coordinator then asked students to share their rationale; the course coordinator also discussed the coordinator's response and rationale to each question. Although perceptions of the SCT as an assessment method were not evaluated in prior course surveys, when students in this elective were asked to rate the SCT in terms of effectiveness to their understanding of psychiatric illnesses, all students rated the activity as somewhat effective or effective based on a 5-item scale (other ratings included ineffective, somewhat ineffective, and unsure/neither).1

Once validated, the test could be used repeatedly, at least until it is determined that the information would need to be updated to current practice or include new treatments. It is not clear whether single cases of 3 questions could be validated individually and interchanged with outdated questions and still maintain overall test reliability and validity. It is possible that multiple-choice questions could accurately assess clinical competence as previously cited in the literature although the process for validation and reliability may be more labor-intensive, possibly requiring peer review for well-constructed stems and answer choices.3 With the SCT, there is still a possibility for faculty to test what they teach and what they perceive to be important rather than what students should actually know for general practice. To account for this, Lubarsky et al4 recommends that 2 authors develop SCT items and that a preliminary test be sent to 2 to 3 independent reviewers for comments and suggestions.

Other professions or specialties have recently used SCT questions to evaluate clinical reasoning in a reliable manner with the ability to discriminate between various levels of expertise.8-11 In the future, it may be of interest to ask students who have not taken the course to complete the SCT to characterize how they may perform and potentially identify the robustness of this exam in determining psychiatric pharmacy knowledge. There was one question for which all content experts selected the same answer. This question was retained in the adjusted scoring as it did not necessarily represent poor quality. According to Lubarsky et al,4 these questions would then perform no better than a multiple-choice question. A limitation of the exam in its current form is that current questions might need to be modified or new questions created and validated to replace the poorly performing questions that were discarded. Dory et al7 report that about 25% of questions and, in some instances, up to 70% of questions are discarded in prior review of SCT studies evaluating internal consistency. Comparatively, this study had a discard rate of approximately 13%. There was no time limit for the exam for either students or pharmacists with the consideration that individually completed SCTs were to be turned in by a prespecified deadline (at least 1 week from when the SCT was given). In the literature, time limits are not routinely set or required for the SCT. A disadvantage of no time limit is that students may be given more time to look up any information and select an answer at their leisure than would be granted in true clinical practice.

Psychiatry is a specialty area that encompasses many gray areas in which disease states are not fully understood and the disease response to medications is not robust. Multiple-choice exams are used ubiquitously in the core curriculum at this school of pharmacy to assess student knowledge and skills; well-written multiple-choice exams may be able to test clinical competence. Similar to multiple-choice exams, the SCT was easy and quick to grade once the rubric had been created. The purpose of this psychiatry-focused elective course was to expose students to real-world aspects and issues patients with psychiatric illnesses face, such as stigma, transgender health, suicide, or co-occurring disorders that students may not cover in depth in the core curriculum. Faculty also wanted to make the topics accessible and clinically relevant to the practicing general pharmacist regardless of the type of pharmacist the student might choose to be in the future. Using SCT was a way to test a student's ability to accept complexity, uncertainty, and ambiguity as part of clinical practice while also testing factual psychotropic knowledge. Although the SCT should certainly not replace multiple-choice questions, it could be considered as a supplemental way to test student knowledge in a validated and reliable way. With a Cronbach alpha of 0.94, this exam performed as expected. In other studies, SCTs consisting of 20 cases, 60 questions, albeit with a 1-hour time limit, have resulted in Cronbach alpha values greater than 0.75.5

Conclusion

Script concordance test questions of psychiatry cases and topics were validated by practicing psychiatric pharmacists and used to test student knowledge after a psychiatric pharmacy elective. The development of validated test questions was feasible and could accurately test pharmacy student knowledge involving clinical decision making and critical thinking skills in a multiple-answer choice format.