The efficacy of peer assessment in objective structured clinical examinations for formative feedback: a preliminary study.

PURPOSE: We sought to determine the impact of medical students' prior experience of assessing peers in the objective structured clinical examination (OSCE) on their clinical performance.
METHODS: Forty-two year 4 medical students participated in an OSCE comprised of three 10-minute stations (syncope, hemoptysis, and back pain). Each student took part in two iterations of the three-station OSCE as either the examiner or examinee, and student performance was assessed using a checklist by a medical faculty member and a student simultaneously. Students were randomly assigned to two groups and their OSCE scores were compared. Students in the control group were tested at a station first and then participated at the same station as a peer examiner, and those in the intervention group participated as a peer examiner first and then as an examinee. Moreover, student OSCE scores rated by peer examiners were compared with those awarded by faculty to evaluate the accuracy of peer assessment. Following the test, students completed surveys on their perceptions of the usefulness of this formative OSCE.
RESULTS: Student overall OSCE scores did not differ between groups. Students in the study group performed better at the hemoptysis station (p<0.001), but poorer at the syncope station (p<0.01). Student performances at the back-pain station were similar in these two groups (p=0.48). OSCE scores rated by faculty and peer examiners were moderately negatively associated at the hemoptysis station (p<0.05), but no such association was observed at the other two stations. This trend was similar in peer examiners who were high-achievers and low-achievers in OSCEs. Students showed positive perceptions of their experience with this OSCE.
CONCLUSION: Student experience as peer assessor offers a feasible means of providing them greater access to OSCEs without consuming more resources, although its impact on enhancing performance in the OSCE is likely to differ across stations.

RCT Entities:   Population Interventions Outcomes

Introduction

Objective structured clinical examinations (OSCEs) were devised to address a growing need in basic medical education to improve medical student competencies in real-world situations [1]. The OSCE method is now well-recognized globally as a tool that substantively meets this requirement [2]. Past studies have shown that OSCEs provide a valid and reliable assessment tool when they are designed and implemented appropriately [1-3], although the reliability may differ across stations and depends on the number of stations and examiners [3].

Originally designed to enable experts to assess student competence, OSCE is also often employed to enable peer assessment and feedback [4]. OSCEs can used for both formative and summative purposes [1], and several research has been done on the effectiveness of OSCEs for formative feedback. Bernard et al. [5] found medical students’ review of formative OSCE scores were associated with their performance in subsequent summative OSCEs. Also, Daniels et al. [6] found from a study of internal medicine residents that having them immediately review their score sheets in OSCEs had a positive impact on their learning by giving timely feedback. Still, a study of year 2 UK medical students by Chisnall et al. [7] indicates formative OSCEs are associated with improved performance in subsequent summative OSCEs only for identical stations.

Peer assessment is often employed in formative OSCEs. Peer assessment in OSCEs has the potential to promote learning by encouraging students to understand expectations and strategies for the test [4,8]. Peer assessment in OSCES give students an opportunity to be exposed to the scoring rubrics, and research indicates the disclosure of scoring rubrics in OSCEs enhance student test scores, particularly in history taking and physical examination scores [9]. A scoping review by Khan et al. [4] illustrated the benefits and challenges of peer assessment in OSCEs. Specifically, they found peer assessment promotes learning, but the scorings rated by students tended to be unreliable as compared with faculty scores, which suggested students need to be trained on how to perform peer assessments in OSCEs. Moreover, research indicates the accuracy of selfassessment of clinical performance is affected by such factors as gender, task familiarity, and years in medical study [10-12]. On the other hand, some studies have shown medical students can act as effective peer examiners [8,13-17] and that student-led OSCEs constitute a sustainable cost-effective approach [15].

Despite the effectiveness of peer assessment for formative OSCEs, students frequently have no opportunity to practice OSCEs other than in the high‐stakes examination itself [8] and empirical evidence is lacking on the impact of student experience as peer examiners on their performance in OSCEs. We considered student experience of OSCE as a peer examiner would benefit their OSCE performances by giving them opportunities to observe other students, therefore to learn from others and also enhance their understanding of scoring rubrics in OSCEs, which would help reflect on their performance and come up with better test strategies.

In this study, we explored the efficacy of a revised format of OSCEs, whereby students were exposed to each station twice, that is, once as an examinee and once as a peer examiner, by investigating the impact of this approach on student performance. In addition, we investigated the association between student OSCE scores assessed by standardized patients (SPs) and peer assessors to examine the accuracy of peer assessment. Summarizing, our study hypotheses were: (1) students who undergo OSCEs after experiencing the role of examiner perform better at that station than those without such experience; (2) students who perform better in clinical performance tests better assess the clinical performances other students.

Methods

1. Study setting and procedure

The study participants were all of year 4 medical students (n=42) at a mid-sized private medical school in South Korea. Students participated in an OSCE comprised of three stations (syncope, hemoptysis, and back pain). Testing followed a general OSCE format, that is, each student was asked to perform in a simulated clinical situation by interacting with a SP by interviewing the patient, history-taking, conducting physical examinations, and arriving at a possible diagnosis and treatment plan. The time allocated at each station was 10 minutes. During this test, medical faculty members specializing in the field related to specific OSCE stations played the role of a SP. In this test, a peer examiner was also present in the station for assessment in addition to the SP. Student performance at each OSCE station was assessed by a SP and a peer examiner simultaneously. Students were assessed for patient interview and physical examination using a checklist format. The maximum possible score at each station was 100 points, which was an average of patient interview and physical exam scores. Both SPs and peer examiners used the same scoresheet.

For this experimental study, students were randomly divided into two groups. Students in the control group (n=21) were first tested at a station and then participated at the same station as a peer examiner, whereas students in the study group (n=21) first underwent the station as a peer examiner and then were tested at the same station. Only SP scores were counted towards student’s final mark and the scores awarded by peer assessors did not contribute to the final assessment mark as the peer assessment was offered for formative purposes. Student OSCE scores were disclosed after a few days of the test. Students were surveyed regarding their perceptions of this formative OSCE format using a questionnaire comprised of three Likert-type questions and one open-ended question at the wrap-up session at the completion of the test. Students responded to the Likert-type questions using a 10-point scale, where 1 represented “strongly disagree” and 10 “strongly agree.” The open-ended question solicited students’ overall experiences of the modified OSCE.

2. Data analysis

OSCE scores of students in the control and study groups were compared. The independent t-test was used to compare student OSCE score between groups. Pearson’s r coefficients were used to establish relationships between student test scores as assessed by faculty and peer examiners to analyze reliability of peer assessment. To compare the accuracy of peer assessment across students who were high performers and low performers, the 42 students were dichotomized by overall OSCE scores into high- and low-achiever groups. The assessment scores allocated to peers by members of these two groups were compared with scores allocated by faculty SPs. Descriptive statistics was used to analyze questionnaire data. The analysis was performed using IBM-SPSS ver. 20.0 (IBM Corp., Armonk, USA) and statistical significance was accepted for p-values <0.05.

3. Ethical considerations

Institutional review board (IRB) approval was not requested for this study, because it was part of the annual survey of students that pertain to their learning outcomes, which fell under the general exemption from our IRB for educational outcomes data. Participation was voluntary and consent was implied with the return of the survey as responses were collected anonymously.

Results

1. Student performance at objective structured clinical examination stations

Table 1 summarizes student performances at the three OSCE stations for group A (tested first) and group B (acted as a peer examiner first). Overall OSCE scores were similar in these two groups (t=0.562, p=0.575). Group A performed better at the syncope station (t=3.02, p<0.01) but poorer at the hemoptysis station (t=0.66, p<0.001). Group performances at the back-pain station were similar (t=0.71, p=0.48).

Table 1.

Student Performances at Objective Structured Clinical Examination Stations

Stations	History-taking scores			Physical exams scores			Total scores
	Control group	Study group	p-value	Control group	Study group	p-value	Control group	Study group	p-value
Syncope	77.3	61.2	0.004	75.6	66.1	0.190	74.0	66.7	0.004
Hemoptysis	84.8	78.3	<0.001	45.2	72.0	<0.001	60.0	76.3	<0.001
Back pain	83.4	84.2	0.428	80.1	83.6	0.672	83.9	81.8	0.481
Total	78.1	75.8	0.303	70.6	70.4	0.942	72.6	77.9	0.575

Students in the control group were tested at a station first and then participated at the same station as a peer examiner, whereas students in the study group were exposed to the station as a peer examiner first before they were tested in that station. The maximum possible score at each station was 100 points, which is an average of the sum of patient interview and physical exam scores.

2. Associations between standardized patient and peer examiner performance scores

Table 2 illustrates associations between student OSCE scores as allocated by SPs and peer examiners. OSCE scores awarded by SPs and peer examiners were moderately negatively associated at the hemoptysis station (r=-0.33, p<0.05), but no association was observed at the other two stations.

Table 2.

Associations between Student Objective Structured Clinical Examination Scores as Awarded by Standardized Patients and Peer Examiners

	Stations
	Syncope	Hemoptysis	Back pain
Pearson’s r	-0.042	-0.334[*]	-0.062

p<0.05.

Table 3 summarizes associations between student OSCE scores as allocated by SPs and by peer examiners in the high- and low-achieving groups. Student OSCE scores awarded by high achievers were not associated with scores awarded by SPs at any of the three stations. Student scores of history-taking skills in the hemoptysis station awarded by low achievers were moderately and negatively associated with scores awarded by SPs (r=-0.44, p<0.05). Still, scores awarded by low achievers were not associated with SP awarded scores at the other two stations.

Table 3.

Associations between Student Objective Structured Clinical Examination Scores as Awarded by Standardized Patients and Highand Low-Achieving Peer Examiners (Pearson’s r)

	Stations
	Syncope			Hemoptysis			Back pain
	History taking	Physical exams	Total	History taking	Physical exams	Total	History taking	Physical exams	Total
High achievers	-0.221	-0.330	-0.328	-0.306	-0.251	-0.336	-0.471	-0.240	-0.408
Low achievers	0.213	-0.105	0.163	-0.442[*]	-0.219	-0.377	0.104	0.299	0.153

p<0.05.

3. Student perceptions of the usefulness of formative objective structured clinical examination

Students generally agreed with the statement that experience of OSCE as a peer examiner would be of benefit in their own exams (mean=8.44, standard deviation [SD]=1.82), and disagreed with the statement that the presence of a peer examiner at an OSCE station made them feel uncomfortable (mean=2.44, SD=2.34). Several students reported they felt they received more feedback by acting as a peer assessor in this OSCE format. Some typical students’ comments were as follows:

“I got far more feedback from this OSCE, because it allowed me to learn by watching other students and see how they do differently from me. I hope to have more opportunities to do OSCEs like this.”

“OSCE scores had not been enough for me to explain what I did not do well. In this OSCE, I better understood my weaknesses by understanding what I was assessed in that station in more detail by seeing the assessment criteria in the scoresheet.”

“It was helpful because I learned some expressions of other students when they talked with the patient that I felt were useful, but I had never used.”

Discussion

Our study shows student exposure to OSCE stations as peer examiners may have a positive impact on their performance at some, but not all, stations, in terms of test scores. Student positive perceptions of this OSCE format indicate several benefits for learning. First, it is likely that student experience as peer examiner gave them an opportunity to improve their knowledge, skills and attitudes in interacting with the patient by learning from how other students perform. Second, students were also likely to learn how to perform physical examinations more accurately from the experience of assessing their peers by being exposed to the scoring rubrics. These findings are consistent with the literature that supports the benefits of formative OSCEs [4,8].

Yet, only weak associations were found between assessment scores rated by faculty members and peer examiners regardless of whether peer examiners were high or low achievers. This finding suggests highperforming students do not necessarily better assess student performance in OSCEs than their lowperforming peers. This is in line with the findings of previous studies that peer examiners are probably less reliable assessors than faculty members and that they should be trained to perform as effective peer assessors [4,16]. Therefore, our OSCE format may be adequate for formative assessments, but not for summative purposes. Nevertheless, student perceptions of this formative OSCE format were positive, which provides another evidence of its efficacy. Furthermore, there were a negative association between student OSCE scores as awarded by SPs and peer examiners in the hemoptysis station. It is speculated this finding is related to student task familiarity with the station. It can be assumed that the hemoptysis station was more difficult than others as the student OSCE scores were the lowest in that station. As previous studies indicates the accuracy of student self-assessment of clinical performance in OSCEs is related to task familiarity [12], student inaccuracy in assessment in the hemoptysis station may be related to task familiarity or difficulty of that station.

Although our study shows limited effectiveness of peer assessment in this OSCE format, it is still worthwhile to explore its feasibility and to enhance its effectiveness as it allows students to practice OSCEs more frequently than in the conventional format. Offering medical students opportunities to practice OSCEs is limited by financial, faculty, and administrative constraints [15]. The described formative OSCE format allowed students to experience OSCE stations twice without increasing resource requirements, but its educational impact on student clinical performance probably differ across stations. Futures research is warranted on which factors in the OSCE format made differences in the outcomes across stations to enhance its educational impact. Future research is recommended for more in-depth understanding of what students learn from their experience of the OSCE as a peer examiner and whether and how it changes their behaviors that would result in an improvement in their clinical performance. Such studies will better inform us on how to make student experience of OSCEs as peer assessor a more effective educational opportunity.

We acknowledge that the present study was a preliminary study and thus has several limitations and warrant future studies. First, we examined student performance at three OSCE stations only, and the study was conducted using a small number of medical students in their final year. Future larger-scale study is warranted to enhance the generalizability of our findings. Second, we studied students who were in the final year in medical school. The impact of experience of peer-based assessment on student performance in OSCEs may be depended on time spent on a medical program, and thus, we suggest research be conducted on medical students during their earlier years.