Validity of OSCE Evaluation Using the FLEX Model of Blended Learning.

BACKGROUND: For OSCE (Objective Structured Clinical Examination) scoring, medical schools must bring together many clinical experts at the same place, which is very risky in the context of the coronavirus pandemic. However, if the FLEX model with the properties of self-directed learning and offline feedback is applied to OSCE, it is possible to provide a safe and effective evaluation environment for both universities and students through experts' evaluation of self-video clips of medical students. The present study investigated validity of the FLEX model to evaluate OSCE in a small group of medical students.
METHODS: Sixteen 3rd grade medical students who failed on OSCE were required to take a make-up examination by videotaping the failed items and submitting them online. The scores between original examination and make-up examination were compared using Paired Wilcoxon Signed Rank Test, and a post-hoc questionnaire was conducted.
RESULTS: The score for make-up examination was significantly higher than those for original examination. The significance was maintained even when the score was compared by individual domains of skills and proficiency. In terms of preference, students were largely in favor of self-videotaped examination primarily due to the availability of self-practice.
CONCLUSION: The FLEX model can be effectively applied to medical education, especially for evaluation of OSCE.

INTRODUCTION

Blended learning, defined as a teaching method conjoining traditional lectures and various technologies, is rapidly being introduced in the field of education along the coronavirus disease 2019 (COVID-19) crisis. Online education first appeared in the 1990’s under the name of E-learning as an ancillary instrument to traditional lectures. However in 2010’s, it started to integrate itself with offline education in order to overcome the shortcomings of one-sided, lecture based education, and started being called blended learning. The COVID-19 crisis accelerated adoption of online learning into the field of education, and along with it a contemplation regarding relative weight, target, and integration of offline and online education has become greater than ever.12

Representative models of Blended Learning include a supplemental model that utilizes online learning activities to supplement online classes, a flipped learning model that performs offline cooperative learning after watching a lecture video, a rotation model that circulates various learning activities by mixing online and offline, And there is a flex model that conducts self-directed learning activities online and gives individual feedback offline.3 In medical education, clinical skills education was traditionally carried out in an offline setting. Conventionally, the whole curriculum would be carried out offline, but when online lecture is necessary, flipped learning model of blended learning proposed by Garrison et al. would normally be used. This refers to in-advance online learning such as watching video clips offered by e-learning consortium (http://www.mededu.or.kr/), followed by offline practice. Prior studies conducted in college of nursing that utilized such model have reported improved student satisfaction.456789

Meanwhile video clips used in the clinical skills education can be divided into tutorial video by the instructor, and self-video by the student themselves. Both the tutorial video and self-video are assumed to have similar educational effect, and researches that advocate the use of self-video taken by smart phones are gaining attention.4567810 And it is very difficult to recruit clinician experts in charge of scoring OSCE (Objective Structured Clinical Examination) in medical school, so rather than meeting the schedule of busy clinician experts, we think the evaluation using these medical students’ self-video clips seems to be very useful in a medical school.

In the light of such prior research, we designed a study where students would self-videotape their own OSCE and receive feedback, in order to validate the applicability of FLEX model in learning clinical skills of medical education. This is because self-video recording has a common context with self-directed learning of the FLEX model. Additionally, due to the elongation of COVID-19 crisis, there is a growing interest in FLEX model which puts more emphasis on online education but little research has been conducted using this model.

METHODS

Sixteen 3rd grade students who failed on OSCE in the year 2015 were asked to take a make-up examination using self-taken video clip. Students were required to upload video clips for the failed items at the school homepage within 7 days. The time limit of 5 minute was identically applied as the original examination. These students were not randomly selected by the researcher, but were students who did not reach the standard score in the tests conducted in class.

Self-taken video clips were reviewed by three independent examiners, all of whom had over 10 years of experience in medical education. The checklist was identical to the original examination, which was divided into skills domain and proficiency domain. The only difference was the ‘not identified’ column that was added in addition to ‘yes’ and ‘no’ for skills domain. There was no difference in the checklist item between the original and the make-up examination.

Evaluation of proficiency domain was carried out using 5-point Likert scale. The score for skills domain was converted into total of 80, and the score for proficiency domain was converted into total of 20, adding up to 100. Students were considered to have ‘passed’ the item when the converted score was equal to or greater than 70. This cutoff was identical to the cutoff applied for the original examination.

The Paired Wilcoxon Signed Rank Test was used to investigate whether there is a statistically significant difference between the results of original examination and make-up examination. Also, post-hoc questionnaire was conducted to investigate the following; number participants required for videotaping, number of videotaping trials, preference between conventional examination and self-videotaped examination, and the reasons behind the preference.

Ethics statement

This study is about the results of tests conducted during regular class hours, and the subject of analysis is not students but anonymized test scores. The post-mortem survey was also conducted anonymously. Therefore, this study is a study in which there is no reason to presume the subject’s refusal to consent, and the risk to the subject is extremely low even if consent is waived. The study was approved by the Kangwon National University Hospital Institutional Review Board (KNUH-2022-02-031-002).

RESULTS

A total of 16 students submitted 33 video clips. There were 11 video clips for cardiopulmonary resuscitation (CPR) item, 10 for endotracheal intubation (ET) item, and 12 for male Foley insertion (FI) item (Supplementary Table 1).

The results of the make-up examination can be seen in Table 1. Students were given credit for each checklist item when 2 or more examiners marked ‘yes.’ In all three items, an increase in the score was noted in the make-up examination compared to the original examination, and the Paired Wilcoxon Signed Rank Test showed a significant difference between the two. For each of CPR, ET, and FI item, the average original examination score was 57.8, 50.8, 51.1, but the average make-up examination score was 73.7 for CPR and 78 for FI, surpassing the cutoff. When comparing the result by domains, the score for both the skills domain and the proficiency domain showed a significant increase in make-up examination compared to the original examination.

Table 1

Comparison of scores between original examination and self-videotaped examination

Item	Original examination				Self-videotaped examination
	Skill	Proficiency	Total	Passa	Skill	Proficiency	Total	Passa
Cardiopulmonary resuscitation	50.9	6.9	57.8	0	62.7**	11.0**	73.7**	6
Endotracheal intubation	44	6.8	50.8	0	57.0**	10.6**	67.7**	4
Foley insertion	44.1	7	51.1	0	68.2**	9.8**	78.0**	10**

aCut-off value of pass is 70 points; **P < 0.01 by paired Wilcoxon signed rank test.

The results of post-hoc questionnaire can be seen in Tables 2 and 3. Among the 16 students, 15 responded. The average number of videotaping trials exceeded 2 for all three items, and the maximum number of videotaping trials for each item was 10, 5, and 5. Eleven of 15 responders reported subjective improvement in OSCE skills, and another 11 out of 15 students reported preference towards self-videotaped examination over conventional examination. The main reason behind the preference towards self-videotaped examination was availability of self-practice without time limit, and students who chose conventional examination pointed out lack of tension in self-videotaped examination.

Table 2

Average number of pre-practice and videotape trials

Items	Pre-practice trials	Videotape trials
Cardiopulmonary resuscitation	2.1	2.4
Endotracheal intubation	4.11	2.78
Foley insertion	2.38	2.3

Table 3

Result of post-hoc questionnaire among 15 medical student responders

Subjective improvement in OSCE skills	Preferred examination type	Reasons to prefer it
Yes	Self-videotaped examination	Repeated practice
Yes	Self-videotaped examination	-
Yes	Self-videotaped examination	Focused practice
Yes	Self-videotaped examination	Practice many times
Yes	Self-videotaped examination	Autonomous practice, without time limit
Yes	Self-videotaped examination	Save time
Yes	Self-videotaped examination	Tension when filming, feedback
Yes	Self-videotaped examination	Self-practice, self-feedback
Yes	Self-videotaped examination	Practice
No	Self-videotaped examination	Self-practice
-	Self-videotaped examination	Feedback between students
Yes	Conventional examination	Lack of tension
Yes	Conventional examination	More tension, similar to the real examination
No	Conventional examination	Conventional examination helps prepare harder
-	Conventional examination	Similar to the real examination

DISCUSSION

The findings of the present study indicate that online education followed by small group offline feedback is capable of improving students’ clinical skills and proficiency in OSCE. In addition, a majority of students reported subjective improvement in their OSCE skills and preference towards self-videotaped examination. Therefore, the researchers have reached a conclusion that FLEX model may be applied in skill learning of medical education and is effective.

The FLEX model has an advantage both in cognitive and affective area. In the cognitive area, the FLEX model may promote academic achievement, problem-solving skills, critical thinking, and comprehension. In the affective area, the FLEX model may promote satisfaction, academic interest, learning attitude, and motivation. In this study’s setting, students can videotape and review their own skills and knowledge and interact with other students by assisting them and giving feedbacks, thereby achieving the above mentioned effects. Video clips have an additional advantage of enabling a comparison between student’s skills along the time course due to the fact that it may be saved and accessed afterwards. Also, utilizing smart phones for video clips can reduce the psychological and technical barrier to making video clips.

In the perspective of education provider, self-videotaped examination requires less educational resources. Normally, even a single round of OSCE examination requires scheduling of examiners, in-test student management, maintenance of consumables and dummy models. In such background, the results by Casey et al.,11 which showed no significant difference between self-assessed remediation and faculty-guided remediation is worth noting. If self-assessed remediation suffices, medical schools may put aside manpower and material resources.

This study has certain limitations. First of all, not having been designed as a comparative study, the score difference between original examination and make-up examination may have been impacted by multiple factors. Specifically, the reason behind the improvement of performance is confounding, potentially attributable to repeated pre-practice or change in learning method, and cannot be assessed individually.

Vivekananda-Schmidt asserted the usefulness of using videotaped OSCE for Joint Examination of medical students, but the study utilized fixed camera and thus the quality, angle, and the voice recording of the video was identical among one another.12 A study by Christoph Kiehl which also argued the usefulness of videotaped OSCE utilized tripod-mounted digital video camera as well, and the quality difference between the videos were negligible.13 However in this study, video clips were taken by student themselves, and thus three independent examiners were required to evaluate video clips in order to minimize the effect of quality difference between one another.

This resulted in another limitation of compromised equivalence in evaluation between the two examinations. In original examination, students were evaluated by one examiner, whereas in the make-up examination, they were evaluated by three examiners. Despite the differences among the two examinations, the present study adopted a form of self-videotaped examination and condoned resulting difference in the number of examiners so that the meaning of offline feedback may be maximized without renouncing the quality of evaluation.

Finally, some checklist items were marked ‘not identified’ due to the physical limitation of a video clip. Major reasons behind ‘not identified’ were small object size, unobservable fine movement of students, unobservable fine movement of dummy models, and unevaluable visual inspection of students. Examples for the suggested reasons are as follows; evaluating a stylet tip of the endotracheal tube, pulling the penis in right angle when inserting a Foley catheter, chest compression in CPR, and visual inspection of sclera to confirm jaundice.

To overcome such limitations, further study comparing the results of self-videotaped examination and conventional examination in the same examination round is required. In addition, we suggest following guidelines in order to minimize the physical limitations that have been confirmed in the present study. First, smart phone cameras need to film the target in close proximity when filming small objects or body surface. Second, if possible, it is recommended to infer the movement of a small object by the movement of large objects when evaluating video clips. Third, a concrete checklist for each item should be prepared in advance. If a certain checklist item is ambiguous, then removal of such checklist item is encouraged. Finally, a monitor and a sensor should be clearly shown on the video clip as ancillary devices.

In spite of above mentioned limitations, the present study validated the applicability of FLEX model of blended learning in OSCE, which is yet to be widely used. Blended learning is a teaching method that will not fade away with the remission of COVID-19 crisis, but a method that will be applied extensively in the coming future. Medical schools therefore need contemplation on how to apply blended learning in their curriculum, and the present study may serve as one indicator in doing so. In particular, since the OSCE curriculum, which has been conducted as a “face-to-face education,” has verified effectiveness of blended learning, the present study demonstrates a very helpful basic material for the medical school curriculum in the future.