Changing Student Perception of an Online Integrated Structured Clinical Examination During the COVID-19 Pandemic.

BACKGROUND: The COVID-19 pandemic has created a hiatus in in-person clinical assessments due to safety and logistical concerns. We aimed to evaluate student perception and utility of an online Integrated Structured Clinical Examinations (ISCEs) during the pandemic.
METHODS: Final-year medical students from a single institution were offered an online mock ISCE through a student-to-student ("near-peer") teaching-programme. A questionnaire-based cross-sectional study was conducted pre- and post-online mock ISCE.
RESULTS: Sixty-four students completed the study. Pre- and post-data showed an increase in confidence (p<0.0001), less worry regarding the online format (p<0.0001) and less anxiety about excelling in ISCEs (p<0.001). Students felt that having done the mock, an online format would more positively affect their overall performance (p=0.007).
CONCLUSION: This study demonstrates a positive change in student perception and confidence in online ISCEs. Online ISCEs are thus feasible, though sole reliance on this format may provide an incomplete assessment of student's overall clinical competency.

Plain Language Summary

Following the COVID-19 pandemic, we studied the use and student perception of performing clinical examinations on a virtual platform. We found conducting the online mock clinical examinations very feasible and our results showed that students felt less anxious, more confident and had a positive experience following the session. We believe that virtual clinical examinations may have a place in the undergraduate curriculum in assessment of learning.

Background

Integrated structured clinical examinations (ISCEs) are a widely utilised modality for assessment of clinical competency and professional ability at British medical schools.1 It is an adapted form of the objective structured clinical examinations (OSCEs), which assesses a broad range of clinical skills such as history taking, physical examination, communication skills and critical thinking. By integrating different elements of these skills within a clinical case, ISCEs provide a global assessment of the candidate’s clinical aptitude.2 These examinations are traditionally performed in-person with an observing examiner and a patient (either simulated or individuals with clinical signs to identify) under standardised conditions.3

The COVID-19 pandemic has caused a disruption in the delivery of these integral clinical assessments, which aim to ascertain that graduated medical students are able to work within the remits of a junior doctor in the National Health Service (NHS).4 In the United Kingdom, multiple medical schools, with approval from the General Medical Council (GMC), decided to suspend final year OSCEs and ISCEs for the 2020 graduating cohort due to these logistical and safety concerns associated with increased transmission of the virus, especially with significant asymptomatic transmission rates.5,6 This is of particular relevance for clinical examinations, which involve repeated physical interactions between different students and patients who may have significant medical comorbidities. Furthermore, lack of available clinical staff due to increased clinical commitments and self-isolation rates were also cited as factors.7 While the exceptional circumstances necessitated these adjustments, this compromise may have direct long-term consequences for the future workforce and patient safety.

An avenue to overcome the limitations of in-person clinical examinations during the pandemic is delivery of these summative assessments through an online platform.8 Multiple postgraduate training programmes and medical schools have conducted OSCEs and assessments of telemedicine skills through virtual platforms and it seems likely that these skills may be increasingly required for continuous assessment of students and also routine clinical practice.9–12 There are however notable drawbacks if virtual clinical examinations, such as ISCEs or OSCEs, were to substitute in-person examinations. These include difficulties in assessing physical examinations or procedural skills, while concerns remain regarding potential dishonesty among students when conducting assessments off-campus.13

Despite these challenges, balancing the everchanging needs and technological advancements of undergraduate medical curricula during the pandemic remains vital.14 Online ISCEs/OSCEs may provide an opportunity to supplement other forms of summative clinical competency assessments during the COVID-19 pandemic and this may follow through beyond.15 The opportunity for enhanced access, flexibility and cost, especially during the pandemic where healthcare services are strained, may provide a stepping-stone to progress our armamentaria in medical school assessments.9

However, in order adequately implement these examinations, student perspectives regarding the online format need to be assessed. Students have previously expressed significant benefits from online teaching programmes, an innovation that has also greatly advanced since the beginning of the pandemic.16,17 The acceptability of this modality for curricula assessment among students however remains largely unknown despite studies showing that online OSCEs are able to test clinical acumen and communication skills.18 Students undertaking these high-stakes medical examinations are key stakeholders19 and although these clinical examinations primarily aim to assess if students are safe and sufficiently competent to perform the tasks of a newly graduated doctor, students’ perceived acceptability and concerns in medical assessments are an important aspect of the examination’s utility.20 If students do not perceive an assessment to be acceptable and deem that it causes anxiety, its face validity and confidence in use are reduced.21,22

This study aims to address the gap in the literature by evaluating student perception of online ISCE during the COVID-19 pandemic. Given the relatively contemporary use of this modality for assessment, we assessed the change in perception and concern using this modality at baseline and after an online mock ISCE.

Methods

Delivery of Online Mock ISCE

We designed a free one day online mock ISCE using an online platform (Zoom). This was organised as part of a student-initiated peer-to-peer teaching programme involving multiple student medical societies. All stations were designed by individuals who had previously sat for the universities’ in-person final ISCEs, based on guidelines provided by the School of Medicine.

The online mock ISCEs were advertised on social media and mailing lists of the student societies. Every final year (fifth year) medical student due to sit for ISCEs was eligible to take part in the session. Spaces were allocated on first-come first-serve basis. Station examiners consisted of final year student volunteers who had previously completed their final year ISCEs and doctors, including junior doctors until Core Trainee level (5-year post-graduate experience). Simulated patients consisted of student volunteers from second year through to fourth year students.

The full protocol of how we optimised the Zoom platform with breakout rooms, examiner and candidate briefings are provided in the Supplementary Data ().

Questionnaire Design and Dissemination

To register interest in the online mock ISCE, final year students were required to complete a questionnaire (using Google Forms, Google, USA) to determine their baseline characteristics, previous experiences of online examinations and their perception of online ISCEs. Students were given one-week prior to the online mock ISCE to complete these questionnaires. Questions evaluating factors such as worry about an online format (along with contributing factors), anxiety about excelling in online ISCEs, confidence regarding an online format and positivity of an online ISCE experience were designed with a Likert scale. Factors associated with student anxiety and worry of an online format were a combination of pre-set question with binary evaluations and free-text modalities. These were utilised in pre- and post-online mock questionnaires.

Conducting the Online ISCE

Following registered interest, we organised 4 different stations for 64 medical students (total of 256 online mock stations). The online mock ISCE was designed to mimic the real ISCE experience but using an online platform ie, only lasting 15 minutes with standardised examiner questions and mark schemes (in accordance with guidelines provided by the medical school), simulated patient scripts/briefing and no verbal feedback provided at the end of each station. Written feedback was emailed to students after the mock ISCE.

Stations consisted of a combination of history taking (from a patient or collateral), drug explanation, forming a differential diagnosis, interpreting clinical data (such as X-rays, blood tests, ECGs) and describing the management plan for the simulated patient. Practical skills were not directly tested due to the constraints of the online platform. However, candidates were required to describe in full how they would perform clinical skills (for example, the process of administering a subcutaneous injection). Any information to be interpreted by the candidates was shared on-screen by the simulated patients. The “host” of each Zoom meeting acted as timekeeper for different sections of the station and moved candidates between each station ().

Statistical Analysis

Summary statistics were presented as appropriate for baseline characteristics. Normality of distribution of all continuous data was tested using the Shapiro–Wilks test with a threshold of >0.05 considered to indicate normal distribution. The non-parametric paired Wilcoxon rank sum test with continuity correction was utilised to analyse pre- and post-mock ISCE Likert measures (1-Low, 5-High) of anxiety, confidence and experience of an online platform. Binary proportion comparisons of pre-determined potential factors associated with worry from an online ISCE format (eg, body language, clarity of speech, etc) was determined using a McNemar’s Chi-squared test. One-way interaction effects ANOVA were utilised to determine the interaction (Pinteraction) between gender, age and previous online interview or examination experience. For all statistical analyses, a p-value threshold of <0.05 was considered to indicate statistical significance. Statistical analysis was conducted using R statistical programming (version 4.0.2) and each result was independently verified by two study authors.

Ethical Considerations

This project was an evaluation of the effectiveness of a near-peer teaching programme and aimed to improve future teaching sessions. According to advice obtained from the NHS Health Research Authority’s online decision tool, the study did not require formal ethics committee approval.23,24 Data was collected in voluntary manner. Students were informed on the questionnaire that through completion, they were consenting for use of their anonymised data in future publications. No incentives were offered for completing the questionnaire and students were cognisant that completion of the questionnaires did not guarantee a spot for the online mock ISCE, as this was offered on a first-come first-serve basis and examiner availability. Our study complies throughout with the Declaration of Helsinki.

Results

Population Characteristics

Population characteristics are summarised in Table 1. The University’s final year cohort due to sit final ISCEs comprises 202 individuals. 84 out of these 202 students completed the pre-online ISCE questionnaire (41.6% response rate). Additionally, 61 out of 64 students who attended the mock ISCE completed the post-questionnaire (95.3% response rate). The median age of the participants were 22 years [interquartile range (IQR 23–25)] and 79.8% were female. 98.8% (83/84) of students had never attended any previous Online Mock ISCEs while 71.4% (60/84) have never had formal online interviews, where “formal” means under exam conditions. 98.8% of participants had never come across any guidance (books/online/courses) on how to maximise their performance in an online ISCE compared to 10.7% for an in-person ISCE (p<0.0001).

Table 1

Baseline Demographics and Student’s Previous Experiences with Online Format Clinical Examination

Baseline Characteristics	Participants (n = 84)
Age (Median, IQR)	22 (23–25)
Gender
Female (n, %)	67 (79.8)
Male (n, %)	16 (19.0)
Prefer not to say (n, %)	1 (1.2)
Students who never attended any previous online ISCEs or mock online ISCEs (n, %)	83 (98.8)
Students who have never had formal online interviews (n,%)	60 (71.4)
Students who had never come across any guidance (books/online/courses) on how to maximise their performance online (n,%)	83 (98.8)
Students who had never come across any guidance (books/online/courses) on how to maximise their performance in-person (n,%)	9 (10.7)

Abbreviation: IQR, interquartile range.

Baseline Perception of Online ISCEs

At baseline, students appeared to be worried about an online format of ISCEs [median: 3 out of 5-point Likert scale (IQR: 3–4)] and were more confident performing an in-person ISCE compared to online [median: 3 (IQR: 3–4) vs 3 (IQR: 2–3), respectively, p=0.0007]. Students also felt that an online format will affect their anxiety [median: 4 (IQR: 3–4)] and confidence [median: 3 (IQR: 2–3)] in excelling in ISCEs. Students also felt that an online format was a relevant contributor if they did not well their ISCEs, [median: 3 (IQR: 3–4)]. Reliance on devices (such as laptops/tablets/phones) to attend online ISCEs caused significant worry among participants [median: 4 (IQR: 2–5)].

Change in Perception Following Online Mock ISCE

Pre- and post-Online mock ISCE data (Table 2) showed less worry regarding the online format [median: 3.5 (IQR: 3–4) vs 3 (IQR: 2–3), respectively, p<0.0001], less anxiety about excelling in ISCEs [median: 4 (IQR: 3–4) vs 3 (IQR: 2–4), respectively, p<0.001] and an increase in confidence in excelling in ISCEs [median: 3 (IQR: 2–3) vs 3 (IQR: 3–4), respectively, p<0.0001]. Study subjects felt that having done our mock, an online format would more positively affect their overall ISCE performance [median: 3 (IQR: 3–3) vs 3 (IQR: 3–4), respectively, p=0.007]. There was less relevance placed on the online format of ISCE, if students deemed themselves to have not done well [median: 4 (IQR: 3–4) vs 3 (IQR: 2–4), p=0.02] but more relevance contributed by the online format if students did well by their own standards [median: 3 (IQR: 2–3) vs 3 (IQR: 2–4) respectively, p=0.02]. Gender, age and previous formal online interview experience were not related to the change in perception observed above (Pinteraction>0.05 for all comparisons).

Table 2

Pre and Post Online Mock ISCE Questionnaire

Question	Pre-Median (IQR)	Post-Median (IQR)	p-value
(Having done our online mock ISCE) How worried would you be if ISCEs were online? (1- Not at all, 5- Extremely worried)	3.5 (3-4)	3 (2-3)	p<0.0001
(Having done our online mock ISCE) To what extent would an online format of ISCEs affect your anxiety about excelling in these exams? (1- Much less anxious, 5- Much more anxious)	4 (3-4)	3 (2–4)	p<0.001
(Having done our online mock ISCE) To what extent would an online format of ISCEs affect your confidence about excelling in these exams? (1- Much less confident, 5- Much more confident)	3 (2-3)	3 (3-4)	p<0.0001
(Having done our online mock ISCE) To what extent would an online format of ISCEs affect your overall exam performance? (1- Much more negatively, 5- Much more positively)	3 (3-3)	3 (3-4)	p=0.007
(Having done our online mock ISCE) If you do not do well (by your standards) in your ISCEs, to what extent do you think that an online format could be a contributor? (1- Very irrelevant factor, 5- Very relevant factor)	4 (3-4)	3 (2–4)	p=0.02
(Having done our online mock ISCE) If you do well (by your standards) on your ISCEs, to what extent do you think that an ONLINE format could be a contributor? (1- Very irrelevant factor, 5- Very relevant factor)	3 (2-3)	3 (2–4)	p=0.02

Abbreviations: IQR, Interquartile range; ISCE, Integrated Structured Clinical Examination.

Factors Associated with Worry Following Online ISCE

Pre- and post-factors associated with worry due to an online format of ISCEs are summarised in Figure 1. There were fewer total pre-determined potential factors causing worry to students following mock online ISCE [pre-median total: 5 (IQR: 3–7) vs post-median total: 4 (IQR: 3–6, p=0.019]. Despite the numerical decrease in proportion of individual factors post mock online ISCE compared to baseline, individual changes for these factors were not significant. Word cloud analysis () was conducted on other factors causing worry and perceived advantages of online ISCEs.

Figure 1

Shows pre- and post-factors associated with worry due to an online format of ISCEs.

Discussion

To the best of our knowledge, this is the first study assessing the quantitative change in student perception of an online format of ISCE/OSCE following a mock session. In particular, we found that at baseline, final-year students were worried and had lower confidence in excelling in their ISCE due to an online format. Having done our online mock ISCE, students had significantly less worry about the online format, more confidence and less anxiety in excelling in these examinations.

Student perception of an assessment format is vital. Lack of acceptability and negative perception leads to poorer examination performance, distracted focus and reduced motivation, where students study for the examination rather than mastering the course content.19,25 At baseline, almost all students had never completed an online mock ISCE or come across guidance to maximise their performance in an online format suggesting that online assessments are relatively new. This may be especially relevant to assessments encompassing practical/viva-based examinations. In-person examinations represent the norm and how students expect their final examinations to proceed and like many other universities, students at our institution had previously done summative ISCEs in second year.26 Hence, given the rapidity of consideration for online ISCEs at our institution, students may feel more unprepared for the change in format beyond the format itself.27 The timing of our mock examination to the final year examination at our university was also around five weeks. A multicentre-cross sectional study clearly demonstrated that the proximity to final exam clearly affected the mental health of students and suggesting this change in format may further affect their reporting of anxiety and confidence.28

Despite this, multiple factors were noted to be associated with worry from an online format in more than 70% of students, including internet connection (82%), conveying body language (74%) and clarity of speech (72%). Internet and local network capabilities are a concern depicted in a national report of high stakes online assessments in the United Kingdom.29 It may be that universities consider upgrading or maintaining these facilities on campus and if students were to be off-campus, fail-safes could be put in place such as alternative devices or on-campus locations. In addition, clear and reassuring communications with students that they will not be penalised for these issues should they arise, will affect student’s acceptability of this modality.30 Interestingly, use of software was perceived as the least worrying factor for students (48%). This could possibly be due to the already vast advancements of the online modality in medical education encompassing online learning at our institution and these observations are noted throughout the UK, United States and also developing countries.17,31

Importantly, through an online mock ISCE, we were able to quantitatively show a significant change in student perception. Students found our online session useful (), similar to the results of the study by Chisnall et al assessing usefulness of formative OSCEs.32 Given that this was the first experience for almost all candidates, a notable change was probable, and this change was consistent in the questions of worry about the format, anxiety and confidence in excelling in the examinations. There were also significantly less predetermined total factors causing worry post-online mock ISCE (p=0.019), while biggest changes were noted in perception of examiner unfamiliarity (Δ15%), clarity of speech (Δ13%) and surrounding factors (Δ13%; all, shown in Figure 1). These results are in keeping with the qualitative analysis conducted by Palmer et al suggesting that TeleOSCEs were of good video quality, while clarity of communication between student, examiner and interviewer was acceptable.9 Our study and that of Major et al18 also describe the online testability of clinical skills such as rapport building, diagnostic reasoning and clinical acumen, and together this may explain why Lara et al11 showed that there was no difference of mean scores or failure rates among participants doing in-person and TeleOSCEs. Hence, showing a change in student worries associated with these factors, which potentially affect student performance, is vital.

A previous study also interestingly showed that while a change in student’s confidence is associated with their exam performance, their level of anxiety, tested physiologically (salivary cortisol) or perceived subjectively, was not associated with their overall examination performance.33 Our mock examinations had the ability to improve subsequent exam performances,3,26,32 and our results further demonstrate that change in perception may be an underlying factor. Furthermore, in our study, less relevance was placed on an online format of ISCE if students deemed themselves to have not done well, but more relevance contributed by the online format if students did well following our mock ISCE. This shows that students believe a negative performance was less dependent on this new format and other factors such as their own knowledge, preparedness and confidence/anxiety of an assessment may have a greater influence.34 Furthermore, if students did well, the experience and possible practice effect from doing the online mock was a contributor. Not only does this show the importance of exposure to the format but also the significance of mock examinations in improving perception and potentially student performance.

Limitations

This was a study at a single institution with a generally small sample size (n=84), hence may limit the generalisability of these results. It is important to consider that although all final year students at our institution were eligible for this mock ISCE, the sign-up sheets were distributed on our student society mailing lists and social media groups. This might have introduced elements of selection bias to those who may be more engaged in the student life and near-peer teaching programme. 41.6% eligible students signed up for the study, creating a response bias especially towards those who may feel motivated/prepared enough to attend out mock ISCE five weeks before the planned final year examinations. Majority of the students in our study were also female (79.8%) and females are more likely to report test anxiety,35 although proximity of exams seems to affect exam anxiety of males and females equally.28

Future Implications and Conclusions

Online ISCEs/OSCEs represent a novel platform to assess medical students that has been evaluated further following the COVID-19 pandemic.36 We have shown significantly positive changes in student perception using this modality, which presents an alternative to cancelling these assessments during the pandemic. However, that sole reliance on the online format for clinical skill assessment may provide an incomplete assessment of a student’s global clinical competency, as physical examinations and clinical skills cannot be directly tested. Instead, online ISCEs are a great supplement to assess students on a media that has cost benefits, and is convenient and robust for other aspects of testing clinical acumen, as has been described previously.9,11,18 Incorporation may also necessitate change in other parts of undergraduate medical curricula assessment. For example, online ISCEs may form part of newer assessment formats gaining traction following the COVID-19 pandemic, such as programmatic assessment, where regular longitudinal lower stakes examinations are performed.37 With the enhanced convenience and cost, online ISCEs could serve as regular low-stake assessment modality for students (both formative and summative), while physical examination skills may be assessed while students are in their clinical placements. Furthermore, incorporation of these skills and examination format may be necessary, as telemedicine may represent the future of patient management pathways.12