Challenges and possibilities for board exams in the Covid-19 era: experience from the Fellowship Committee of European Board of Nuclear Medicine.

Introduction

The Covid-19 pandemic outbreak, starting at the end of 2019, has been far more than a healthcare crisis due its effect on the economy, education and social life. It has globally affected health care services, medical education and research activities including the field of nuclear medicine (NM). The particular impact of pandemic in NM departments has been recently documented by several international surveys [1-5] reporting an overall decline in diagnostic and therapeutic activities. These studies reported a reduction in the number of NM procedures performed resulting in disruption of usual education activities. The decreasing number of teaching staff, internal or external rotations of physicians, the reduction of exchange programmes and the adaptation of scientific meetings from face-to-face formats to on-line virtual platforms have negatively affected NM training in many institutions, particularly in the training of young residents. Social distancing also has had a negative influence on clinical daily reporting sessions, multidisciplinary tumour boards and interactive lectures that are vital for academic training. On the other hand, this pandemic environment highlighted the importance of digital technology and communication to create new solutions for education and training. Most of the academic institutions and professional organisations developed supplemental materials for education based on e-learning platforms, online programs, continuing medical education sessions or webinars. Also, many international organisations such as European Association of Nuclear Medicine (EANM), Society of Nuclear Medicine and Molecular Imaging (SNMMI), British Institute of Radiology (BIR) or International Atomic Energy Agency (IAEA) quickly implemented different solutions for providing easy access to virtual training tools [6, 7].

While adaptation of these resources into online education possibilities was relatively fast, the issue of medical assessment after completing training or board examinations appeared to be challenging. According to the survey conducted by European Union of Medical Specialists–Council for European Specialists Medical Assessments (UEMS CESMA) in 2021, some European Boards postponed or changed their formal exam structure to online models due to Covid-19 in 2020 [8]. The restrictions in travelling affected exam organisations, resulting in decreased number of submissions, some also related to the shifted priorities in these difficult times. While the usual exam models were varying depending on the requirements of each medical discipline such as multiple-choice questions (MCQ), written, oral or clinical exams, most of the European Boards envisaged online solutions for the future [8]. Furthermore, the transformation of exam models into fully online version is definitely more complicated than the evolution of training into online classes. The idea of implementing remote examination obviously raised important discussions about possible technical difficulties and security problems, with most concern about keeping the high-quality standards of European Board certification or UEMS diplomas.

As there was a lack of published material providing guidance for setting up a remote oral exam for medical boards, we aimed to present the first experience of the Fellowship Examination Committee of European Board of Nuclear Medicine (EBNM) during Covid-19 pandemic. The transformation of a usual face-to-face oral exam into an online model and the steps of this exam setting process were presented in order to provide insights to our colleagues before embarking in an online oral exam.

Pre-COVID EBNM Fellowship Examination

The EBNM was established in 1993 and the first Fellowship Examination was performed in 1996 [9]. So far, nearly 300 NM physicians from the across the globe have been awarded the title of Fellow of European Board of Nuclear Medicine (FEBNM), which is considered a quality recognition, proof of knowledge and skills, attesting that a NM physician meets European standards [10, 11].

Since its inception, the EBNM Fellowship Examination was designed as a 2-step exam, firstly through MCQs followed for successful candidates by an oral exam. The scientific content of the exam is prepared by the Fellowship Examination Committee members, who also hold previously been awarded the FEBNM. The Committee is responsible for creating the annual board exam that is based on the published curriculum and is updated annually. The committee is also responsible for the examination of the candidates. This process focuses on producing a reliable and objective medical assessment, ‘good quality’ MCQs and clinical cases for oral exam, followed by an analysis of the post-test results in line with the UEMS guidance [12]. For the written part of the exam, 140 ‘type A’ questions are created covering basic and clinical NM topics, including clinical case presentations or images. For the oral part, several sets of 5–6 clinical cases are produced to evaluate the scientific knowledge and clinical judgement skills of the candidates. Both the MCQs and the cases used on the oral examination test the whole range of nuclear medicine scans and therapies. The oral exam is conducted by 2 examiners and it aims to reproduce daily case reporting session using the Objectively Structured Clinical Examination (OSCE) model. In order to maintain a standard and reproducible examination, an additional document is prepared with typical questions on clinical cases and expected answers from the candidates, along with marking recommendations for examiners.

The applications of candidates for the exam are reviewed by the board members focusing on the adequacy of training according to ‘training requirements for the specialty of nuclear medicine’ by UEMS/EBNM Education and Syllabus Committee following guidance and rules set out by the UEMS [13]. Those candidates in the final year of their residency or those with completed training and holding the title of specialist in their countries are accepted if they have achieved the training requirements. The information about the application process, fees, dates of the exam sessions and other relevant information is published on the official website (https://uems.eanm.org/). The MCQ was taken using English as was the oral examination.

Previously, a preparation session was organised before the exam. This session was aiming to provide information on UEMS/FENM and the examination itself, enabling the candidates to feel more familiar with the exam format and answering any doubts. This session has been well-attended by the majority of the candidates.

The written exam is used to be organised in two sessions per annum in spring and autumn in the EANM headquarters in Vienna and the oral exam is used to occur during the Annual Congress of EANM.

During-COVID EBNM Fellowship Examination

In 2018, our committee decided to integrate the latest digital developments for an online delivery of MCQ so that it would be easily accessible for colleagues from all over the world with low cost. In May 2019, just a year before the Covid-19 pandemic, a big step forward for the Fellowship Examination Committee was achieved, and the first online MCQ was successfully initiated. The online MCQ offered the possibility to connect to the exam platform during a 24-h period. This connecting flexibility was particularly appreciated by candidates from outside Europe working in different time zones. Once logged onto the examination platform, the candidate has 150 min to answer 140 questions. This evolution from a real written exam into an online MCQ session made us well prepared for the pandemic era. The FEBNM was therefore more prepared than many other medical organisations in being ready to offer the board exams without interruption during the pandemic. The EBNM MCQ online exam was performed on the 12/13th of September 2020 with 22 candidates and on the 29/30th of May 2021 with 35 candidates. Despite the global effects of Covid-19 pandemic, the number of applications for the MCQ 2020 Fellowship exam was similar to the previous years. This was very inspiring for us to see that young trainees were motivated to add the EBNM certification, as a quality recognition to their career goals even during Covid-19 period, though this board exam is not an obligation for their practice in nuclear medicine.

Unfortunately, in 2020, the unpredictable circumstances did not allow to offer a reliable alternative solution for remote oral exam, so this part of the board exam was shifted to 2021. Meanwhile, we started to search for possible options to replace usual face-to-face oral exam into online exam. Considering that there was no official guidance for a remote oral exam by European organisations and that there were only individual attempts from some societies both in the USA and Europe, the situation was discussed between the FEBNM Examination Committee members. After some brainstorm online meetings about the actions to be taken, it was decided to have several test trials for online oral exam using video meeting applications widely available (Zoom, MS teams, GoToMeeting, etc.). We tested different applications to be able to choose the most appropriate and user-friendly. We also wanted to check the clarity of clinical cases presentations in the platform. We had to be sure that the online platform would allow the maintenance of the high quality of the exam content and ensure security and appropriate communication with the candidate during the exam. Also, we wanted to familiarise ourselves with the platform. The results of the trials were satisfying, and active planning was started with the support of UEMS/EBNM office. We decided to use GoToMeeting web conferencing software and to accept maximum of 24 applicants for the exam to make the organisation and scheduling in a more manageable manner. In order to have the highest administrative support and to have inter-connection among the examiners and office staff against any possible failure, we decided that the exam venue for examiners should be EANM office. Five examiners were in the office, while one was connected from home. All the candidates were connected from home and all the exams were conducted by two examiners. This hybrid model of online oral exam for FEBNM was successfully performed in this very first experience on the 6–7th of November 2021. Although this was the first online exam experience, we were pleased that all the places for this board exam were occupied soon after the examination was announced. Of the initial 24 candidates, 23 took the exam and 17 of them passed. The six candidates who failed were encouraged to apply the following year. The success rate of this remote oral exam was comparable with the pre-Covid sessions. While some minor problems were noted as in a real face-to-face exam, we, examiners, consider this online version reliable and sustainable. The feedback received from the candidates was also positive, encouraging us to continue this online option for future, even after the Covid-19 pandemic ends.

The process of online oral exam setup can be defined in 3 stages (Fig. 1) as follows:

Fig. 1

Flow chart summarising the online oral exam settings

As the safety of the online exam was one of the major problems, the preparation of examination rules and instructions manual, the administrative and organisational issues, identification checks of the candidates and individual online appointments for each video call were arranged in this pre-exam period. The candidates were invited to attend a ‘system check session/connection trial’ on the 27th of October 2021, to test whether their technical settings were adequate. This session also allowed us to get familiar with the candidates, decreased the feeling of remoteness and created a friendly atmosphere. The second part of this online meeting was used to give further information to the applicants about the exam. The exam structure and sample cases were presented to the candidates interactively, in a familiar way similar to our previously organised preparation session, which was very well attended by most of them. For those who were not able to attend this connection trial, another link was sent to check their connection and device settings. Verbal and written consent was obtained from all candidates to follow the exam rules and for the agreement of exam session recordings also. This exam setting period can be considered the most important stage. When planning every detail, one should take in account possible problems or unexpected events, such as interruptions of the internet feed, might happen during the exam, so the alternative solutions should be prepared in advance.

A timetable was created for all candidates; the individual time slots of 50 min for each exam were distributed over 1.5-day period. Exam links were individually provided for each candidate and sent to them few days before the actual exam. The exam started with sound and image quality, identification of the candidate and exam room checks, followed by clinical cases’ presentations at the scheduled time of appointment. All exam sessions were recorded. The exam was performed in the normal way using an OSCE format by the examiner teams for each of the candidates. Final assessment for pass or fail was obtained by the consensus of both examiners after the exam session, using the evaluation sheets as usual.

This stage was the review of recorded exam sessions. All the examiners together reviewed the recorded sessions to evaluate the candidate’s performance in case of borderline answers and to decide together whether the candidate’s knowledge was sufficient to pass the exam. This helped all the examiners to review the exams and provided consensus and consistency for the final decision about the result. Moreover, the revision of the sessions allowed examiners to get familiar with the performance of each candidate, and to be able to make a comparative and more homogeneous assessment. Also, in case of an appeal, these recorded sessions would be helpful as an evidence of the candidate’s performance.

The challenges and possibilities

One of the most important issues of a remote exam is the verification of the candidate’s identification. This should be considered a safety measure of exam at the registration stage and on the exam day. Additionally, it is important to establish a certain number of applications for the online oral exam, guaranteeing the quality of the exam. A virtual meeting with all the candidates before the exam appeared to be helpful to initiate an open social connection with them, giving some exam instructions and answering their questions to overcome the feeling of remoteness. During the planning, it should be also considered that technical problems or connection disruptions may occur. Therefore, it is recommended to invite candidates to check their equipment’s compatibility and settings before the exam. During the exam, we had no major problems with internet connection or technical issues, except minor sound problems in a single session only. This problem was temporary and the chat window was used for writing when we had difficulty in hearing. Cheating has been another worrying aspect for this type of remote exams, though remote exam proctoring software could be used, as suggested by some medical boards, despite the increase in cost. Therefore, we decided not to use this kind of external application; we defined the exam rules and regulations in detail instead. Furthermore, the exam room was checked, and applicants were previously informed that any possible action related to cheating would exclude the candidate from the exam. A detailed manual was uploaded on the website and sent to the applicants after registration in line with the exam terms and conditions document [14]. It is our opinion that recording the exam also provides a safety check in case of a suspicion of cheating.

This first online experience stimulated us to improve the visibility of the EBNM Fellowship Examination and Fellowship Committee in social media and develop social network links such as LinkedIn and Facebook to encourage active participation of younger colleagues.

Finally, although missing the face-to-face interaction and in-person communication, this remote option has some advantages, because it is less expensive by avoiding travel and accommodation costs, it is easier to include in the daily schedule, it avoids jet-leg, it saves time, it is environmentally-friendly and it requires less administrative bureaucracies for those candidates who may need to apply for visas to travel to an examination site.

Conclusion

The current Covid-19 pandemic stimulated us to offer digital solutions for both teaching and assessing the knowledge. We think that remote online exam for board exams has a potential to offer a flexible and reliable alternative with easy access and low cost. Despite the inherent limitations of a virtual examination, if adequately prepared, remote oral exam can be organised using the widely available video conferencing applications, while optimising the specific needs of the medical field. This will enable medical boards or societies to continue to deliver online examinations even in difficult times such as Covid-19 pandemic outbreak and, most importantly, widening the perspective for future possibilities. We consider that the pandemic situation has pushed board members, educators and examiners to collaborate and share knowledge, experiences and skills that will reshape medical teaching and proficiency knowledge assessment models for next generation of NM physicians.