Lessons to Learn From a Successful Virtual Mock Oral Examination Pilot Experience.

oronavirus (COVID-19) has caused marked impact on graduate medical education for all medical specialties. Radiation Oncology and the American Board of Radiology have also had to rapidly adapt to converting education and examinations to virtual platforms. We describe our small pilot experience in transitioning our in-person mock oral examinations to a virtual platform. Survey-based assessment revealed excellent feedback regarding ease of use and educational usefulness. Our mock oral examinations pilot experience adds to evidence that virtual mock oral examinations are an important considerationfor Radiation Oncology education and a feasible alternative to an in-person oral examination.

Introduction

Novel coronavirus disease 2019 has had a marked effect on clinical care and graduate medical education, including in radiation oncology (RO). Nearly all RO physicians face challenges balancing personal and professional obligations amid unprecedented difficulties. Appropriately, the American Board of Radiology (ABR) postponed the oral and written board certification examinations for 2020.

One of the most daunting aspects of any radiation oncologist’s career is facing the oral certifying examination (OE). The American Board of Medical Specialties (ABMS) has been phasing out OEs from board certification; the ABR recently terminated the diagnostic radiology OE. Despite recent controversy related to administration of the written examinations, OE in RO board certification has been well-accepted in recent editorials by the American Society for Radiation Oncology and the ABR addressing RO board certification reform.,

How best to prepare for successful completion of the OE is undefined and nonuniform; there exists no standardized method to prepare for the OE. A recent survey reported mixed results, with 73% of residency directors indicating that their faculty conducts mock oral examinations (MOEs) ≥50% of the time; however, only 56% of the corresponding chief residents reported the same finding.

The benefits and logistics of implementing a rigorous MOE curriculum have been documented in other specialties represented in the ABMS.4, 5, 6 In general surgery, Fingeret et al reported the correlation between MOE performance and first-time pass rate of OEs, as well as the educational value of sequential MOEs. Given these purported benefits of MOE and the necessity of social distancing, we conducted a virtual MOE in lieu of in-person MOE. We report here our pilot experience with a virtual MOE to highlight its ease of implementation for RO graduate medical education and to demonstrate its feasibility as an option for the ABR’s OE.

Virtual MOE Format

In the National Capital Consortium RO residency, we conduct annual MOEs for all residents each April that mimic the published guidelines for the OE by the ABR. We additionally incorporate a minimum of 3 to 5 minutes of feedback per each of the eight different clinical categories. All residents participate in the MOE, with the particular site and number of sites required dependent on level of training. We also welcome recent graduates from our program and regional board-eligible junior staff to participate in this experience. Owing to coronavirus disease 2019 constraints, we decided to conduct the examination using an interactive web-based software meeting platform (Zoom Communications Inc, San Jose, CA).

Eight examinees (6 residents, 2 junior attendings) and 8 examiners representing 4 academic institutions participated in this MOE. All were offered an optional test run in advance of the MOE to explore features of the virtual platform. “Breakout rooms” allowed for multiple simultaneous examinations to run within 1 larger meeting. “Screen share” and “annotate” allowed for anatomy review, contouring, and treatment plan evaluation. Patient-identifying information was not permitted in any examination materials.

On the day of the virtual MOE, we asked all participants to join a brief orientation. The residency coordinator proctored the examination and switched examinees between each site (breakout room) every 30 minutes, after a 25-minute examination and 5 minutes of feedback. The examination was conducted successfully, with all examinees rotating through their sections in the allocated time frame. Screenshots with diagrams of the examination administration are shown in Figures 1 and 2.

Figure 1

Example of an anatomy review (prostate MRI anatomy image courtesy of Prostadoodle) using screen share.

Figure 2

Example of interactive contouring (of a salvage prostate case) using screen share and annotate features.

Postexamination Survey and Survey Results

A survey was developed using Google Forms (Alphabet Inc, Mountain View, CA) by the chief resident and the residency program director. It aimed to assess the preparation, administration, and overall MOE experience for both examiners and examinees. Survey items to assess each of these areas were developed with most questions either using a 5-point scale (not at all, minimally, somewhat, moderately, and very confident) or a yes/no answer format. Most questions also allowed for a free text input if there was feedback that could not be captured appropriately by the survey questions.

After the examination, we conducted a postexamination survey for both examiners and examinees. We had an excellent response rate of 100% for examinees and 87.5% for examiners.

The majority of examinees and examiners (87.5% and 85.7%, respectively) reported that the virtual platform (Zoom) was “easy to use” or “very easy to use”; 87.5% of examinees and 100% of examiners (who participated in test session) reported the test run left them “very well prepared”; 100% of examinees and 85.7% of examiners reported that the web-based software meeting platform was “very adequate” or “somewhat adequate.” There was a free-text section in the survey regarding any technical issues encountered during the virtual MOE; however, none of the responses reflected any technical difficulties, but they did reflect that it was felt to be a fairly seamless experience.

Compared with their previous in-person MOE experience, 87.5% of examinees and 83.3% of examiners reported that the virtual MOE experience was either easier or the same for ease of understanding the cases. One hundred percent of examinees and 100% of examiners reported the preparation for virtual MOE to be equivalent or less time-consuming than preparing for in-person MOE. Seventy-five percent of examinees and 71.4% of examiners reported that they would “definitely use” or “consider using” a video recording of each encounter to enhance learning/teaching.

In regard to the use of unauthorized resources (ie, use of any reference material that is not available during in-person OE) during the examination, examiners reported that 57.1% were “not concerned at all,” whereas 28.6% reported they were “somewhat concerned.” When asked if examinees should have access to certain resources (“open book”) during the MOE and OE, 25% and 37.5% of examinees reported that they should “definitely” and “maybe” have access to such resources, respectively; 37.5% of examinees reported being “unsure.” When the same question was asked of examiners, 42.9% and 14.3% reported that examinees should “definitely not” and “likely not” have access to such resources, respectively.

This pilot experience and survey were conducted before the ABR’s official announcement of a move to a virtual platform for the 2020 OEs. When asked how strongly they felt the ABR should move to a virtual OE, 50% of examinees and 42.9% of examiners preferred virtual OE, 25% of examinees and 42.9% of examiners had no preference, and only 25% of examinees and 14.3% of examiners preferred in-person OEs. Nevertheless, 87.5% of examinees and 85.7% of examiners reported a virtual MOE was a feasible alternative to an in-person MOE.

Discussion

In our small pilot virtual MOE, we found the virtual platform allowed for high reported rates of quality and adaptability of virtual MOE. There appeared to be little to no learning curve to using the web-based software meeting platform and all participants found this format to be no more time-consuming than an in-person MOE. These findings are expected given the numerous applications of virtual platforms routinely in the clinic (ie, virtual grand rounds, virtual chart rounds, virtual treatment planning).

Additional benefits to virtual MOE and OE can include a video recording for both an appeal process of pass/fail decisions and for quality assurance of examination administration. In our survey, approximately 70% to 75% of respondents reported they would consider using this as a tool for their learning and/or teaching.

One common concern with virtual MOE is the possibility of the use of unauthorized resources. Our examiners reported a small majority reporting no concern of the use of such resources. In an era when some have questioned the modernity of RO board certification,, we also surveyed the use of open resources on the test. Examinees reported a slight majority favoring use of open resources, whereas examiners reported a slight majority against the use of open resources. This is subject to bias in both cohorts with respect to experiences preparing for and/or having taken the OE.

Although this is the first report of virtual MOE in RO, it is not unique in its effort to successfully implement the latest technology to bridge gaps in RO education, with multiple successful international RO education initiatives dating back to 2007.10, 11, 12 Additionally, a small randomized trial of in-person versus virtual MOEs for emergency medicine residents found no differences between the groups with respect to competency evaluation and quality of assessment. This study also found that the examinees preferred the virtual format and found it to be less intimidating.

There are numerous potential advantages to virtual OEs: increased accessibility, decreased costs, decreased time away from family, increased quality assurance and feedback using video recordings, and decreased psychological intimidation. Another significant concern that can be mitigated is the potential discomfort of an examinee in the closely cramped quarters of an unmonitored examination room. These come at the cost of minimal disadvantages of virtual MOEs, nearly all of which can be easily accounted for: increased potential for use of unauthorized resources and potentially altered examination quality because of virtual environment. These can be mitigated with the use of proctored, remote testing centers near each examinee, allowing appropriate monitoring while maintaining the advantages of virtual OE. Of note, the case for virtual OEs is clearly demonstrated by the shift of 2 members of the ABMS from in-person OEs to a virtual OEs for 2020: the American Board of Surgery and the American Board of Ophthalmology.

There was an overwhelming response to the ABR’s initial decision to maintain an in-person certification OE, to take place in December 2020, by the Association of Residents in Radiation Oncology, American Society for Radiation Oncology, and Society of Chairs of Academic Radiation Oncology Programs. In light of consulting with these stakeholders and various organizations, the ABR has appropriately decided to create a virtual OE that will occur in 2021. Our small virtual MOE pilot experience adds evidence that virtual MOEs are an important consideration for RO education in the technological age and that a virtual OE is a feasible option.