Exploring the role of open book high-stakes examinations in 2021 and beyond.

Performance on medical licensing examinations has been previously shown to be predictive of performance in practice. However, licensing examinations are closed-book and real-world medical practice increasingly requires doctors and patients to consult resources to make evidence-informed decisions. To best assess the ability of physicians and physicians-in-practice to avail themselves of point-of-care clinical resources and tools, open-book components may have an emerging role in high-stakes examinations.

Introduction

Performance on medical licensing examinations has been repeatedly demonstrated to be predictive of performance in practice.[1]-[3] Adverse events during medical care are significant,[4] and high-stakes examinations have an important ‘gatekeeping’ function in ensuring that physicians and surgeons have the necessary knowledge and skills to care for the Canadian public. Historically, licensing and certification examinations in Canada have been closed-book, i.e., the candidate is expected to ‘possess’ the requisite knowledge to succeed the examination. When examinations targeted knowledge recall, this approach was arguably concordant. However, the licensing and certification processes are now aimed at higher levels of Miller’s pyramid–ranging from interpretation and application of knowledge to demonstration of skills, and even real-world performance—with the aim of identifying those who are (in)competent for clinical practice. At the same time, the medical knowledge base is rapidly expanding and much of this information is available, imperfectly, to our patients.[5] Thus, the time is opportune for a re-evaluation of the place of open-book examinations in licensure and certification.

Knowledge and online resources in clinical practice

While doctors must have core knowledge that they draw on in everyday practice, the medical literature is estimated to double every 73 days.[5] It is thus unrealistic to think that any doctor’s knowledge base will be sufficient to answer every clinical question within their scope of practice. To facilitate the application of knowledge during clinical care, a broad range of evidence-based tools designed to be used at point-of-care are readily accessible to learners and physicians in practice.[6] While many doctors read relevant primary literature, knowledge syntheses ranging from systematic reviews to clinical guidelines and clinical decision support databases such as Dynamed® and Up-to-Date® help doctors access knowledge just-in-time. Apps that allow doctors to enter data and calculate scores that guide management—from the Ottawa Ankle Rules score recommending whether an ankle x-ray is indicated after traumatic injury, to the Pneumonia Severity Index that recommends inpatient or outpatient treatment for pneumonia, to the Well’s Criteria for DVT—are commonplace, open-access, and available on computers, tablets and smartphones. Learners and physicians are encouraged to use clinical risk calculators in their assessment and document this in the patient’s dossier. In hospitals and clinics, patient order sets and computerized order entry programs often integrate clinical decision aids, suggesting dosages, highlighting the costs of various treatment options, and grouping multiple interventions together to encourage evidence-guided care and reduce errors of commission and omission.

In parallel, patients commonly use the internet to gather information in advance of medical appointments.[7] Doctors often have to help patients navigate a web of (mis)information, countering false claims with reliable evidence accessed during the visit and guiding patients to reliable websites outlining their surgical pathway or information on their condition.[8]

Thus, during clinical training and in practice, medical students and doctors are encouraged to look up what they don’t know, engage with clinical decision-making tools, and help patients navigate information sources to provide safe and effective high-quality care. Yet, these authentic physician tasks are assessed poorly with closed-book examination formats.

Medical education beyond medical expertise

While currently well-known and widely accepted, the Royal College of Physicians and Surgeons of Canada (RCPSC) CanMEDS Framework[9] postdates the establishment of most Canadian licensure and certification practices. While the CanMEDS Framework was formally adopted by the RCPSC in 1996,[10] its widespread uptake began in 2005 following the integration of the CanMEDS Framework into educational standards for RCPSC-accredited residency programs. CanMEDS-FM was adopted by the College of Family Physicians of Canada (CFPC) in 2009. Accordingly, by 2010 there was explicit recognition in all medical specialties in Canada that while medical expertise remains at the heart of the doctor’s role, knowledge and technical skills are insufficient. Since the adoption of CanMEDS and CanMEDS-FM for postgraduate medical education (PGME) and continuing professional development (CPD), the CanMEDS Framework has also been used to guide the medical education objectives of many Canadian undergraduate medical education (UGME) programs.

The objectives of the Medical Council of Canada (MCC) examinations[11] (are structured according to the CanMEDS Framework, and the examinations of the RCPSC and CFPC likewise may assess any or all of the CanMEDS roles. In addition to influencing the content of high-stakes examinations, the CanMEDS Framework could inform our assessment processes. For example, the CanMEDS Scholar role speaks to the importance of doctors engaging in lifelong learning and integrating best available evidence into practice. In parallel, the MCC Scholar objectives speak to, among other competencies, the need for candidates to: “Retrieve information from appropriate sources” and “Integrate retrieved information into clinical practice.” Thus, accessing and applying information is as much as a physician skill as developing a core knowledge base of medical expertise and is a legitimate target for high-stakes assessment.

While the potential to assess the Scholar role during an open-book examination is perhaps most obvious, other CanMEDS roles may also be assessed more authentically through integration of open-book components within licensure and certification. In an OSCE or long-case oral examination with a standardized patient, we could conceivably assess how candidates navigate doctor-patient communication alongside the use of open-book resources, and whether their use enhances or detracts from the clinical encounter. Similarly, open-book examinations might afford a window into whether a candidate uses information to empower patients to make choices to enhance their health. One can use nutritional information to demonstrate that a muffin has more calories than a donut,[12] or a risk calculator to illustrate how smoking cessation or treating blood pressure would decrease the patient’s 10-year risk of myocardial infarction or death.[13] Open-book elements in licensure and certification examinations could, thus, plausibly offer authentic assessment of multiple CanMEDS roles.

Evolving literature regarding open-book examinations

In 2016, Durning et al. published a systematic review comparing open-book and closed-book examinations.[14] Despite reviewing upwards of 4000 papers published prior to 2013-2014, only 37 met criteria for inclusion in the review. Of these, only nine were directly related to medical students or physicians in practice, and only two were related to assessments that could be considered high-stakes. The authors considered exam preparation (learning strategies and time spent studying), test-related anxiety, student performance, time to complete the examination, and test-enhanced learning. Overall, the authors concluded that there was insufficient evidence to favour exclusively open-book or closed book examinations. They found that both formats were associated with test anxiety and test-enhanced learning, though closed-book examinations were associated with more time studying and higher test scores in some studies and open-book examinations tended to take longer for students to complete.

Since that systematic review, there has been a multitude of papers describing the use of open-book examinations in the health professions. In the context of the COVID-19 pandemic, many education programs had to pivot their learning and assessment activities to an online format. Given this context, the expanding body of evidence largely relates to remote online open-book examinations implemented out of concern regarding remote proctoring. Many of these are descriptive single-institution experiences or opinion pieces, and few refer to high-stakes examinations or oral examination formats.

Several papers since January 2020 are worthy of note. Sam et al. describe a high-stakes, end-of-year medical examination successfully implemented as a remote open-book online exam, with examination scores being similar to prior closed-book administrations.[15] Jervis & Brown describe a student’s perspective of the challenges of preparing for the less-familiar open book format.[16] Zagury-Orly & Durning ask us to challenge our reluctance to implement more open-book examinations in medical education and to consider the contexts in which they may prove useful.[17] The literature does not give a unified message regarding when to implement open-book examinations, but does provide insight into how to implement open-book examinations by highlighting potential benefits and pitfalls.

Using the theoretical rationale for integrating open-book elements within high-stakes examinations to build on the existing literature, the opportunities and cautions can be conceptualized as outlined in Table 1.

Criteria[18]	Opportunities	Cautions
Validity / Coherence	Improved authenticity vis-à-vis current clinical practicePossible to assess multiple CanMEDS rolesSimilar scores to closed-book examinations, i.e., application of knowledge can be assessed	Risk of construct-irrelevant varianceRequires new items and item banksRequires mechanism to prevent communication with other people despite access to online resources
Reproducibility / Consistency (Reliability)		Impact on psychometric properties of exam format (MCQ, SAQ, OSCE) not fully demonstrated
Equivalence		Risk of inequity (experience with, knowledge of, and access to resources)Risk of technical failure if resources online
Feasibility	Written exams do not require proctoringCould be integrated into various testing formats (MCQs, SAQs, OSCEs)	Longer testing timeRisk of longer marking time if items not in single-best answer format
Educational Effect	Demonstrated test-enhanced learningImplicitly message that it is impossible to ‘know’ everythingAllow for technology-enhanced assessmentPotential to influence educational programs to enhance teaching and in-training assessment of self-directed learning, integration of evidence into decision-making, and use of references and tools at point-of-care	Candidates need additional guidance on how to prepareImpact of candidate’s stage of training or practice not yet known
Catalytic effect	Meaningful feedback can be given	Requires identification of what feedback areas are most appropriate
Acceptability		Emerging

Conclusion

Given the importance of authentic assessment in the health professions,[19] it is opportune to consider how open-book components might be integrated within medical licensure and certification examination processes. There are strong theoretical reasons why open-book examinations may have a role in high-stakes assessment. Alongside closed-book examinations that assess the candidates’ ability to apply their knowledge base to clinical scenarios, open-book examinations can promote test- enhanced learning while more closely approximating the use of information resources in actual clinical practice. With diligent planning to mitigate against the cautions associated with this approach, an open-book component within a licensing or certification examination provides opportunities to enhance authentic assessment of multiple CanMEDS roles.