A Survey on Changes to the Canadian Anatomical Pathology Certification Examination Due to Coronavirus Disease 2019 and Implications for Competency-Based Medical Education.

The coronavirus disease 2019 pandemic resulted in a dramatic change in the Royal College of Physicians and Surgeons of Canada assessment process through elimination of the oral and practical components of the 2020 Anatomical Pathology examination. Our study sought to determine stakeholder opinions and experiences on these changes in the context of the 2019 implementation of competency-based medical education. Surveys were designed for residents and practicing pathologists. In total, 57 residents (estimated response rate 29%) and 185 pathologists (estimated response rate 19%) participated across Canada; 67% of pathologists disagreed with the 2020 Royal College examination changes, compared with 30% for residents (P = <.00001). When asked whether the Royal College examination should be eliminated, 95% of pathologists indicated they would be against this, compared to only 34% of residents (P = <.00001). Perceptions on changes to and importance of different components of assessment in competency-based medical education were similar between pathologists and residents, with participants perceiving assessment practices to have changed fairly little since its implementation, with the exception of more frequent feedback. Analysis of narrative comments identified several common themes around assessment, including the need for objectivity and standardization and the problem of failure-to-fail. However, residents identified numerous elements of their performance that can be assessed only through longitudinal evaluation. Pathologists, on the other hand, tended to view these aspects of performance as laden with bias. Our results will hopefully help guide future innovation in assessment by characterizing different stakeholder perspectives on key issues in medical education.

Introduction

In the last 2 years, large-scale changes in the evaluation and assessment of Canadian Anatomical Pathology residents have occurred. The first was the shift to competency-based medical education (CBME) in 2019 and the second, the coronavirus disease 2019 (COVID-19) pandemic in 2020.

Of the first change, the move to a CBME model was rolled out by the Royal College of Physicians and Surgeons of Canada (RC) and implemented as the Competency by Design (CBD) program. The purpose of a CBME approach is manifold, but one overarching goal is to adopt a more programmatic approach in assessing resident competencies with: (1) an increased emphasis on direct and indirect assessment, (2) more frequent assessment, and (3) many low-stakes observations of key clinical tasks.

The second change was reactive to extraordinary circumstances and aimed at preserving some element of assessment at the RC examination while maintaining safety in a global pandemic. This change resulted in the RC examination consisting solely of a written component. Ordinarily, the RC examination would otherwise include a written component consisting of short answer questions, a practical component including digital slides, gross and forensic images, and an oral component with a panel of examiners and the candidate.

This change in the RC examination precipitated by the COVID-19 pandemic raises several questions, particularly in light of CBD. Implementation of CBD in Canadian pathology residency training was intended to produce a multifaceted and longitudinal assessment of trainee performance over time which includes multiple, low-stakes assessments or “multiple biopsies” of a resident’s performance. On the other hand, the RC examination represents a high-stakes, single assessment of fitness for independent practice—comparatively and, in a sense, a “resection and gross examination” of the trainee’s abilities. These 2 methods of evaluation are disparate while serving the shared purpose of assessing a trainee’s readiness for independent practice.

This dichotomous model raises an important question: What is the optimal balance within the CBD framework to prepare for the final examination as well as independent practice, and can this be realistically achieved? It has been stated that competence is what a doctor demonstrates in a test situation, but that performance is what a doctor demonstrates in real clinical practice. Knowledge and practical skills do not necessarily translate to competence and while practicing pathologists believe ourselves to be able to assess knowledge, assessment of performance is not necessarily achieved by assessment of knowledge alone.

Miller’s pyramid proposes a hierarchy of assessment, with “knows” at the bottom and “does” at the top. With the unprecedented changes to the RC examination for 2020, the portion of the examination that assessed how the trainee “shows how” to the examination committee was eliminated. How these actions were perceived by the Canadian Anatomical Pathology community has not been surveyed explicitly, nor has the question of whether this change should be regarded as a “one-off” or as a springboard to changing methods of assessment moving forward at the level of the RC.

In this study, Canadian practicing pathologists and Anatomical Pathology residents were surveyed with respect to the 2 significant changes in the landscape of Canadian Anatomical Pathology residency training. Our survey sought to determine the use of and changes in assessment since the implementation of CBD in Anatomical Pathology residency programs in addition to soliciting insights into what aspects of assessment work best to evaluate residents. Our study further sought to determine stakeholder opinions and experiences surrounding the RC examination, with respect to the 2020 change due to COVID-19, as well as the utility of the examination moving forward.

Materials and Methods

Survey Design

Surveys were developed that focused on the changes to CBD assessment and to the RC examination, stemming from the COVID-19 global pandemic. The question-based surveys consisted of 12 and 11 questions distributed to Canadian Anatomical pathologists and residents, respectively (Supplemental File 1), and included both multiple-choice and free-text questions. Definitions of terms, including CBD, and information regarding the changes to the RC examination format for the 2020 cohort were provided. Individual identifying information was not collected and the data were compiled in aggregate. This survey was developed as an opinion-based survey for quality improvement in medical education and did not require institutional ethical review under Article 2.5 of the Tri-Council Policy statement.

The survey for pathologists consisted of a set of demographic questions including number of years in practice, type of practice (community or academic), and number of hours spent teaching residents. The survey for residents consisted of a set of demographic questions including resident level of training, structure of their program (academic, community, or both), and number of hours engaged in teaching with a pathologist (includes time with the resident at a multiheader microscope, academic half days, rounds, etc). Subsequent questions for both pathologists and residents focused on how (if at all) assessment in their program had changed since CBD implementation, how residents were assessed, and which elements of assessment should be emphasized in a CBD model. Respondents were also asked for their opinion regarding the 2020 RC examination format of written examination only and whether they perceived a need for the RC examination in CBD.

The electronic survey was designed using Qualtrics and distributed to Canadian pathologists and Anatomical Pathology residents via the Canadian Association of Pathologists and Ontario Association of Pathologists mailing lists, Anatomical Pathology department chairs and program directors across Canada and informally via word of mouth and social media platforms (Facebook, Twitter). As a result of this distribution strategy, we were blind to the total number of surveys distributed.

Analysis

Descriptive statistics were used to summarize participant characteristics. Responses were compared between pathologists and residents, and statistically significant difference between groups was determined by chi-square testing. Thematic analysis was performed on narrative, free-text based questions. The participant responses were grouped into themes with new categories developed until all participant responses were allocated to a theme.

Results

Demographics

In total, 57 residents and 185 pathologists across Canada participated in the survey. For residents, the estimated response rate was 29%, assuming a maximum of 200 Anatomical Pathology residents across Canada based on available residency spots. For final year residents, the response rate was 50%. For pathologists, the estimated response rate was 19% based on the Canadian Medical Association’s reported number of 960 practicing Anatomical pathologists in 2019. Narrative responses from 130 pathologists and 38 residents were collected.

Participant demographic information is summarized in Figure 1. There were a range of participants in terms of years in practice for pathologists and postgraduate year level for residents. The respondents were predominantly from academic centers, with 51% of pathologists practicing in academic centers and all of the residents training at least in part at an academic center; 14% of residents had a component of training take place in a community setting. The majority of pathologist participants (52%) reported that they spent less than 5 hours per week teaching residents, and in this group, 76% of pathologists were from community sites; 51% of community participants (n = 27) reported they were not involved in resident teaching to any degree.

Figure 1.

Survey participant demographics. The percentage of responses from each respondent category (pathologist and resident groups) is shown. PGY indicates postgraduate year.

Changes in Assessment With Implementation of Competency by Design

How implementation of CBD has changed resident assessment was examined. Resident and pathologist responses were similar. The largest proportion of respondents (93% of residents and 88% of pathologists) indicated day-to-day assessment at the microscope played a role in resident assessment; 54% of pathologists and 46% of residents felt resident teaching was prioritized in their department. With respect to how CBD has changed (or is expected to change) the assessment of resident competencies at their institution, both residents and pathologists perceived local written and practical examinations to have changed fairly little since CBD implementation, although it was believed that residents receive more frequent verbal and written feedback (Figure 2).

Figure 2.

How respondents perceive CBD has changed (or is expected to change) the assessment of resident competencies at their institution in terms of frequency of different aspects of evaluation.

Respondents were asked to what degree they felt assessment components should be emphasized in CBD; 85% of residents and 82% of pathologists felt that day-to-day assessment should be emphasized a great deal. Comparatively, only 17% of residents and 18% of pathologists felt that written examinations should be emphasized a great deal. However, there was a statistically significant difference (χ2 1 [N = 228] = 28.978, P < .00001) between the proportion of residents (26%, n = 14) compared to pathologists (3%, n = 5) who felt that written examinations should have no emphasis (Figure 3).

Figure 3.

To what extent respondents felt assessment components should be emphasized in CBD. Asterisks indicate which differences between respondent groups were statistically significant comparing the distribution of all response categories (P value < .05).

With regard to the utility of various types of assessments in CBD, significant differences were found in pathologist and resident perceptions regarding the importance of the RC examination, practical components of assessment, and a combination of components (Figure 4). In terms of standardized assessment, 60% of pathologists felt the RC examination evaluated resident competencies very well, compared to 17% of residents (χ2 1 [N = 234] = 49.8902, P < .00001). No pathologist respondent felt the RC examination poorly assessed resident competence, compared to 11% of resident respondents. Day-to-day assessment was perceived to be most important in CBD with 70% of residents and 85% of pathologists indicating it assessed residents very well.

Figure 4.

How well respondents perceive each assessment component evaluates residents. Asterisks indicate which differences between respondent groups were statistically significant across all response categories (P value <.05).

Opinions and Perspectives on the Anatomical Pathology Royal College Examination in 2020

Participants were asked whether they agreed with the 2020 decision by the RC to eliminate the oral and practical components of the Anatomical Pathology certification examination. There was a significant difference of opinion between residents and pathologists with 67% of pathologists disagreeing with the 2020 RC decision compared to only 30% of residents disagreeing (χ2 1 [N = 186] = 22.6579, P = <.00001).; 44% of residents agreed with the RC decision to eliminate the oral and practical examination components and 68% of final year resident respondents who had written or were eligible to write the examination agreed with the decision. A large proportion of residents (26%) were neutral regarding their opinion with the largest group being first year residents (56%).

Regarding eliminating the RC examination entirely, 95% of pathologists indicated they would be against this, compared to only 34% of residents (χ2 1 [N = 206] = 78.7427, P = <.00001).

In addition, 74% of pathologists felt that there should be no change in RC examination format and it should continue to be comprised of the 3 components (oral, practical, and written). Of the 16% of pathologists who believed it should change, 21% supported having an oral and practical component only, 18% supported having a written component only, 7% supported the idea that there should be no national examination; and the majority (54%) indicated it should change in a different way. Of those indicating a different change, respondents specified suggestions such as administering the examination in-house or online to avoid travel, eliminating the oral component only and keeping the practical component, testing practical skills rather than minutia, and making the exam more focused on knowledge needed for practice.

In addition, both groups acknowledged the extraordinary circumstances surrounding examination administration in 2020 and the perceived need to make unprecedented changes. However, many participants also expressed frustration with the changes, citing examples and solutions for administering the examination virtually and otherwise.Both residents and pathologists indicated the need for the practical component rather than solely the written component. Participants from both groups noted that the written component tested “minutia” and focused on memorization, which is less useful practically.Possible stigma relating to the 2020 cohort specifically as a result of the change in the RC examination format was evident from the survey responses. Participants expressed doubt around the examination’s utility to test competency and highlighted potential impact on job prospects for 2020 graduates.Analysis of pathologist narrative responses resulted in the identification of 4 inter-related themes around the utility of the RC examination: (1) objectivity and standardization, (2) public trust, (3) failure-to-fail, and (4) bias in local evaluations.

I think the pathology oral examination could have been easily administered virtually through online access to scanned slides for a set time period, followed by virtual video conference with examiners. (for example, the Royal Conservatory of Music is doing piano/violin/flute examinations through this methodology). (Pathologist)

The written exam is much more esoteric than the reality of the average daily pathology practice. Much of what is tested on the written exam can be referenced in reality. The written component only accounts for a proportion (likely less than half) of the knowledge/skill needed to be a functional practicing pathologist. (Pathologist)

Eliminating the practical component for the 2020 cohort will always leave doubts as to their competency. (Resident)

I would not be comfortable hiring from this assessment. As we get so little time with prospective candidates we interview I rely heavily on the fact that I know they have met a minimum standard in a practical exam setting. (Pathologist)

…if we hire someone from the 2020 cohort, I would put in place a period of time with enhanced oversight of their work as I am not confident of a written exam only. (Pathologist)

1. Objectivity and standardization

Pathologists expressed a belief that formal examination, such as the RC examination, offered an element of objectivity that is lacking in assessments at the residency program level.

The Royal College examination is a baseline assessment that all trainees should pass. It eliminates the possibility that residents would become licensed based on adequate evaluations effected by local bias. (Pathologist)

While it is true that most knowledge and skills are acquired during residency, relying solely on evaluation of the programs for certification would not ensure competency for the following reasons - lack of standard (each program is a bit different), lack of incentive for trainees to learn (the exam is a very good goal and pushes residents to study), lack of assessment of minimum level of competency. (Pathologist)

2. Public trust

Pathologist participants believed the current pathway to certification is integral to public trust in pathologists in Canada.

Going from a single national certification authority (RCPSC) to multiple certification authorities would result in a complete loss of standardization, loss of testing accountability, and public and professional loss of faith in the medical training system of Canada. (Pathologist)

This independent assessment for licensing is crucial not only to have competent pathologists, but also to maintain a relationship of trust and confidence for Canadian-licensed physicians by the public. (Pathologist)

3. Failure-to-fail

Pathologists expressed distrust in a program’s ability to adequately assess and, if necessary, to fail trainees. This was based in part on personal experience with residents.However, participants also acknowledged that the RC examination itself is not a panacea for this issue.

I have no faith in the ability of the residency program directors to deliver an objective final assessment. They are at less than arm’s length…I don’t believe I’ve ever seen a really bad final assessment even when it was a really bad candidate. (Pathologist)

The exam is already too easy as it is, as I have seen many people pass who were not ready for independent practice. (Pathologist)

4. Bias in local evaluations

This theme mirrored the first theme of objectivity and standardization. In contrast to the RC examination, program evaluation and assessment was seen as influenced by factors that introduce bias to assessment.Residents also expressed views on the utility of the RC examination, and 4 themes were identified among resident participants. Although there were some similarities between the 2 groups, unlike pathologist perceptions, many narrative responses focused on the possible negative aspects of the RC examination.

Evaluations and assessments can become complicated by personal interactions, relationships, local politics and can even become litigious when a candidate’s ability to practice may be in doubt. (Pathologist)

The *huge* problems with CBME will be getting the message out to all the clinical teachers in the community. It will work OK in the big academic centres, but you’ll find that for the community-based preceptors it will be business as usual. (Pathologist)

1. Utility of the content tested, particularly within the written component of the examination

Residents felt their learning was impaired by the need to spend time memorizing content and felt the utility of this was limited. Pathologist respondents in practice also questioned the utility of the examination as a negative experience with respect to learning.

Pathology practice is open book, we should and do look up books and resources. If there is no exam, residents will learn better and in a relaxed environment. (Resident)

I think that an official national exam is an antiquated way of testing that fosters a highly stressful environment for already stressed and overworked physicians and imposes an extraordinary toll on mental and physical wellbeing. Moreover, it tests your ability to take a test and not your ability to make accurate and safe diagnosis in a caring and professional manner in a real-life setting. (Pathologist)

2. Competency should be assessed longitudinally

A second theme relayed concern regarding how competency should be determined by multiple factors and that performance assessment is multifaceted. Participants reiterated their opinions that competence needs to be assessed longitudinally and based on real-life performance, and not necessarily determined in one day or point in time, as with the RC examination. This contrasted with pathologists’ perceptions of a lack of objectivity and the presence of bias in program assessments. Residents viewed assessment of factors other than medical expert knowledge, such as work ethic and time management, as a component of their competence, which is not assessed on a knowledge-based examination.

A program sees how residents perform and improve over a long period of time, see how they deal with day to day cases and scenarios, and are better suited to determine a resident’s competence for independent practice. (Resident)

Some resident work hard in rotations, they are ready to be a staff, get great evaluations but will not remember the classifications, syndromes or first 10 chapters of Robbins etc. (Resident)

3. Objectivity and standardization

Some residents, similar to pathologists, perceived that the examination offered significant benefit in terms of standardization, given the difference in training across Anatomical Pathology residency programs in Canada. However, other residents perceived residency programs to be in the best position for trustworthy evaluation.

Path residency programs can be trusted and should be able to evaluate residents quite well. (Resident)

4. Failure-to-fail

Similar to pathologist participants, a final theme addressed the issue of “failure-to-fail,” but skepticism in CBD remained.

I believe we need an impartial evaluation of the competencies of the residents before declaring them fit for practicing on their own. I have seen residents finish their residency program where the program let them pass their rotations when they were clearly not ready to graduate. I believe that our profession has to keep a certain standard of expertise that we should not leave behind. (Resident)

While in theory programs should be capable of assessing residents and only promoting them to the next year in the appropriate setting, I know this absolutely does not occur in my program. Residents are unfortunately promoted to the next year regardless of abilities or inabilities. (Resident)

Discussion

This national survey-based study on pathologist and resident perceptions around the changes to the RC examination due to the COVID-19 pandemic and changes related to assessment in CBD in general shows a significant contrast between pathologist and resident views.

Our findings on changes related to CBD suggest formal assessment practices such as oral, written, and practical examinations have changed little, however there is more frequent feedback. Feedback is an important cornerstone of CBD, and our study results align with studies in other specialties which have also reported an increase in feedback with implementation of CBD principles. Increased feedback and competency-based assessment have multiple benefits including better identification and remediation of residents in difficulty. In Canada, implementation of assessment in CBD is through the evaluation of Entrustable Professional Activities (EPAs) and milestones. As respondents did not indicate that formal assessment became less frequent, this suggests overall assessment increased with the addition of increased feedback. Other studies in nonpathology settings have shown that implementation of CBD increases the frequency of EPA-based assessments, compared to the previous system which often relied on a single end-of-rotation evaluation. Respondents did disagree to the extent certain assessment components should be emphasized, with more pathologist respondents indicating written, oral, and especially practical examinations should be emphasized. Our study did not explore whether the nature of examinations have changed in order to align with CBD; however formal assessment practices can and should change to reflect more programmatic assessment and to enhance the practice of assessment for learning. Our study therefore highlights a potential area for improvement in Anatomical Pathology education in the CBD era—namely, with respect to formal assessment practices. Pathology is a unique field where real-world assessment tasks are easily portable to examination settings—for instance, correct diagnosis of common cases. As such, it can be argued that in the Anatomical Pathology setting, formal practical assessment is a necessary complement to workplace-based assessment, in particular for cases which may not be encountered frequently in daily practice. However, prior studies have shown that CBD implementation has increased administrative and assessment burden, for both educators and residents. Resources for enhancing formal assessment practices may therefore be limited as significant changes are made to evaluate EPAs and provide frequent feedback. Nonetheless, competency frameworks emphasize not only medical knowledge domains but behavioral competencies such as collaboration, communication, and professionalism. These behavioral competencies are best assessed in unstandardized practice settings through the lens of expert judgment. This is borne out in our results indicating day-to-day evaluation is felt to be important in a CBD assessment program and that this modality evaluates residents very well. Our findings on changes to assessment in CBD overall mirror other studies in terms of finding an increase in feedback, without an increase in formal assessment. Respondent views on formal assessment differed, which ties into the second part of our survey.

A striking finding was the divergence between resident and pathologist perceptions around the RC examination. Firstly, with respect to the changes in the 2020 administration of the RC examination, resident respondents were more likely to agree with the elimination of the oral and practical components compared to pathologists. Residents were also more likely to support elimination of the RC examination overall compared to pathologists. This difference may be related to differing perceptions around the role of assessment—longitudinally versus a one-time assessment in a formal, high-stakes examination. Residents identified numerous elements of their performance that can be assessed only through longitudinal and multifaceted observation and evaluation. Pathologists, on the other hand, viewed these aspects of performance as laden with bias. Residents viewed the examination through the lens of their own performance, whereas practicing pathologists tended to see the examination as a type of quality assurance. Pathologists saw standardized examination as an objective evaluation of residents.

The question of the role of a certification examination is an interesting one and intersects with discussions centered in CBD. Assessment is known to impact how learners learn and should be seen not only as a tool for measurement but also as a driver for improvement. Assessment is important to monitor to understand how it drives learning and to promote desirable learning behaviors. In addition, assessment may drive negative effects. For instance, if the primary content in assessment measures factual knowledge, learners will concentrate on retaining facts rather than developing clinical reasoning skills. Our findings suggest that learners perceive negative consequences to the RC assessment. Several resident participants expressed frustration with the idea that the RC examination contained an emphasis on elements that are perceived as arbitrary by both groups. Trivialization of assessment is an issue insofar as it drives trainees to learn in a way that is ultimately not beneficial for the stated goal of graduating competent doctors. Examination content that prioritizes testing of isolated facts has been criticized for contributing to trivialization of assessment. This is true for both the RC examination and formal assessment practices within residency programs.

At the same time, formal summative evaluations are important for assessment of medical knowledge, and our findings suggest formal summative evaluations have not increased in the CBD era. There are implications with respect to the changes to the 2020 RC examination. Particularly, pathologists were more likely to disagree with the change to the format than residents were, and a few participants cited unanticipated negatives, such as consideration for job opportunities. Studies on skills needed of newly trained pathologists indicate medical expert knowledge is a component of readiness to practice, however a multitude of other skills are required, such as judgment in requesting consultation and in ordering ancillary testing. Nonetheless, medical expert knowledge is rated highly in hiring decisions. Work ethic and professionalism are crucial and are assumed to be built on a base of diagnostic excellence. As such, this is an important factor to consider with respect to the RC examination and any changes moving forward. With respect to CBD, our study found respondents valued day-to-day assessment in residency training. This has value as assessment of a number of cases by multiple observers is more objective than traditional evaluations completed by a single assessor. However, as practical, oral, and written assessments were not perceived to increase in frequency, this suggests that the RC examination may still have an important role to play in the CBD era.

Failure-to-fail is a critical and complex problem and has been well recognized by the medical community. Two considerations on this topic were revealed in our survey. The first is a lack of trust in residency programs in terms of adequately remediating or dismissing residents who perform below standards. The second is entrusting certification bodies like the RC, or other forms of standardized assessment, as a means of mitigating failure-to-fail. Performance in clinical settings is frequently collaborative and interdependent on multiple team members, making assessment of independent trainee behaviors challenging. As such, standardized examination does have a role to play within assessment of competency. Prior studies have demonstrated that medical expert knowledge is an area where residents may require remediation. Through our study, we found that residency programs used a combination of assessment strategies in their evaluation of residents. Practical, oral, and written examinations administered by the program were reported to be frequent, however no change was seen as a result of implementation of CBD. These components provide data on at least the medical expert competency. However, the reality of clinical practice indicates it is important to assess and consider a trainee’s performance in a team setting, with the elements of subjectivity and interdependence that exist in these settings. The RC examination is not designed to adequately test other domains that are relevant to pathology practice, such as professionalism and laboratory management, given its “one-time, do-or-die” application. To adequately assess these competencies, longitudinal observation by multiple observers and multiple assessments are required for a comprehensive assessment of resident performance over time. The increase in feedback and the importance of day-to-day assessment adds an important complement to formal assessment practices.

Factors that impact the ability to adequately identify and deal with trainee incompetence include institutional factors such as limited support for remediation as well as supervisor factors such as social concerns with respect to failing residents. Utilizing the RC examination for the purpose of identifying incompetent residents is problematic as it may not leave adequate time for possible remediation. Certainly, high-stakes summative evaluation is not a cure-all against failure-to-fail and should not absolve residency programs of their duty to effectively train, remediate, and in some cases dismiss residents. Our findings raise a potential lack of confidence in residency training programs to tackle the problem of failure-to-fail, which is consistent with other studies noting that residency programs struggle with remediating and potentially failing residents. However, in all, early identification of residents in difficulty is essential for remediation and mitigating the problem of failure-to-fail should be addressed at the program level.

Finally, in terms of the future of the RC examination, the COVID-19 pandemic has served as a catalyst for potential widespread changes in medical education, from the restructuring of certification examinations to increasing utilization of online learning. Overall, given the almost unanimous disagreement with respect to eliminating the RC examination, we anticipate that despite the change to a CBD training model in Canada, this examination will continue to play a role in trainee assessment and licensing, particularly if formal assessment practices do not increase to compensate. As formal assessment practices within residency programs add further administrative and educational burden, and given that significant resources must be put toward implementing other aspects of CBD, it is possible that enhancing formal written, oral, and practical examinations will fall by the wayside. Our findings suggest robust efforts are needed to maintain the practical and oral components of the RC certification examination in Anatomical Pathology specifically. Technology should be employed to facilitate virtual examinations, including a practical and oral examination component. Although this was not utilized for the 2020 examination, a virtual practical component was implemented for the 2021 examination in Anatomical Pathology.

One limitation to this survey study was that the exact response rate could not be determined due to broad distribution methods. In order to maintain anonymity, identifying information, including IP address, was not collected, and as such, we are unable to assess whether an individual submitted multiple surveys. Of note, there was no evidence of multiple identical survey submission and an individual was prevented from completing the survey multiple times on the same computer. Second, our study was not designed to determine the specific content administered in programmatic formal assessment; however if and how these assessments have changed with CBD implementation would be an interesting avenue for further study.

Competency by Design invites us to ask how one can determine whether we are achieving its stated purpose—that is, to train competent pathologists. Although our survey was not designed to determine the overall quality of feedback and assessment in CBD, it does suggest: (1) the need for multimodality assessment and highlights the perceived importance of day-to-day assessment, (2) the perceived value of the RC examination in the CBD era, (3) the ascertained problem of failure-to-fail in Canadian Anatomical Pathology residency training programs, and (4) problematic gaps in pathologist and resident perceptions.

It is worth exploring how we can improve the CBD process to optimize the positive impact of the examination while mitigating trainees’ negative appraisal of it. Our study suggests the need for future innovations in order to deliver components of assessment most valued by the Canadian Pathology community. The implementation remains to be seen; however, this study indicates that the Anatomical Pathology community values practical examination at the level of the national certification examination and day-to-day longitudinal evaluation at the level of the residency training programs. Future efforts can be made to strengthen these 2 components in assessment of trainees in our specialty going forward.

Click here for additional data file.

Supplemental Material, sj-pdf-1-apc-10.1177_23742895211060711 for A Survey on Changes to the Canadian Anatomical Pathology Certification Examination Due to Coronavirus Disease 2019 and Implications for Competency-Based Medical Education by Katherina Baranova, Emily A. Goebel, Jason Wasserman and Allison Osmond in Academic Pathology