Real-world impact of the COVID-19 pandemic on the assessment of anaesthesiology residents.

Editor—The coronavirus disease 2019 (COVID-19) pandemic has forced a sudden and drastic change in anaesthesia resident training, with significantly reduced case volumes and frequent trainee redeployment to other clinical areas. Additionally, the increased demands related to applying COVID-19 perioperative protocols and personal protective equipment recommendations, social isolation, and concerns of getting ill and infecting family members are all factors impacting the health and well-being of both residents and faculty. Many postgraduate anaesthesia training programmes worldwide have shifted to competency-based medical education curricula, the success of which are contingent upon adequate clinical exposure and timely workplace-based assessments (i.e. ‘what doctors do in practice’) with learner feedback and with close monitoring to gauge learner progression.

A paucity of formative assessment opportunities during this pandemic is a significant concern for competency-based programmes; however, the real-world impact of this pandemic on trainee experience and assessment has not been documented. As the largest Canadian anaesthesia residency training programme, located in one of the Canadian epicentres for COVID-19, we were interested in gauging the pandemic's effect on learner assessment. We conducted a study that aimed to compare the quantity and quality (as determined by information richness) of resident assessments completed during the COVID-19 pandemic (March 15 to May 15, 2020) with the same time frame in 2019. The primary outcome was the number of assessments completed. Secondary outcomes included time to assessment completion and richness of information provided (entrustment ratings and comments).

Concurrent with the launch of a competency-based programme in 2017, the Anesthesiology Department at the University of Toronto created and implemented a novel ‘assessment dashboard’ that captures and visualises resident workplace-based assessments to facilitate reflective learning (i.e. assessment for learning). Assessment data related to entrustable professional activities (EPAs), essential tasks of a discipline, were extracted for residents in postgraduate years 1–2, including completion rates, time to completion, assessment type (direct observation vs case review), entrustment rating, and faculty comments. We excluded data from postgraduate years 3 and above, as they lacked comparative EPA data for 2019. Ethical approval was obtained from the University of Toronto. Data were analysed using descriptive statistics, with continuous variables summarised using means and standard deviations, and discrete variables using proportions. Significant differences between years were examined using two-sample t-tests for continuous variables and χ2 analyses for discrete variables (α=0.05).

In 2019, 32 residents requested 350 EPA assessments, whereas 36 residents requested 276 assessments in 2020 (Table 1 ). Whilst the number of assessments requested per resident declined from 2019 to 2020 (P=0.046), the number of assessments completed per resident did not change (P=0.069) significantly. The number of assessments completed in the moment, as opposed to after the learning encounter, decreased by 17% from 2019 to 2020 (P<0.001). However, the mean time to assessment completion for those completed after the learning encounter also decreased by nearly 2 days from 2019 to 2020 (P=0.013). Whilst there was no change in the assessment type (direct observation vs case review), there was a reduction in the proportion of anaesthesia vs acute care EPAs assessed (2019: 78.7% anaesthesia; 2020: 66.9%; P=0.003). There was no difference in the overall number of assessments rated as ‘entrusted’ between years (P=0.56), nor in the overall entrustment categories assigned (P=0.30). The number of responses to open-ended questions seeking resident strengths and weaknesses was unchanged between 2019 and 2020 (P=0.072). Similarly, the word counts for these responses were unchanged (P=0.10).

Table 1

Overview of assessment data. Data presented as n (%), or mean [SD].

Variable	2019	2020	P-value
Number of assessments requested, n (%)	350	276	—
Postgraduate year 1	244 (69.7)	151 (54.7)
Postgraduate year 2	106 (30.3)	125 (45.3)
Number of assessments completed, n (%)	301 (86.0)	242 (87.7)	—
Postgraduate year 1	212 (86.9)	141 (93.4)
Postgraduate year 2	89 (84.0)	101 (80.8)
Number of assessments requested per resident	10.94 [8.11]	7.67 [4.93]	0.046
Number of assessments completed per resident	9.41 [7.21]	6.72 [4.61]	0.069
Number of faculty completing assessments, n (assessments/faculty)	147 (2.05 [1.72] assessments/faculty)	131 (1.85 [1.36] assessments/faculty)	0.29
Assessment type, n (%)			0.77
Direct observation	196/301 (65.1)	172/242 (71.1)
Case review	74/301 (24.6)	70/242 (28.9)
Not specified	31/301 (10.3)	0/242 (0)
Timing of the assessment, n (%)			<0.001
In the moment	214/301 (71.1)	131/242 (54.1)
After the learning encounter	87/301 (28.9)	111/242 (45.9)
Mean time to completion (for assessments completed after the learning encounter)	4.27 [6.81] days	2.34 [3.76] days	0.013

The COVID-19 pandemic has led to considerably decreased anaesthesia clinical volumes (65–80% decrease across educational sites based on data from a survey of anaesthesia department chiefs across academic sites in Toronto, ON, Canada) and, based on our data, has negatively impacted residents' requests for assessments and the variety of workplace-based assessment data obtained, with fewer anaesthesia and more acute care EPAs being completed. A decrease in one to two EPA assessments requested per month, particularly if primarily anaesthesia based, could have an educational impact should the pandemic period be prolonged. However, despite fewer assessment requests and less EPAs completed ‘in the moment’ of the learning activity, the number of assessments completed per residents remained stable and the time to completion improved. This is important, as studies have shown that prompt documentation of clinical performance improves the accuracy and effectiveness of feedback. Additionally, despite the decreased anaesthesia clinical exposure and correspondingly decreased assessment requests, our data suggest that the qualitative feedback provided did not deteriorate. This was shown by the consistency of written comments pertaining to strengths and weaknesses in the EPAs completed. Formative assessments are more meaningful to residents when they provide rich narrative feedback.

The results of this study highlight the importance for learners and residency programmes to monitor closely formative assessment data. For residents, a learner performance dashboard that analyses current assessment data in a visual format can be a powerful metacognitive tool, as it makes them aware of their performance and behaviour, supporting critical reflection. Assessment data can also serve to shape learning behaviours and encourage residents to direct their own learning and develop goal-directed learning actions, which can be collaboratively supported by an academic coach or advisor, for a successful educational outcome. In a disruptive environment, longitudinal coaching relationships can stimulate reflective dialogue and help residents generate personalised learning plans. For programmes, an assessment dashboard can facilitate identification of missing evidence of competence and identify clinical gaps and changing trends in learning curves, which can be targeted with individualised rotation or curriculum changes as restrictions ease.

In addition to longitudinal coaching, this study draws attention to the need for programmes to adapt effectively and efficiently residents' day-to-day curricula to maximise their learning and their ability to acquire necessary competencies. Whilst some training experiences could be replaced with simulation sessions during the pandemic, workplace-based assessment requires authentic clinical scenarios. Approaches to assessments could be more intentional and clinical experiences purposefully organised to facilitate evidence of achievement of competencies. Many competency-based programmes, including ours, use assessment tools that quantify a faculty member's entrustment of the resident in patient-care-related tasks. Because meaningful supervisor–trainee relationships have been shown to increase the likelihood of entrustment, strategic pairing of residents and faculty could potentially enable determination of entrustment over a shorter time period.

The COVID-19 pandemic has impacted anaesthesia residents' clinical experience and their workplace-based assessments intended to support learning. As such, programmes and residents will have to be more deliberate to create a bespoke learner experience.

Funding

.

Declarations of interest

The authors declare that they have no conflicts of interest.