Simulation-based Assessment to Measure Proficiency in Mechanical Ventilation among Residents.

Background: Mechanical ventilation (MV) skills are essential for clinicians caring for critically ill patients, yet few training programs use structured curricula and appropriate assessments. Objective structured clinical exams (OSCEs) have been used to assess clinical competency in many areas, but there are no OSCE models focused on MV. Objective: To develop and validate a simulation-based assessment (SBA) with an OSCE structure to assess baseline MV competence among residents and identify knowledge gaps.
Methods: We developed an SBA using a lung simulator and a mechanical ventilator, and an OSCE structure, with six clinical scenarios in MV. We included internal medicine residents at the beginning of their rotation in the respiratory intensive care unit (ICU) of a university-affiliated hospital. A subset of residents was also evaluated with a validated multiple-choice exam (MCE) at the beginning and at the end of the ICU rotation. Scores on both assessments were normalized to range from 0 to 10. We used Cronbach's α coefficient to assess reliability and Spearman correlation to estimate the correlation between the SBA and the MCE.
Results: We included 80 residents, of whom 42 also completed the MCE examinations. The final version of the SBA had 32 items, and the Cronbach's α coefficient was 0.72 (95% confidence interval [CI], 0.64-0.81). The average SBA score was 6.2 ± 1.3, and performance was variable across items, with 80% correctly adjusting initial ventilatory settings and only 12% correctly identifying asynchrony. The MCE had 24 questions, and the average score was 7.6 ± 2.4 at the beginning of the rotation and 8.2 ± 2.3 at the end of the rotation (increase of 0.6 points; 95% CI, 0.30-0.90; P < 0.001). There was moderate correlation between the SBA and the MCE (rho = 0.41; P = 0.002).
Conclusion: We developed and validated an objective structured assessment on MV using a pulmonary simulator and a mechanical ventilator addressing the main competencies in MV. The performance of residents in the SBA at the beginning of an ICU rotation was lower than the performance in MCE, highlighting the need for greater emphasis on practical skills in MV during residency.

Mechanical ventilation (MV) is a lifesaving intervention, and several randomized trials have shown that specific strategies can reduce mortality and complications, decrease the duration of MV and intensive care unit (ICU) stay, and reduce costs (1–5). However, it is associated with complications and high mortality rates (6–10), and its implementation requires complex integration of knowledge and critical thinking (11).

Management of MV is a core competency in critical care training and should be part of the undergraduate and internal medicine residency curriculum (12). In many countries, including the United States (12–13) and Brazil (14), only a minority of patients under mechanical ventilation are managed by intensivists. More recently, the coronavirus disease (COVID-19) pandemic has highlighted the importance of teaching basic MV skills for most medical specialties early in medical training. Yet, few medical training programs use a structured curriculum in MV (13), and there is considerable variation in curricula among training programs (11). As a result, trainees report low confidence in managing patients on MV (14–16). In addition, assessing learning has always been a challenge. Written examinations are commonly used and can be useful for verifying theoretical knowledge, but they do not assess cognitive and psychomotor skills, nor can they predict whether trainees are able to implement such skills at the bedside (17). Practical examinations, on the other hand, have proved to be superior in assessing competence of complex skills, especially when simulations are used (18–21). Objective structured clinical exams (OSCEs) are standardized, competency-focused practical examinations designed to test clinical skill performance and assess competence using structured clinical scenarios (22). OSCEs have been used for training and assessment of competence in many areas of critical care medicine, such as invasive procedures, advanced cardiac life support, professionalism, and data interpretation, but there are no OSCE models focused on MV. There is a need for residency and critical care fellowship programs to implement competency-based curricula in MV and properly assess competency and skills as trainees advance in their programs.

Therefore, we developed and validated a simulation-based assessment (SBA) with an OSCE structure to assess baseline MV competency among internal medicine residents and identify the most important knowledge gaps. Secondary aims were to compare residents’ performance in the SBA with their performance on the rotation’s multiple-choice standard assessment at baseline and compare residents’ performance in the multiple-choice assessment at the beginning and at the end of the ICU rotation. Some of these findings have been previously reported in the form of an abstract (23).

Methods

Study Design and Setting

The study was conducted at the MV laboratory and respiratory ICU of an academic hospital affiliated with the University of Sao Paulo medical school between September 2017 and December 2018. The project was submitted and approved by the Research Ethics Committee (CAAE 53091516600000068), and all participants signed informed consent.

The inclusion criterion was to be a second-year resident of internal medicine rotating in the respiratory ICU during the study period; exclusion criteria were unavailability of the resident on the dates stipulated for data collection or refusal to participate and to sign informed consent.

In Brazil, a 2-year internal medicine residency is a prerequisite for several subspecialties, including critical care fellowship (up until 2022, when critical care fellowship became direct access). At the University of Sao Paulo, there are 60 internal medicine residents each year, starting in March, organized in 12 groups of 5 residents, and 12 rotations lasting approximately 4 weeks. In Year 1, there is one medical ICU rotation, and in Year 2 there are two ICU rotations: the respiratory ICU and the medical ICU. For all but two groups, the respiratory ICU rotation precedes the medical ICU rotation. Residents are assessed in each rotation with a combination of objective and subjective assessments to generate a rotation final grade, with a minimum passing grade of 7, on a scale of 0 to 10.

The respiratory ICU rotation was 4 weeks long, and residents spent 60 hours weekly in the ICU, including night and weekend coverage. Residents were responsible for caring for patients under the supervision of two second-year pulmonary medicine fellows rotating in the ICU and attending physicians. Teaching opportunities included four didactic lectures, discussions of each case at the bedside during morning rounds, and bedside discussions of MV adjustments according to patient needs. Didactic lectures lasted between 50 and 60 minutes, were mandatory, and were given by one of two senior pulmonary–critical care fellows. The lectures focused on ventilatory modes, acute respiratory distress syndrome (ARDS), MV for patients with obstructive lung disease, and MV liberation, with standardized scripts. The curriculum did not include simulation.

Development and Validation of the SBA

The scenarios and questions were developed in accordance with a competency list, shown in Table 1 (24). The first version had 36 questions, organized in six clinical scenarios following the format of an OSCE. Some of the items were open-ended conceptual questions addressing the most likely diagnosis, and some required that participants adjust ventilatory parameters in the ventilator. We used a modified Delphi system to assess the content validity of the SBA (25). A first version of the assessment was sent individually to six experts—three physicians and three respiratory therapists—from different institutions across Brazil, with >10 years of experience in MV. The experts had access to the scenarios, questions, programming of the simulator, initial settings on the ventilator for each scenario, and a rubric with correct answers. They reviewed the six proposed scenarios in the SBA with respect to clarity of the clinical presentation and tasks that examinees were being asked to perform, the importance of the competence measured by each item, and content validity of assessment (26). Feedback from the experts in each round was used to modify the scenarios, and the modified version was sent for another round of feedback. After two additional rounds of review, a satisfactory degree of agreement was reached (set as >90%), resulting in the final version of the SBA with 32 items divided into six scenarios. Figure E1 in the data supplement shows scenario 3 as it was presented to participants, and Figure E2 shows the examiner’s OSCE checklist for that same scenario.

Table 1.

Competency list of skills measured with the simulation-based assessment

Item	Competency Assessed
Scenario 1: pulmonary mechanics
1	Recognize volume-controlled mode by observing ventilator waveforms
2	State the formula for respiratory system compliance
3	Demonstrate how to measure respiratory system compliance
4	Demonstrate how to measure respiratory system resistance
Scenario 2: postoperative adjustment of basic modes of MV
5	Recognize pressure-controlled mode by observing ventilator waveforms
6	List the appropriate Vt by IBW according to the clinical context
7	Describe appropriate initial settings to start MV
8	Set inspiratory pressure in PCV mode to deliver desired Vt
9	Adjust FiO2 and PEEP according to the clinical context
10	Describe and monitor physiological goals of MV
Scenario 3: patients with obstructive disease
11	Interpret an arterial blood gas
12	Describe the ventilatory approach for patients with hypercapnia
13	Set initial parameters for a patient with obstructive lung disease
14	Recognize auto-PEEP by observing ventilator waveforms
15	Demonstrate how to measure auto-PEEP
16	Adjust MV parameters to correct/minimize auto-PEEP
17	List benefits and disadvantages of applied PEEP
Scenario 4: MV in patients with COPD and asynchrony
18	Recognize PSV mode by observing ventilator waveforms
19	Recognize the occurrence of patient–ventilator asynchrony
20	List strategies to minimize patient–ventilator asynchrony
21	Adjust ventilator parameters to correct patient–ventilator asynchrony
Scenario 5: MV in patients with ARDS
22	Diagnose ARDS
23	Identify particularities of ventilatory management in patients with ARDS
24	Demonstrate PEEP adjustment in ARDS
25	Adjust respiratory rate in the context of low Vt protective ventilation
26	Adjust Vt to offer protective MV in ARDS
27	Describe management of refractory hypoxemia
Scenario 6: liberation from MV
28	Describe the liberation strategy for patients on MV
29	Set the ventilator to perform a spontaneous breathing test
30	Identify signs and symptoms that indicate that a patient is ready for extubation
31	Demonstrate how to initiate NIV in patients with hypercapnic respiratory failure
32	Demonstrate how to initiate NIV in patients with cardiogenic pulmonary edema

Definition of abbreviations: ARDS = acute respiratory distress syndrome; COPD = chronic obstructive pulmonary disease; FiO = fraction of inspired oxygen; IBW = ideal body weight; MV = mechanical ventilation; NIV = noninvasive ventilation; PCV = pressure-controlled ventilation; PEEP = positive end-expiratory pressure; PSV = pressure support ventilation; Vt = tidal volume.

We pilot tested the SBA in seven intensivists and five second-year residents (not the same who participated in the study) to evaluate the clarity of the questions and estimate the time required to complete each scenario. The mean score among intensivists was 8.2 ± 0.7, whereas among residents it was 6.7 ± 0.7. A duration of 8 minutes for each scenario and 50 minutes to complete the evaluation was set based on the average time needed by participants during this pilot testing.

Additional steps for validation of the SBA included relationship to other variable validity, by comparing the performance in the SBA with another assessment (described below), and response process validity, by developing standardized instructions for the examinees and standardized scoring OSCE checklists (Figure E2). Two respiratory therapists were trained to apply the SBA. They followed a predefined printed script and performed three of the pilot examinations together to standardize the methodology. Interrater agreement was not calculated because of the small number of paired observations.

Data Collection

Residents completed the SBA on the first day of the respiratory ICU rotation. The experimental setup consisted of a mechanical ventilator (Servo-I, Maquet Getinge Group) coupled to a pulmonary simulator (ASL 5000, IngMar Medical) (Figure 1). The ventilator is the same used in our respiratory ICU, so residents were familiar with the interface. Following an OSCE model, participants were assessed individually at the MV lab. Each scenario had an explicit execution time and objectives, presented to the participant at the beginning of the evaluation, in a predetermined sequence.

Figure 1.

Experimental setup of the simulation-based assessment. 1) Intensive care unit ventilator was connected to the 2) ASL5000 simulator, which is controlled by 3) a computer.

In addition, a subgroup of residents starting their second year of residency in March 2018 completed a multiple-choice exam (MCE) adapted from a validated assessment (27). The MCE was introduced in the beginning of a new academic year, substituting a previous written evaluation routinely used to provide an objective assessment of learning. These evaluations were not developed as part of this research project; they were used for routine learning assessment. However, our project and informed consent included the use of assessments performed during the rotation as a comparison with the performance in the SBA, as an additional step to validate the SBA (relationship with other variables validation) (26). Because the previous evaluation had different items and scoring, we report data for those residents who completed the MCE applied as of March 2018. The MCE consisted of 24 multiple-choice questions, applied using a web-based platform (Research Electronic Data Capture) (28) at the beginning and at the end of the rotation. Table 2 shows the competency measured by each item in the MCE and its correspondence with the SBA items. Because the MCE was not developed for this research project, not all items had a correspondence, and even the ones that measured the same competence might approach it differently.

Table 2.

Competency list of skills measured with the multiple-choice examination

Item	Competency Assessed	Corresponding Item in SBA
1	Demonstrate ability to interpret arterial blood gases correctly	11
2	Indication of NIV in acute respiratory failure in COPD	31
3	Identify auto-PEEP	14
4	Ventilation strategy with bag-valve mask	No correspondence
5	Describe ventilatory management approaches to the patient with hypercapnia	12
6	Recognize patterns of resistance and compliance of the respiratory system and their relationships with prevalent diseases	3, 4
7	Recognize volume-controlled mode by observing ventilator waveforms	1
8	List risk factors for difficult airway	No correspondence
9	Diagnose ARDS	22
10	Describe pathophysiological mechanisms of hypoxemia in ARDS	No correspondence
11	Identify particularities of ventilatory management in patients with ARDS	23
12	List strategies for securing the airway in a patient who is difficult to intubate	No correspondence
13	Describe the advantages and risks of rescue measures for refractory hypoxemia and their indications	27
14	Apply concept of compliance and its relationship with tidal volume during mechanical ventilation	2
15	Identify that a patient is ready for an SBT	29
16	Adequately indicate the need for ventilatory support in patients with acute respiratory failure due to neuromuscular disease	No correspondence
17	Identify the impact of excess secretions on airway pressure	No correspondence
18	Describe volume-control ventilation, in terms of the trigger, limit, and cycle variables	1
19	Identify patients with high risk for extubation failure	30
20	Use of NIV in patients with cardiogenic pulmonary edema	32
21	Identify signs and symptoms of pulmonary embolism in patients under mechanical ventilation	No correspondence
22	Interpret airway waveforms—respiratory system resistance	4
23	Recognize the occurrence of patient–ventilator asynchrony	19
24	Adjust ventilator parameters to correct patient–ventilator asynchrony	21

Definition of abbreviations: ARDS = acute respiratory distress syndrome; auto-PEEP = auto–positive end-expiratory pressure; COPD = chronic obstructive pulmonary disease; NIV = noninvasive ventilation; SBA = simulation-based assessment; SBT = spontaneous breathing trial.

Statistical Analysis

Each of the 32 items in the SBA was rated as done correctly or incorrectly and assigned 1 or 0 points, respectively. A total score for the SBA was calculated as the sum of the points divided by 32 and multiplied by 10, resulting in a standardized score ranging from 0 to 10. The total score on the MCE was calculated as the number of correct answers divided by 24 and multiplied by 10, with a standardized score ranging from 0 to 10.

There were no articles in the literature to guide us in the sample calculation; therefore, we used a convenience sample of 80 participants.

Continuous variables were presented as mean and standard deviation or median and interquartile range. We used the Shapiro-Wilk test to assess the normality of these continuous variables. Categorical variables were presented as proportions and 95% confidence intervals (95% CIs).

To assess the reliability of the SBA, we used Cronbach’s α coefficient, which measures the internal consistency of items in a test, survey, or assessment. It calculates the correlation among all the items, in every combination, and is a common method to assess reliability (26). We considered values >0.7 as reliable (29).

We used paired t tests to compare the residents’ performance in the MCE at the beginning (pretest) and at the end of the internship (posttest). The Spearman test was used to test the correlation between the SBA and MCE. All hypothesis tests are two-tailed with a significance of 0.05 and performed using the R software (R Core Team, 2016).

Results

A total of 81 participants were eligible for the study between September 2017, and December 2018 (Figure 2). One resident did not participate because he was absent on the date of the evaluation. Thus, 80 participants were included in the analysis of the SBA performance. Of these, 49 completed the MCE pretest and 42 completed both the pretest and posttest. The mean age of participants was 27 ± 3 years, and 74% were male.

Figure 2.

Study participant flow. Flow of potentially eligible participants in the study and final numbers included and analyzed. MCE = multiple-choice exam; SBA = simulation-based assessment.

The reliability of the SBA was high, with a Cronbach’s α coefficient of 0.72 (95% CI, 0.64–0.81).

Mean standardized score in the SBA was 6.2 ± 1.3. Sixty-eight (85%) participants scored at least five (>50% answers correct), and 20 (25%) scored at least seven (>70% answers correct). Table 3 shows the percentage of correct answers for each item. Performance varied considerably across items. Some items had a ceiling effect, with most participants scoring one point, such as recognizing the mode pressure support or diagnosing ARDS, whereas others had a floor effect, with very few participants scoring, such as calculating respiratory system resistance (Table 3). The item assessing use of protective tidal volume (Vt) for ARDS had 67% correct answers. The appropriate indications for the use of noninvasive ventilation after extubation was correctly answered by 62% of the participants, and only 12% of participants recognized the presence of patient–ventilator asynchrony on the ventilator screen (Table 3).

Table 3.

Percentage of participants scoring correctly for each item in the simulation-based assessment

Item	Competency Assessed	Correctly Done (%)
18	Recognize PSV mode by observing ventilator waveforms	99
22	Diagnose ARDS	99
23	Identify particularities of ventilatory management in patients with ARDS	99
10	Describe and monitor physiological goals of mechanical ventilation	95
5	Recognize pressure-controlled mode by observing ventilator waveforms	87
11	Interpret an arterial blood gas analysis	87
1	Recognize volume-controlled mode by observing ventilator waveforms	84
6	List the appropriate Vt by IBW according to the clinical context	82
7	Describe appropriate initial settings to start MV	80
32	Demonstrate how to initiate NIV in cardiogenic pulmonary edema	79
29	Set the ventilator to perform a spontaneous breathing test	74
28	Describe the liberation strategy for patients on mechanical ventilation	72
24	Demonstrate PEEP adjustment in ARDS	71
8	Set inspiratory pressure in PCV mode to deliver desired Vt	70
9	Adjust FiO2 and PEEP according to the clinical context	69
12	Describe the ventilatory approach for patients with hypercapnia	69
14	Recognize auto-PEEP by observing ventilator waveforms	69
26	Adjust Vt to offer protective MV in ARDS	67
31	Demonstrate how to initiate NIV in hypercapnic respiratory failure	65
30	Identify that a patient is ready for extubation	62
25	Adjust respiratory rate in the context of low Vt protective ventilation	54
2	State the formula for respiratory system compliance	51
15	Demonstrate how to measure auto-PEEP	51
17	List benefits and disadvantages of applied PEEP	46
16	Adjust MV parameters to correct/minimize auto-PEEP	41
20	List strategies to minimize patient–ventilator asynchrony	40
3	Demonstrate how to measure respiratory system compliance	31
21	Adjust ventilator parameters to correct patient–ventilator asynchrony	31
27	Describe management of refractory hypoxemia	24
13	Set initial parameters for a patient with obstructive lung disease	12
19	Recognize the occurrence of patient–ventilator asynchrony	12
4	Demonstrate how to measure respiratory system resistance	9

Definition of abbreviations: ARDS = acute respiratory distress syndrome; FiO = fraction of inspired oxygen; IBW = ideal body weight; MV = mechanical ventilation; NIV = noninvasive ventilation; PCV = pressure-controlled ventilation; PEEP = positive end-expiratory pressure; PSV = pressure support ventilation; Vt = tidal volume.

Mean standardized score on the MCE was 7.6 ± 2.4 in the pretest and 8.2 ± 2.3 in the posttest. The mean change was 0.6 (95% CI, 0.30–0.90; P < 0.001).

The SBA score and the MCE pretest score showed a positive correlation of moderate intensity and statistically significant (rho = 0.41; P = 0.002), with the SBA mean score being lower than the MCE score.

Discussion

In this study, we developed a SBA with an OSCE structure addressing the main competencies in MV and applied it to 80 internal medicine residents at the beginning of their rotation in the respiratory ICU. We found that the reliability of the assessment was high and that the mean score of residents at the beginning of the rotation was 6.2 ± 1.3 on a scale of 0 to 10. We also found that performance on a multiple-choice assessment was higher than that on the simulation assessment and increased significantly at the end of the rotation.

The reliability of the practical assessment was good, with a Cronbach’s α coefficient of 0.72. The implementation of the SBA for all the groups of residents was feasible, and examiners usually spent one morning in the MV lab to assess a group of five residents.

The mean SBA score was 6.2 ± 1.3, or 62% correct answers, but only 25% of participants scored 7 or higher (>70% correct answers) and 15% scored less than 5 (>50% incorrect answers). However, the examination was applied at the beginning, not the end, of rotation; therefore, we expected that there would be knowledge gaps, and mastery was not expected. We chose to apply the SBA at the beginning of rotation because one of our aims was to identify the most common knowledge gaps, to address them during the rotation. Our finding is similar to that of a recent study (30) in which investigators used simulation to assess baseline skills, as a teaching tool in the form of deliberate practice, and to assess end-of-rotation skills. They found that at the beginning of ICU rotation the median score was 52%, and it increased to 86% at the end of rotation. We did not repeat the OSCE simulation in our study because we were concerned that residents would be able to recall the clinical scenarios, overestimating the scores. However, using the SBA as a pre- and posttest is feasible, and we believe that the SBA can be used in the future for several purposes: 1) to identify gaps for individual learners and address them during a rotation, providing a personalized approach to teaching; 2) to identify common gaps in larger groups of learners and address them with appropriate curriculum adjustments; and 3) to assess learners at the end of a rotation or during their progression in a training program.

We found that performance on important clinical topics was variable. There was a high percentage of error for the item that evaluates recognition of patient–ventilator asynchrony, in agreement with previous studies emphasizing the difficulty health professionals have in recognizing and managing asynchrony (31, 32). Considering that asynchrony is a common phenomenon and is associated with extubation failure (33) and mortality (34, 35), this finding underscores the need for intentional training (32). The item requiring participants to measure respiratory system resistance also had a high percentage of error, possibly reflecting a lack in familiarity with performing such measurements at the bedside.

On the other hand, the item that required Vt setting for protective ventilation in ARDS had 67% correct answers, slightly higher than the result found in a previous study (13). This is an important finding, considering that adherence to protective ventilation remains low (36, 37) despite overwhelming evidence of its impact on survival (6, 38). Performance on the item evaluating noninvasive MV was also moderate, with 65% correct answers, similar to what was previously found (13) and higher than self-perceived competence in a survey of recently graduated physicians (14).

The scores on the MCE had a small but significant increase between the pre- and posttest, which suggests that residents improved their knowledge about MV during the rotation, as previously shown (27). However, the increment was small, and it is difficult to speculate the relationship between performance on an MCE and independent patient care. In addition, the contribution of each activity, such as discussion at the bedside, didactic lectures, and the morning rounds discussion of cases with multidisciplinary team, is unknown.

Scores were higher in the MCE than in the SBA, which is in agreement with previous studies that suggest that learners tend to have higher scores in written exams than in simulations and practical tests (17). This discrepancy in performance may reflect that learners may have adequate theoretical knowledge but are less able to apply such knowledge during practical exams that simulate clinical situations they might encounter in clinical practice.

The correlation between the scores of the MCE and SBA was positive and of moderate intensity. Although written assessments such as the MCE can be used to assess learning, SBAs require that learners apply knowledge when they operate a mechanical ventilator in a scenario that mimics real clinical scenarios, providing a more complete assessment of competence. Previous studies showed that simulation in MV can provide lasting knowledge and prepare residents to safely manage ventilated patients (30, 39, 40), and SBAs have been successfully used to identify learning gaps and provide remediation (30).

Strengths and Limitations

Our study has several limitations. First, the study was performed in a single center and the residents’ performance may therefore represent our teaching strategy and our resident population. In addition, the choice of a convenience sample may have included participants with heterogeneous characteristics, the timing of the assessment was not the same for all participants, and residents rotating in the ICU later in the academic year could have been exposed to other rotations where they managed ventilated patients, which may have interfered with the result. Second, we did not repeat the SBA at the end of rotation, so it could not be used to assess improvements related to the ICU rotation in our study. We were further limited by the fact that not all of the participants completed the MCE, as it was implemented later in the study; because our main outcome was the SBA, the impact in our main results was not significant. Third, the SBA assessment has not been externally validated; however, we used a modified Delphi method, standardized scripts, and pilot testing to improve validity. Moreover, SBA may not be easily generalizable to other settings, as it depends on the availability of the simulator and a ventilator and requires time availability from both the learner and raters. However, simulation in MV has been increasingly used as it becomes part of the teaching tools in critical care (30, 40). Fourth, we did not formally survey participants regarding their experience and perceived advantages and disadvantages of the SBA, compared with traditional evaluations; in addition, we did not provide feedback to participants because the assessment was not validated yet, and we wanted to prevent residents from discussing the scenarios with one another on the day of the evaluation. Finally, we did not assess knowledge and skill retention; therefore, it is unclear if the improvement in MCE scores after the ICU rotation translates into sustained knowledge and skills or improved patient outcomes.

Our study also has strengths. The SBA was standardized and followed OSCE guidelines; we had buy-in from attending physicians and residents, and only one eligible resident did not participate; there was moderate correlation between traditional and simulation assessment; using simulation-based evaluations for baseline assessment provided a safe environment, so that trainee competence could be assessed in a realistic clinical scenario without putting patients at risk; and the SBA can be used to gauge baseline knowledge on several important competencies in MV, identify gaps, and provide a personalized guide for learner progression during ICU rotations.

Conclusions

We developed and validated a reliable SBA that addressed the main competencies in MV. The performance of residents in the SBA at the beginning of an ICU rotation was worse than the performance in a multiple-choice test. Implementation of an objective measurement of skills on MV was feasible and can be used to provide a safe environment for assessing clinical competency in MV, in realistic clinical scenarios, without putting patients at risk. These findings highlight the need for greater emphasis on practical skills in MV during internal medicine residency and for assessments that measure clinical skill performance and competence.