Multi-source feedback following simulated resuscitation scenarios: a qualitative study.

Background: The direct observation and assessment of learners' resuscitation skills by an attending physician is challenging due to the unpredictable and time-sensitive nature of these events. Multisource feedback (MSF) may address this challenge and improve the quality of assessments provided to learners. We aimed to describe the similarities and differences in the assessment rationale of attending physicians, registered nurses, and resident peers in the context of a simulation-based resuscitation curriculum.
Methods: We conducted a qualitative content analysis of narrative MSF of medical residents in their first postgraduate year of training who were participating in a simulation-based resuscitation course at two Canadian institutions. Assessments included an entrustment score and narrative comments from attending physicians, registered nurses, and resident peers in addition to self-assessment. Narrative comments were transcribed and analyzed thematically using a constant comparative method.
Results: All 87 residents (100%) participating in the 2017-2018 course provided consent. A total of 223 assessments were included in our analysis. Four themes emerged from the narrative data: 1) Communication, 2) Leadership, 3) Demeanor, and 4) Medical Expert. Relative to other assessor groups, feedback from nurses focused on patient-centred care and communication while attending physicians focused on the medical expert theme. Peer feedback was the most positive. Self-assessments included comments within each of the four themes. Conclusions: In the context of a simulation-based resuscitation curriculum, MSF provided learners with different perspectives in their narrative assessment rationale and may offer a more holistic assessment of resuscitation skills within a competency-based medical education (CBME) program of assessment.

Introduction

Competency-based medical education (CBME) requires an increased focus on direct observation and formative feedback to help learners advance their learning.[1] Canadian specialty postgraduate medical residency programs implemented a CBME assessment program known as Competence By Design (CBD). CBD is composed of entrustable professional activities (EPAs) that are predetermined centrally by each program’s national specialty committee.[2] Most CBD programs require one or more EPAs focused on the identification and resuscitation of acutely unwell patients.

The assessment of resuscitation focused EPAs is usually performed by attending physicians. However, the observation of resuscitations in the clinical environment is difficult for attending physicians due to their unpredictable and time-sensitive nature. Multisource feedback (MSF) may address this challenge and complement a competency-based program of assessment as it is workplace-based and involves the direct observation of learners. Furthermore, it is an effective method to assess broad competencies that are relevant to resuscitation such as teamwork, communication, and interpersonal skills [3]-[5]. MSF has the potential to leverage the various perspectives of an interprofessional team and may provide a more holistic assessment. It is unclear if and how the focus of assessment varies between members of a resuscitation team (e.g. registered nurses, resident peers, attending physicians).

The objective of this study was to describe the similarities and differences in the assessment rationale of attending physicians, registered nurses, and resident peers in the context of a longitudinal, interdisciplinary simulation-based resuscitation curriculum. This will further our understanding of how MSF may contribute to the competency-based assessment of resuscitation.

Methods

We conducted a qualitative content analysis study of narrative multisource feedback of medical residents in their first postgraduate year of training participating in a simulation-based resuscitation course.

Setting and participants

This study took place at Queen’s University (Kingston, Ontario) and the University of Saskatchewan (Saskatoon, Saskatchewan) in Canada. Postgraduate medical residents in their first year of training were invited to participate in the Nightmares course[6] during the 2017-2018 academic year and were recruited for this study. The Nightmares course faculty included attending physicians and registered nurses with an interest in resuscitation medicine and simulation-based teaching. Participating nurses had a background in critical care or emergency medicine and at least 10 hours of simulation experience acting as a confederate. Attending physicians were all involved in simulation-based teaching within their respective training programs (emergency medicine, intensive care medicine, anesthesiology, and internal medicine). The institutional research ethics boards of Queen’s University and the University of Saskatchewan approved this study and all participants provided informed consent.

Course design

The Nightmares course is a simulation-based curriculum created to prepare first-year residents to respond to acutely unwell patients during independent on-call shifts.[6],[7] Specifically, it is designed to teach and assess an EPA common to all training programs: “recognizes an acutely unwell patient, calls for appropriate help, and initiates a basic assessment and management plan.” The course took place at both institutions between August and December 2017. Over this period, each resident participated in four simulation sessions, once every four weeks. Each session at the Queen’s University site was 90-minutes and involved three scenarios. Each session at the University of Saskatchewan site was 180-minutes and involved four scenarios. Scenarios were based on the most common calls to the Kingston General Hospital’s Rapid Assessment of Critical Events team (Appendix A). Four to six residents attended each session and took turns leading a scenario, so that over the duration of the course each resident participated in 12 (at the Queen’s University site) or 16 (at the University of Saskatchewan site) scenarios and was the leader for two to four of them. Residents did not know the case details in advance of the session. One or two registered nurses participated in each scenario as confederates and facilitated scenario flow. One or two attending physicians directly observed the scenario and led a debrief. The Kingston site and one of the Saskatchewan sites (Saskatoon) used the high-fidelity SimMan 3G manikin (Laerdal, Toronto, Canada), the other Saskatchewan site (Regina) used the Gaumard manikin (Hal S3000 Tetherless Patient Simulator, Gaumard, Florida, USA).

Data collection

Following each scenario, the leader completed a self-assessment and was assessed by fellow residents, nurse(s), and attending physician(s). Assessment forms were based on the Queen’s Simulation Assessment Tool.[8] All assessor groups (residents, nurses, attending physicians) completed the same assessment form, and only the wording of the ‘anchor’ statement at the top of the form was specific to each group. An example of the attending physician and self-assessment forms can be found in appendices B and C. All forms included a five-point entrustment score (the O-SCORE)[9],[10] and space for participants to explain their rating and provide written narrative assessment rationale. All participants were aware that assessment data was being used for research purposes, but no further detail was provided. A debrief occurred after all assessment forms were submitted.

Data analyses

We transcribed the narrative assessment rationale from each assessment form and uploaded them into NVivo (version 12) for analysis. We used an inductive analysis approach to code the content thematically.[11],[12] At the beginning of the coding process, two researchers coded a diverse sample of the data independently and then met to compare coding to ensure shared meaning and address inter-coder reliability. The diverse sample consisted of transcripts from both institutions with varying lengths of narrative feedback content. These researchers were not physicians or assessors but were well versed in the educational program and assessment process of the Nightmares course. The comparison of coding consisted of the two researchers reviewing the coded transcripts line by line. They discussed the meaning of each coded segment and reported their assigned code. During instances where the meaning of the code was the same, but the name of the code differed, the two researchers discussed until they agreed upon the same code name. This process was repeated across the sample of transcripts. During this process, one researcher recorded the number of times that they had the same segments coded with the same meaning and the number of times that they disagreed and had coded them differently. They agreed on 95% of the codes. The remaining 5% of codes were discussed until consensus was reached. This preliminary coding process generated a consensus-built codebook that was used by the same researchers to code the remainder of the content with the focus on identifying patterns across data.

Each transcript was coded in full, by reviewing each sentence, line-by-line. Each sentence or segment of text was assigned a code from the consensus-built codebook described above. The individual ‘code’ was the smallest unit of analysis. The researchers made annotations throughout the coding process to capture questions, repeated findings across the transcripts, connections between the codes, potential biases, and preliminary interpretations of the data. Once each transcript was coded once, the two researchers used the codebook and developed categories by grouping similar codes together. Once all codes had been organized into categories, the researchers discussed how categories could be grouped into themes to represent broader patterns across the content. This process was repeated until all categories were grouped into themes which were used as a framework for the constant comparative method.

We used a constant comparative method[13] consisting of open-coding to identify emerging patterns across assessor groups (registered nurses, peers, attending physicians, and self) and feedback categories (e.g. communication). The focus for this analysis was on identifying similarities and differences across assessor groups. The final themes were identified from common and frequent patterns across assessor groups.

The full research team then met to discuss the codes, and proposed themes and their practical implications. During this process, a few theme and category names were revised to better represent the data and align it with the clinical context. These changes occurred because preliminary theme and category names were proposed by the educational researchers who had coded the data but had no clinical expertise. This was an iterative process with meetings involving all team members in order to assist with the interpretation of the data and designed to address reflexivity and mitigate bias.[14],[15] All quotes were also tagged to indicate whether the content originated from self, nurse, peer, or attending physician assessment. Similarities and differences between the feedback provided on each type of assessment were explored descriptively.

Research team and reflexivity

The research team consisted of three Emergency Medicine physicians with interests and training in medical education (TC, AS, BT) and three health education researchers (HB, ND, RE) with experience in conducting mixed methods studies. All three of the health education researchers were external to the EM departments. The researcher leading the coding process maintained a journal to note her thoughts, potential biases, preliminary findings, and any other considerations related to this study. This journaling was conducted to maintain reflexivity.

Results

Across both institutions, the 2017 Nightmares course was offered to and attended by a total of 87 first-year postgraduate medical residents from 14 training programs (anatomic pathology, anesthesiology, diagnostic radiology, emergency medicine, general surgery, internal medicine, neurology, obstetrics and gynecology, ophthalmology, orthopedic surgery, physical medicine and rehabilitation, psychiatry, radiation oncology, urology). All 87 (100%) provided written consent to participate in this study. A total of 33 attending physicians and 11 nurses participated and all (100%) provided consent. A total of 223 narrative assessments were completed and included in our analyses.

Overall themes

Four themes were identified from the narrative assessment data across all assessor groups: 1) Communication, 2) Leadership, 3) Demeanor, and 4) Medical Expert. The first section of the results outlines these four themes across all participants with positive and constructive feedback embedded within each theme. Table 1 presents the overall themes and affiliated categories. The second section of the results outlines the similarities and differences in feedback from each assessor group.

Table 1

Emergent themes and categories

Theme	Categories
1. Communication	Interacting with patients and families Calling for help and resources Sharing mental models
2. Leadership	Awareness Delegation
3. Demeanor	Perception of comfort Perception of control
4. Medical Expert	Content knowledge Diagnosis and treatment

Theme 1: Communication

Communication was a common focus of feedback across all assessor groups and included three categories: interacting with patients and their families, calling for help and resources, and sharing mental models. See Table 2 for selected quotations within this theme.

Table 2

Selected quotations for Theme 1: communication

Category	Selected Quotations
Interacting with patient and families	“Interacted well with patient” (RN5, Institution1)“Communicated clearly to patient and wife” (RN2, Institution1)“Don't forget to check in with patient” (RN5, Institution2)“Great introduction to patient” (F13, Institution2)
Calling for help and resources	“Asked for help early and managed appropriately until help arrived” (Peer12, Institution1)“Did call appropriate resources” (RN1, Institution1)“Clear consult summaries” (Faculty6, Institution2).“If unsure, call consultant sooner” (Peer5693, Institution2)
Sharing mental models	“Solicited opinion from team members as needed” (Peer11, Institution1)“Clearly communicated thought process” (Peer10, Institution1)“Shared the mental model with colleagues” (F20, Institution2)“Good mental model discussions with team members” (N5, Institution2)

When interacting with patients and their families, narrative comments described the importance of clear communication between resident and patient (e.g., clearly introducing themselves or having a discussion prior to cardioversion). Comments within the second category, calling for help and resources, included consulting cardiology and clearly describing the scenario over the phone of a patient suffering a myocardial infarction, or directing requests to specific team members to ensure mobilization of resources (e.g., monitors, antibiotics). The third category underscored the importance of a shared mental model. Feedback focused on the extent to which resident leaders listened to other team members and solicited their opinions. Constructive comments within this theme emphasized the need to interact more with the patient, call for help earlier in the scenario, and verbalize their decisions to the team.

Theme 2: Leadership

This theme included the categories of awareness and delegation. Comments related to awareness referred to either situational awareness or self-awareness. Situational awareness was conceptualized as being conscious of the physical environment during the resuscitation or knowing when to pay attention (or not) to stimuli. For example, studying an electrocardiogram while a hemodynamically unstable patient with sepsis has no intravenous access. Self-awareness was referred to as being aware of one’s limitations as leader.

Comments related to the category of delegation focused on the extent to which the leader assigned tasks (e.g., establishing roles clearly and early). There was less constructive feedback within the theme of leadership compared to the others. When constructive feedback was provided it tended to focus on the need to be more assertive, delegate, and assign roles. See Table 3 for selected quotations within this theme.

Table 3

Selected quotations for Theme 2: Leadership

Category	Selected Quotations
Awareness	“Aware of his surroundings” (Faculty8, Institution2)“Knew when to seek help” (Faculty13, Institution2)“Able to prioritize appropriately” (RN1, Institution1)“Difficult to prioritize interventions with one nurse” (R35, Institution1)
Delegation	“Good job delegating so that you were available to lead the team” (Peer5248, Institution2)“Directed friend to do primary assessment” (Faculty7, Institution2)“Consider delegating responsibilities (e.g. calling for tests) to other team members” (Peer4030, Institution2)“Did a good job utilizing team and resources” (Peer53, Institution1)

Theme 3: Demeanor

Comments within this theme related to the assessor’s observation of the leader’s outward manner and subsequent attribution of a quality. Within the theme of demeanor there were two categories: perceptions of comfort and control. The perceived level of control was mentioned by all assessor groups and qualities were ascribed to a leader who appeared to be in control such as composure and confidence.

Constructive feedback within this theme largely related to uncertainty on the part of the leader. This was perceived as a lack of comfort and/or control. Qualities that were attributed to a perceived lack of control included looking flustered, overwhelmed, and unsure. See Table 4 for selected quotations within this theme.

Table 4

Selected quotations for Theme 3: Demeanor

Category	Selected Quotations
Perception of comfort	“Confident in my ability to stabilize patient” (R51, Institution1)“I am actually more confident with this case after the scenario than before” (R7368, Institution2)“Did attempt to very cautiously intervent with IV fluids for hypotension” (RN1, Institution1)“Communication: Too quiet”(Faculty2, Institution2)
Perception of control	“Confident and comfortable” (Peer5207, Institution2)“Resident appeared comfortable with assessment and asking for vitals/monitoring” (RN4, Institution1)“[name of resident] was calm and in control of room” (Peer7413, Institution2)“She was very calm throughout the procedure” (Peer24, Institution1)

Theme 4: Medical expert

The final theme identified in our analysis was the concept of medical expertise. Within this theme, there were two categories: content knowledge, and diagnosis and treatment. Comments relating to content knowledge focused on a specific topic such as cardiac arrythmias or medication doses. Positive feedback related to delivering appropriate medication, diagnostics, and interventions.

Constructive feedback focused on similar areas, suggesting reading around specific medications or interventions. The timing of the diagnostic actions and interventions were also of interest to assessors as they identified when a resident leader should have initiated these earlier in the scenario. See Table 5 for selected quotations within this theme.

Table 5

Selected quotations for Theme 4: Medical Expert

Category	Selected Quotations
Content knowledge	“Strong medical knowledge base” (Faculty6, Institution2)“Solid understanding of arrhythmia” (Faculty1, Institution1)“Recognize need for repeat epi” (Faculty15, Institution1)“In summary label patient as “Delirium” + “Septic shock” more clearly” (Faculty6, Institution2)
Diagnosis and treatment	“Unsure on how and when to use meds vs. shock patient” (RN2, Institution1)“Some delay intervening on abnormal vital signs and initiating treatment” (Faculty9, Institution2)“Remember initial doses of Atrovent and Ventolin” (Faculty6, Institution2)“Needed help from the [senior resident] with Ativan dose” (RN4, Institution2)

Similarities and differences in feedback from each assessor group

. Relative to other assessor groups, the feedback from nurses focused on patient-centred care and within-team communication. For example, nurses commented on the interaction between resident and patient, and the extent to which residents solicited opinions from their team members. For example, “Solicited opinion from team to ensure everyone was on the same page” (RN3, Institution2). Nurses focused on more nuanced behaviours such as making eye-contact and using names when addressing team members. Similar to other assessor groups, nurses were interested in the timeliness of seeking additional help. In relation to leadership, nurses commented on the prioritization of multiple orders, reminding the leader that the nursing skillset is also a resource that must be considered. Generally, nurses commented far less on the medical expert theme when compared to attending physicians. When nurses did provide feedback on this theme, it was focused on the extent to which the leader required prompting or help, such as, “Needed help from senior resident with ativan dose” (RN4, Institution2).

. Within the communication theme, peers provided feedback related to specific conversations such as discussing goals of care. For example, “There was no mention of goals of care as the case progressed” (Peer24, Institution1). Peers also focused on the promptness of calling for help and seeking additional resources. Within the leadership theme, peers identified the team as a resource and the organizational skills required to delegate in order to reduce the leader’s cognitive load. Peers also commented on the perceived level of control, ascribing a relaxed or calm demeanor to control of the scenario. During instances when peers provided feedback relating to the medical expert theme, they focused on the identification and labeling of diagnoses. Peers provided more positive narrative comments compared to nurses and attending physicians, and a similar amount compared to resident self-feedback.

Attending physician feedback was overwhelmingly focused on the medical expert theme, specifically on the category of diagnosis and treatment. For example, “If you have a patient with a STEMI, put the pads on early” (F13, Institution 2) or “Needed prompting for treatment of hyperkalemia and transfer to ICU. 2ml of 10% calcium gluconate, unsure how to pace” (F9, Institution 1). Attending physician feedback directed at communication skills primarily related to discussions between the leader and the patient, rather than family members. Attending physicians also commented on the extent to which residents verbalized their thinking and summarized it for their team members. When the leader appeared to be comfortable, attending physicians provided praise. Some attending physicians did suggest when additional confidence was needed: “You know your stuff but lack a bit of confidence!” (Faculty19, Institution2).

Generally, the self-assessment rationale was balanced, and residents reported positive aspects of their performance as well as areas for growth. Self-assessment was scattered across all four themes but did frequently comment on confidence. For example, “I definitely needed the cardiology fellow to guide me through and had no confidence in [my] management” (Resident7380, Institution2). Such comments were classified within the theme of demeanor given their relationship with this construct. The self-assessment narratives also focused on emotions experienced upon completion of a scenario, with ‘unsure’ being one of the most common. As well, resident leaders mentioned perceived gaps in their knowledge and included statements such as “must know dose of epi[nephrine] in anaphylaxis” (Resident33, Institution1). Similar to nurses, residents identified when they required additional help or prompting.

Discussion

We identified four themes (communication, leadership, demeanor, and medical expert) within the narrative assessment rationale provided to residents during a simulation-based resuscitation curriculum. The focus varied between each assessor group (nurses, peers, attending physicians, and self) suggesting that MSF may have value as part of a holistic assessment of resuscitation skills.

It is not surprising that ‘medical expert’ was a prominent theme as it is the core of clinical medicine; however, it is not sufficient for team-based scenarios. Feedback on the other themes identified in our analysis (communication, leadership, demeanor) was provided more frequently by assessor groups other than attending physicians, suggesting that MSF may be an important method of assessment for these skills. This is in keeping with prior literature that suggests the strength of MSF lies in the assessment of broad-based competencies.[16] Competence in these domains are outlined in competency frameworks published by the Accreditation Council for Graduate Medical Education and the Royal College of Physicians and Surgeons of Canada.[17] It is not surprising that two of these themes (communication and leadership) are included in the Crisis Resource Management skillset, a set of non-clinical abilities that are required for successful team function.[18] The final theme, demeanor, is interesting as it is ascribes qualities to a leader based upon their expressions, behaviour, or attitude. Although demeanor can be directly observed, the attribution of qualities is an assumption on the part of the assessor and fraught with potential biases and inaccuracies. This has the potential to either over or underestimate the actual abilities of the learner. Prior work has found demeanor to be a common theme within the narrative feedback provided by peer and attending physicians to medical students for competencies relating to teamwork.[19]

We found several key differences between the feedback provided by each assessor group. Attending physicians focused largely on medical expertise, whereas nurses commented on communication more so than other groups. This is in keeping with prior literature[20] suggesting that nurses are well positioned to provide feedback on this skill. If nurses were not involved, important perspectives and feedback may not have been provided to residents. Similarly, peers provided a focus on leadership and demeanour that may have otherwise been lost. Finally, self-assessment narrative was diverse and spread across several themes. This is in keeping with the findings of Jong et al[21] who studied MSF in the context of an adult simulation scenario and found that inter-rater reliability improved when self-assessment was removed. Our results would suggest that self-assessment, although an important skill to be fostered in CBME curricula, may not add value to MSF in the assessment of resuscitation skills.

Resuscitations in the clinical environment can be difficult to directly observe and assess as they are time-sensitive and occur at unpredictable intervals and times. Usually, a supervising attending physician, who is also the treating physician in most cases, attempts to observe and provide assessment and feedback. As such, overlap in the content of the feedback provided by the four assessor groups may have been a desirable finding as it would have supported the use of any single team-member’s assessment as providing a similar focus of narrative assessment. However, we found that each assessor group provided a different focus and relying on a single perspective may result in an incomplete picture of the resident’s performance. This is substantiated by prior work that found incongruence between the assessment rationale provided by the attending physician assessor and that of the learner.[22] Our findings suggest that MSF can play an important role as an adjunct to, rather than a replacement for, the supervision and assessment provided by attending physicians.

MSF has ‘face-validity,’[23] particularly for team-based EPAs such as the resuscitation of an acutely unwell patient. As Crossley and Jolly stated, “For the same reason that no single assessment method can encompass all of clinical competency, it is clear that no single professional group can assess it either.”[24] Our finding of different areas of focus for each assessor group is in keeping with this statement and prior research in this field.[25] The feasibility and implementation of MSF into a program of assessment will be challenging and is an active area of research at our institutions. Future work will investigate the use of MSF only for certain key clinical scenarios or partitioning the focus of feedback amongst assessor groups to maximize the value of the feedback and minimize the assessment burden placed on any single assessor group.

Our study has limitations. First, reception and provision of MSF is contextual, and our findings may not be generalizable to programs of assessment within different institutions or countries. Second, it assessed only junior residents in relatively simple resuscitation scenarios. It is unclear whether our findings would transfer to more senior residents. Third, assessment occurred in the context of a simulation-based curriculum in which learners and observers were primed to provide feedback. It is possible that the feedback from nurses and resident peers in the clinical context will differ in its composition and usefulness. Finally, the assessment tool (QSAT) presents domains to consider when assessing performance that may have influenced the narrative feedback provided. However, the same tool was used by each assessor group.

Conclusion

In the context of our simulation-based resuscitation curriculum, MSF provided learners with different perspectives in their narrative assessment rationale. This may offer a more holistic assessment of resuscitation skills within a CBME program of assessment.

Session	Scenario	Topic
1	1	Acute pulmonary edema
	2	Pneumonia
	3	Pulmonary embolism
	4 (Saskatchewan only)	Asthma exacerbation
2	1	Bradycardia
	2	Hypertensive emergency
	3	Seizure
	4 (Saskatchewan only)	Supraventricular tachycardia
3	1	Ventricular tachycardia
	2	Hyperkalemia
	3	Sepsis
	4 (Saskatchewan only)	Gastrointestinal bleed
4	1	Myocardial infarction
	2	Anaphylaxis
	3	Opiate intoxication
	4 (Saskatchewan only)	Hyperkalemia

Primary Assessment
· Ensures monitors are applied & vital signs obtained (incl glucose + Temp)· Establishes appropriate vascular access· Conducts a focused assessment of airway & breathing	· Assesses level of consciousness/disability· Simultaneously performs initial diagnostic & initial therapeutic/resuscitative actions· Allocates & utilizes resources appropriately

Diagnostic Actions
· Performs a targeted history & physical exam· Exposes the patient appropriately to complete exam· Orders appropriate blood work	· Performs rhythm analysis/ECG as indicated· Orders appropriate imaging

Therapeutic Actions
· Prioritizes critical or time sensitive therapies· Performs/directs necessary resuscitative maneuvers· Manages airway & ventilator support as needed	· Orders IV fluids or blood products as appropriate· Orders appropriate medications as required· Coordinates disposition & specialist involvement

Communication
· Uses clear, directed, closed loop communication· Clearly assigns & articulates leadership· Shares mental model & verbalizes priorities	· Solicits opinion from team members, experts, & consultants as needed· Involves patient & family in decision-making· Prepares & debriefs team as time permits

ENTRUSTMENT DECISION
1 I had to do	2 I had to talk them through	3 I needed to prompt	4 I needed to be there just in case	5 I didn’t need to be there

PLEASE EXPLAIN YOUR RATING:

Primary Assessment
· Ensures monitors are applied & vital signs obtained (incl glucose + Temp)· Establishes appropriate vascular access· Conducts a focused assessment of airway & breathing			· Assesses level of consciousness/disability· Simultaneously performs initial diagnostic & initial therapeutic/resuscitative actions· Allocates & utilizes resources appropriately
Diagnostic Actions
· Performs a targeted history & physical exam· Exposes the patient appropriately to complete exam· Orders appropriate blood work			· Performs rhythm analysis/ECG as indicated· Orders appropriate imaging
Therapeutic Actions
· Prioritizes critical or time sensitive therapies· Performs/directs necessary resuscitative maneuvers· Manages airway & ventilator support as needed			· Orders IV fluids or blood products as appropriate· Orders appropriate medications as required· Coordinates disposition & specialist involvement
Communication
· Uses clear, directed, closed loop communication· Clearly assigns & articulates leadership· Shares mental model & verbalizes priorities			· Solicits opinion from team members, experts, & consultants as needed· Involves patient & family in decision-making· Prepares & debriefs team as time permits
ENTRUSTMENT DECISION
1 The attending had to do	2 The attending had to talk me through	3 The attending needed to prompt me	4 The attending needed to be there “just in case”	5 The attending didn’t need to be there

PLEASE EXPLAIN YOUR RATING:

Table 6

Additional supporting quotations for all themes

Theme	Categories	Additional Quotations
1. Communication	Interacting with patients and families	“Always remember to talk through things with the patient if they're conscious” (Peer36, Institution1)“More clear disclosure, attention to patient perception of adverse event” (F13, Institution1)“Disclosed medical error to patient.” (F13, Institution2)“Very well done with communication with patients family - diffused situation well.” (Peer5207, Institution2)
	Calling for help and resources	“Called for help early and appropriately” (F12, Institution1)“Called for help early - code 99 - I appreciated that as I feared the patient was impending code.” (RN3, Institution2)“Consulted senior and appropriate service” (RN5, Institution1)“Called Sr. help early” (R7378, Institution2)
	Sharing mental models	“Shared mental model.” (RN1, Institution2)“Shared thought process.” (F6, Institution2)“Was able to verbalize a good summary and let us know what she was thinking of doing.” (Peer5211, Institution2)“Spoke thoughts to team” (Peer22, Institution1)
2. Leadership	Awareness	“Recognized when to ask for help and to stabilize patient” (R17, Institution1)“Identified needed help and escalated level of care” (RN1, Institution1)“Strong medical knowledge base + understanding of limits” (Faculty6, Institution2)“Realized the complexity of anti-coags in this patient” (Faculty10, Institution2)
	Delegation	“Great organization - assigned roles quickly + effectively” (Peer5248, Institution2)“Utilized team effectively to assess pt [patient] and manage pt [patient] when was talking on the phone” (RN3, Institution2)“Stayed calm and organized” (Peer 5232, Institution2)“Organized” (Peer13, Institution1)
3. Demeanor	Perception of comfort	“Unsure how to proceed when not responding to therapies” (Faculty1, Institution1)“Was not sure of asthma exacerbation” (Peer35, Institution1)“I feel like you became flustered when you were unsure of dosage” (RN4, Institution2)“Calm demeanor helped this case run smoothly” (Peer5214, Institution2)
	Perception of control	“I felt like I was in control” (R33, Institution1)“Good control of the room” (Peer53, Institution1)“Felt you were calm and in control” (RN3, Institution2)“From start to finish was in control” (F11, Institution2)
4. Medical Expert	Content knowledge	“Maybe identify the bradycardia of RX a little sooner” (Peer7378, Institution2)“Good assessment + recognition of shock” (Peer4051, Institution2)“Quickly identified PE” (RN4, Institution2)“Did not identify ETOH withdrawal despite several questions” (RN1, Institution1)
	Diagnosis and treatment	“Unsure of treatment to initiate in symptomatic bradycardia” (R20, Institution1)“Did attempt to very cautiously intervent with IV fluids for hypotension” (RN1, Institution1)“Periodically reassessed the patient's response to treatment” (Peer5207, Institution2)“Don't have opioids + Narcan running at the same time. Consider titrated low dose Narcan” (F6, Institution2)