Training fellows in neonatal tele-resuscitation using a simulation-based mastery learning model.

BACKGROUND: Neonatal tele-resuscitation uses real-time, audio-video telemedicine to connect neonatologists with community hospital care teams during advanced neonatal resuscitations. While telemedicine continues to expand, best practices for training fellows in tele-resuscitation are not known.
OBJECTIVE: We aimed to develop a neonatal tele-resuscitation curriculum using a simulation-based mastery learning model that provides neonatal-perinatal medicine (NPM) fellows with the knowledge, skills, and behaviors needed to lead tele-resuscitations.
METHODS: Using technology-enhanced simulation education and a mastery learning model, we developed a longitudinal pilot tele-resuscitation curriculum. From 07/2018 to 03/2021, NPM fellows participated in the curriculum, which included individualized telemedicine learning, observing and leading simulated tele-resuscitations, and finally, performing clinical (non-simulated) tele-resuscitations. A performance assessment tool was developed to assess competency through eight questions mapped to the Accreditation Council for Graduate Medical Education (ACGME) core competencies, with responses on a 1 to 5 scale (1 = critical deficiencies; 5 = competence of an expert).
RESULTS: Four NPM fellows participated in the curriculum, progressing through the curriculum at an individualized pace. Median scores on the three learning modules were 96-100%. Fellows participated in variable number of simulated tele-resuscitations based on when mastery was achieved (2-3 supervised simulations per fellow, 1-4 unsupervised simulations per fellow). In total, eighteen simulated tele-resuscitations (eight unsupervised, 10 supervised) and one clinical tele-resuscitation were conducted. Twenty-five performance assessments were completed. Assessment scores across the ACGME competencies were consistently high, with mean scores ranging from 4.2-4.6, with 4 equating to 'ready for unsupervised practice' and 5 equating to 'competence of an expert'.
CONCLUSIONS: As telemedicine use continues to expand, curricula that improve learners' comfort with and proficiency in tele-resuscitation are essential. A simulation-based mastery learning model may be one approach that affords learners gradual exposure to and mastery of complex tele-resuscitation skills and behaviors.

Introduction

In response to the COVID-19 pandemic, the Accreditation Council for Graduate Medical Education (ACGME) implemented program requirements permitting residents and fellows to use telemedicine to care for patients . Despite differences between virtual and in-person care 2., 3., limited evidence exists on how to effectively train fellows in telemedicine and assess their performance. Specialty-specific milestones and competencies defining the knowledge, skills, and attitudes required to provide high quality telehealth care are in their infancy 4., 5., 6..

In the United States, neonatology is one of the top five specialty services in pediatric telehealth . Neonatal tele-resuscitation, a real-time, audio–video telemedicine service, connects remote neonatologists with community hospital care teams during advanced neonatal resuscitations. Neonatal tele-resuscitation improves the quality of resuscitation and reduces patient transfers to the neonatal intensive care unit (NICU) 9., 10.. To successfully lead a team remotely, the consulting neonatologist must be knowledgeable in advanced neonatal resuscitation and have the necessary telemedicine technical and communication skills. Best practices for training neonatal-perinatal fellows in tele-resuscitation and assessing their performance have not been described.

To address these teaching and learning gaps, we developed and studied an innovative neonatal tele-resuscitation curriculum. We hypothesized that a simulation-based mastery learning model would allow trainees to achieve proficiency in leading remote neonatal resuscitations via telemedicine. Here we report our approach to curricular design, assessment tools used, and our preliminary outcomes.

Methods

We developed a novel, proof-of-concept curriculum using our center’s clinical neonatal tele-resuscitation program that afforded neonatal-perinatal medicine (NPM) fellows increased exposure to and leadership of remote neonatal resuscitation.

Setting and participants

The study was conducted from July 2018 to March 2021, coincident with our implementation of a longitudinal neonatal tele-resuscitation curriculum for NPM fellows at Mayo Clinic. The fellowship program includes one fellow per year with a maximum of four fellows. Fellows attend high-risk deliveries and rotate at Mayo Clinic’s 34-bed Level IV regional NICU and 24-bed Level III NICU . Four Mayo Clinic Health System hospitals participated in the tele-resuscitation curriculum. These hospitals had Level I well newborn (n = 3) or Level II special care (n = 1) nurseries with annual delivery volumes ranging from 250-700 live births/year. Nine NRP instructors formed the ‘local NRP instructor’ group and were familiar with the telemedicine technology used for neonatal resuscitation. The NRP Provider Courses included learners who were pediatricians, family physicians, nurses, and respiratory therapists completing either their initial or renewal training session.

Intervention

Using the simulation-based mastery learning model 13., 14. and instructional design principles of technology-enhanced simulation education , we designed a three-year, longitudinal curriculum with the goal of training NPM fellows to successfully lead neonatal resuscitations using telemedicine. As such, simulation was integrated across the curriculum in a goal-directed, systematic manner.

The process for simulated neonatal tele-resuscitations was the same as our process for actual clinical consultations , including community hospitals requesting a consult via the organization’s admission and transfer call center, the neonatologist or NPM fellow connecting to the community hospital team via telemedicine, and rapidly orienting the neonatologist/fellow to the resuscitation scenario before assuming the team leader role. The wireless proactively monitored telemedicine device in the community hospital had a dual camera system controllable by the neonatologist/NPM, a directional echo-cancelling microphone, and two speakers (Fig. 1A). The neonatologist/NPM connected to the device from a workstation in the NICU using the provider access software. The software includes a ‘multipresence’ feature that permitted the fellow to experience the consult exactly as experienced by the neonatologist, including views of the mannequin and community hospital team, interpersonal communication, and control of the telemedicine cameras (Fig. 1B). At the end of the simulation, fellows participated in the group debrief that was led by the local NRP instructor. Participation in the debrief allowed for bidirectional feedback between the resuscitation team and the remote leader.

Fig. 1

(A) Depiction of NRP participants utilizing a tele-resuscitation consult from a neonatologist. (B) Neonatal-perinatal fellow conducting a supervised, simulated teleneonatology consult. The supervising neonatologist (left), simulated resuscitation (upper center), and trainee (lower center) are all visible on the screen using the multipresence feature.

The curricular approach is detailed in Fig. 2. During their first year, fellows completed Neonatal Resuscitation Program (NRP) instructor training. They viewed three learning modules related to the unique features of neonatal tele-resuscitation, i.e., workflow considerations, professionalism and communication, and legal and compliance considerations. Fellows also completed telemedicine technical training that included operating the two cameras (e.g., panning and zooming) to obtain adequate view of the infant, monitors, and remote team members as well as manipulating the audio controls. Fellows had unlimited access to demonstration devices such that they could practice these skills at any time during training.

Fig. 2

Curricular components for teaching neonatal tele-resuscitation using the conceptual framework of simulation-based mastery learning. Though all fellows begin the curriculum at the same time, progression through it is variable and based on mastery of each level, not based on time in training. Higher level learning tasks in simulated and clinical consultation are paired with frequent, individualized reflection on action and feedback.

Second-year fellows observed simulated tele-resuscitation consults, where the neonatologist guided a community hospital team through a complex neonatal resuscitation that was conducted during NRP training. After completing the observations, fellows assumed the role of expert consultant in two to three simulated tele-resuscitation consults that were supervised by a neonatologist. After each simulation, fellows participated in a brief reflection on action and tailored, individualized learning with the Teleneonatology Medical Director (JLF). Fellows progressed to unsupervised simulated tele-resuscitation consultations as competency increased. This was determined both by achieving a mean score of 4 or more for each performance item during supervised consultations, as well as reported readiness for unsupervised consultation by the fellow. If fellows did not report comfort with leading remote resuscitation, they were offered additional opportunities for supervised consultation and feedback.

Third-year fellows continued to perform unsupervised simulated consults. Using the same telemedicine platform, fellows who mastered simulated consults were permitted to perform clinical tele-resuscitation consults supervised by a neonatologist using the multipresence feature.

Outcomes measured

Because there were no published assessment tools for neonatal tele-resuscitation training, we developed instruments to evaluate the knowledge, skills, and behaviors that are critical to lead a remote, often unfamiliar, team through neonatal resuscitation. Content experts in neonatal tele-resuscitation (JLF) and health care delivery (JLF, BLK) developed written tests that were administered after fellows completed the learning modules. Experts in tele-resuscitation (JLF, BLK) and graduate medical education (GME; WAC) developed a performance assessment tool for observed simulated and clinical tele-resuscitation consults (Appendix 1). The tool assessed competency through eight questions mapped to the six ACGME core competencies, with responses on a 1 to 5 scale (1 = critical deficiencies; 5 = competence of an expert) that included descriptions for raters (i.e., the supervising neonatologists and local NRP instructors). The assessor could select “Unable to Assess” and provide individualized written feedback.

Assessments were requested from both the supervising neonatologist and the local NRP instructor for all supervised consults (n = 10), from the local NRP instructor for the unsupervised consult (given no supervising neonatologist was present, n = 8), and from the supervising neonatologist for the clinical tele-resuscitation (as there was no NRP instructor at the delivery, n = 1), leading to 29 possible assessments.

Analysis of the outcomes

Descriptive statistics were performed on the written tests and performance assessments. All responses were reported as percentages, median (range), and mean (SD) as appropriate. Given the small sample size, no comparative analyses were done.

Results

Four NPM fellows participated in the neonatal tele-resuscitation curriculum. They successfully completed NRP instructor training, the three learning modules, and the telemedicine technical training. Median (range) scores on the modules were 96% (91–100%) for Module 1: Introduction to Teleneonatology; 100% for Module 2: Professionalism and ‘Webside’ Manner; and 96% (92–100%) for Module 3: Legal and Compliance Considerations.

Fellows completed two to three supervised simulated consultations before achieving scores and reporting confidence to progress to unsupervised simulated consultations. Fellows performed one to four unsupervised simulated consults during the study period. Only one fellow progressed to the level of supervised clinical consultation during the pilot study, which occurred after four unsupervised simulation experiences. In total, the cohort of four fellows completed 18 simulated tele-resuscitations and one clinical tele-resuscitation.

Performance assessments were requested for the 19 tele-resuscitations. One assessment was sent to the wrong person, and three assessments were not completed (two by local NRP instructors during unsupervised consults, and one by the local NRP instructor during a supervised consult), resulting in 25 returned (86% of available) performance assessments.

The 25 completed assessments provided a total of 190 rated responses and ten “unable to assess” responses to the eight questions (Table 1). Mean scores across the six ACGME competencies ranged from 4.2 to 4.6, on a 1 to 5 rating scale, with 4 equating to ‘ready for unsupervised practice’ and 5 equating to ‘competence of an expert’. Written feedback provided to the fellows included suggestions related to both neonatal resuscitation key behaviors and telemedicine-specific knowledge and skills. Selected examples are presented in Table 2.

Table 1

Aggregate neonatal-perinatal fellow performance of neonatal tele-resuscitation consults. Performance assessment scores ranged from 1-5 (1 = critical deficiencies; 2 = behaviors of an early learner; 3 = advancing and demonstrating improvement; 4 = ready for unsupervised practice; 5 = competence of an expert).

ACGME Core Competency	Brief Description of Question	Responses*N	Score Mean (SD)
Patient Care and Procedural Skills	Develops and carries out management plans	21	4.5 (0.7)
Medical Knowledge	Evaluates and applies evidence to patient care	23	4.4 (0.6)
Systems-Based Practice	Works in interprofessional teams to improve care	23	4.6 (0.6)
Practice-Based Learning and Improvement	Educates other health professionals	23	4.5 (0.8)
Professionalism	Demonstrates trustworthiness that makes colleagues feel secure	25	4.5 (0.6)
Professionalism	Provides leadership that enhances the learning environment	25	4.4 (0.7)
Interpersonal and Communication Skills	Communicates effectively	25	4.4 (0.8)
Interpersonal and Communication Skills	Acts in consultative role to others	25	4.2 (0.9)

*Maximum of 25 responses. The N may be less than 25 if the evaluator selected “unable to assess” for that question.

Table 2

Selected written feedback provided to fellows on tele-resuscitation performance. N = neonatologist feedback, LI = local instructor feedback.

ACGME Core Competency	Examples of feedback provided to fellows
Patient Care and Procedural Skills	Used all of the technology available (boom [camera] for vitals, was able to turn and zoom main [camera]). –LIWhen possible, take even greater advantage of the camera by zooming in for detailed views, e.g. quality of chest compressions, and listen for other details, e.g. HR provided to team via metronome. – N
Medical Knowledge	[The fellow] has at least a training level-appropriate degree of knowledge to lead advanced resuscitations. –N
Systems-Based Practice	Ready and able to lead a more robust debriefing… [The fellow] demonstrated this ability…by asking the local team about which services comprise their delivery-response personnel. –NAcceptable to reinforce key NRP principle of delivery of effective ventilation prior to initiation of chest compressions. –N
Practice-Based Learning and Improvement	Provide feedback once the simulation is complete, highlighting what went well and opportunities for improvement. –N…Stayed for debriefing and had relevant adequate input for the scenario. –LI
Professionalism	Excellent bedside manner, communication and assisting, very collaborative and flexible with resources available. –LIIt can be challenging to identify the leader in an emergent situation and to assert oneself as the leader (esp. when one is a remote provider). We discussed strategies for identifying/asserting leadership… – N
Interpersonal and Communication Skills	Introduce yourself to the team when you join the consult. Ask [for] an update on what is going on with the patient. –NVoice sounds a bit timid at times - always speak with confidence. –NContinue to use encouraging, re-affirming language as applicable. Pace and prioritize your requests. – N

Discussion

This report provides early evidence that a curriculum utilizing a simulation-based mastery learning model allowed trainees to achieve proficiency in neonatal tele-resuscitation. Given the expansion of telemedicine services , effective training for GME learners is essential. Our institution has a well-established regional, neonatal tele-resuscitation program that includes scheduled simulations with our community hospital sites . These simulations provided a safe, supportive learning environment where fellows gained familiarity with the technology, processes, skills, and behaviors needed for tele-resuscitation.

This curriculum presents a longitudinal approach based on principles of simulation-based mastery learning models and deliberate practice . Design elements associated with improved long-term learning and transfer, including repetitive and distributive practice, cognitive interactivity, individualized learning, feedback, and clinical variation , were incorporated into the training. Fellows were increasingly exposed to authentic clinical experiences using telemedicine, first by developing their skills as NRP instructors, then as remote consultants during NRP simulation, and ultimately as a remote consultant during actual neonatal resuscitations.

Given the high standards needed for physicians to conduct tele-resuscitation, we chose to incorporate multiple points of assessments from both neonatologists and experienced local NRP instructors over time. Written and performance assessment scores suggest that fellows comprehend key elements of tele-resuscitation, including the workflow and inter-professional communication needed to provide high quality care at a distance. By leading neonatal tele-resuscitations, fellows experience the complex interplay between medical knowledge, communication skills, and trust affecting their ability to lead an interprofessional team. Feedback provided to the fellows after tele-resuscitation was highly relevant to the key behavioral skills emphasized in NRP training, including communicating effectively, maintaining professional behavior, using available information and resources wisely, allocating attention wisely, and clearly identifying a team leader. Feedback also reflected the need to develop skills specific to tele-resuscitation, such as agility with camera placement, voice intonation, and pacing of recommendations.

Our pilot curriculum has limitations. First, general telemedicine competencies and validated assessment tools for GME learners are currently in their infancy and do not yet exist for training fellows in tele-resuscitation. Core competencies and development of subsequent Entrustable Professional Activities of telehealth consultation will be necessary to guide future training and assessment. For this reason, we developed a pilot assessment tool guided by local experts in neonatal telemedicine and graduate medical education that will require further validity evidence in the future prior to widespread use. Our telemedicine use was limited only to acute settings, and our training may not generalize to non-emergent consultation. Additionally, during the pilot period, only one fellow progressed to the level of supervised clinical consultation, so further evaluation of fellow performance during clinical tele-resuscitation consults is needed. Lastly, feedback was collected from neonatologists and NRP instructors but not NRP students; this may provide an important perspective that helps NPM fellows improve communication and team-based performance.

Future studies should assess the patient impact of learner-led tele-resuscitation. Our pilot data suggest that fellows who progress through such a curriculum are ready to perform clinical consultation, but further work is needed to ensure patient safety. While several studies on telemedicine training have reported learners’ reactions to curricula and perceived knowledge gains 18., 19., there is only one report on the impact of telemedicine training on patient outcomes that compared time to tPA between fellow-led and attending physician-led telestroke consultations. In this program, fellows progressed through telemedicine training through graduated exposure and entrustment until independently performing clinical telestroke consultation – similar to our pilot tele-resuscitation curriculum . Our tele-resuscitation curriculum has integrated higher assessment outcomes , i.e., fellows’ demonstration of skill in real and simulated practice, but could be strengthened by comparing patient outcome variables attained by fellows and attending neonatologists in our clinical tele-resuscitation program.

Finally, general barriers to telemedicine training must be overcome in any telemedicine program, including the lack of reliable telemedicine infrastructure, difficulty identifying experts to serve as instructors, and the need for additional licensing and credentialing . Our curricular model and assessment tools should be validated in additional learner populations utilizing telemedicine for acute or emergency care before widespread adoption within graduate medical education.

Conclusions

As telemedicine use continues to expand, curricula that improve learners’ comfort with and proficiency in tele-resuscitation are essential. A simulation-based mastery learning model may be one approach that affords learners gradual exposure to and mastery of complex tele-resuscitation knowledge, skills, and behaviors.

Ethical statement

The Mayo Clinic IRB deemed this study exempt as education research. All learners provided informed consent to use of their assessment data in this study.

CRediT authorship contribution statement

Stephanie C. Mavis: Writing – original draft, Formal analysis. Beth L. Kreofsky: Project administration, Data curation. Melody Y. Ouk: Project administration, Data curation. William A. Carey: Conceptualization, Supervision, Formal analysis. Jennifer L. Fang: Conceptualization, Methodology, Supervision, Formal analysis, Writing – review & editing.

Declaration of Competing Interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Jennifer L. Fang and William A. Carey have licensed intellectual property with and can earn royalties from Teladoc Health. Dr. Fang also holds stock in the company.