Virtual Deliberate Practice Module for Tracheostomy Change Training: An Application of Educational Design Research.

Background: The pandemic poses challenges for in-person procedural skills training. We developed a virtual module for teaching hands-on tracheostomy skills. Objective: To develop and evaluate a virtual module prototype grounded in deliberate practice using tracheostomy change as an example.
Methods: After identifying desirable features of a virtual module by surveying stakeholders, we designed a prototype using VoiceThread, a multimedia-based collaborative learning platform. We created an asynchronous module accessible to learners for repeated skill practice and for video upload of individual performance on a tracheostomy task trainer using personal devices. This virtual module provided a four-step coaching (demonstration, deconstruction, formulation, and performance) to practice tracheostomy change. Two instructors reviewed the learners' performance videos, providing timely feedback for further refinement of skills.
Results: Sixty-four residents completed the module, System Usability Scale, and self-efficacy survey. All residents rated the module, with a mean System Usability Scale score of 68.6 ± 18.4 (maximum score of 100). Two independent instructors rated performance videos using a 12-item checklist with mean interobserver agreement of 88.1% (standard deviation, 9.7) and mean performance checklist score (n = 40) of 10.1 (standard deviation, 1.2) out of 12. After training, residents reported high confidence in their ability to list and perform procedural steps, with improvement in median (interquartile range) comfort levels from 1 (1-2) to 4 (3-4) out of 5 (P < 0.0001).
Conclusion: We developed an asynchronous deliberate practice module on a virtual platform using tracheostomy change as an example. Residents evaluated the module favorably using system usability and learner self-efficacy surveys with improvement of skills.

Because of the pandemic and implementation of social distancing, educators have faced unprecedented challenges to instruct trainees in hands-on skills (1). Many educators have an increased workload with limited time for hands-on skills training owing to social distancing restrictions for in-person simulation. A number of training programs, particularly those with limited resources, cannot sustain personnel and financial support for optimal training (2, 3).

Previous studies highlight that pediatric and emergency physicians lack experience in managing tracheostomy emergencies (4–6). Studies show that the use of simulation (4, 7) and videos (8) to teach management of tracheostomies has positive effects on learners’ acquisition of knowledge (5), with some using social media for dissemination of resources (9). Trainees at our institution are required to learn and perform tracheostomy change and suctioning. With the challenges of social distancing and reduced in-person training, we identified a great need for robust methods for hands-on skills training using virtual instruction.

We describe an innovative approach for teaching and learning procedural skills grounded in educational theory and evidenced-informed instructional methods using tracheostomy change as an example. We propose the design principles of asynchronous deliberate practice (DP) on a virtual platform for medical educators to adopt and expand for teaching other procedural skills beyond the pandemic.

Design and Methods

Setting and Participants

This innovation was developed at Texas Children’s Hospital, a teaching site for 140 pediatric residents of Baylor College of Medicine. We conducted tracheostomy change training for first-year residents in a 36-bed intensive care unit for children with medical complexities and technology dependence. Our previous instructions included video-based instruction followed by didactic lectures and simulation (8). This study was approved by the Baylor College of Medicine Institutional Review Board and was implemented in June 2020.

Designing an Innovation

We used the educational design research (EDR) framework (10–12) to guide the development of the module. We went through a rapid prototyping in 4 weeks, with a plan for ongoing refinement over the next month given the emerging demands imposed by the pandemic.

Analysis and Exploration

We surveyed stakeholders involved in tracheostomy education and engaged in informal discussions with medical trainees, instructors, and directors of medical units to gain insights about ideal characteristics of the virtual module (Figure 1). We scanned the literature and identified the theory of DP to serve as a framework for teaching procedural skills (13). DP starts by analyzing and deconstructing exemplary performance by identifying the mental structures and skills of experts. The approach emphasizes the importance of having a specific goal for training; access to immediate, detailed feedback; and opportunities for repetition and successive refinement of skills until mastery is achieved.

Figure 1.

Desirable features of a virtual module as described by needs assessment (n = 15).

Design and Construction

Our design (Figure 2) was, in essence, the synergistic integration of a DP approach with four-step coaching, including a video blogging tool and platform that harnessed the social media–like environment, enhancing virtual learning. We used Community of Inquiry to guide the design of the virtual environment (14). Three interrelated elements, cognitive, social, and teaching presences, are essential for creating an effective online learning experience.

Figure 2.

The design framework for the virtual deliberate practice module. The three interrelated elements of the “community of inquiry” (cognitive, teaching, and social presence) guide the design and construction of the module. Peyton and Walker four-step coaching includes 1) demonstration: the instructor demonstrates the procedure at a normal pace with no comments; 2) deconstruction: the instructor repeats the procedure, this time describing all necessary steps in detail; 3) formulation: the learner explains each step while the instructor follows the learner’s instructions; and 4) performance: the learner performs the entire procedure without guidance. DP = deliberate practice.

For cognitive presence, we selected the “Peyton and Walker’s Four-Step Approach,” an instructional method for DP (15, 16). This four-step coaching (demonstration, deconstruction, formulation, and performance) aligns with educational neuroscience for manual-skills training in many fields and is gaining attention in medical education (17, 18). This approach helps learners generate mental representation and motor imagery, the essence of DP, of a specific skill through the steps described in Figure 2.

For social presence, we emulated the popular social media platform (i.e., TikTok) to foster a collaborative and psychologically safe learning climate in which learners can project their authentic self to engage with learning experiences (19). This skill challenge platform is used for uploading user-generated videos for sharing content, thereby allowing repeated practice via tutorials and interactivity with comments and feedback (20, 21).

For teaching presence, we used VoiceThread (VT; VoiceThread, LLC) for educators to organize, facilitate, and direct the instructional processes. VT is a multimedia-based platform that allows users to upload, share, and discuss digital content. Users can react to content using microphone, webcam, text, or phone with easy access through a computer or mobile device. For learners to record their performance videos, we used the Osmo3 smart phone stabilizer given its functions of enhanced video quality for recording and ability for facial recognition and tracking of movements during video recording. Using an animated video-maker platform (Powtoon Ltd.), we created orientation videos to guide residents to navigate through the learning platform.

Evaluation and Reflection

We tested the prototype to gain insights from learners on its usability and effectiveness at the completion of the module. We used the System Usability Scale (SUS; see the data supplement), a 10-item, 5-point Likert scale providing a global view of subjective assessments of usability of a product (22) and interventions (23). We used a 4-item, 5-point Likert scale survey for learners to assess perceived self-efficacy in listing and performing critical steps of tracheostomy change and to compare comfort level before and after the module. Two investigators independently rated the performance videos uploaded by learners using a 12-item performance checklist created with expert consensus of essential steps for performing a tracheostomy change (see Checklist in the data supplement). Reflection on the processes and evaluative findings offered further understanding of the theoretical framework and informed the design principles. Educators sought feedback from learners during in-person training from our first group of residents and from chief residents. We simplified our module by minimizing the number of hyperlinks sent to residents. Introduction videos on accessing the module were grouped to one hyperlink, and videos demonstrating the four steps for tracheostomy change were accessible on a separate link.

Data Analysis

The SUS scores were calculated according to a recommended formula (22). We used median (interquartile range) to summarize the responses and Wilcoxon signed rank test for paired comparisons of the Likert scale measurements. Interobserver agreement (IOA) and Cohen’s Kappa were used to assess the interrater reliability of the performance checklist. Prevalence-adjusted bias-adjusted kappa (PABAK) was used in case of Kappa paradox (24, 25) (high or low prevalence of a given response leading to an artificially low level of reliability despite high IOA). We calculated the mean and standard deviation (SD) of the performance scores.

Results

The Innovation

The final prototype is an asynchronous DP module on a multimedia-based collaborative learning platform. The module has three video clips: 1) Demonstration, 2) Deconstruction, and 3) Formulation. Learners may view the first two videos repeatedly until they have memorized all the procedural steps to narrate the step-by-step procedure performed by the instructor in the formulation video. On VT, learners can practice the narration repeatedly before submitting the audio file for review. The instructors, prompted by an auto-generated email notification from VT, can provide prompt feedback on the narration using digital conversation capabilities. This swift communication enables a collaborative learning environment among learners, allowing for the opportunity to post clarifying questions for instructors to address promptly (Figure 3). All digital conversations were shared only within a small group of learners in each rotation.

Figure 3.

Screenshot from the module showing learners’ narrations (step 3: formulation) indicated by blue dots on the bottom, with the learner initials on the left side of screen. Highlighted in the text box is the feedback from the instructor to a learner.

Learners then progressed to recording their individual performance videos with a tracheostomy care manikin using a mobile device. This step can be done either asynchronously or with assistance from an instructor or facilitator, depending on the logistics or availability of the manikins. Learners can practice repeatedly before uploading their videos for review and feedback. The module is available at https://bcm.voicethread.com/myvoice/browse/threadbox/6052.

The Module Evaluation

The SUS provides “usability” scores ranging from 0 to 100, with a score of 68 representing an average based on an analysis of 500 studies (26) across a variety of applications, websites, and systems. The mean SUS score of 64 participating residents was 68.6 (SD, 18.4). We observed that some learners scored the same number for every question despite contrasting meaning across items—the SUS has mixed positive and negative statements. For assessment of self-efficacy, 64 participants reported an increase in their confidence level, measured by median (IQR), about their ability to list (4 [3-4]) and perform (4 [3-4]) critical steps of tracheostomy change. A retrospective pre- and postmodule comparison showed significant improvement in confidence with tracheostomy change, from 1 (1–2) to 4 (3–4) (P < 0.0001; Wilcoxon signed-rank test).

For the performance scores, 40 videos uploaded by participants were rated by instructors. Across the 12 checklist items, the mean IOA was 88.1% (SD, 9.7). Given the Kappa’s paradox (Table E1), we used PABAK to determine the interrater reliability. The mean PABAK κ statistic was 0.76 (SD, 0.19). The highest IOA was 100% for questions regarding handling of the tracheostomy. The two lowest IOAs were for neck roll placement (65.0) and checking tightness of tracheostomy ties (77.5). The mean performance score was 9.8 (SD, 1.2) out of 12 by rater 1 and 10.4 (SD, 1.2) by rater 2.

Discussion

We developed a module prototype grounded in DP principles for teaching hands-on skill, using tracheostomy change as an example. This module provides an asynchronous, self-directed DP instruction on a virtual platform, fostering a community of inquiry in a social media–like format while enabling effective digital conversations and timely feedback. Guided by four-step coaching to promote DP, our residents rated the module favorably for usability, learned to perform critical steps, and gained confidence in performing the procedure. Educators who seek an effective approach to teaching clinical skills may adapt this module for coaching beyond hands-on skills (e.g., communication or history taking). The following reflection represents the final stage of the EDR. This reflection helps conceptualize the final design principles and informs how the chosen theory (i.e., deliberate) should be used with some practical suggestions for application. Key components of DP include well-defined learning objectives, precise assessment of performance, and timely feedback, in addition to intense focus on practice (13). Our module sets a clear objective of learning to perform tracheostomy change and offers ample opportunity for repeated practice, allowing intense focus for acquiring new skills, and timely assessment and feedback to the learner, enabling refinement of skills toward mastery (13, 27, 28). A unique benefit of asynchronous, self-directed virtual learning is its allowance for individualized practice, with focused repeated practice on one’s own device at one’s own pace. Our module fulfills all critical elements of DP, unlike traditional, in-person, procedural skills teaching using simulation or task trainers that are limited by stipulations of learner numbers and availability of resources.

An innovative aspect of this module is the synergy of evidence-informed educational practices and a social media–type environment grounded in a Community of Inquiry framework (19). Our module guides learners, in four steps, to generate mental representation and motor imagery of a specific skill (cognitive presence) within a collaborative and psychologically safe environment (social presence). The platform allows instructors to facilitate and direct learning through instructions and prompt feedback (teaching presence). We recommend educators consider maintaining this framework as they adapt our model for teaching other clinical skills.

Though the SUS of the module meets the industrial standard (>65), there is still room for improvement. These scores reflected both the technological aspects and the steps that users had to follow to successfully complete the module. We emphasize the importance of providing clear instructions to help learners navigate the module. Learners experienced technical difficulties uploading performance videos, so we added clear, step-by-step instructions and had a facilitator available to upload these videos for completion.

This study has a few limitations. Although our module, by design, is completely asynchronous, some learners faced technical difficulties. Having a facilitator available to address potential technical issues, providing direct guidance, can enhance engagement. This additional support is critical to enable the adoption of new technologies. Although we are encouraged by the positive feedback about the module and the educational outcomes, the evaluative data include self-reported measures with inherent subjectivity and can be influenced by social desirability. Aligning with EDR, we designed an educational module that unfolded in an authentic learning environment without an experimental comparison, and thus our findings have limited generalizability. Our design principles, however, are transferable concepts for others to adopt and adapt to their local contexts.

Conclusions

We developed a virtual DP module for teaching procedural skills. Our learners acquired and practiced skills through four steps on an asynchronous, self-directed virtual platform designed specifically to foster collaborative learning and interaction to optimize feedback.