A simulation approach to measure critical safety behaviors when evaluating training methods for respirator education in healthcare workers.

BACKGROUND: The N95 respirator is the most common safety tool used in hospitals to protect health care workers (HCW) from inhaling airborne particles. Focusing on HCW behavior related to respirator use is an effective route to improve HCW safety and respiratory health.
METHODS: Participants were asked to perform the donning and doffing of an N95 respirator to camera. Then they were randomized to a video alone or a reflective practice intervention. After the intervention they repeated the donning and doffing to camera. A critical safety behavior scoring tool (CSBST) was developed to compare the performance of the participants over time at pretest, post-test and 1 month later for follow-up.
RESULTS: The reflective practice intervention group was found to have significantly higher scores on the CSBST at post-test and follow-up than the video alone group. In the reflective practice intervention group, the participants perceived they were better at performing the N95 donning and doffing than the experts scored them.
CONCLUSIONS: The CSBST is a tool to measure the performance of HCWs on a specific targeted safety behaviors. The addition of a reflective practice intervention may result in a measurable and sustained improvement in the safety behaviors demonstrated when using the N95 respirator.

The N95 respirator is the most common safety tool used in hospitals to protect health care workers (HCW) from inhaling airborne particles. This became even more evident during the COVID-19 pandemic. The challenges of maintaining an effective respiratory protection program (RPP) are noted in the literature in statewide and regional evaluations.1, 2, 3 One of the problems is that HCW do not use the respirators properly. The Centers for Disease Control and Prevention (CDC) focuses on the use of N95 filtering facepiece respirators as the common tool for HCW protection from pandemic and infectious respiratory illness. There is a continued need for more tolerable respirator designs in healthcare, but innovation in this area is slow. An educational research focus on HCW behavior related to respirator use is an effective route to improve HCW safety and respiratory health at this time.

Outside the health care setting, recent studies funded by the National Institute for Occupational Safety and Health (NIOSH) have analyzed training, evaluated different types of training methods, and measured persistence of learning related to respirator use for both filtering facepiece N95 respirators and dual cartridge half facemask elastomeric respirators.6, 7, 8 Findings from a component analysis found training of respirator users improved knowledge, but there were common challenges in performance related to seal checking and avoiding facemask contact with removal. When comparing training methods, video training that included illustrations, text, and demonstration was found to be a superior training method to printed materials or computer-based training with additional review questions, both of which were self-paced. Thework included an evaluation of the persistence of respirator donning and doffing techniques at 6 months, and performance scores became significantly worse.

Educational research additionally supports video review as an effective tool for learners with various levels of experience, but it emphasizes the importance of evaluation metrics or specific benchmarks. , Behavior related to respirator use may be improved by structuring observations through specific objective measures instead of simply reviewing video of performance. Effectiveness of video feedback is enhanced when a standard evaluation form is provided to the trainee to focus their concentration on specific targeted behaviors.

A systematic review in 2019 found that simulation studies regarding personal protective equipment (PPE) use need to focus on increasing the number of study participants and determining best training methods. Recent studies have found poor doffing behaviors in isolation care using trained observers in the clinical environment. Simulation has become a feasible means to evaluate PPE infection control behaviors in HCWs. , Simulation has also been used to measure the potential for fomite-based transmission during donning and doffing maneuvers of respirators. A detailed analysis of respirator-associated behaviors from video of simulated patient care scenarios was used to identify critical safety behaviors while using a respirator. This study compared 2 N95 respirator training methods using a simulation-based approach. One group of participants completed an evaluation metric while the second group did not.

Methods

The Institutional Review Board approved study occurred in 2 locations between August 2018 and April 2019 – a 700-bed tertiary care teaching hospital and a 100-bed hospital within the same hospital system. To aid in evaluation, 2 video cameras were set up outside of a doorway leading into a patient care room of an unoccupied section of the hospital, providing a front and side view of the study participant. A third camera was set up on the other side of the doorway giving a view from the patient room (Fig 1 ). An isolation cart or bedside table was stationed in front of the room with hand sanitizer, gloves, and the appropriate sized N95 respirator. A wastebasket was available on both sides of the door for disposal of the respirator.

Fig 1

Camera set up.

A repeated performance method was used (Fig 2 ) for 2 intervention groups: Video Alone (VA) or Reflective Practice (RP). All participants began by completing a demographic questionnaire which asked about role, age, gender, and years of experience. They were then instructed to “please apply and remove the respirator as if entering and exiting a patient room under airborne isolation.” Study personnel started the recordings and participants performed the donning and doffing of a respirator to camera for a pretest measure as a simulation. A randomized educational intervention followed in the VA or RP option. The participant was then asked to repeat the donning and doffing to camera immediately after the intervention for a post-test measure. Participants returned approximately 1 month later for follow-up measure and repeated the video-recorded donning and doffing simulation.

Fig 2

Study design.

A randomization schedule was developed before the study began. Participants were randomized through simple urn randomization, ensuring equal sample sizes. The randomized educational intervention was either (1) VA, in which participants watched the first 4 minutes and 40 seconds of a CDC training video on donning and doffing an N95 respirator and performing a seal check (available at: https://www.cdc.gov/niosh/npptl/topics/respirators/disp_part/donningdoffing.html), or (2) RP, in which the participants watched the same CDC training video segment and then watched the video of their own baseline performance while self-scoring using the critical safety behaviors scoring tool (CSBST), described in Figure 3 . The self-scoring intervention employs reflection to improve professional performance. This additional intervention was hypothesized to improve performance both for the post-test and 1-month follow-up. The VA option took approximately 5 minutes, and the RP option took between 8 and 10 minutes.

Fig 3

Critical safety behaviors scoring tool.

The authors, with backgrounds in industrial hygiene, infection control, and research design, developed the 10-point CSBST for N95 respirator use (Fig 3) from previous study findings. There were 6 measures for donning and 4 for doffing. Each point is equally weighted. Hand hygiene was a critical component of both donning and doffing processes, accounting for 4 of the 10 total points. Correct strap placement and performing a proper seal check were key behaviors noted in donning. Doffing safety behaviors included limiting touch to the respirator straps for removal and gently moving the respirator from the face to the wastebasket. These CSBST items were covered well in the CDC video, but the video did not demonstrate the conduct of hand hygiene. Prior to data analysis, each video-recorded simulation was reviewed by a member of the research team and point values were assessed using the CSBST. During the video scoring process, the research team was blinded to which educational intervention was implemented for the participants. Interobserver agreement was assured by systematically reviewing videos independently and then comparing scores until sufficient agreement was reached. Video scoring was also randomly assessed for consistency for the videos assessed by individual scorers. No discrepancies were noted.

The targeted study sample size was 30 in each intervention group. A sample of 30 participants per group is considered sufficient in a pilot study for aims involving between group differences, estimating effect sizes, and using the results to perform a power analysis for a future, fully-powered study. HCWs who are required to enter airborne isolation rooms for their work were recruited from clinical and ancillary services in the hospital. The participants worked at least part-time at a hospital with an active RPP. Participants were recruited through group emails and personal contacts with managers or unit educators throughout the hospital from August to November of 2018 which resulted in a convenience sample. Each participant consented individually and could refuse to participate at any time without penalty. A cafeteria meal ticket was given to participants as compensation for each of the 2 study visits required to complete the research. Study personnel worked with managers and unit educators to identify a day and time for the 1-month follow-up; when unavailable, study personnel reached out via email to participants directly to schedule a time for the 1-month follow-up recording. The study received expedited Institutional Review Board approval (#346-18-EX) as social and behavioral science.

Prior to statistical analysis, total scores were calculated on the 10-point CSBST from the expert scoring results as previously described and entered into an electronic spreadsheet (Microsoft Excel), along with results from the demographic questionnaire. To evaluate normality, skewness and kurtosis statistics were reviewed. The ratios of skewness and kurtosis values to their respective standard errors did not exceed 2 for pretest, post-test, or follow-up scores, so parametric statistics were used in all analyses involving total scores. Repeated measures analysis of variance models were used to compare overall changes between groups, with significant time by group interactions indicating significant differences in change. Independent groups t tests were used to compare scores at each time point, and paired samples t tests were used in post-hoc tests to assess changes between adjacent time points within each group.

Results

Sixty-two HCWs consented to participate, but only 48 participants (79%) completed the study (ie, returned for the 1-month follow-up) and will be described in this report. Randomization of the participants resulted in 24 participants in each intervention group. At least 8 of those that did not complete the follow-up were no longer hospital employees for multiple reasons not captured in the study. Based on demographic data collected, participant roles included nursing (n = 33), environmental services (n = 10), respiratory therapy (n = 3), radiology (n = 1), and patient-care technicians (n = 1). Participants ranged in age from 24 to 66 (average = 36.8) and comprised 44 (92%) females and 4 (8%) males. Years of experience averaged 10.1 (median 9.5). While participants in the VA group were significantly older than those in the RP group (40.8 years vs 32.8 years, P = .01), there was not a significant difference in years of experience in health care (11.9 years in VA group; 8.4 years in RP group). The 2 groups had equal distributions of gender (92% female) and having experienced a needle stick (20.8%). The majority of participants were white/Caucasian (81.3%), with 75% white/Caucasian in the VA group and 87.5% in the RP group (P = .267).

The longitudinal follow up did have some variation. The time since post-test ranged from 21 to 245 days. The median and mean were 35 days and 53.2 days, respectively. Eight of the 48 participants returned more than 2 months later. Although the increased latency for these participants could affect any decreases in total scores, a Mann-Whitney U test for days between visits indicates that the 2 groups were not significantly different in time between post-test and follow-up. The total video files analyzed for this longitudinal study sample was 432, with 3 video recorded performances and 3 views for each of them.

Table 1 details the percentage of critical safety behaviors performed for the completed participants at pretest, post-test, and follow-up. The highest total score prior to the interventions was 6 of 10. All of the perfect scores post-test were from the RP group (7 of 24 RP participants). The total scores were averaged and plotted in Figure 4 . An independent groups t test indicated that the VA group and the RP group were not significantly different at pretest [t(46) = 1.43, P = .161]. The RP group was significantly higher than the VA group post-test (P < .001) and at follow up (P < .001). Repeated measures analysis of variance models showed that both groups significantly improved from the pretest score to post-test, but the RP group improved more significantly [F(1,46) = 34.93, P < .001]. Both groups declined from post-test to follow-up, but the VA group at follow-up was not significantly higher than their pretest score [paired t(23) = 1.79, P = .088]. The RP group was still significantly higher at the follow-up than they were at pretest [paired t(23) = 6.12, P < .001].

Table 1

Number and percentage of critical safety behaviors performed by participants who completed the study at pretest, post-test and follow-up by intervention group(n = 48)

	Pretest		Post-test		Follow-up
	Control	Treatment	Control	Treatment	Control	Treatment
Donning hand hygiene before	8 (33.3%)	12 (50%)	5 (20.8%)	20 (83.3%)	8 (33.3%)	22 (91.7%)
Apply nose clip	22 (91.7%)	20 (83.3%)	23 (95.8%)	23 (95.8%)	22 (91.7%)	22 (91.7%)
Top strap	14 (58.3%)	16 (66.7%)	17 (70.8%)	22 (91.7%)	16 (66.7%)	16 (66.7%)
Bottom strap	11 (45.8%)	14 (58.3%)	17 (70.8%)	23 (95.8%)	18 (75%)	17 (70.8%)
Seal check	1 (4.2%)	1 (4.2%)	18 (75%)	14 (58.3%)	5 (20.8%)	7 (29.2%)
Hand hygiene after	0 (0%)	2 (8.3%)	0 (0%)	16 (66.7%)	4 (16.7%)	10 (41.7%)
Doffing hand hygiene before	1 (4.2%)	1 (4.2%)	0 (0%)	17 (70.8%)	3 (12.5%)	15 (62.5%)
Only strap	10 (41.7%)	7 (29.2%)	17 (70.8%)	23 (95.8%)	8 (33.3%)	14 (58.3%)
Gentle movement	11 (45.8%)	12 (50%)	14 (58.3%)	22 (91.7%)	10 (41.7%)	14 (58.3%)
Hand hygiene after	7 (29.2%)	13 (54.2%)	6 (25%)	21 (87.5%)	8 (33.3%)	20 (83.3%)

Fig 4

Total scores averaged and plotted (scored by research team).

In the RP intervention group, expert scoring was compared to the self-scoring of the participant. The participants perceived they were better at performing the N95 donning and doffing than the experts scored them. The difference in scoring was significant at P = .002 using a paired samples test.

Discussion

This study investigated respiratory protection education in the context of HCW safety using 2 randomized educational interventions measured both immediately and longitudinally. The approach in this study is novel because of the health care worker population studied (ie, ancillary services in addition to nurses) as noted by recent systematic reviews in this area. The study was designed to be a “no lose” situation as either intervention would be practical and relatively easy to implement.

The CSBST is a tool to measure the performance of HCWs on a specific targeted safety behaviors. The addition of a RP intervention made a measurable and sustained difference in demonstrated safety behavior, while only taking participants 5 minutes longer than the VA intervention. As expected, the performance quality in both groups did decrease at follow-up which provides information on skill degradation at an earlier time point than previous studies. The findings build on Harber's work which found video to be the marginally better learning tool when training on respirator use. Our population was specifically HCWs who had some level of confidence and previous experience using N95 respirators for patient care, so a pretest allowed for the evaluation of typical behaviors without training.

The video used in the study focused on safely using a filtering facepiece N95 respirator but did not include detailed context for the health care setting specifically. The CSBST used for video scoring included several metrics on hand hygiene that may have impacted the participants’ reflections and ultimately their performance scores. This may indicate that scoring tools for clinical skills and other health care training programs should very specifically include the critical moments of hand hygiene and rationales for them.Ultimately, a study design where the educational materials are directly patterned off of the scoring tool would best demonstrate if the RP intervention truly made the difference in behavior. Nevertheless, the VA alone intervention reflects the currently available materials for most hospital RPPs at this time.

Specific to filtering facepiece N95 respirators, some additions to the CSBST may be warranted for future studies. Applying a respirator correctly requires personal hand contact with the face and ungloved hands to feel for air leaks. The hands should not be considered clean enough to provide patient care afterward. Removal can be complicated by several factors, but clean hands should be used for removal, and hands should be cleaned after contact with the respirator straps. Additional considerations from the video scoring might include donning location, doffing location (eg, performed inside or outside of patient room), hair management, and facial hair status. These data are more complex and may need to be captured as a different level of measurement than a simple point value.

There were several limitations of our study. The study participants were from one Midwestern hospital network. Our sample included only 4 males and no physicians. The only data collected on behavior was based on a simulated experience and did not include a component of real-life patient care in the clinical setting. The study used an existing educational video from the CDC on the use of N95 respirators. Follow-up visits were targeted for 1 month after the initial visit, which limits the long-term retention conclusions which can be made from the study. Our study provides more information about the time period after training when performance decreases, expanding the work by Harber.

Health care has become more safety focused with increased attention to novel infections, pandemic, hazardous drugs, and nosocomial transmission in recent years. This study was conducted prior to the COVID-19 pandemic. The lack of critical PPE supplies internationally presented unique challenges. Unexpected educational needs arose with the pandemic including the need for training on extended use and limited reuse of N95 respirators. While the content of instruction may be altered by this global experience, the educational strategies demonstrated in this study would likely remain effective.

This study design could be altered to be applicable to several clinical tasks requiring good personal protection skills with the development of a similar CSBST. Further and more systematic investigation is needed with representative samples of multiple health care professions to determine when the knowledge loss occurs on these clinical tasks, perhaps with weekly recorded simulation activities over a longer period. Future studies should implement and assess the feasibility of a RP intervention by examining clinical practice outcomes related to safety behaviors to determine if these behaviors in simulation are better or worse in the construct of clinical practice demands. These practical interventions and tools to measure performance may substantially improve the safety of our health care workforce in the care of patients with infectious respiratory pathogens as well as other clinical tasks that require use of the N95 respirator.