The association between self-perceived proficiency of personal protective equipment and objective performance: An observational study during a bioterrorism simulation drill.

BACKGROUND: The recent Ebola virus disease outbreak emphasized the potential misuse of personal protective equipment (PPE) by health care workers (HCWs) during such an event. We aimed to compare self-perceived proficiency of PPE use and objective performance, and identify predictors of low compliance and PPE misuse.
METHODS: An observational study combined with subjective questionnaires were carried out during a bioterror simulation drill. Forty-two observers evaluated performance under PPE. Mistakes were recorded and graded using a structured observational format and were correlated with the subjective questionnaires and with demographic parameters.
RESULTS: One hundred seventy-eight HCWs from community clinics and hospitals were included. The mean self-perceived proficiency was high (6.1 out of 7), mean level of comfort was moderate (4.0 out of 7), and mean objective performance was intermediate (9.5 out of 13). There was no correlation between comfort and objective performance scores. Self-perceived proficiency was in correlation with donning and continuous performance with PPE but not with doffing. Clinic personnel performed better than personnel in hospitals (40.3% vs 67.8% with 3 or more mistakes, respectively; P = .001). Demographic characteristics had no correlation with objective or self-perceived performance.
CONCLUSIONS: Self-perceived proficiency is a poor predictor of appropriate PPE use. The results suggest poor awareness of the possibility of PPE misuse.

Protection of health care workers (HCWs) by personal protective equipment (PPE) use and hand hygiene has become a standard practice in modern health care. These measures are stretched to an extreme in the uncommon events of severe contagious diseases outbreaks, which can be the result of a deliberate attack or a natural outbreak.1, 2 Examples of such outbreaks that occurred since the beginning of the 21st century are the severe acute respiratory syndrome epidemic, the H5N1 avian influenza, the 2009 H1N1 influenza pandemic, the recent Ebola virus disease (EVD) outbreak, the novel influenza A H7N9 virus, and the Middle East respiratory syndrome corona virus outbreak.

HCWs are subject to increased risk of infection during an outbreak, but can be well protected by PPE. Nevertheless, the 2009 H1N1 influenza pandemic taught us that knowledge and self-reported compliance to recommended PPE use are suboptimal among HCWs. More than 850 confirmed HCW infections were reported from Guinea, Liberia, and Sierra-Leone in the recent EVD outbreak with more than 500 reported deaths (see http://www.cdc.gov/vhf/ebola/outbreaks/2014-west-africa/index.html). Three cases of nosocomial transmission among apparently protected HCWs in Spain and the United States heightened concerns among health care professionals and decision makers.9, 10

Several studies demonstrated that overall adherence to appropriate PPE use while providing care for patients in hospitals was modest, and PPE misuse was frequent.11, 12 Factors found to be associated with appropriate PPE use were knowledge, training, perception of being afflicted with life-threatening diseases, and personal comfort.13, 14 Nevertheless, most studies were conducted in a hospital setting and were based on self-reported questionnaires without matched objective observations. Moreover, studies were not conducted in settings of a severe contagious outbreak in which participants need to combine contact, droplet, and airborne precautions.

We aimed to observe PPE use among HCWs in hospital and outpatient clinic settings to determine whether self-perceived proficiency of PPE use in the setting of a severe contagious outbreak influences objective adherence to protocols and to identify predictors of low compliance and PPE misuse.

Material and methods

Participants and setting

This was an observational study combined with subjective questionnaires. The platform was the Israeli “Orange Flame” exercise, a national preparedness buildup project conducted by the Israeli Ministry of Health aimed at improving national preparedness for large natural and bioterrorism-associated outbreaks.2, 15 During the exercise, HCWs in various health settings provide care for hundreds of patient-actors while using PPE, including a disposable gown, face shield, N95 respirator, and nonsterile gloves.

The study was conducted during November 2014 and included HCWs from 2 tertiary medical centers, 8 civilian community-based primary care clinics, and 2 military primary care clinics. Forty-two fourth-year nursing students observed and evaluated donning, doffing, and continuous performance under PPE in the different health care settings. The observers had no other tasks during the drill, and each trainee was followed by 1 observer. The observers received comprehensive education on PPE use, in accordance with the Centers for Disease Control and Prevention (CDC) guidelines (see http://www.cdc.gov/ncidod/dhqp/pdf/isolation2007.pdf) using a detailed checklist (Supplementary Table S1, available on request). The premise of using nursing students as observers is that the students are reliable judges: standard precaution and infection prevention and control (IPC) are included in the nursing core curriculum and are considered to be obligatory skills.

All trainees were HCW professionals, including physicians, nurses, medics, directors, logistics staff, and administrative staff directed to work under contact, droplet, and airborne precautions. The selection of which HCWs to observe was random. The trainees knew about the observers' participation in the drill but did not know which of them observed aspects of infection control.

The study was approved by the Institutional Review Board of the Israel Defense Forces Medical Corps and exempted from informed consent requirements because all participants were aware of the observation and could refuse to participate in the exercise or to fill out the questionnaire.

Data collection and analysis

Data were collected from participating HCWs using a structured self-administered questionnaire. This included demographic parameters, number of lectures on infection control in which the trainee participated during the past 3 years, and a subjective assessment of PPE discomfort and self-perceived proficiency of PPE use based on a Likert-type scale (where 1 = low and 7 = high). The participants were also asked to comment on their own motivational factors and obstacles to correctly use PPE (Supplementary Table S2, available on request).

The performance scores were based on the objective competency checklist and included donning score, doffing score, and working under PPE score. HCWs received 1 point for every step they correctly performed and zero if a step was incorrectly executed. This scoring technique is based on the Skills Competency Checklist for Contact Precautions of the American Association of Nurse Assessment Coordination (see http://www.aanac.org/docs/2015-ltc-leader/n-coley_capstonefinal.pdf?sfvrsn=2) and adjusted to contact, droplet, and airborne precautions requirements during a severe contagious disease, in accordance with CDC guidelines as mentioned earlier. The ranges of these scores are 0-12, 0-6, and 0-13 for donning score, doffing score, and working under PPE score, respectively. Overall, every individual participant received 3 different performance scores (for donning, doffing, and working under PPE) and each performance score was based on the sum of steps that were properly completed by the participant (Supplementary Table S3, available on request).

Statistical analysis

Data were analyzed using BMDP Statistical Software (Statistical Solutions Ltd, Boston, MA). Between-group differences of discrete variables were analyzed using the Pearson χ2 test or Fisher exact test, as applicable. Because the PPE scores did not have Gaussian distributions, they were compared using the nonparametric Mann-Whitney U test. Correlations were computed using Spearman's correlation. P ≤ .05 was considered significant.

Results

Characteristics of participants

Overall, 178 HCWs were observed in the study. Characteristics of the participants are presented in Table 1 . The average time of working with full PPE was 73.3 minutes (range, 17.0-156.0 minutes). Sixty-five percent worked in tertiary medical centers (the hospital group) and 35% worked in primary health care settings (the clinic group).

Table 1

Characteristics of participants

Characteristic	Hospitals (n = 116)	Health maintenance organizations (n = 49)	Military clinics (n = 13)	Total (n = 178)
Gender: Male	49 (42.6)	13 (26.5)	6 (46.2)	68 (38.2)
Age (y)	39.9 ± 11.4	41.6 ± 11.9	22.3 ± 5.5	38.9 ± 12.2
Physicians	18 (15.6)	12 (24.5)	2 (15.4)	32 (18.0)
Nurses	60 (51.7)	15 (30.6)	0 (.0)	75 (42.1)
Medics	1 (0.9)	1 (2.0)	8 (61.5)	10 (5.6)
Managers	3 (2.6)	3 (6.1)	3 (23.1)	9 (5.1)
Administrative staff	11 (9.5)	10 (20.4)	0 (0.0)	21 (11.8)
Logistics staff	20 (17.2)	7 (14.3)	0 (0.0)	27 (15.2)
Unknown profession	3 (2.6)	1 (2.0)	0 (0.0)	4 (2.2)
Health care workers with direct patient contact*	79 (69.9)	28 (58.3)	10 (76.9)	117 (67.2)
Health care workers without direct patient contact†	34 (30.1)	20 (41.7)	3 (23.1)	57 (32.8)
No. of participants in more than 1 lecture during past 3 years	58 (50.0)	21 (42.9)	11 (84.6)	90 (50.6)

NOTE. Values are presented as n (%) or mean ± standard deviation.

Physicians, nurses, and medics.

Managers, administrative staff, and logistics staff.

PPE scores and objective parameters

One hundred seventy-seven HCWs were observed while donning and working under PPE and 166 were observed while doffing PPE. The mean PPE scores ± standard deviations were 9.82 ± 2.63 (out of 12; median = 11), 3.69 ± 2.06 (out of 6; median = 4), and 9.49 ± 2.69 (out of 13; median = 10) for donning, doffing, and working under PPE, respectively. The most common errors regarding PPE misuse were the N95 respirator flexible bands were not fastened to the nose bridge (37.2%), gloves did not cover the wrists (26.9%), and N95 respirators did not cover the nose (20.4%). In addition, 26% of HCWs with long hair did not collect the hair while donning PPE, and 41.2% did not change gloves between patients. Moreover, doffing the PPE not according to CDC guidelines was observed in high rates with all PPE items (26.5%, 42.8%, 41%, and 35.5% for gloves, face shields, gowns, and N95 respirators, respectively). Fifty-five percent used designated placards while donning and 46% used designated placards while doffing. Using the placards improved donning and doffing sequences but did not influence the PPE scores. Hand hygiene and disinfection were observed in 164 out of 178 HCWs. Proper hand hygiene protocol was followed by 51.8% (disinfection and then drying of the hands), 16.5% disinfected the hands but did not properly dry them, and 31.7% did not follow the hand hygiene protocol at all. Proficiency scores were significantly higher for participants in the clinics group compared with the hospital group, as reflected by fewer mistakes observed in donning, doffing, and working under PPE scores (P ≤ .001) (Table 2 ).

Table 2

Personal protective equipment (PPE) scores in relation to the organizational affiliation of participants. The hospital group includes participants in tertiary health care settings. The clinic group includes participants in primary health care settings

Score	Hospital group	Clinic group	P value
Donning score (n = 177)
12 (no mistakes)	30 (26.1)	32 (51.6)	< .001
10-11 (1-2 mistakes)	37 (32.2)	21 (33.9)
0-9 (≥3 mistakes)	48 (41.7)	9 (14.6)

NOTE. Values are presented as n (%).

No association was found between the PPE scores and gender or number of prior lectures (data not shown). In addition, no correlations were found between age and PPE scores (r = 0.03, r = 0.03, and r = −0.05 for donning, doffing, and working under PPE, respectively). Moreover, there was no significant difference between the PPE scores of HCWs with direct patient contact (ie, physicians, nurses, and medics) to those without direct patient contact (ie, directors, administrative staff, and logistics staff) (P = .16, P = .08, and P = .13 for donning, doffing, and working under PPE, respectively).

PPE scores and subjective parameters

Overall, the subjective assessment of PPE discomfort was moderate with a mean of 4.0 ± 1.6 (median = 4) out of 7.0 (1 = highly discomfort and 7 = highly comfortable), and the self-perceived proficiency of PPE use was high, with a mean of 6.1 ± 1.2 (median = 7) out of 7.0. No correlations were found between PPE discomfort and PPE scores (r = 0.09, r < 0.001, and r = 0.16 for donning, doffing, and working under PPE scores, respectively). Moreover, we found a moderate correlation between self-perceived efficacy of PPE use and PPE scores for donning and working under PPE (r = 0.21 [P < .05] and r = 0.26 [P < .05], respectively). No correlation was found between self-perceived efficacy and doffing score (r = 0.08).

The primary subjective motivational factor for the correct use of PPE was personal and family safety. The primary subjective obstacle for the correct use of PPE was the discomfort of using a N95 respirator (Table 3 ).

Table 3

Primary motivational factors and obstacles for using personal protective equipment (PPE) correctly

Motivational factor	n (%)	Obstacle	n (%)
Perception of contracting a life-threatening disease	101 (56.7)	N95 respirator discomfort	45 (25.3)
Appropriate guidance	19 (10.7)	General discomfort	34 (19.1)
Understanding the importance of practice for treatment quality	18 (10.1)	The participant mentioned that there were no obstacles	19 (10.7)
Compulsion of the directing echelon	11 (6.2)	Face shield discomfort	14 (17.9)
Willingness to strengthen knowledge and skills	8 (4.5)	Crowdedness and stress while donning PPE	9 (5.1)
Prevention of illness from patients and other health care workers	6 (3.4)	PPE donning order	6 (3.4)
Importance of public health	6 (3.4)	No former experience	6 (3.4)
Supporting atmosphere	2 (1.1)	Gown discomfort	4 (2.2)
Other	7 (3.9)	Gloves discomfort	4 (2.2)
		Lack of supportive atmosphere	4 (2.2)

Discussion

HCW protection is essential for emergency response, preparedness, and efforts to build a resilient health system in emergencies such as a severe contagious outbreak (see http://apps.who.int/iris/bitstream/10665/171823/1/WHO_EVD_SDS_REPORT_2015.1_eng.pdf?ua=1&ua=1). This includes not only medical personnel, but also nonmedical staff within the health care setting, such as managers, administrative staff, and logistics staff. Despite recommendations issued by international organizations dealing with infectious diseases such as the World Health Organization, there is still insufficient evidence to draw conclusions regarding the comparative effectiveness of various types of PPE.

It seems clear that even after implementing PPE and other barrier measures during a severe contagious outbreak, there are still cases of transmission. Possible shortcomings include insufficient training and supervision on proper practices such as donning and doffing PPE, with emphasis on safe removal of contaminated clothing, shortage and improper use of PPE, extensive working hours due to shortage of medical personnel, and spread outside the patient care setting.16, 17, 18 Using adequate PPE during patient care is important to prevent HCW infection, to allow proper patient care, and also to ensure comfort and safety of caregivers, because wearing PPE increases the risk of heat stress and the loss of dexterity.13, 14, 16

Several studies examined PPE use by nurses; physicians; and other personnel, including students, assistants, laboratory workers, housekeeping, and administrative staff. In most cases these studies were not designed to evaluate proper PPE use, and the depiction of PPE protocol is limited. Important aspects, such as specific characteristics and quality of the PPE, disposability, quantity of protective equipment used simultaneously, methods of donning and doffing, and adherence to recommendations, are usually missing.

Gozel et al studied the compliance of HCWs with wearing PPE while treating Crimean-Congo hemorrhagic fever patients. They found the highest compliance in the infectious diseases ward. They also found a low rate of seropositivity among HCW and concluded that it was related to both the compliance and to regular education programs implemented. Although adequate PPE is important, correct donning and doffing are essential in preventing infection. Thus, lack of adequate training is an important factor for disease transmission.

In the case of EVD, studies suggested that protocol violations and inadequate training are risk factors for HCW infections, and led to transmission of the disease to HCWs.17, 20 In the recent EVD outbreak, providing training to HCWs in affected regions was identified as a “key strategy” for preventing transmission. Ongoing guidance and monitoring of HCWs through the donning and doffing procedures were implemented in Medicine Sans Frontier Ebola treatment centers.16, 18 A dedicated person was stationed for guiding the HCWs through each step of PPE removal, regardless of their previous experience.

The use of PPE is only 1 factor of infection prevention and other important factors include hand and environmental hygiene. As was shown in our study, these measures were not kept in an acceptable manner, with more than 30% of HCWs not following the hand hygiene protocol as mandated. Another relevant factor is implementing a strong safety climate by HCWs. This was shown to be a powerful tool to promote the appropriate use of PPE, including support for safety programs; communication and feedback about safety; and senior management support for safety, training, and education.

We present the results of a thorough examination of PPE use. This unique study takes advantage of an extensive national drill intended to improve and evaluate preparedness for a severe contagious outbreak. This setting allowed real-time observation of a large number of HCWs in both hospitals and outpatient clinics. The drill's scenario necessitated extreme infection control precautions, and therefore allowed evaluation of multiple key tasks, including PPE donning, doffing, prolonged use, and hand hygiene.

Based on our findings, self-perceived efficacy of protection among HCWs was high, but was a poor predictor of appropriate PPE use. PPE use was not influenced by subjective discomfort, age, gender, profession, or number of prior lectures. Placard use improved donning and doffing order, but was not followed by the correct use of PPE. Performance in PPE by primary-care HCWs was significantly better than that of tertiary-care staff. Hospital staff are more familiar with PPE, yet this may bring a sense of overconfidence while overlooking small but important details. For the community health clinic staff, this exercise dealt with a more unfamiliar scenario, and they probably paid more attention to details. The results suggest poor awareness to the possibility of PPE misuse, emphasizing the importance of a trained observer while working under PPE in the setting of a severe contagious outbreak.

Several insights for future training and exercises are suggested based on our study. More focus should be put on techniques and identified pitfalls. The N95 mask was rated as the most problematic item for trainees. It may result from the fact that it is not used in everyday work, and therefore proper use should be emphasized in future drills, and more comfortable alternatives should be sought by future research and development efforts. Trainees following the placard instructions kept the right order of donning and doffing, but still did not perform it well and PPE breaches were found. This may be the result of the way the placards are built. People usually read only the titles, and do not look thoroughly at the sentences that follow. Using larger fonts and simple illustrations may improve performance. Demonstrations are important, with emphasis on short videos using more accessible tools such as smartphones and social and organizational networks. HCWs should not rely on their self-perception, no matter how experienced they are. Simple measures, such as a mirror, can have an important influence on the way HCWs use PPE. Active observation by a designated staff member who comments on PPE use is imperative. It is important to use motivational factors, such as personal safety. Emphasizing personal and family members' safety would probably have a greater influence than emphasizing the importance of infection control and public health.

There are several limitations to our study. The exercise was preplanned and rehearsed for a few months. The trainees were well aware of being observed, and therefore might not have acted as in a real-life event. Nevertheless, we could still observe PPE misuse quite frequently. PPE was relatively at hand, probably more than in a real scenario. Despite all of that, results show troubling attitudes of HCWs, raising concerns on the way they will handle a real outbreak event.

The participating clinics were reinforced with designated personnel in preparation for the exercise. In the reality of an outbreak, we should anticipate a relative shortage in HCWs and other professionals.

The PPE used in the exercise was in accordance with the CDC guidelines for contact, droplet, and airborne precautions. However, it was not consistent with CDC recommendation for care of EVD patients, which is more stringent and was not reflected in our study. Therefore, when trying to draw implications for EVD PPE recommendations, this limitation should be considered. Nevertheless, most of our insights can still be implemented in preparedness training programs for potentially fatal infections such as EVD.

Interrater reliability in the observational phase was not discussed and is considered as another limitation of our study.

Using nursing students as observers is an advantage of our study design. For the students, the observation is actually a learning from an error process: It is a structured learning process of self and external feedback on donning, doffing, and working under PPE, helping to identify the causes leading to operational errors, understanding the relationships between actions and outcomes, and at the end of this process, it will help them implement and improve their behavior and decision-making process. Therefore, it is recommended to extend the involvement of nursing and medical students in such experiences during their professional education on IPC. It would be interesting to learn whether using such strategies could improve their future compliance with IPC—implementing it not only in drills, but also in a continuous learning process.