The effects of trained observers (dofficers) and audits during a facility-wide COVID-19 outbreak: A mixed-methods quality improvement analysis.

BACKGROUND: In response to a facility-wide COVID-19 outbreak, our tertiary acute care hospital implemented an evidence-based bundle of infection control practices including the use of audits and trained observers "dofficers" to provide real-time constructive feedback.
METHODS: We trained furloughed staff to perform the role of dofficer. They offered support and corrective feedback on proper PPE use and completed 21-point audits during a 4-week intervention period. Audits tracked appropriate signage, placement and availability of supplies (equipment), correct PPE use, enhanced environmental cleaning, along with cohorting and social distancing rates. Audit data was used to provide weekly quality improvement reports to units.
RESULTS: Nine hundred and sixty two separate audits recorded 36,948 observations, over 7,696 observer-hours. The most common errors were with environmental cleaning and PPE use; the least common were with regards to equipment availability and cohorting and social distancing. Mean error rates decreased from 9.81% to 2.88% (P < .001). The largest reduction, 22.57%, occurred in the category of PPE doffing errors.
CONCLUSIONS: Dofficer led audits effectively identified areas for improvement. Feedback through weekly reports and real-time correction of PPE errors by dofficers led to statistically significant improvements; however, error rates remained high. Further research is needed establish if these relationships are causal.

Background

Since January 3, 2020, the COVID-19 pandemic has affected over 115 million individuals leading to over 2.6 million deaths globally. The SARS-CoV-2 virus is highly infectious; causing outbreaks in numerous settings including acute and long-term healthcare facilities. SARS-CoV-2 is transmitted by respiratory droplets, aerosols, and contact (fomites). Public health and infection control measures targeting these mechanisms of transmission may help to stemming the spread of the virus. Such steps include appropriate physical distancing, the appropriate use of personal protective equipment (PPE) (including the use of face masks), enhanced environmental cleaning, and hand hygiene. , However, these relatively simple interventions can be challenging to implement and operationalize. One study demonstrated that training, understanding, and consistency of cleaning in hospitals is often sub-optimal. Hospital crowding and aging infrastructure often makes physical distancing in healthcare facilities difficult. Staff errors have also commonly been noted during the donning and doffing of PPE. ,

Bundled audit tools have been shown to improve practice with regards to many infection prevention measures, as well as improve guideline-compliant practice. To date, however, many bundles have not included trained observers (dofficers) - who can assist in guiding health care workers in the donning and doffing of PPE with the goal of preventing errors leading to self-contamination. The trained observer role was first developed in response to viral haemorrhagic fever outbreaks, however, observers may have an important role in the management of COVID-19 as well.

In response to a facility-wide COVID-19 outbreak at our acute care hospital, we developed an evidence based bundled audit which was performed by dofficers who also offered real-time corrective feedback on proper PPE use. In this evaluation, we present our process for the implementation of dofficer auditors and examine the audit data to determine which items were most amenable to influence through direct observation and feedback.

Methods and methods

Intervention setting

This intervention was conducted at a 310-bed tertiary academic suburban hospital located in Edmonton, Alberta, Canada. On June 21, 2020, several hospital-acquired outbreaks of COVID-19 were declared in the facility, resulting in multiple unit closures. The rapid rise in case numbers (58 facility wide) and furloughed staff led to a complete closure of the hospital to admissions and transfers, and cancelation of all elective procedures and outpatient encounters. Patients remained admitted and could be discharged as appropriate, and urgent surgical procedures for admitted patients continued as deemed necessary by their attending physicians. As part of the response to the outbreak, starting July 10, 2020, the hospital administration team implemented a program to train multiple dofficers and also implemented a combined evidence-based audit tool to evaluate infection control practices of healthcare staff on the multiple outbreak-affected patient-care units. Audits were conducted from July 13-August 11, 2020.

Training of dofficers

Training materials used for dofficers were adapted from the Centers for Disease Control Ebola Virus Disease trained observer programme. Training of dofficers was conducted by eight clinical nurse educators. Training consisted of a 60-minute didactic session that reviewed multiple key topics including PPE donning and doffing, providing corrective feedback, and performing infection prevention and control audits (see Supplement 1A for dofficer training checklist). The final component of training was participation in donning and doffing simulations. Dofficer training competency was confirmed using a quiz evaluation, on which dofficer trainees had to achieve a score of ≥90% (Supplement 2). Active feedback was provided to the dofficer trainee during marking of their quiz. Those receiving a score <90%, were asked to re-review training material and re-take the quiz the next day. The healthcare workers training as dofficers consisted of facility staff re-deployed from their other roles as clinical nurse educators, dieticians, medical device re-processing staff, nurses, occupational therapists, physiotherapists, respiratory therapists, social workers, transition coordinators, unit clerks, and unit managers. The primary role of the dofficers was not to train other healthcare workers but to be direct observers to monitor and actively provide feedback for errors noted in the PPE donning and doffing process when seeing patients with suspect or confirmed COVID-19.

Audit training consisted of an orientation to the audit, definitions for what constituted a pass or fail (any breach in a 4-hour period for most items), and how to submit completed forms. Following the didactic sessions, the learning objectives were re-reviewed and staff were offered the opportunity to seek clarification and practice what they learned. Audits were meant to be performed by dofficers of the unit they were deployed to on their shifts.

Audit tool and process

The audit tool (Supplement 3) was developed in collaboration with the hospital's Outbreak Response Team (ORT). The ORT consisted of individuals from the hospital infection prevention and control (IPC) team as well as managers representing all the different departments in the hospital (eg, Emergency, Medicine, Women's Health, Surgery, Critical Care, etc.). The tool was developed after a comprehensive review of literature.5, 6, 7, 8 , 11, 12, 13, 14, 15, 16 Assessment categories that were supported by this review included ensuring appropriate signage, PPE equipment supplies, PPE use, cleaning, as well as cohorting and social distancing. “Signage” is meant to ensure that posters indicating a patient is on precautions are clearly posted, as well as graphics that take staff through the individual PPE donning/doffing steps are available at each PPE station. “Appropriate PPE equipment” refers to ensuring that PPE stations are set up correctly and supplies readily available and regularly re-stocked. “PPE use” refers to ensuring the appropriate steps for donning and doffing of PPE are used as well as consistent use of continuous masking and eye protection in non-patient areas as well. The “cleaning” category audited whether enhanced cleaning schedules and proper terminal cleans of patient rooms was conducted in accordance with institutional IPC guidelines. Lastly, “Cohorting and social distancing” valuates if patients with suspected COVID-19 were preferentially placed in single rooms, and that appropriate social distancing principles were followed for patients in multi-bed rooms and by staff in non-patient care areas whenever possible. Supplement 4 includes information regarding which specific audit points are categorized within each of these five categories.

The audit tool adopted a dichotomous (pass/fail) assessment of 21 items (18 of which were assessed every four hours; three of which were assessed every eight hours). A single breach during the assessment interval resulted in the recording of a fail. All 21 fields had a section provided for additional comment. Staff members working in the capacity of a dofficer were to complete an audit form that recorded the date, auditor name, time of the shift, and the unit on which they were working with all forms submitted to the ORT office at the end of each shift.

Data analysis and statistical evaluation

Audit data were extracted and tabulated using Microsoft Excel (Microsoft Corporation, Redmond, USA). Dichotomous variables were binary coded and any free-text was copied verbatim. Data analysis was performed using IBM SPSS Statistics Version 25 (IBM Corporation, Armonk, USA). Descriptive statistics are reported as means and standard deviations. Welch's ANOVA with Games Howell post hoc testing were used to compare means for repeat measures. Post hoc testing was limited to items that were found to be statistically significant on ANOVA testing. A chi-square goodness of fit test was performed to examine remaining non-repeat measure categorical variables (namely, whether the difference between observed night-time and day-time audit sheet completions was statistically significant). All testing was two-tailed with predefined significance set at P < .05.

Qualitative data analysis

Narrative comments collected during the first week of the intervention period were thematically analyzed by 2 authors. Continuous open coding of the earliest returned audit forms was performed until 20 consecutive audit forms were extracted without generating any additional codes. Primary codes were then grouped into 9 error categories and 14 provider codes. These codes were used for all subsequent coding (Supplements 5 and Supplement 6).

Real-time feedback and administrative action

Audit forms were assessed by one of the authors who also worked in ORT. Quantitative audit metrics were analyzed in real time and dashboards with data visualizations of deficiency types was generated on a weekly basis over the implementation period. Data were disseminated to frontline care providers through a designated ORT member who liaised with each unit. The ORT liaison assisted each unit to develop specific improvement targets with concrete outcomes and specified timelines for improvement. Lack of improvement led to additional follow-up and increasingly prescriptive guidance. Feedback about providers’ behaviors were handled by their respective leadership teams. For example, physicians received feedback from the site IPC medical lead, unit staff from their managers, and ancillary staff from their respective departments.

Ethics approval

Approval for use of the data from this intervention implementation and evaluation was obtained from the Human Research Ethics Board at the University of Alberta (study identifier Pro00103046).

Results

Trained observers ‘dofficer’

All staff who successfully completed training and passed the quiz without remediation became dofficers. In total, 186 dofficers were recruited, trained, and deployed on hospital units to perform audit and feedback. All dofficers completed at least one audit form during the intervention period; some audit forms were completed by more than one dofficer. A total of 962 forms were returned.

Baseline audit data

Over the evaluation period, 36,948 observations were recorded on 962 audit forms (Table 1 ). The mean number of assessment items completed per form was 38.41 (SD 12.62) with a mean of 1.95 (SD 2.27) comments per audit form. On average, 32 (SD 10.69) forms were received per 24-hour audit period. Cumulatively, dofficers conducted 7696 hours of auditing during the implementation period. Significantly fewer audits were done during the evening shift (1901-0700hrs) compared to the daytime shift (0701-1900hrs) [423 vs 539, P < .002].

Table 1

Description of auditors and audit forms

Item	Value (SD)
Number of dofficers who returned at least one form	186
Total number of forms returned	962
Number of observation hours	7696
Mean daily return rate for forms	32 (10.69)
Mean number of audit points observed per form	38.41 (12.62)
Mean number of comment fields completed per form	1.95 (2.27)
Mean number of words per shift	32.85 (44.53)
Daytime shift audits	539
Evening shift audits	423

Abbreviations: SD, standard deviation

Deficiency rates

The most commonly noted error categories were noted with proper PPE use and environmental cleaning (Table 2 and Fig 1 ). The least common sources of error were in equipment availability and cohorting and social distancing (Table 2). There were significant reductions in global error rates (9.81% to 2.88%; P < 0.001) over time. The most significant error reductions were noted in PPE doffing which saw error rates reduced by a mean of 22.57% (SD = 3.78%) (Supplement 4). However not every individual audit item saw significant improvements (Supplement 7).

Table 2

Mean error rates by category

	Week 1		Week 2		Week 3		Week 4		Total Mean	P-value
Error Category	N	Mean % (SD)	N	Mean % (SD)	N	Mean % (SD)	N	Mean % (SD)	(%) (SD)
Signage	146	6.21 (10.68)	195	3.76 (6.95)	307	2.69 (9.14)	313	1.88 (5.55)	3.18 (8.11)	<0.001
Equipment	146	4.01 (9.17)	195	1.44 (5.20)	307	2.13 (6.69)	313	1.51 (5.77)	2.07 (6.64)	0.013
PPE use	147	7.18 (8.09)	195	5.26 (7.63)	307	5.01 (7.18)	313	3.23 (6.43)	4.81 (7.30)	<0.001
Cleaning	125	19.50 (34.16)	187	4.65 (18.00)	282	4.77 (19.17)	293	1.39 (9.42)	5.70 (20.26)	<0.001
Cohorting and social distancing	135	7.60 (19.26)	192	4.25 (13.54)	292	1.68 (8.70)	296	1.07 (6.57)	2.90 (11.66)	<0.001

N, refers to number of observations in that category; SD, standard deviation.

Figure 1

Grouped errors over time.

Narrative comments

A total of 981 narrative fields were examined. Most staff who offered narrative feedback did so in one of 2 ways: the first was to briefly describe the nature of the error using one or two words. Typically, these entries addressed the nature of the error and read as follows “nurse did not tie waist tie.” The second way narrative feedback was offered was through a thorough description of overall performance which occurred in the additional comments section. These had greater amounts of text and thematic or provider codes (Supplement 8).

Discussion

The implementation of a dofficer combined with an infection control audit were used to identify, evaluate, and communicate potential deficiencies in day-to-day clinical procedures with the goal of positive reinforcement and continuous quality improvement. This was done in the setting of a facility-wide COVID-19 outbreak that led to hospital closure and cessation of all ambulatory, emergency, and elective operations. Through this quality improvement intervention, we were able to identify a number of key areas in which significant improvements were measured over the 4-week time period.

Although previous studies have examined PPE donning and doffing error rates and the effect of dofficers on doffing breaches, most studies take place in simulated environments, use small samples, focus on specific tasks, and assess materials not used with COVID-19.18, 19, 20 The majority of studies examining the utility of dofficers are associated with caring for patients with Ebola virus disease (EVD). These studies examined fewer than 50 interactions in simulated EVD settings. , While they offer insights into where doffing errors are likely to occur, they are limited in their generalisability because assessments were conducted in controlled simulation environments. By comparison, our intervention examined 962 audits using 7696 dofficer hours in a real-life clinical setting. The doffing episodes observed in our evaluation reflect true-to-life uncontrolled clinical realities such as wandering patients, competing workloads, and working with night-shift staffing levels. Healthcare providers in this intervention wore droplet and contact precautions PPE (eg, surgical mask, gown, gloves, face shield or goggles) as opposed to EVD PPE which makes our results more applicable to typical hospital conditions. We believe this evaluation offers insight into the pragmatic effects of dofficers on PPE doffing errors during clinical workflow, something that has not been done in relation to COVID-19. BecauseCOVID-19 transmission may be linked to fomite transmission, , efforts aimed at reducing PPE doffing error rates will likely limit iatrogenic spread to providers and patients alike.

Despite significant improvements over the time of the intervention, there were persistently high rates of errors while donning and doffing PPE (with rates being ≥ 10% for both procedures) (Supplement 4). High error rates ranging from 39-90% have been noted in a number of other studies evaluating doffing of PPE for non-COVID-19 conditions (such as respiratory viral infections or antibiotic resistant organisms). , Our intervention however did demonstrate more than 50% reductions in the errors noted for both donning and doffing processes. Furthermore, our data highlights the importance of dofficers to help direct the PPE donning and doffing process. This is critical given previous published data highlighting self-perceived proficiency is a poor predictor of proper PPE use and use of dofficers providing verbal directions in COVID-19 settings is important to help support safety and quality of care. ,

Previous research has found that a bundled approach to addressing COVID-19 transmission on a respiratory medicine unit at a Singaporean hospital can be successful. The study implemented a bundle including infrastructure enhancements, PPE audits, social distancing requirements, masking of patients when not eating, and enhanced environmental cleaning protocols. Our COVID-19 bundle included many of these interventions but also recommendations regarding appropriate signage and scheduled cleaning of staff workspaces. Critically, we implemented active, real-time feedback to improve deficiencies which has not previously been described in the COVID-19 outbreak response literature.

The major strength of this intervention lies in the large number of audit points that were assessed across multiple departments at all times of the work day, and during all aspects of patient care on multiple unit types (eg, medicine, surgery, rehabilitation etc.). The bundled nature of the intervention allowed us to examine individual components that were deficient and provide feedback that could be tailored and constructive – highlighting the deficiency while providing positive reinforcement based on the successes.

Our intervention has several limitations. Given the rapid need to respond to the facility-wide COVID-19 outbreak, instructional materials for dofficer training were not pre-tested on staff, and instructor standardisation was not formally assessed. Eight instructors delivered training to a range of providers, thus the data was collected by a broad range of observers, which may have introduced heterogeneity in how the audit tool was applied and reported. Furthermore, given the nature of our intervention, we are unable to determine if observed error rates were causally linked to intervention or due to changes in infection rates or fomite transmission. A number of audit forms had data missing or were illegible, which may decrease the reliability of the audit results. Because multiple episodes of PPE donning and doffing could occur in each 4-hour interval, we cannot establish a per episode error rate and our results only reflect whether the intervention yielded an improvement. Likewise, the grouped nature of the donning and doffing process precluded us from determining where in the process the errors occurred (example.g., with hand hygiene versus equipment).

Conclusions

In conclusion, we implemented a COVID-19 dofficer training program in conjunction with a dofficer-driven audit and feedback tool. Overall, the implementation noted a significant decrease in healthcare worker error rates associated with posting of signage to indicate need for precautions and provision of graphics to provide instruction, PPE provision and station setup, PPE donning and doffing processes, cleaning in patient and staff areas, as well as social distancing of staff and patients when possible. The use of dofficers, aside from helping to decrease donning and doffing errors, may offer additional protection to both staff and patients. Additional research is required to determine if this additional protection is clinically significant and if the model is financially sustainable.

Declarations

Approval for use of the data from this intervention implementation and evaluation was obtained from the Human Research Ethics Board at the University of Alberta (study identifier Pro00103046).

Funding

This work did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors. Resources and funding to implement and evaluate the intervention presented were provided in the context of hospital response to a COVID-19 outbreak.

Conflicts of interest

All authors have filled out an ICMJE form to make any declarations. None of the authors have conflicts of interest relevant to this manuscript to declare.

Credit author statement

Christopher Picard – Conceptualization, methodology, formal analysis, investigation, data curation, writing – original draft, writing – review and editing, project administration, resources management, and correspondence.

Meghan Edlund – Data curation, formal analysis, writing – original draft, methodology, formal analysis, and project administration, Writing – review and editing.

Candice Keddie – Writing – review and editing, project administration, resources, supervision, and funding acquisition.

Leyla Asadi – Validation, formal analysis, resources, data curation, writing – review and editing.

Domhnall O'Dochartaigh – Conceptualization, Writing – original draft, writing – review and editing.

Richard Drew – Writing – original draft, writing – review and editing

Matthew J Douma –Writing – review and editing.

Conar R O'Neil – writing – review and editing

Stephanie Smith – writing – review and editing

Jamil N Kanji – validation, writing – original draft, writing – review and editing