Using a Second Stakeholder-Driven Variance Reporting System Improves Pediatric Perioperative Safety.

Despite recognizing the occurrence of variances, we noted a low rate of reporting with the established computer variance program. Therefore, we developed and introduced a simple, handwritten variance reporting system. The goal of this study was to compare our pediatric perioperative handwritten variance cards to our established computerized variance reporting system.
METHODS: We developed a handwritten variance card program through a stakeholder-driven quality-improvement initiative. We collected variances from handwritten cards in 4 perioperative locations and also from the established computerized variance system. We analyzed the variances and categorized them into 6 safety domains and 5 variance categories.
RESULTS: Over 6 consecutive years, 3,434 variances were reported (687 computerized and 2,747 handwritten). For safety domains, the computerized system was more likely to capture adverse events and near-misses (8.7% vs. 1.1%, P < 0.001; 23.5% vs. 8.6%, P < 0.001, respectively) while the handwritten system was more likely to identify the safety process and other non-safety issues (20.1% vs. 38.3%, P < 0.001). Both systems addressed policy/process issues most often, with 37.9% of the handwritten cards and 66.6% of the computerized variance reports. Of the handwritten cards with a patient identifier (n = 1,407), only 5.1% (n = 72) also had a computerized variance filed about the same event. Thus, staff reported >1,300 additional variances that were not identified with the computerized variance system alone.
CONCLUSION: The handwritten, stakeholder-driven variance reporting system was essential to identify local and system issues that would not have been identified by the computerized variance reporting system alone.

INTRODUCTION

Medicine is a high-risk profession where adverse events occur in up to one-third of hospital admissions.[1] Other high-reliability organizations, such as aeronautics and nuclear power, have established comprehensive processes to sustain remarkable levels of industrial safety. The healthcare industry has not achieved the same success. In 1999, the Institute of Medicine released the landmark report To Err is Human: Building a Safer Health System, which stressed that adverse event reporting and recognition of near-miss events are an essential part of improving patient safety.[2] While a majority of hospitals in the United States use a centralized incident reporting system, multiple studies show that both adverse events and near-misses remain vastly underreported.[3-5]

Recent studies have shed light on barriers to effective use of an incident reporting system.[6] In a recent manuscript, 11 international experts on incident reporting noted 5 persistent system challenges—including inadequate reporting processes, lack of adequate physician engagement, insufficient action, inadequate funding and institutional support, and a failure to capture evolving health information technology.[7] Even with a robust variance reporting system in place, individual barriers to submitting variances also exist. These may include time commitment, fear of personal or professional repercussions, lack of knowledge about what is appropriate to report, lack of feedback, and disbelief that the system will change as the result of reporting.[8-12]

In addition to reporting of adverse events, incident reporting systems can also be used to evaluate near-misses, defined as unplanned events that could have harmed a patient, but did not due to chance or mitigation.[13] Like adverse events, near-misses are also vastly underreported. In our previous study, with direct observation of 211 surgical procedures, we identified 137 near-misses. While employees did report some near-misses, our hospital variance system did not identify the vast majority of the observed near-misses.[14] Constructing systems that will promote the effective use of variance reporting of both adverse events and near-misses can allow healthcare organizations to both identify and prevent variances that can lead to medical errors.

Although we already had a computerized variance reporting system in place as part of the hospital-wide safety program, we noted many adverse events, process problems, and near-misses were not being captured in the computerized system by employee self-reporting. We identified barriers to incident reporting and then devised a simple reporting system to overcome some of these obstacles. We implemented a stakeholder-driven, handwritten variance card system to increase the self-reporting of adverse events, near-misses, and other safety observations. We hypothesized that our local, handwritten variance reporting system would capture additional incidents that would not be reported in our computerized variance system.

METHODS

The Children’s Memorial Hermann Hospital is an affiliate of McGovern Medical School at the University of Texas Health Science Center in Houston. It is the only pediatric hospital in a 16-facility hospital system. Approximately 6,500 operations are performed annually in this academic, 278-bed children’s hospital.

First, a multi-disciplinary focus group of surgeons, anesthesiologists, nurses, and surgical technologists who work in the perioperative setting met to identify barriers to incident self-reporting at our hospital. The 2 most commonly identified barriers were a lack of time and a fear of repercussions secondary to a lack of anonymity. Through a quality-improvement initiative that addressed both of these issues, we developed a confidential handwritten variance card system. We placed 4 Health Insurance Portability and Accountability Act-compliant, locked boxes into select pediatric perioperative locations—the operating room hallway, the post-anesthesia care unit, the preoperative area, and the patient registration area. Handwritten cards asked, “What happened?” “Where did this occur?” and “What would you do to fix this?” There was space for open-ended answers and an option to leave a name and contact if the individual wanted to be included in the improvement process. All perioperative caregivers, including attending physicians, residents, fellows, medical students, nurses, and surgical techs were educated about the purpose of the handwritten cards and had access to the variance cards at all times. Participants who wrote variance cards could choose to maintain anonymity, be identified, or volunteer to participate in the quality improvement process. A non-biased clinical research coordinator transcribed handwritten variance cards to a spreadsheet for analysis, keeping variances anonymous. Every week, a multidisciplinary committee consisting of a surgeon, an anesthesiologist, an operating room nurse, a preoperative or recovery room nurse, a surgical technologist, the surgical quality coordinator, and our clinical coordinator reviewed the handwritten cards. We classified each variance into variance categories and safety domains during this 1-hour, weekly meeting. We developed the variance categories and safety domains at the onset of the handwritten card project to have a general overview of the types of variances that staff submitted. Each variance was sent to various departmental leaders and surgical leadership. For example, variances about availability or timing of blood products were sent to the manager of the blood bank. Variances about availability and condition of surgical instruments were sent to the manager of the sterile processing department, while performance and behavioral variances were sent to individual managers. We sent timely feedback to those who wish to be involved and gave monthly reports to the perioperative staff. This approach helped to provide loop closure at the local level.

Safety-related events described on the handwritten variance cards were further discussed at our weekly Surgical Safety Council, which is a multidisciplinary group represented by surgery, anesthesia, nursing, pediatrics, neonatal intensive care unit, pediatric intensive care unit, pediatric emergency room, and hospital leadership. We developed many of our quality improvement projects stemming from the variance cards at this meeting. During the most recent year of the program, we also began to enter the handwritten variances into the computerized variance system. This documentation allowed tracking of the types of variances submitted throughout the hospital and provided loop closure at the hospital level. It required a weekly commitment of 2 hours of meetings, 1–2 hours to enter the variances, and time for managers to investigate the variances. All perioperative leadership supported this investment of time and effort.

For the electronic variances, we used the hospital-wide VRS variance reporting program (RL Solutions, Cambridge, Mass.). This computer program allows every person with access to a hospital computer the ability to report variances in multiple categories. This reporting system requires the inclusion of a medical number for the patient involved and requires more than a dozen fields to be filled for each variance. The variance reporter may click a box to remain anonymous if desired. These reports are sent electronically to a centralized pediatric risk manager, and the director of pediatric patient care also reviews all electronic variances in the children’s hospital. From there, the electronic variances are categorized into a quality review, filter committee review, and others. Nursing leadership performs the quality review and brings issues back to nursing management. More serious, select electronic variances are analyzed a weekly multidisciplinary filter review committee comprised of physicians, nurses, and hospital leadership. There is no formal feedback system, and only the individuals directly involved in the variance are part of the analysis.

For this project, we analyzed and compared the pediatric perioperative computerized variances to the handwritten perioperative variance cards. We analyzed and categorized both the handwritten and electronic variances into 5 variance categories, including Adverse Event, Good Catch/Near-Miss, Safety Process, Non-safety, and Indeterminate. We classified all electronic and handwritten variances into 6 safety domains including Equipment/Supplies, Knowledge/Attitude, Policy/Process, Environment, Operations, and Unable to Categorize (due to lack of information). Tables 1 and 2 explain each category and provide examples. These categories were developed before the implementation of the handwritten cards, to allow tracking and trending to the types of variances being submitted. Differences between groups were assessed using Chi-square tests. All analysis was performed using Stata 13.1 (College Station, Tex.).

Table 1.

Examples of Variances by Variance Category

Table 2.

Examples of Variances by Safety Domains

RESULTS

Over 6 consecutive years (2012–2017), 3,434 variances were submitted in the pediatric perioperative area, with 687 (20%) computerized variances and 2,747 (80%) handwritten variances. The handwritten system was sustainable throughout the 6 years.

For each year of the program, the number of handwritten cards surpassed the number of electronic cards. There was little overlap of variances reported in the handwritten and electronic variance systems (Fig. 1). Only 72 of the 1,407 (5.1%) handwritten cards that included a patient medical record number also had a computerized variance submitted about the same incident. Thus, at least 1,335 variances were captured by the handwritten variance system that may have otherwise gone unreported.

Fig. 1.

Comparison of handwritten and electronic variances, with and without medical record numbers (MRN).

For safety domains, the computerized system was more likely to capture adverse events than the handwritten system (n = 59, 8.7% vs. n = 30, 1.1%, P < 0.001). The computerized system also captured a higher proportion of near-misses (n = 162, 23.6% vs. n = 232, 8.6%, P < 0.001). On the other hand, the handwritten system was more likely to identify safety process issues than the computerized variance system (n = 1,052, 48.4% vs. n = 138, 20.1%, P < 0.001). There was no difference in the percentage of non-safety cards in the handwritten card vs. electronic system (n = 1,306, 48.4% vs. n = 315, 45.9%, P = 0.23). Regarding the types of variance categories, the handwritten cards were more likely to address equipment issues (n = 383, 14.5% vs. n = 49, 7.3%, P < 0.001) (Fig. 2). The handwritten variances were also more likely to address knowledge/attitude variances (n = 978, 37% vs. n = 131, 19.6%). The electronic variances were more likely to be about policy/process issues (n = 458, 68.5%, vs. n = 1,040, 39.4%).

Fig. 2.

Comparison of types of handwritten and electronic variances.

As a result of our stakeholder-driven, handwritten variance reporting system, we implemented many new safety programs within our children’s hospital. Some of our initial cards alerted our surgical safety council to equipment shortages that affected patient safety. Other handwritten variance cards alerted our surgical safety council to patient care issues that resulted in policy change within the hospital. Table 3 highlights examples of concrete changes in policy and protocol. As an example, several preoperative nurses submitted handwritten cards, which noted that parents were unhappy that their children were not allowed to eat or drink (NPO) for an extended period. This concern led to an audit of the time patients were NPO before their surgical procedure. We found that many children were NPO for longer than necessary. Thus, we created an “NPO per pediatric guidelines” order set in the electronic medical record. This order set allowed consumption of clear liquids up to 2 hours before the planned surgery as opposed to the previous “NPO after midnight.”

Table 3.

Examples of Quality Improvement Projects Developed from the Handwritten Variance System

DISCUSSION

With our stakeholder-driven incident reporting system, we intended to break down some of the traditional barriers and improve the safety culture in the pediatric perioperative setting. The existing computerized variance reporting system was designed at the system level and was analyzed by hospital quality leadership, predominantly individuals who did not work in the perioperative area. Staff were introduced to the computerized variance system during their orientation, and the nurses were required to report adverse events that occurred while a patient was in their care. Interestingly, physicians do not receive training in the electronic variance system as part of their hospital orientation. There was a lack of trust in the electronic system, a fear of repercussions for reporting, and a lack of time to fill out the lengthy required components of the electronic variance report. We created the handwritten variance cards to supplement the existing computerized variance reporting system in a way that would foster trust and promote variance self-reporting. By including input from various stakeholders within the perioperative setting—including nurses, physicians, technologists, and hospital leadership—we were able to assess some factors specific to the perioperative staff that would decrease variance self-reporting. Fear of repercussions for reporting, which has also commonly been shown to be a common deterrent to voluntary incident reporting, was common in our stakeholders.[11,15] To address this fear, we established trust with the participants in the program, ensuring participants that every card would be reviewed and would remain confidential. Individuals who submitted handwritten variance cards could choose to be a part of the improvement process or prefer to remain anonymous. Most chose to remain anonymous. To further foster transparency, we also invited interested individuals to observe the analysis of the handwritten variance cards confidentially. The perioperative staff on the safety card committee also continually changes, allowing more individuals to become involved in the process. While some of the content of the variance submissions resulted in individual consequences or policy change, no person who submitted a variance was penalized for submitting a handwritten variance card.

As commonly stated in the literature, our stakeholders noted that one of the biggest barriers for incident reporting was that the process took too much time.[6,16,17] A successful variance reporting system must be simple, allowing easy reporting that can be completed by all staff. Although simple, variance reporting must be thorough enough to identify unsafe conditions.[12,18] Our electronic variance reporting system was time-consuming, requiring a computer login and completion of more than a dozen required fields for each variance. The simple handwritten cards were quick to complete, requiring as little as 1 sentence.

With the handwritten variance card system, we also addressed 2 other common barriers to incident reporting, including a lack of feedback and a disbelief that the system would change as a result of variance reporting.[19,20] In a report of 527 residents at a major urban hospital, 63% of residents reported exposure to adverse events and 77% of residents’ exposure to near-misses. Only 43% had ever filed a computerized incident report.[17] Some staff members did not believe that submitting computerized variances would either change the system or prevent recurrence of problems. With the handwritten card system, individuals who identified themselves on the variance cards received direct feedback from their immediate supervisor about the actions generated from the submitted card. We regularly reported summaries of outcomes from the handwritten variance system at our weekly surgical safety council, quarterly surgical council meeting, weekly operating room staff report, and at our biannual multidisciplinary surgical safety workshops. We kept the analysis, feedback, and resultant policy and process changes at the local level and continued to involve the stakeholders. In contrast, the computerized variances were distributed by email to various managers within the children’s hospital. No formal feedback system existed for this system, leading to the belief that little resulted from computerized incident reporting.

While we have had success in capturing many more safety concerns in our hospital, variance reporting remains severely underreported nationwide. In an analysis of observed and reported adverse events on the surgical service of an academic hospital, staff reported only 25% of complications and 75% of inpatient deaths.[21] In our institution, the number of reported variances has greatly increased with the implementation of the handwritten variance reporting system. The vast majority of computerized incident reporting at our hospital and other facilities is by nursing staff.[9,11,22] In contrast, a combination of nurses, surgeons, anesthesiologists, surgical techs, and others submitted handwritten safety cards. This system has resulted in a change of culture in our perioperative services, allowing more individuals to participate in perioperative quality and safety. Traditionally, physicians do not receive training in variance reporting. Instead, they report complications and adverse events at their departmental morbidity and mortality conference, not the hospital variance system. Physicians may also not understand what types of unplanned events in their practice merit submission of a variance. We have included physician education about variance reporting at our biannual surgical safety conference and recently added it to our physician onboarding process for the operating room. Patient safety and quality improvement have now also been incorporated into the medical school and residency curriculum. We hope that early training in patient safety and quality improvement will begin to change the culture of safety among physicians.

To facilitate the coexistence of the 2 programs, we strived to find a way to have loop closure for variances and allow for analysis of trends in hospital variances. Thus, we continued our local follow-up and also have recently begun to enter the data from the handwritten variance cards into the electronic system. In addition to the review by the safety card committee and perioperative management, each handwritten variance then undergoes the same analysis by risk management, quality improvement, and hospital management as previously done for the electronic variances. The combination of a local, stakeholder-driven variance process, combined with the system-wide computerized variance system, has allowed the analysis of variances at both the local and hospital level.

This study has demonstrated that >1 incident reporting system may be beneficial to capture as many variances as possible. The hand-written safety card system has been a sustainable way to allow simple, quick, anonymous reporting of adverse events and near-misses that would not have been captured in our computerized reporting system. We created this stakeholder-driven quality improvement program that arose from a perceived need in our perioperative environment. The development, implementation, improvement, and maintenance of the handwritten variance system by the stakeholders themselves have been essential in its success. There is a strong group dynamic and leadership commitment to creating the safest possible environment for children, with individuals taking responsibility for their actions and reporting both positive and negative variances. A similar program also now exists for pediatric inpatient services. This program can easily be applied to other services within the hospital but does require a commitment of time and leadership support.

The most important consequences of the handwritten variance card program were the resultant programs, processes, and changes in safety culture that have occurred since the implementation of the program. We have been able to involve surgeons, anesthesiologists, and surgical technicians in a variance reporting system that had previously only involved nurses. Most importantly, numerous quality improvement projects have come as a direct result of information that we have received from our handwritten card system. The addition of a handwritten variance reporting system has been 1 step in our journey of improving our culture of safety and becoming a high-reliability organization.

DISCLOSURE

The authors have no financial interest to declare in relation to the content of this article.