Implementation of a Neonatal Audit Tool to Drive Quality Initiatives.

Audit tools optimize the delivery of healthcare to patients. A network of 11 neonatal intensive care units (NICUs) affiliated with a large urban pediatric care institution implemented an audit tool for use on daily patient rounds. The article reports findings collected from 2011 to 2016.
METHODS: Primary drivers for implementation were (1) engagement from local providers; (2) identification of local improvement needs and improvement progress; (3) ability to customize audit questions based on local needs; (4) encouragement of information sharing between NICUs; and (5) improving measurable outcomes in neonatal care delivery. The Level IV NICU managed and refined a centralized process for managing site-specific tools, data analysis, and reporting. Each NICU customized the number and wording of action questions based on their needs. Answer choices were limited to "yes" or "no," which corresponded to favorable or unfavorable actions toward the patient. Users also answered, "Was action taken as a result of an unfavorable response?"
RESULTS: Plan-Do-Study-Action cycles were completed to refine the tool and its implementation process. Adherence was variable among and within each network site. Across the network, 11.4% of actions tracked by the audit indicated improvement over time.
CONCLUSION: Generalized use of the Audit Tool resulted in limited optimization of care actions addressed in the NICUs. Success depended on multi-disciplinary, multi-professional teamwork, respect for local autonomy, and leadership support. Increasing the use of the Audit Tool likely depends on the team's ability to evolve the tool's intrinsic value from a reminder to a monitoring system.

INTRODUCTION

Background Information

Children’s Hospital of Philadelphia’s Newborn Care Network consists of 11 neonatal intensive care units (NICUs) located in the Greater Philadelphia area. The units vary in size from 4 to 46 beds with a wide range of clinical care capabilities and variations in practice workflows.

Audit tools contain “housekeeping” tasks, both routine and maintenance, such as age-appropriate vital sign alert parameters, vaccine adherence, or medication reconciliation. These items, while not requiring daily conversation, may have dire consequences such as medication errors[1-4] if not addressed promptly. In addition to reducing errors and improving outcomes, rounding audit tools improve the provider’s job satisfaction, reduce stress, and result in a generous reduction in health care dollars spent over time.

Problem Description

Clinical care rounds in the NICU should be efficient as well as effective in optimizing patient care and safety. In a large unit, the care team can rarely spend more than a few minutes reviewing patient data, examining the patient, and devising a care plan for even the most medically complex patients. Routinely, the care team prioritizes discussion toward the patient’s active and acute problems, while often responding to unanticipated changes in patient acuity as well as unavoidable interruptions.

In a setting with highly complex patients and care delivery, rounding practices may fall short of addressing less urgent but important care items related to patient maintenance and safety, such as medication reconciliation, vaccinations, and checking prenatal screen results. Through the use of quality improvement (QI) initiatives, NICUs often introduce “better practice protocols,” designed to optimize the quality and safety of health care delivery. However, in times of high stress and fatigue, the provider’s ability to remember protocol details may fade. Such a distracted focus can lead to decreased proficiency, an increase in judgment errors, and failure to observe established protocols or procedures. A pilot initiative at the main hospital NICU estimated that corrective action was taken 8% of the time when the staff used the Audit Tool during rounds. The SMART aim was to increase corrective actions as a result of Audit Tool use across the Network by the same 8% within one year.

METHODS

Context

We conducted this initiative within a mid-Atlantic regional neonatal care network associated with a quarternary NICU at a tertiary children’s hospital in an urban center from 2011 to 2016. A multidisciplinary team composed of members from the network convened to plan and operationalize the tool system across sites.

The focus of the project was to implement a sustainable Audit Tool System during patient rounds across the network. Each site used the tool to monitor and measure their quality and safety needs. Both at the individual site and the network level, the outcome measure was the frequency that the clinical team responded to an audit question with an answer that was favorable to the patient.

The challenge of implementing such a tool across a heterogeneous NICU network was to create a system that standardized the question structure and audit process while still addressing the care quality and safety needs of each network site. The Institute for Healthcare Improvement’s model provided a continuous learning framework that permitted incremental improvements to the system over time as the impact on quality and safety was measured.

Creating the Tool

The Project Team introduced the concept of using an Audit Tool to medical directors from each NICU site at the 2011 Annual Planning Meeting. Over subsequent months, the Project Team conducted site visits to gain a deeper appreciation and understanding of site-to-site variations in size; workflows; staffing models; local administrative/hospital priorities; resource support; safety culture of nursing and medical staff; and QI competencies.

Through local site visits, the Project Team became aware that: (1) sites achieve common goals with different processes shaped by their environments; (2) implementation should preserve each site’s clinical rounding workflows; (3) sites agreed to be transparent and share results; and (4), sustained use of the tool depended on how valuable it was to each site, and not the result of a “top-down” mandate to use it.

Four key drivers for improving outcomes in neonatal care delivery using the Audit Tool were identified (Fig. 1):

Fig. 1.

Key driver diagram.

Robust engagement from local providers.

Identification of local improvement needs and progress.

Ability to customize audit questions based on local needs.

Encouragement of information sharing among NICUs.

Initial interventions associated with the drivers were:

Develop a network-wide tool system, easily customizable, which included standardized answers allowing for data aggregation and estimation of care plan impact.

Build a central data repository to produce actionable reports for individual sites and the network as a whole.

Provide opportunities (eg, site visits, just-in-time discussions, web conferences) for sites to develop appropriate questions for their tools and to understand how they might integrate the use of the tool in their clinical care rounds.

Identify a site tool administrator who could attend to many of the administrative tasks embodied in using the tool, such as adding questions and entering data into the central database.

Promote extrinsic and intrinsic motivation for using the tool.

Through the use of Plan-Do-Study-Action (PDSA) cycles, interventions to implement the tool were either adopted, adapted, or abandoned based on whether they were successful in impacting the key drivers. This work identified processes that we could standardize across the network as well as processes that could allow for variation to accommodate individual site characteristics. Each Site adhered to the following requirements: (1) Audit Tool was implemented as a physician-driven part of patient rounds; (2) questions chosen and used needed to be pertinent to the nursery’s patient-related quality and safety items/issues; (3) data needed to be reviewed on a regular (monthly or quarterly) basis; (4) based on regular review, determinations were made regarding questions that needed to be retired; and (5) structure of the questions was “yes”, “no” or “action taken.”

Each NICU site had a “Site Champion” who assumed responsibility for submitting and revising the site’s Audit Tool questions as well as reporting results to colleagues. The local providers developed their “question bank” by identifying any gaps in care noted through previous assessments or quality work. Criteria for selecting questions focused on the following categories: (1) relevance to local practice, (2) content aimed at improving best practices, and (3) questions focused on decreasing variation in practice. Many questions referred to process (eg, “Did you update the parents today?”) while others focused on specific safety improvement projects (eg, “Are the central lines still needed?” or “Have you converted medications to an enteral route?”). Site Champions submitted questions with a “yes” or “no” response, and indicated which response would be the preferred “favorable” one concerning patient care. For example, many sites submitted the question, “Is the daily weight documented?” for which “no” would be the unfavorable response requiring follow-up action. Questions were to be concise, requiring only 1–2 minutes to answer during patient care rounds. On the tool next to each question, the clinician could check a box labeled “Action Taken” as an indicator that asking the question influenced the care plan for that patient. Standardizing use of these constraints allowed the Project Team to aggregate answers across common categories of questions and sites.

Variation, as refined through PDSA ramps during implementation across sites, was necessary for successful adaptation of the tool. Using PDSA cycle ramps, sites refined how they decided to retire and replace questions that were meaningful to their quality initiatives. The network’s priority in newborn care quality and safety reassured leadership within network administrations that the hospital’s data transparency was not only critical for accelerating improvement but was also better for both patients and providers. Empowering the sites to implement the Audit Tool with autonomy for interpretation of results was critical to stakeholder “buy-in” and enhanced respect for the capabilities and the needs of the local NICUs. For instance, some centers selected questions relevant to particular patients on rounds, while other centers determined the focus question and selected patients within the unit to whom the questions were applicable. Based on site preference, questions were answered in various ways: (1) in real-time during rounds; (2) completed their questions immediately after rounds; (3) answered the questions as a group; and (4) allowed a single physician to complete the questions.

Implementation

In the first PDSA cycle, a project coordinator at Children’s Hospital of Philadelphia’s central facility created a paper tool with the site-specific Audit Tool questions and tracked the questions and answers in a database. Clinicians at each site had the autonomy to determine how to implement the tool, ie, during patient care rounds, separate “QI Rounds” or teaching rounds. The number of questions asked during rounds varied depending on site-specific needs. Some sites asked 2–3 questions per patient, while others asked 5–10 questions on 1 specific patient. These questions could be asked during morning or evening rounds on weekdays or weekends. Each center implemented the tool to fit their workflow.

The Program Coordinator entered information from each site’s completed audit tool sheets into a central system that calculated 2 ratios for each question:

Favorable Response Rate = No. of favorable responses/No. of times a question was asked.

Action Taken Rate = No. of actions taken/No. of unfavorable responses.

For example, Site A asked the question, “Are the oxygen saturation alarm settings appropriate for this infant?” 20 times during a single month. The number of unfavorable “No” answers was 4. For these 4 answers, the care team checked 3 of the “Action Taken” boxes. Thus this question’s favorable response and action taken rates were 80% (16/20) and 75% (3/4), respectively, for this site in 1 month. The Coordinator distributed a transparent quarterly report of individual and aggregate results. All sites were able to access an electronic dashboard created on a secure share site that organized the results, transparently, by category, site, and responses. The favorable response rate and action taken rates demonstrated the usefulness of the question, not just the number of times that the clinicians asked a question. Due to variability in unit size, staff size, and several questions on each Audit Tool, the specific number of questions did not reflect the improvement over time.

The system calculated a “Hit Rate” and an “Action Rate” for each question to help site teams decide whether to keep individual questions or retire them:

Hit Rate = No. of unfavorable responses/No. of times question was answered.

Action Rate = No. of actions taken/No. of times a response was unfavorable.

We assumed that the more times a question was asked, the fewer unfavorable responses there would be and thus a lower “Hit Rate.” If a question had a low hit rate, the local quality improvement team had the option to retire the question and replace it with a new question so they could continue to build on their success.

RESULTS

The project began in 2011 with 134 initial audit questions collected from the 11 sites in the network. By 2016, the network had added 248 new questions and discontinued 202, for a final count of 180 questions. The number of questions in each sites’ question bank ranged from 14 to 55. The audit tool questions fell into 10 major care categories (Fig. 2):

Fig. 2.

Audit tool checklist questions by category (top 10).

During the period 2011–2016, the 11 NICU sites collectively asked their questions 77,576 times (Fig. 3), of which 9,507 produced an unfavorable answer, yielding a network hit rate of 12.2%. Of these unfavorable answers, 5,926 resulted in an action taken, yielding a network action rate of 62.3%. Control charts demonstrated improvement for 11% of questions from the network’s question bank (see calculation below). The top 5 question categories that prompted action were also the most frequently asked categories: (1) Communication/Documentation, (2) Nutrition, (3) Discharge Planning, (4) Care Management, and (5) Monitoring.

Fig. 3.

Number of questions asked per site from 2011 to 2016.

Since >1 clinician could direct questions to the same patients, not all data points were independent of one another. Accordingly, the Project Team elected to apply run charts to assess the impact of implementing the audit tool on care improvement efforts. The average time questions were used by each site ranged from 7.9 to 39.3 months. A question was associated with sustained improvement when its run chart indicated a centerline shift in a favorable direction (6 data points above or below the centerline) (Fig. 4). Across the Network, 11.4% of questions in the question bank demonstrated shifting centerlines in a favorable direction with a positive impact.

Fig. 4.

SPC chart example: weekly nutrition plotting question.

To keep the audit process brief, the Audit Tool did not ask care teams to explain why unfavorable responses did not prompt actions taken. Instead, periodic interviews with care teams showed that in a majority of cases, there were valid reasons for no action taken. As one example, a question about vaccines being up to date for a particular patient might generate a “no” response, which would be unfavorable and thus require a follow-up action; however, in some cases, the care team felt it was not appropriate to take that action (eg, would not administer vaccine because the patient was septic).

DISCUSSION

The Project Team experienced several key successes in the first PDSA cycle that spotlighted network collaboration, local level fortitude, commitment to quality, and outstanding teamwork among all caregivers. Early success in implementing the Audit Tool was partially due to linking physician participation to their annual evaluation process and incentive metrics. This mandate acted essentially as an extrinsic motivator that resulted in high usage of the tool. As extrinsic motivation naturally wanes over time, it becomes critical to build intrinsic value. One way to achieve this intrinsic motivation is by evolving the Audit Tool System into a QI monitoring system. As individual NICUs utilized their audit tools to advance quality initiatives, the utility of the questions increased provider/site participation. A future goal for the Project Team is to grow the intrinsic value of this system to the point that will sustain its usage. The Newborn Care Network achieved complete transparency enabling sites to share their questions and opportunities for improvement.

Throughout the process, the Project Team had difficulty building a system that allowed sites to examine their data in real-time. While an online dashboard was available for sites to use, access to the dashboard was cumbersome due to institutional firewall and security measures. The quarterly spreadsheet reports sent by the Project Team may not have been frequent enough to help local teams identify failing interventions quickly. Sites that benefitted the most were able to either monitor their progress locally or access the dashboard regularly. In essence, while the Audit Tool served well as a reminder system, it failed as an effective monitoring system for local site use. Most successful sites within the Network accessed their quarterly reports and reviewed the effectiveness of specific audit questions at monthly staff meetings. This approach allowed these teams to continue questions encouraging improvement and retire questions that local sites no longer required due to newly hardwired initiatives. Local teams replaced those retired questions with new audit questions that advanced new initiatives.

A safety and quality culture grew at each site where site champions and care teams identified gaps in care. The Audit Tool functioned consistently as one catalyst for “closing the care gaps.” Consistent use of the tool during rounds provided a forum for multiple stakeholders, including nursing staff and front line clinicians, to participate in the unit’s QI activities. Sites with the most effective use of the audit tool implemented questions from the top 5 categories: (1) Communication/Documentation, (2) Nutrition, (3) Discharge Planning, (4) Care Management, and (5) Monitoring in real-time during patient care rounds. This timely reminder eliminates easily overlooked items during rounds and increases compliance with improvement initiatives. With quarterly reports on improvement provided by the Project Team, local sites were enabled to report QIs to their local hospital administrations.

Variations in the completion of questions from site to site posed challenges for data analysis. While limiting answers to “yes/no” was helpful, it became clear that local interpretation of the questions could vary among providers, and the Project Team was therefore forced to “wordsmith” (or reword) the questions thoroughly before publishing. During the implementation phase, the Network site care teams varied in how they answered the questions depending on their respective rounding processes. In some cases, the attending physician answered the questions as rounds proceeded; in other cases, the care teams answered the questions together as a group. While site autonomy was permitted to a certain degree to respect the rounding workflows established at each site over time, it became clear that sites that were filling out the forms after rounds, rather than during rounds, required frequent coaching and clarification of the goals and processes. The Project Team found it challenging to maintain a balance between process standardization and site autonomy. This fact alone underscores the importance of this system in providing palpable value for QI efforts at each site. Future work by the Project Team involves encouragement of successful sites to share their experiences at monthly network meetings while allowing struggling sites to seek input for improvements in audit tool usage. This information sharing and collaboration among sites within the Network aims to advance the use of the Audit Tool and local site quality initiatives.

CONCLUSIONS

The amount of time and effort invested in fully integrating the Audit Tool into clinical care practice has been both impressive and worthwhile. A multi-discipline, multi-professional Project Team was essential to implementation success. Hospital and department leadership provided the extrinsic motivators that jump-started the project—which now must continue to build its intrinsic value for sustainability. While question content remains locally driven, increased standardization of the answer structure will be useful for comparative analysis.

In Phase 2, the Project Team will focus on evolving the Audit Tool into an “improvement monitoring tool,” which can track QI progress. Currently, the team has built a new dashboard with easier access. This dashboard will display run charts for each question for each site, complete with the response and the rates of action taken. The auditing process during rounds is now electronic to shorten the time between data entry and data visualization while eliminating previous security and firewall issues. The Project Team has engaged Site Champions to implement a more collaborative sharing of audit findings, and also to encourage sites to use the Audit Tool System for data collection aspects of local QI projects.

ACKNOWLEDGMENTS

Assistance with the study: Jeffrey Gerdes, MD, MBA, Jacquelyn Evans, MD, Phyllis Dennery, MD and The Children’s Hospital of Philadelphia Newborn Care Network physician teams.

DISCLOSURE

The authors have no financial interest to declare in relation to the content of this article.