A Longitudinal and Sustainability Assessment of Pediatric Interfacility Transport Handover Standardization.

INTRODUCTION: Standardization of interfacility transport handover is associated with improved shared mental model development, efficiency, and teaming. We sought to build upon previously published data by evaluating 1-year follow-up data, assessing face-validity, and describing sustainability.
METHODS: We performed a pre-post, retrospective cohort study in a stand-alone, tertiary, pediatric referral center for children 0-18 years of age transported to our pediatric intensive care unit, neonatal intensive care unit, or emergency department from October 2016 to November 2017. Handover was standardized using multidisciplinary checklists, didactics, and simulation. Data were collected for three 8-week periods (preintervention, postintervention, and 1-year follow-up). Outcomes included shared mental model index (shared mental model congruence expressed as an index, percent congruence regarding healthcare data), teaming data (efficiency, attendance, interruptions, interdependence), and face validity (5-point, Likert scale questionnaires). Statistics included 1-way analysis of variance, Kruskal-Wallis, chi-square, and descriptive statistics.
RESULTS: One hundred forty-eight handovers (50 preintervention, 50 postintervention, and 48 at 1-year) were observed in the emergency department (41%), pediatric intensive care unit (45%), and neonatal intensive care unit (14%). No differences were noted in demographics, diagnoses, PIM-3-ROM, length of stay, mortality, ventilation, or vasoactive use. Sustained improvements were observed in shared mental model congruence expressed as an index (38% to 82%), physician attendance (76% to 92%), punctuality (91.5% to 97.5%), interruptions (40% to 10%), provision of anticipatory guidance (42% to 85%), and handover summarization (42% to 85%, all P < 0.01). Efficiency was maintained throughout (mean duration 4.5 ± 2.1 minutes). Face validity data revealed handover satisfaction, effective communication, and perceived professionalism.
CONCLUSIONS: Enhancements in teaming, shared mental model development, and face validity were achieved and sustained 1-year following handover standardization with only minimal reeducation during the study period.

INTRODUCTION

Insufficient or poor-quality handover has been linked to in-hospital sentinel events, therapeutic errors, inappropriate laboratory testing, increased hospital costs, and prolonged lengths of stay.[1-5] It is also well known that communication errors increase as the number of handovers increase. As a result, organizations such as the United States Joint Commission, World Health Organization, Air Medical Physician Association, and Children’s Hospitals Solutions for Patient Safety Children’s Healthcare Network have focused on handover standardization as a priority to improve patient care and reduce medical errors.[6-8] Critically ill children undergoing interfacility transport represent a uniquely high-risk population having undergone several transitions of care and medical interventions before arrival at an accepting institution.

A healthcare worker’s ability to determine the importance of environmental stimuli, distinguish and prioritize workplace relationships, and generate expectations for clinical performance is regulated by internal, cognitive frameworks known as mental models.[9,10] Throughout handover, individual participant cognitive models are transformed into a shared mental model regarding historical context, existing patient status, current posthandover healthcare plan, and anticipatory guidance.[11-14] In a preceding publication,[14] our research group was able to show that greater degree of shared mental model congruence expressed as an index (SMMi) can be achieved by using simple checklist-based handover standardization and can enhance other quality outcomes.[15-17]

Like many quality improvement initiatives, we found initial pre-post assessments to yield significant improvements. However, a significant portion of quality improvement projects fail to show persistent improvements after longitudinal assessment.[18,19] In this article, we refocus on the concept of quality improvement project sustainability by using a descriptive and comparative analysis to identify lasting improvements in SMMi development, teaming, handover process, and participant face validity by matching our previously published pre/post assessment with 1-year follow-up data. Our findings continue to add to the ongoing dialogue regarding the importance of standardization of interfacility transport handover,[20,21] but subsequently bring to light a fresh discussion on mechanisms that maintain lasting quality improvement.

METHODS

Clinical Setting and Study Design

Research was conducted within a 259-bed, university-affiliated, tertiary care pediatric referral center where a dedicated pediatric transport team performs approximately 1,300 interfacility critical care transports yearly. We performed a retrospective descriptive and comparative cohort study of children 0–18 years of age transported to our pediatric intensive care unit, neonatal intensive care unit, or emergency department from October 2016 to November 2017. Data from transports to the cardiac intensive care unit (ICU), general care pediatric unit, or non-ICU subspecialty services were excluded from study as process improvement interventions did not occur simultaneously for all hospital units during initial plan-do-study-act (PDSA) cycles (representing approximately 25% of interfacility transports during the study period). In addition, we excluded encounters where survey data (described below) were not completed by handover participants. This study was reviewed and approved by our local institutional review board.

Study Outcomes and Definitions

The primary study outcome was SMMi measured at 1-year poststandardization defined as percent congruence among handover participants regarding key patient healthcare data including: (1) patient identification; (2) primary diagnoses; (3) transport team interventions; (4) posthandover immediate care plan; and (5) anticipatory guidance. Secondary outcomes were participant face validity assessed by responses to posthandover questionnaire items including assessment of perceptions of (1) handover efficiency; (2) degree of interruptions/distractions; (3) clarity of healthcare data exchange; and (4) overall satisfaction with handover on 5-point, parametric Likert scales (range: strongly disagree to strongly agree). Other study outcomes were handover and teaming metrics including objective efficiency (duration of handover), attendance, interruption frequency, team member inclusion, prompts for questions and clarification, and handover comprehensiveness. We defined efficiency as mean handover duration determined by initial time out called by the transport team and cessation of handover discussion between providers. Handover comprehensiveness was measured by frequency of reference to specific historical data, transport team interventions, and anticipatory planning. Team members were expected at the bedside when the transport team arrived, and tardiness was defined as participant arrival to handover after transport team began oral handover. Additional descriptive data included demographics, comorbidities, general patient outcomes (length of stay and mortality), and severity of illness metrics including Pediatric Index of Mortality 3 Risk of Mortality (PIM-3-ROM), frequency of invasive and noninvasive mechanical ventilation, and vasoactive administration.

Intervention Description

Study intervention was standardization of handover using a combination of a checklist tool, didactic lectures, and simulation education introduced after baseline data collection. The handover checklist (Fig. 1) was derived by modifying an existing, validated pediatric postoperative handover tool and altered using contributions from subspecialty disciplines including transport medicine, critical care medicine, emergency medicine, neonatology, respiratory care, and critical care nursing.[14,22] In addition to facilitating a structured handover procedure, the checklist outlined individual roles and expected professional behaviors. Transport and hospital unit-based staff—such as nursing, respiratory therapists, emergency medical technicians, bedside nursing, advanced practice providers, and attending physicians within each division—received didactic and simulation education before postintervention data collection to improve familiarity with study interventions. Staged examples of “poor prestandardization” and “ideal standardized” handover were recorded in our simulation center using research staff members. During a 4-week education period, night and day preshift staff meetings (aka “huddles”) included printed educational material and viewing sessions of educational videos. Concurrently, staff members participated in simulation with debriefing. All simulation was performed after baseline data collection within each study unit. New staff members hired after the initial study-education period received didactic training but were not required to undergo simulation. PDSA cycles were planned every 3 months over the 1-year study period to assess and correct areas of potential process improvement.

Fig. 1.

Handover checklist tool used as scripting assistance to handover participants.

Data Collection

One-year follow-up observational and face validity data were compared against our original pre- and postintervention data previously published.[14] As outlined in our original publication, posthandover survey data were collected for three 8-week study periods (preintervention, postintervention, and now at 1-year follow-up). A 4-week period without data collection following the preintervention period was provided for education and familiarity with standardized handover process as described above. Observational handover data were recorded prospectively as part of an ongoing process improvement initiative by an independent, nonclinical transport staff member including timing, participant characteristics, handover content, teaming metrics, and behavioral interactions. Data were stored within a hospital-maintained transport medicine database queried by our research personnel retrospectively after institution review board approval.

Shared Mental Model Index (SMMi)

SMMi was calculated using responses from a brief, 5 question immediate posthandover survey. Survey questions were as follows: (1) Who is the patient? (eg, “one-liner” with age and primary indication for transfer); (2) What are the primary medical diagnoses?; (3) What interventions were performed during transport or prior to handover?; (4) What are the key features of the immediate healthcare plan as specified during handover?; and (5) What anticipatory guidance was given and who should be contacted if concerns were to arise? Responses were independently reviewed by 2 research team members with interrater reliability calculated. A minimum of 2 survey responses were required for assessment of SMMi and missing or inadequate data resulted in encounter exclusion from study. Total SMMi and SMMi for each individual question were independently examined.

Face Validity Assessment

Face validity was assessed using responses to posthandover survey statements ranked on a 5-point, parametric Likert scale. These statements were as follows: (1) Handover was conducted quickly and with minimal interruptions or distractions; (2) Patient history and healthcare data were clearly presented without misunderstanding; and (3) I am satisfied with the transport handover process.

In addition, face validity by participant discipline was assessed for variation.

Sustainability Assessment

Project sustainability was qualitatively measured by assessing 1-year follow-up data from completed study surveys and observer data collection. In addition, we retrospectively used the National Health Services Sustainability Model, an assessment tool used to assess long-term sustainability of proposed quality improvement endeavors before implementation.[23] These data are primarily presented in the discussion section as they were not included in the initial methodology of this quality improvement project.

Statistical Analysis

Descriptive data are reported as proportions with percentages, means ± SD (SD), or medians [interquartile range (IQR)] depending on data type and variance. Statistics included 1-way analysis of variance, Kruskal-Wallis, chi-square, and descriptive statistics. SMMi data calculated from posthandover surveys had interrater reliability calculated using Cohen’s Kappa. For all tests, type I error was set at 0.05. Statistical analyses were completed using Stata v15.1 software (Stata, College Station, Tex.).

RESULTS

PDSA Cycles

Baseline data were collected in the 8-week preintervention period starting October 2016. Immediately following, planned PDSA cycles were completed every 3 months over the 1-year period. PDSA content focused on potentially modifiable handover processes including: (cycle 1) initial roll out of handover standardization, (cycle 2) reeducation of attending physician participants emphasizing provision of anticipatory guidance and summarization at the completion of handover, (cycle 3) implementing a mandatory prearrival provider notification call tree through our transfer center, and (cycle 4) inclusion of education and training for resident physician providers. The cycle 3 call tree notification alteration included a page to the charge nurse, bedside staff, and physician to begin handover preparation 15 minutes before scheduled arrival.

Sample Characteristics

During the study period, an additional 48 handovers were observed at PDSA cycle 4 upon 1-year following transport standardization to compare with data previously assessed from PDSA cycle 1 where 50 pre- and 50 postintervention handovers were evaluated. General cohort demographics, transport disposition, and patient characteristics can be found in Table 1. There were no observable differences in demographics, admission diagnoses, PIM-3-ROM, length of stay, mortality, respiratory failure requiring invasive/noninvasive ventilation, or vasoactive use when comparing PDSA cycle 4 subjects to initial pre-post encounters. An equal proportion of hospital units received transported patients for the study periods and were predominantly represented by the ED (41%) and pediatric intensive care unit (45%).

Table 1.

Sample Characteristics Including Demographics, Transport Disposition, Primary Diagnoses and Metrics of Severity of Illness

Handover Process and Teaming Assessment

Handover efficiency, process data, and teaming metrics can be found in Table 2. Efficiency was sustained throughout the study with 1-year follow-up handover durations noted at 4.5 ± 2.1 minutes. A median of 3 handover participants were present for each encounter (range, 2–7) and included a transport nurse, transport respiratory therapists, bedside clinical nurse, and/or accepting provider in 83% of all encounters. At 1-year follow-up, lasting improvements in attendance were noted for all participant disciplines, but significantly for attending physicians (76–92%; P = 0.01). Similarly, a sustained reduction in tardiness was noted (8.5–2.5%; P < 0.01). All handover process and teaming metrics showed sustained enhancement at 1-year. Notably, we observed significant reductions in handover interruptions (40–10%), greater team member attentiveness (82–100%), provision of anticipatory guidance (42–85%), and accepting provider handover summarization (42–85%, all P < 0.01).

Table 2.

Pediatric Inter-facility Handover Duration, Attendance, Process, and Team Science Data

During the study period, a total of 448 posthandover surveys were evaluated for SMMi calculation. Interrater reliability was excellent between the 2 study team members (Cohen’s Kappa = 0.9; P < 0.05). Data regarding total SMMi, subcategory SMMi, and SMMi by participant count are found in Table 3 and reveal sustained improvements at 1-year. Mean total SMMi markedly improved from 38% ± 20 to 82% ± 17 (P < 0.01). Furthermore, lasting gains were observed regardless of total number of handover participants following standardization.

Table 3.

Shared Mental Model Index before and after Standardization of Inter-facility Transport Handover

Face validity data are found in Table 4. Generally, we observed a high degree of participant satisfaction, perception of effective communication, professionalism, and efficiency following handover standardization. These findings were sustained at 1-year with 98% responding positively [agree (19%) or strongly agree (79%)]. When analyzed by individual discipline, we noted improved perceptions of handover standardization by all participant disciplines, but significantly for attending physicians (data not shown).

Table 4.

Face Validity Data by Responder Group on a 5-point, Parametric Likert Scale

DISCUSSION

In this pre-post, retrospective descriptive and comparative cohort study, we noted considerable enhancements in teaming metrics, handover process, and shared mental model development 1-year following interfacility transport handover standardization. These gains were consistent with our original findings and sustained using an intervention that primarily consisted of a standardized checklist with only minimal retraining throughout the study period.[14] The addition of face validity and longitudinal data strongly suggests an intuitive checklist tool can promote lasting qualitative and quantitative enhancements to a quality improvement project without interfering with workplace efficiency.

Checklists and scripting to promote handover structure and behavioral norms improve data exchange, enhance efficiency, and yield general provider satisfaction in several studies evaluating postoperative and intrafacility unit transfer handover.[2,5,17,22,24-29] Our findings add to transport handover literature[20,21] and supplement our initial publication describing short-term handover standardization results.[14] The primary outcome, SMMi, remained highly congruent at the 1-year mark despite minimal interventions to preserve “herd” standardization and quality outcome gains.

A shared mental model, as the primary cognitive output of healthcare teaming and transitions of care, represents an essential framework from which a provider can explain, predict, and interpret situational and clinical expectations.[13,14] Cognitive analyses in pediatric critical care medicine have determined that insufficient shared modeling may result in preventable harm.[11,12] Theoretical barriers to model development include system factors, such as inadequate data exchange, lack of shared language, or guarded teaming, and individual participant qualities including individual cognitive capacity, limited expertise, and preexisting teaming skills.[14,30] In our sample, these factors did not seem to result in process deterioration, decreased model development, or poor team performance.[14,30] We noted consistent improvement in attendance, professionalism, provision of anticipatory guidance, and data summarization that, taken collectively, suggest that handover standardization became integrated as part of daily practice in our intensive care unit.

A significant percentage of quality improvement projects fail to show sustained, longitudinal impact.[18,19] Limited dialogue has focused on identifying process and contextual factors that contribute to outcome sustainability after initial project enthusiasm has faded. Major quality improvement organizations such as the Institute for Healthcare Improvement, Agency for Healthcare Research and Quality, and the National Health Services have created evidence-based guides for predicting sustainability before implementation of quality improvement projects drawing from industrial organization, behavioral psychology, and team science literature.[23,31,32] The National Health Services offers an assessment tool for organizations and researchers to assess the potential long-term sustainability of proposed quality improvement endeavors before implementation.[23] The tool utilizes weighted scores for responses to questions categorized by process (benefits, credibility, adaptability, systems monitoring capacity), staff (involvement, behaviors toward change, leadership engagement), and organizational (alignment with strategic aims, infrastructure for change) contextual factors that may influence sustainability on a scale from 0 to 100 (where scores >55 have a high likelihood of successful long-term impact). Our quality improvement project scored an 82.2 that is consistent with our study findings (data not shown).

Specific circumstantial factors associated with successful quality interventions have been identified.[18,19,30-34] Stone et al.[18] concluded that sustainable projects should plan to achieve sustainability before implementation during process development. For example, they conclude sustainable efforts (1) build a clear, shared mental model for change; (2) are energetically supported by local and institutional leadership; (3) actively engage staff; (4) provide transparent processes for implementation and reassessment; (5) routinely share process change data; and (6) integrate quality improvement seamlessly into routine practice. Our intervention was developed collaboratively with a multidisciplinary team of individuals from each participating discipline and hospital unit. This team included organizational leadership who routinely sought staff member feedback. Once baseline data were collected, the 4-week educational period created a palatable integration of transparent quality processes into daily workflow. With exception of PDSA cycle 1 that included intensive education via didactics and simulation, all subsequent cycles were focused initiatives requiring only minimal labor from research team members. These qualities taken in sum likely account for our observed lasting and successful impact on interfacility transport handover standardization in our center.

Limitations

This study represents a single center experience and thus limits its generalizability as handover practices may vary between institutions. However, it is likely our data are applicable to similar sized, tertiary care pediatric referral centers. We excluded transports to the cardiac ICU, general pediatric ward, or non-ICU subspecialty services as no baseline data were collected for comparative analysis. Although handover content may differ for populations transported to these destinations, it is unlikely that our findings would be altered by their inclusion as subsequent PDSA cycles have now incorporated all transports to our facility. Postintervention handover participants received ongoing education regarding process improvement and checklist times, but remained blinded to calculation of SMMi and face validity. The median number of handover participants and survey responders were 3 per handover encounter. Although participants were encouraged to fill out surveys, we could not mandate this and thus may represent selection bias. Face validity responses were on a parametric Likert scale, but it is plausible that interpretation of qualitative scales could vary by survey responder and lead to misrepresentation of perceptions on standardized handover.

CONCLUDING SUMMARY

We observed sustained enhancements in teaming metrics, handover process, and shared mental model development at 1-year following interfacility transport handover standardization. All study observations were noted without alteration in handover duration. Participant face validity, longitudinal SMMi, and teaming data suggest that handover standardization with an intuitive checklist tool may yield lasting qualitative and quantitative enhancements to similar quality projects seeking to standardize healthcare transitions without interfering with healthcare efficiency. We recommend that quality improvement endeavors seeking sustainability plan for such before process change by using multidisciplinary collaboration, seeking enthusiasm from leadership, engaging clinical staff, providing transparent and current reassessments, and choosing endeavors that blend into routine care.

DISCLOSURE

The authors have no financial interest to declare in relation to the content of this article.