Addressing Challenges of Baseline Variability in the Clinical Setting: Lessons from an Emergency Department.

To demonstrate methods of adjusting data in quality improvement projects for better learning about interventions over time.
METHODS: A secondary analysis of data from a quality improvement project to improve patient wait times at an urban academic pediatric emergency department using electronic medical data from 2015 to 2018. The primary outcome was the wait times for low-acuity patients. Control charts were used to determine if the interventions were effective in reducing wait times. Two different data adjustment techniques were applied to account for changes in patient volume and seasonal effects on the outcome measure.
RESULTS: We more effectively demonstrated improved patient wait times after adjusting for patient volume or seasonality. Patient wait times decreased from 75.2 to 72.9 minutes after the intervention; a 3% decrease sustained over 18 months. A strong correlation between patient volume and wait times was noted. Process stability was achieved on the control charts after data adjustment, with one centerline shift after data adjustment in contrast to 5 centerline shifts required before data adjustment.
CONCLUSION: Adjusting for seasonality or patient volume created process stability and improved learning from control charts. After adjustment, we sustained decreased patient wait times more than a year out from the original intervention Adjusting by patient volume seems to be a preferred method of adjustment. Our findings support the importance of adjusting for baseline variability affected by seasonality or patient volumes, especially in flow projects, as a high yield method for process improvement.

INTRODUCTION

Background

Methods to understand variation within clinical environments where there is baseline variability are not well developed. There is a lack of literature on quantifying and adjusting data used for quality improvement for the many sources of variation in clinical environments.[1] Improvement methodologies suggest sustainability should be measured as persistence of the process without unexplained variation.[2] However, changes in patient demand in the microenvironment are frequent in most clinical settings. Emergency departments (EDs), in particular, are subject to changes related to seasonality, day of the week, and hourly swings in patient volumes.[3] For example, changes may occur in yearly or monthly cycles or may be related to sudden increases in viral prevalence, or may vary by the specific day of the week.

To address the complexity of learning within this environment, we considered 2 methods of adjusting wait time data to account for these cycles of variation. The first method can be found in the improvement literature and is based on adjustment for seasonal variations in patient wait times. Because seasonal variation depends highly on viral pandemics in the pediatric population, and the exact months of these pandemics vary by year, we created a new method using actual monthly volumes rather than seasonal wait times.

To evaluate these methods, we chose a recent quality improvement (QI) project to improve the wait times of low-acuity patients presenting to our pediatric ED (PED). During the intervention, the QI team struggled with accurately interpreting the changing center line on the control charts (Fig. 1A). Five center lines were present in the original study.[4] Although content experts suspected seasonal effects, the charts still reflected special cause variation. The data provided an ideal opportunity to evaluate adjustment techniques, examine the outcome measures, and reassess the sustainability of the intervention.

Fig. 1.

Original wait time data of low acuity patients presenting to the emergency department. A, Original I chart from the initial study. B, Unadjusted I chart of low-acuity patient wait times. UCL, Upper Confidence Limit; LCL, Lower Confidence Limit.

Process for Decreasing Patient Wait Times

In March 2017, Children’s National Health Systems (CNHS) PED initiated a QI initiative to improve wait times for low-acuity patients presenting to the PED.[4] The QI initiative consisted of streamlining the front end triage and assessment processes. Lean methodology informed the strategy development.[5] Specific barriers to efficient patient flow were identified. The improvement team chose the following top target interventions: adding a provider, placing the provider in triage, and implementing improved nurse triage and assessment processes. Because 55% of our low-acuity patients arrive between 12 pm and 10 pm, the added provider worked from 12 pm to 10 pm daily. The overall aim was to improve patient flow and mitigate ED crowding. The primary outcome measure was patient wait times, specifically patient arrival to physician times. We previously published detailed methods for improving the flow of low-acuity patients in our PED.[4]

Study Objectives

The objective of this study was to demonstrate the application of 2 types of data adjustments for special cause due to seasonality and patient volume. This study had 3 specific aims. First, we quantified the impact of seasonality and patient volumes, respectively, on the outcome measure. Second, we describe the 2 potential data adjustments and explain the pros and cons of each. Third, we assessed the long-term sustainability of the QI intervention after properly adjusting for seasonal and volume variations.

Our global aim was to advance QI methodology to allow improved learning in dynamic clinical settings with numerous sources of significant baseline variability.

METHODS

Study Setting and Population

This retrospective observational study used data from the electronic medical record for patient wait times from January 1, 2015, to November 30, 2018. The setting was a large, urban, tertiary, academic PED, and level 1 pediatric trauma center with approximately 90,000 annual visits. The PED is staffed by fellowship-trained pediatric emergency medicine faculty, fellows, and resident trainees at all times. Board-certified pediatricians, physician assistants, and nurse practitioners provide additional staffing and often practice independently for low-acuity patients. Low-acuity patients presenting to the ED were the target population. We defined low-acuity as Emergency Severity Index triage level 4 or 5. This project was undertaken as a QI initiative at Children’s National Health Systems. It did not constitute human subjects research, and as such, it was deemed exempt from review by the institutional review board.

Data Collection

We extracted all data from the electronic medical record and ED tracking system (Cerner FirstNet, Cerner Corporation, Kansas City, Mo.). Data for low-acuity patients were obtained retrospectively for the 48 months between January 2015 and December 2018. Because a key component of the intervention was an additional provider between the hours of 12 pm and 10 pm, we focused on data from this period. Collected data included the number of patients, time of arrival, time seen by a provider, and time of disposition. Patients missing any of these data points in the EHR, or those patients arriving before 12 pm or after 10 pm were excluded from this analysis. Consistent with our usual practice, we excluded patients implausible data caused by computer entry errors.

Measures

The main outcome measure was wait times for low-acuity patients presenting to the ED between the hours of 12 pm and 10 pm daily. We calculated the time from arrival to the provider by subtracting the time a provider initially saw a patient from the time of arrival. As a second measure used to test the adjustment methodology, we evaluated the length of stay (LOS) defined as the time of disposition minus time of arrival. We used statistical process control methodology to evaluate the effects of the interventions over time. We compared the outcome measures before and after modifications to the control chart based on our adjustments for seasonal variability and baseline variability in patient volumes.

Data Analysis and Results

Between the hours of 12 pm and 10 pm, 102,852 low-acuity patients arrived during the study period-an average of 2,188 low-acuity patients per month. To assess patient wait times, we developed and updated control charts using software package QI Charts V.2.0.23 (Process Improvement Products, Austin, Tex.) for Microsoft Excel 2013 (Microsoft, USA). In the original analysis and publication of these data,[4] I charts were used (instead of Xbar/S charts) because they provided a simpler analysis and were sensitive enough to detect the impact of the intervention. Therefore, for comparison purposes, in this secondary analysis, we continued this approach and used monthly averages to assess the outcome measure of patient wait times. We used data leading up to the implementation of the initial intervention to calculate the baseline centerline and control limits. Significant shifts in the measures (ie, special cause variation) were identified using traditional rules for patterns on statistical process control, including 8 consecutive measurements persistently above or below the mean, 2 out of 3 consecutive data points at the outer third of upper or lower limits, or 6 consecutive points trending up or down. We calculated new control limits and centerline if a sustaining system shift was observed.[6]

Unadjusted Data

Figure 1B is the original I chart of the primary outcome before adjustments and is presented for comparison purposes. The annotation indicates the implementation of the initial change in the triage process. The winter months of 2017–2018 indicate that the gains of the original improvement were not sustained through the successive fall and winter. But note a similar pattern appears between September and February, starting as early as 2015. It recurs in 2016 and continues into 2017. This variation suggests a recurrent process contingent upon the winter months.

Similarly, note the special cause in favorable direction between March and September 2017. The QI team originally attributed this to the success of the intervention, but careful examination of the chart demonstrates that a similar special cause in favorable direction occurred between March and September in 2016, before the QI team even convened. We hypothesized the shift changes were due to seasonal effects or the effects of changes in ED volume.

Model 1: Seasonal Adjustment: Wait Time Data Adjusted by Monthly Average Wait Time

We first considered adjusting the wait time data for seasonal effects, using calendar months to identify “season.” The seasonal effect includes multiple potential causes of variation, including variation in volume or patient acuity that changed over the year.

Step 1: We calculated the monthly wait time deviation from the mean using nearly 4 years of wait time data from January 1, 2015, to November 30, 2018. We computed the average monthly deviation and plotted those specific months (January, February, etc.) as subgroups of 3 or 4 months on an X-bar chart. Standard criteria were used to identify months with special cause variation in wait times.

Step 2: Once we identified the months with significant deviations, we adjusted the mean average patient wait times (adjusted wait time = average waiting time − deviation for the month). We then plotted the adjusted data on a new I chart.

Figure 2 shows the average monthly deviations in wait times. November, December, February, and March were special cause high wait time months, whereas June, July, and August were special cause low wait time months. Average wait times in December and March were 20 and 24.5 minutes longer than the overall mean. Average wait times in July and August were shorter by 22 and 27 minutes, respectively, than the overall mean.

Fig. 2.

Monthly average wait time deviations. UCL, Upper Confidence Limit; LCL, Lower Confidence Limit.

Figure 3 shows the I chart of wait time data adjusted for the month. During the baseline period, the average low acuity patient wait times were 77 minutes (upper limit (UL) of 95.5 minutes and lower limit (LL) of 58.5). Implementation of the new front end interventions resulted in a decrease of wait times to 69.3 minutes, a 10% decrease (UL 89.2 minutes, LL 49.4 minutes).

Fig. 3.

I charts for wait times adjusted by month and adjusted by volume. UCL, Upper Confidence Limit; LCL, Lower Confidence Limit.

Model 2: Volume Adjustment: Wait Time Adjusted by Monthly Patient Volume (Novel Methodology)

Sometimes the timing of seasonal effects is not consistent from year to year. So we considered an alternative approach to adjust the wait time by volume during the month.

Step 1: To determine the impact of patient volume on wait times, we created a scatterplot of wait times versus daily volume using Microsoft Excel 2013 (Microsoft, USA) (Fig. 4). A linear regression provided the equation to summarize the relationship between volume and wait time as shown on the scatterplot. The correlation coefficient squared (r2) describes the percent of the variation in wait times that can be explained by the changes in volume. As expected, wait times increased as patient volumes increased. The correlation coefficient R of 0.8 (√0.633) indicates a strong relationship between these variables. The slope of the line, 0.067 minutes per patient (6.7 min increase in wait times for 100 additional patients) provide the factor needed to adjust wait times to a standard volume. We explored other models including polynomial, logarithmic, and exponential, and found that simple linear models fit the best.

Fig. 4.

Scatterplot of low-acuity patient wait times vs. low-acuity patient volumes. UCL, Upper Confidence Limit; LCL, Lower Confidence Limit.

Step 2: To adjust wait times by patient volume, we computed the average monthly volume (a), the average monthly wait times (b), the overall mean patient volume for the 4 years of data (c) and the adjustment factor (m), identified as the slope of the best fit line calculated previously in the scatterplot. We adjusted monthly average wait times by patient volume using the following formula: m × (c − a) + b

The adjusted wait times were analyzed using an I chart (Fig. 3, I chart adjusted for volume). This chart helps clarify the impact of the implementation of the new front end interventions, which resulted in a decrease of wait times to 72.9 minutes, an improvement of 2.3 minutes (3%). Less variation is present than in the unadjusted data, and the impact of the intervention is sustained through the winter months of 2018. One month is noted as a special cause, in contrast to 8 months in the unadjusted I chart (Fig. 1B). Furthermore, the special cause in winter 2018 noted on the seasonal chart, which content experts suspected was due to a flu epidemic, is eliminated as a special cause on the volume adjusted I chart. Table 1 summarizes the I charts from this study of adjustment methods on the primary outcome, average monthly wait time (Table 1).

Table 1.

Summary of Wait Time Charts

In addition to reducing the special cause signals of the unadjusted chart, both of the adjustment methods reduced the common cause variation (summarized by the value of sigma) on the adjusted charts. This change leads to more sensitive control limits in both of the adjusted charts.

A Second Example of Seasonal and Volume Adjustment for LOS

To validate that these models adjust for the impact of seasonal and volume variation on outcomes, we provide a second example for the total LOS in the ED. See Figure 5, which shows the unadjusted I charts, the I chart adjusted for seasonal effects, and the I chart adjusted for volume. The patterns on the adjusted charts and the reduction in common cause variation are similar to the waiting time charts. The table at the bottom of Figure 5 summarizes the control limits for the 3 charts. The effects of the adjustments on the LOS charts are similar to that seen on the wait time charts.

Fig. 5.

I charts for LOS unadjusted, adjusted by month, and adjusted by volume.

DISCUSSION

The results of this study demonstrate the advantage of adjusting for changes in baseline ED characteristics related to seasonal and monthly changes. The new models account for changes in season and patient volumes and provide a more accurate measurement of the improvement compared to the control charts for the unadjusted data. The results suggest that one should consider adjustments to the wait time data when evaluating the effects of interventions on waiting times measures.

The first method, seasonal adjustment by month, is useful for analyzing processes with anticipated and consistent, seasonal effects. Basic control charts do not address this challenge. In our example of paatient wait time data, the unadjusted I chart shows recurrent special cause in unfavorable directions each winter. Using that chart for QI purposes would suggest that any interventions made were effective during the summer months only and the gains were lost by winter. Also, the unadjusted QI chart suggests that this happens each year routinely. Only after adjusting for the special high and low months can the process be reliably assessed through all the seasons of the year and provide learning opportunities. Content experts can then take a deeper dive and learn from specific data points with special cause signals. A disadvantage of this approach is that the seasonal effect (often due to viral prevalence) may not always be consistent by month from year to year.

The second method, adjusting for monthly patient volumes, allows more timely learning from control charts. Monthly volume adjustments may be preferable to seasonal adjustments because determining the volume correction allows evaluation of the intervention’s success at any point in the year, independent of seasons. Although we used monthly data in this study, other periods such as day or week could be evaluated using a volume adjustment. The adjustment also creates an immediate predictive capacity related to the control chart. For the wait time example, we can expect a 6.7-minute increase in wait times for every increase in monthly volumes of 100 patients. Managers can use this information to determine staffing needs with increased precision to sustain satisfactory patient wait times.

Identifying and quantifying the effects of variation due to patient volume has the potential to inform sustainability plans by helping QI teams better understand the impact of the volume on existing processes and outcomes. QI analysis tools like run charts and control charts of project measures are designed to visualize patterns of variation over time. But if other variables (such as season or volume here) are contributing to these patterns, it makes it difficult to evaluate the impact of interventions that are believed to lead to improvement. The methods of adjustment described in this article can be used to reduce the impact of other variables and make the effect of the intervention visible and clear. This approach is a useful means of validating suspicions of content experts and integrating that knowledge into the QI process.

In this study, we only considered adjustments to wait-time data analyzed on I charts. But we think this approach would also be useful for other types of Shewhart charts. The seasonal adjustment has been used with charts for attribute data,[6] and there is no reason our approach to volume adjustment could not be useful.

Limitations

We performed this study in a single institution, and therefore, the specific effects of volumes and wait times will differ by institution. Nevertheless, the principles of adjusting for these factors would remain the same regardless of institution.

Readers may note that once we demonstrated the method adjusting for volume, there are other factors potentially affecting the outcome measure that could be adjusted, or controlled for when analyzing data on control charts. Broad categories of factors contributing to variability exist in the ED setting. A key example in our ED setting would be nurse staffing shortages on a given day, which impact patient wait times. Every healthcare setting provides independent variables that serve as confounders for the QI processes, and those variables could be identified and measured when monitoring the process. Hopefully, our approach described here will spur more investigations into multivariable adjustments.

Another limitation is that some users may be uncomfortable with seeing adjusted data on control charts, as they do not reflect the actual performance at any given time and they cannot be used to predict actual wait times for future months.[6] An alternative approach (that will be explored in a follow-up paper) is to adjust the centerline and control limits on a control chart for the seasonal or volume effects. This analysis will create control charts of the unadjusted data that can be used for predictions, making the varying expectations for future months clear.

CONCLUSIONS

This QI study is one of the first to address the methods to account for other factors in the clinical setting that affects outcome measures in QI work. Specifically, we found that our initial intervention was sustained, but the impact was different once we properly accounted for variation in patient volumes by individual patients and by months. Our findings support the importance of higher reliability QI methodology, such as identifying and controlling for variables within the control chart baseline data supporting the control charts. Future research should continue to explore adjusting QI methodology for dynamic health care setting with variability as an integral part of their process.

DISCLOSURE

The authors have no financial interest to declare in relation to the content of this article.