Validation of "Personal Protective Equipment Conservation Strategies Tool" to Predict Consumption of N95s, Facemasks, and Gowns During Pandemic-Related Shortages.

We sought to prospectively validate a model to predict the consumption of personal protective equipment in a pediatric emergency department during the COVID-19 pandemic. We developed the Personal Protective Equipment Conservation Strategies Tool, a Monte Carlo simulation model with input parameters defined by members of our emergency department personal protective equipment task force. Inputs include different conservation strategies that reflect dynamic reuse policies. Over the course of 4 consecutive weeks in April and May 2020, we used the model to predict the consumption of N95 respirators, facemasks, and gowns in our emergency department based on values for each input parameter. At the end of each week, we calculated the percent difference between actual consumption and predicted consumption based on model outputs. Actual consumption of personal protective equipment was within 20% of model predictions for each of the 4 consecutive weeks for N95s (range, -16.3% to 16.1%) and facemasks (range, -7.6% to 13.1%), using "maximum conservation" and "high conservation" strategies, respectively. Actual consumption of gowns was 11.8% less than predicted consumption for Week 1, gown resupply data were unavailable on Weeks 2-4. The Personal Protective Equipment Conservation Strategies Tool was prospectively validated for "maximum conservation" and "high conservation" models, with actual consumption within 20% of model predictions.

During the COVID-19 pandemic, healthcare workers and hospitals have experienced shortages of personal protective equipment (PPE), including gloves, face shields, masks, respirators, and gowns.[1] PPE shortages have put providers at an increased risk for infection during patient care. Healthcare workers have high rates of disease exposure, and significant numbers have been infected with severe acute respiratory syndrome coronavirus 2, which is partially attributable to an inadequate supply of PPE.[2-7] Compared with the general population, providers in the United Kingdom and the United States have a 3- to 11-fold increased risk of a positive test, with adequacy of PPE an important factor in modulating that risk.[8] Appropriate allocation of facemasks is especially important to protect against the spread of severe acute respiratory syndrome coronavirus 2, where asymptomatic carriage is relatively common.[9,10]

Mitigating PPE supply shortages requires a multifaceted approach. The Centers for Disease Control and Prevention has recommended contingency protocols for PPE conservation, including limiting the use of some PPE, reusing supplies intended for one-time use, and using supplies beyond their recommended shelf life.[11] Other strategies include activating the Defense Production Act to maximize production and repurposing manufacturing sites to meet demand.[1,12] Local grassroots efforts by independent designers, clinicians, and researchers have also led to innovative solutions such as 3D printing of face shields and fashioning protective masks from scuba diving equipment.[13]

Importance

Recommendations for conservation from the Centers for Disease Control and Prevention, hospitals, and our emergency department (ED) are altered frequently (see Supplemental Digital Content 1, http://links.lww.com/CIN/A128, with daily updates in PPE recommendations from COVID-19 leadership in our ED). With fluctuations in disease burden and evolving definitions of acceptable PPE reuse, governments and healthcare systems must be able to estimate the amount of PPE needed in real time. Accurate estimations of the potential demand for PPE vary depending on a range of potential pandemic scenarios, shifting policies, and utilization behaviors. To meet this demand, a number of PPE “burn rate” calculators have emerged to help estimate needs based on usage.[14,15] These resources use average consumption of PPE to estimate future use, relying on a number of assumptions around disease burden stability. Other simulation models to predict the consumption of PPE, developed concurrently to our own, generate outputs for a hospital, including inpatient and ICU patients.[16,17] Although these models could be modified to focus exclusively on ED PPE consumption, neither could be used to predict PPE consumption based on the most restrictive strategies for PPE used in our setting. Even using the staffing-based calculations of other models, the total number of patients is a required input to predict PPE consumption.[16,17] In our setting, stakeholders recommended restrictive PPE strategies requiring staff to reuse PPE across one or multiple shifts, independent of patient load and length of stay.

Goals of This Investigation

We created the PPECS-Tool (Personal Protective Equipment Conservation Strategies Tool) to compare the consumption of PPE, using various levels of conservation, during the pandemic-related shortage. We sought to prospectively validate the model over the course of 4 weeks of actual PPE consumption during spring 2020 in a large, urban pediatric ED. We hypothesized that actual use would be within ±20% of model predictions. Based on conversations with our ED PPE task force and the COVID-19 incident command team for the ED, this was the a priori threshold for the model to be clinically useful in making decisions about PPE conservation strategies and requisition requests from our hospital's central supply.

METHODS

Study Setting

We performed this study in a large, urban, academic children's hospital ED in the United States with approximately 90 000 pediatric ED visits annually. There are 135 full- and part-time medical providers and 130 nurses and patient care technicians. The study period was 4 consecutive weeks between April 20 and May 17, 2020.

Model Development

We built a set of four models, representing four levels of PPE conservation strategies, to predict daily consumption of N95 respirators, surgical facemasks, and gowns. These models represent increasingly restrictive use patterns, based on strategies considered or actually used in our clinical setting. Pre-pandemic, PPE was not reused between patients, consistent with our model representation of “liberal use.” In the “moderate conservation” model, each healthcare worker reuses each article across every interaction with the same patient. In the “high conservation” model, each healthcare worker is assigned a limited quantity of the PPE article per shift. In the “maximum conservation” model, a subset of healthcare workers reuse a single article across multiple shifts. Each model for the PPECS-Tool uses different input parameters, as displayed in Table 1. Users are required to provide their own estimates for input parameters based on recent patient arrival patterns or epidemiologic prediction models.

Table 1

PPECS-Tool Input Variables for Each Level of Conservation

	Level of Conservation				Input
	Liberal	Moderate	High	Maximum
Variables included in the final tool
Daily patient volume	X	X			Min, max
Patient risk levelsa as proportions of patient volume	X	X			Min, max
No. involved healthcare workers per patient by risk levela	X	X			Min, max
No. interactions per healthcare worker per patient by risk levela	X				Min, max
No. healthcare workers scheduled for the next 24 h by roleb			X		Min, max
Healthcare worker PPE requirements by patient risk levela		X			Yes/no
Probability that healthcare worker uses a second PPE article per patient, by risk levela		X			Constant
No. PPE articles assigned per healthcare worker per shift			X		Min, max
No. healthcare workers needing article of PPE per day by roleb				X	Min, max
Proportion of healthcare workers who bring the article from prior shift by roleb				X	Min, max
Variables excluded from the final tool
Patient PPE requirements by risk levela	X	X	X	X	Yes, no
No. accompanying support people per patient	X	X	X	X	Min, max
Accompanying support person PPE requirement	X	X	X	X	Yes, no

aPatient risk levels: high risk (person under investigation: ill/requiring hospitalization), medium risk (person under investigation: not requiring hospitalization), and low risk (other patients).

bHealthcare worker roles: physicians, nurses, respiratory therapists, and other (eg, technicians).

Each model is a Monte Carlo simulation that produces a probability distribution of real-world outcomes via simple calculations between multiple variables. For a specified conservation strategy, the values of all the variables included in that model are multiplied together. This calculation is repeated across many replications (default, 300). For every replication, new values are randomly generated for each variable, with uniform probability between possible minimum and maximum values, obtained by user input. These inputs are based on recent experience or historical data, excluding outliers. Each variable is independent and identically distributed.

We built the models in Microsoft Excel (v. 2013, Microsoft Corporation, Redmond, WA, USA) with Visual Basic. We performed error-checking and verified that outputs from all four models in our overall model match direct arithmetic calculations using the same inputs. We made an initial version of our model publicly available as the Excel-based simulation on April 1, 2020.[18] We incorporated stakeholder feedback for a subsequent Web application, built in JavaScript using the same formulas, available at https://ppecs-tool.childrensnational.org.

Selection of Model Inputs

On Monday of each week, members of our ED PPE task force met to discuss current PPE supply levels and determine if changes were necessary in PPE conservation strategies. We used the task force determination to select the conservation strategy in our model and make predictions over the following week. Each Monday, we also queried aggregate patient counts from our EHR. These patient counts were used to determine the minimum and maximum number of patients presenting on a single day in the preceding week and to estimate the minimum and maximum number of patients who would be considered a “person under investigation,” defined by the number of patients for whom COVID testing was sent. Staffing was based on average daily staff levels from the previous week. After entering all of the input parameters, based on data from the prior 7 days, we executed the simulation to generate a prediction for the 24-hour consumption of each article over the following 7 days.

Validation Measures

At the end of each week, we determined the mean 24-hour consumption of each article using an in-person count of the remaining inventory (see Supplemental Digital Content 2, http://links.lww.com/CIN/A129, which displays our PPE COVID Inventory Count Guide) and by adjusting for the number of articles provided by our hospital central supply during that period. Specifically, we tracked consumption of surgical masks with elastic ear loops, the standard facemask for healthcare workers in our unit. We did not distinguish between regular and small-size N95s because prediction of different PPE subtypes was beyond the scope of our model. Predicting combined regular and small-size N95 consumption was useful for our unit, where central supply established a maximum weekly limit of total respirators based on hospital supplies and provided the distribution of N95 sizes as requested by our ED PPE task force. Central supply does not keep records on the resupply of disposable gowns to each hospital unit. Actual consumption of gowns could only be confirmed by daily, in-person counts of inventory.

Outcomes

We defined average daily consumption as the difference between the most recent PPE count and the count on the day the model predictions were made, minus the resupply during that period, divided by the number of days between each count. For each of the four study weeks, we compared actual consumption of PPE with model predictions from the preceding week. Although we used historical arrival rates and recent staffing as input parameters, we compared model output with PPE use in the subsequent week, as the goal of the model is to predict future consumption based on a specific conservation strategy and the best available input data. We used this basic evaluation of model accuracy because significantly underestimating or overestimating necessary supplies can have severe consequences during a period of shortage.[19] We assessed whether actual consumption fell within 20% of the predicted consumption for each item for each week in the study period.

This study was acknowledged by the institutional review board as “Not Human Subjects Research” and did not undergo institutional review board review.

RESULTS

Stakeholder Feedback

After publicly posting an early version of the model, we received feedback from one user who thought every model input was necessary to predict PPE consumption, regardless of conservation strategy. To address this in the final Web application, the PPECS-Tool displays only the required inputs for the selected conservation strategy. Beginning on April 17, 2020, our hospital policy changed and visitors arriving to the ED were no longer offered a facemask from our ED supply but rather offered a cloth mask at the hospital entrance. Based on this local experience, we excluded the three user options from the final version of the model that previously allowed users to select options for distributing PPE articles to patients and support persons (bottom three rows, Table 1).

Study Week Characteristics

Staff schedules had already been significantly decreased by the time of initiation of our model validation effort, and staffing remained stable throughout the 4 weeks of validation. Mean daily ED volumes were 85 (Week 1), 97 (Week 2), 101 (Week 3), and 104 (Week 4).

Modeling Results

For each article during each study week, the Excel-based simulation produced an estimated 24-hour consumption rate with a 95% confidence interval. Our use patterns for N95 respirators were stable from week to week, so our input parameters did not change. Model outputs for N95s, including a mean estimate of 17.2 respirators consumed per day, are displayed in Figure 1. During weekly meetings with the ED PPE task force, they progressively estimated fewer ED facemasks would be used per healthcare worker as staff became more familiar with facemask distribution and reuse policies. Based on these model inputs, daily facemask use was predicted to be 86.1 for Week 1 and decreased over the study period to 29.2. Predicted gown use ranged between 148.6 and 185.6 per day. We were only able to compare gown predictions with actual consumption for Week 1, during which model outputs averaged 156.3 gowns used per day.

FIGURE 1

Example model outputs for Excel-based simulation, including estimates that 17.2 N95 respirators will be used in the following 24 hours, with 95% confidence interval of 17.0–17.4. Ninety-five percent confidence interval is based on the number replications and not a representation of real-word precision. These outputs were generated from (1) a user selection of maximum conservation; (2) estimates for the number of healthcare workers who will need to use an N95 in the next 24 hours, that is, between 10–14 physicians, 20–24 nurses, 3–5 respiratory therapists, and 4–6 others (eg, technicians); and (3) an estimate that between 50% and 70% of these healthcare workers would bring an N95 from a prior shift. Abbreviation: HCW, healthcare worker. ©2021 Children's National Hospital & Kelly Na'amah Rickard, image of tool outputs reproduced with authors' permission.

Main Results

Conservation strategies for each article of PPE were stable over our 4-week study period (N95: “maximum conservation”; facemask: “high conservation”; gown: “liberal use”). Table 2 displays average daily consumption of PPE, which was within 20% of model predictions for each of 4 consecutive weeks for N95s (range, −16.3% to 16.1%) and facemasks (range, −7.6% to 13.1%). Average daily consumption of gowns was within 20% of model predictions for Week 1 (11.7% difference) but could not be assessed for Weeks 2-4 because of a lack of resupply data.

Table 2

Predicted and Actual 24-Hour PPE Consumption During 4 Weeks in Spring 2020

PPE (Conservation Strategy)	Predicted 24-h Consumption, No. Articles	Actual 24-h Consumption, Mean No. Articles Over the Collection Period	% Difference
Week 1: 4/20/2020 to 4/25/2020a
N95 (maximum)	17.2	15.4	10.5
Facemasks (high)	86.1	74.8	13.1
Gownsb (liberal use)	156.3	138	11.7
Week 2: 4/27/2020 to 5/4/2020
N95 (maximum)	17.2	14.43	16.1
Facemasks (high)	58.8	63.29	−7.6
Week 3: 5/4/2020 to 5/11/2020
N95 (maximum)	17.2	16.86	2.0
Facemasks (high)	58.8	57.86	1.6
Week 4: 5/11/2020 to 5/17/2020a
N95 (maximum)	17.2	20	−16.3
Facemasks (high)	29.2	26.7	8.6

aBased on the availability of study team to physically count ED PPE inventory, Week 1 and Week 4 validation data included shortened, 5- and 6-day data collection periods, respectively.

bNo resupply data were available for gowns on Weeks 2–4. Analysis of model predictions for gown consumption was limited to the period with daily inventory counts (Week 1).

DISCUSSION

To our knowledge, the PPECS-Tool is the first prospectively validated model to predict consumption of PPE with a simulation model, based on site-specific information and general conservation strategies. Models for “high conservation” and “maximum conservation” were validated, based on our a priori outcome of actual consumption within ±20% of model predictions over the course of 4 consecutive weeks. Models for “liberal use” and “moderate conservation” were not validated in this study. The minimal difference between model predictions and actual consumption of gowns during Week 1 of our study suggests that the “liberal use” model may be appropriate for a general estimate for PPE consumption, but there is not enough evidence to support the use of “liberal use” and “moderate conservation” models to predict PPE consumption. Because only “high conservation” and “maximum conservation” strategies were validated in our study, this model has the greatest value for decision-makers who are considering more restrictive use of PPE to maintain supply during an acute shortage. For stakeholders who are beginning to liberalize use of PPE, past use patterns may provide more reliable estimates of “liberal use” or “moderate conservation” than the outputs from our model.

In a unit with stable PPE protocols and patient population, closely tracking usage and resupply remains the most important strategy to estimate ongoing consumption of PPE. The PPECS-Tool provides an option to explore PPE consumption with different strategies or in the event of a change in resource demands, such as from a surge in the number of patients presenting. Other researchers have been successful in using Monte Carlo simulation for forecasting hospital bed demand during the pandemic, reporting up to 85%-95% accuracy.[20]

The challenges in obtaining and preserving PPE supplies have shifted since the pandemic began more than 1 year ago.[21] Due to efforts to distribute facemasks to patients and family members early in the pandemic, we initially included a model option to distribute PPE to visitors. We eliminated this input from the final model since it was set to zero in each of the 4 weeks of our validation study. The availability of facemasks has greatly increased since the early days of the pandemic, and most Americans now report regular use of facemasks in public,[22] with survey data regarding mask usage supported by direct observations.[23] Although some healthcare decision-makers might still opt to distribute hospital facemasks to visitors who are already using a facial covering, which has not been our practice, and other than occasional facemasks for patients brought in by emergency medical services, no articles of PPE have been distributed to visitors out of the ED supply since mid-April 2020 (see Supplemental Digital Content 1, http://links.lww.com/CIN/A128). Since our model was initially developed in spring 2020, processes to sterilize N95 respirators have provided new opportunities to extend the use of PPE.[24] We did not update our model based on the availability of N95 sterilization processes, and stakeholders who take advantage of such processes would have to consider the impact of N95 sterilization when generating the model input for the “proportion of healthcare workers who bring the article from prior shift.”

We chose to use descriptive labels for conservation strategies rather than the three levels of operational status used by the Centers for Disease Control and Prevention: conventional, contingency, and crisis. The Centers for Disease Control and Prevention made recommendations for contingency and crisis statuses that resemble the “high conservation” and “maximum conservation” strategies in our model (eg, implement extended use of PPE and implement limited reuse of PPE with extended use, respectively).[11] However, each operational status is also associated with a suite of other administrative actions, such as canceling nonemergency procedures and deferring outpatient encounters,[25] which might not be enacted uniformly across institutions and are often outside the control of staff in a particular healthcare unit or department. Still, any data-driven response to a PPE shortage requires a multifaceted response with broader interventions than simply changing PPE use patterns in a single healthcare unit. In addition to the need to dedicate resources to maintaining sufficient supplies of PPE, resources are required for consistent research and training efforts to maximize the effectiveness of PPE for providers.[26]

There are several important limitations to this study. First, we assumed independence between model inputs. Prior research into the impact of ED crowding on hospitalization offers one example for how model inputs are not completely independent in the real world. A greater proportion of high-acuity ED patients may be admitted to the hospital during periods of ED crowding,[27] potentially leading to model underestimates in these circumstances. Other model inputs, such as the number of healthcare workers per patient and the number of interactions with each healthcare worker, are also dependent on ED volume and might act consistent with or counter to the effect of ED crowding on hospitalization. The unpredictable impact of ED crowding on model estimates was untested during our study period. For these reasons, under circumstances of ED crowding, our independence assumption is a major limitation for the “liberal use” model, which was also not validated in the present study because of insufficient resupply data on gowns.

Model outputs are also limited by the need for each user to estimate input parameters. It was not feasible to count the number of healthcare workers who interact with each patient, and we had to estimate that model input based on our clinical experience. We selected ED volumes based on presentation rates over the preceding week, but we were not able to use a predictive algorithm to accurately forecast ED volumes as an input for the model. We expect most decision-makers will have similar limitations in their ability to select input parameters for the model. Other researchers have attempted to address the challenge of burdensome input parameters, most notably the PPE forecasting calculator developed at the University of Pennsylvania, which integrates with a model to predict COVID-related hospitalization rates. Unfortunately, their hospitalization prediction is only applicable to the period prior to peak infections.[17]

Our model performs well for predicting PPE consumption with “high conservation” and “maximum conservation” strategies at our institution, but it is unknown how generalizable these results are to different pediatric EDs, adult EDs, or other healthcare settings with less restrictive access to PPE. In our ED, access to N95s was restricted to a trusted group of senior nurses, only when working in their role as charge nurses. Small allocations of facemasks and gowns were available in highly visible public areas in the ED. Larger stores of PPE were kept in a single storage room in the ED under lock and key. Units with less restrictive access to PPE might be at a greater risk from supply chain diversions, which would lead to model underestimates.

Furthermore, some input parameters may be difficult to generate in a nonacademic setting, especially for the “liberal use” and “moderate conservation” options that were not validated in the present study. All our models rely on uniform probability distributions: if a user were to include outliers in their maximum estimates for input parameters, our use of uniform distributions would cause an overestimate in the consumption of PPE. Furthermore, many decision-makers may not have access to historical data regarding the proportion of patients of specific risk levels, much less an effective way to predict that patient breakdown. We included these risk levels, namely, high risk (person under investigation: ill/requiring hospitalization), medium risk (person under investigation: not requiring hospitalization), and low risk (other patients), because they played a role in the operational use of PPE (gowns) at our institution, but this variable added greatly to the complexity of using the model in our setting even with a team of researchers dedicated to data analytics in our ED. Future efforts in the prediction of PPE consumption will consider the value of model parsimony to ensure good performance and minimize energy in input data generation.

In summary, the PPECS-Tool was prospectively validated for “maximum conservation” and “high conservation” models, with actual consumption within 20% of model predictions during each of 4 consecutive weeks in spring 2020. The accuracy of model predictions depends on the accuracy of model inputs and model assumptions.