COVID-19 Vaccine Clinic Real-Time Throughput Analysis: Development and Implementation of an Innovative Data Collection Tool.

BACKGROUND: The novel coronavirus disease 2019 (COVID-19) pandemic has presented the healthcare system with a plethora of challenges, including implementation of an efficient vaccination strategy. Mass vaccinations have been used during previous pandemics; however, the associated data have largely been limited to theoretical simulations and post hoc analysis.
METHODS: An innovative data collection tool was created to deliver real-time data analysis during a drive-through mass vaccination. Patients were assigned unique identification numbers at the clinic entrance. Using these identification numbers, and the web-based spreadsheet, patients were tracked throughout the vaccination process. Static timestamps corresponding to the entry and exit at each checkpoint were recorded in real time.
RESULTS: Data were collected on a total of 3,744 vehicles over five clinic days. Total time was collected, from entry to exit, on 2,860 vehicles. Registration and vaccination times were collected on 3,111 vehicles. Of the vehicles sampled, 1,588 (42%) had data points associated with all checkpoints.
CONCLUSIONS: This open-source, innovative data collection tool was successfully implemented in our mass vaccination clinic for tracking patients in real time providing actionable data on overall throughput efficiency. This cost-effective tool can be used on a variety of healthcare-related projects to provide data-driven evaluation on the efficiency of care.

Introduction

The novel coronavirus disease 2019 (COVID-19) has quickly become the most critical public health crisis of this century. The current death toll is approximately 5.05 million individuals worldwide and counting.[1] Over the past year, this global pandemic has overwhelmed medical systems, both in the United States and internationally. At the peak of the pandemic, the United States was losing more than 3,000 citizens daily from COVID-19, more than the number of lives lost on D-Day, during the Normandy invasion of World War II or the attacks on September 11, 2001.[2]

Pfizer-BioNTech and Moderna COVID-19 vaccines became available through an emergency use authorization by the US Food and Drug Administration at the end of 2020.[3-5] As vaccinations became more readily available, public health officials, hospital systems, and politicians alike struggled to identify the quickest way to vaccinate the public and curtail the pandemic.

Mass vaccination clinics represent one important arm of the overall vaccination strategy, which also includes brick-and-mortar, mobile, and community-based clinics. However, mass vaccinations are inherently complex and require thorough coordination, planning, and design iteration to be successful. Therefore, efficiency, defined as the ability to achieve a goal with little to no time or material waste, became the primary metric in the country's ability to vaccinate the public quickly. Theoretical simulation models can be implemented to quickly calculate and visualize throughput capacity and improve the overall efficiency of the mass vaccination clinic before putting doses in arms.[6] Although these models are useful in the planning stage, it is paramount to have a process in place that can evaluate clinic efficiency in real time. Measuring the efficiency of the vaccination clinic enables operators to reduce unused doses, promote safety, reduce harm, and focus on vaccinating as many individuals as possible in as short a period as possible.

As a part of the community effort, a large twelve-hospital health system in our state opened a variety of vaccination clinics including fixed brick-and-mortar clinics, community pop-ups, and a drive-through mass vaccination site. The drive-through mass vaccination events occurred in a large parking lot, which was owned by a major league baseball team. A number of mass vaccination sites have been established at sporting stadiums and arenas that were otherwise idle during the pandemic.[7] As one of the largest COVID-19 mass vaccination events to date, this drive-through clinic set new standards for efficiency and volume.[8]

To track patients in real time and provide actionable data on traffic flow and overall throughput efficiency, we developed an open-source, novel, and innovative data collection tool. This low-cost, easy-to-use tool is adaptable to a variety of clinic layouts and showed success in this mass vaccination drive-through clinic.

Methods

Vaccination Clinic Overview

This project was designated as a quality improvement initiative and deemed exempt from review by the institutional review board. The site of implementation was two large outdoor parking lots with a two-lane road connecting them. The primary parking lot was approximately 2,600 feet long and wide enough to support 22 vehicles parked side by side with access roads to the east and west of the parking stalls (Figure 1). The second parking lot to the south had a capacity of 500 vehicles and was used for the 15-minute observation period postvaccination (Figure 2).

Figure 1.

Mass vaccination primary parking lot site plan.

Figure 2.

Mass vaccination observation parking lot site plan.

As of March 31, 2021, five drive-through Pfizer-BioNTech COVID-19 vaccination clinics were held including an initial pilot clinic and four subsequent vaccination clinics on the following weekends. At the pilot event, 982 first-dose vaccinations were administered in approximately 2 hours with patients' second doses being completed at an indoor clinic 3 weeks later because of inclement weather. The subsequent vaccination clinics were completed the following weekend. A total of 9,822 vaccines were administered in 12 hours over the course of two 6-hour days. These patients returned to the drive-through vaccination clinic 3 weeks later for their second vaccine doses.

During the pilot and two first-dose clinics, the layout began with entry to the primary parking lot where vehicles quickly fanned into six queuing lanes preceding the large, central registration tent wide enough to span all six lanes while simultaneously accommodating three cars, bumper to bumper, in each lane. After registration, vehicles were directed to 1 of 16 vaccination cabanas where vaccination and documentation were completed (Figure 1). Once vaccination was complete, vehicles were directed to the observation lots to complete the recommended 15-minute observation period in the presence of a medical observation team (Figure 2).

When patients returned for their second vaccine dose, the clinic layout was modified. Patients continued to enter the large parking lot but were now fanned into eight queuing lanes, increased from six. These queuing lanes fed into a larger central tent that was eight lanes wide with each lane simultaneously accommodating four cars, bumper to bumper. With the modified layout, registration and vaccination were completed at the large, central tent that eliminated the 16 vaccination cabanas from the initial clinics (Figure 3). Once vaccination was complete, vehicles were again directed to the observation lots to complete the recommended observation period in the presence of a medical observation team (Figure 4).

Figure 3.

Revised mass vaccination primary parking lot site plan.

Figure 4.

Revised mass vaccination observation parking lot site plan.

The novel data collection tool was created in Google Sheets (Google LLC, Mountain View, CA), and observers used checkboxes to record static timestamps at various checkpoints throughout the vaccine clinic determined by the layouts described above. As an operational team, we made the goal to track approximately one in every five vehicles that came through the clinic each day. The sampled vehicles were tracked from arrival to departure with four checkpoints—arrival, registration tent, vaccination tent, and observation area—during the pilot and first-dose vaccine clinics, and three checkpoints—arrival, registration or vaccination tent, and observation area—for the second-dose vaccine clinics with the revised layout. The Google Sheets data capture form consisted of three columns: vehicle identification number, the arrival time, and the departure time (Figure 5) corresponding to each checkpoint, with only the vehicle arrival time recorded at the time of arrival, which started the clock on the vaccination process.

Figure 5.

Google Sheets checkpoint data input sheet.

On arrival at the clinic, each vehicle selected for tracking was given a unique identification number printed on an 8.5″ × 11″ fluorescent-colored paper and placed under the drivers-side windshield wiper. To minimize a potential bottleneck, the identification number was placed on the vehicle at the first required stop in the queue rather than at the entrance to the parking lot. As an observer placed an identification number on the vehicle, they used their personal smartphone device to check the checkbox associated with the arrival time for the corresponding identification number. This recorded a static timestamp that served as the arrival time to the vaccine clinic. As the vehicle moved through each checkpoint, observers checked the arrival and departure checkboxes corresponding to the unique identification number on each Google Sheets checkpoint tab. For consistency, observers were educated to record arrival at the vaccination tent on first contact with the vaccination team and departure when the vehicle left the vaccination tent. Volunteers monitoring the observation area checked the arrival time on entry and the departure time once their observation period was complete and the vehicle left the observation lot. Total time in the clinic was calculated using arrival time at the clinic and departure time from the observation area. Safety was ensured by volunteer flaggers, and the observation area was staffed by a medical response team.

Data Collection Tool Development

The Google Sheets template consisted of three columns corresponding to the unique identification number, the arrival time, and the departure time at each identified checkpoint (Figure 5). Each cell in the arrival and departure columns were formatted to include a checkbox (Insert, Checkbox) which was set to the default values of checked = TRUE and unchecked = FALSE. When the box was checked, a static timestamp was reported. To accomplish this function, a script was written in the Script Editor that when a checkbox was checked, a value of TRUE was returned. Then, using the “newDate()” function, a static timestamp was recorded in the Summary sheet corresponding to the respective checkpoint and unique identification number of the vehicle. If an input was recorded incorrectly by a volunteer, removing the inappropriate check resulted in the timestamp being erased. The full script can be viewed in the Supplemental Digital Content 1 (see Appendix, http://links.lww.com/JHQ/A154). This three-column template was copied into multiple rows and duplicated into separate sheets for each checkpoint dependent on the vaccination clinic layout. This allowed volunteers to only work on the tab associated with their assigned checkpoint, decreasing the risk of aberrant data being recorded.

In addition to the sheets corresponding to each individual checkpoint, Summary (Figure 6) and Dashboard sheets were created (Figure 7). The Summary sheet is automatically populated with all arrival and departure timestamps for each checkpoint regarding their unique identification number. This sheet served as the collection of raw data for each sampled vehicle corresponding to each checkpoint. The Dashboard sheet provided interval measurements calculated from the Summary data. With the prepopulated input modeling, each time interval was calculated and could be viewed in real time. Conditional formatting was set for each interval, and colored highlights were triggered when times for each interval fell outside of the target time limit. Initially, for the pilot clinic, these time limits were estimated by working backwards from the maximum allowable time in the clinic in order for the overall vaccination clinic to run successfully. As data were collected at each subsequent clinic day, these time limits were revised to better reflect the true upper limit values associated with each checkpoint in the process.

Figure 6.

Google Sheets summary sheet—reports static timestamps by checkpoint.

Figure 7.

Google Sheets dashboard sheet—calculated intervals in real time.

Results

Time study data were collected on 3,744 vehicles (18%) over the five clinic days (Table 1). This included any vehicle that had a timestamp recorded, and the percentage calculation assumes one vaccine per vehicle, for a maximum of 20,621 vehicles traveling through the clinics. Of the total number of vehicles sampled, timestamps at every checkpoint were collected on 1,588 vehicles (42%). Data points were most likely to be recorded at the registration and vaccination steps. 3,111 sampled vehicles (83%) had both registration and vaccination times collected. 2,860 sampled vehicles (76%) had timestamps corresponding to both entry and exit times collected. Only one accident occurred, and no individuals were injured. No serious allergic reactions occurred that required EMS transportation.

Table 1.

Comparison of Vaccination Clinics With Number of Vaccines Administered and Vehicles Sampled

Date	Vaccines administered	Vehicles sampled (% of vaccines administered)	Entry to exit (% of vehicles sampled)	Registration and vaccination (% of vehicles sampled)
1/24/21	982	225 (23%)	185 (82%)	183 (81%)
1/30/21	4,877	906 (19%)	795 (88%)	777 (86%)
1/31/21	4,945	997 (20%)	716 (72%)	813 (82%)
2/20/21	4,895	768 (16%)	538 (70%)	705 (92%)
2/21/21	4,922	848 (17%)	626 (74%)	633 (75%)
Total	20,621	3,744 (18%)	2,860 (76%)	3,111 (83%)

Analysis of the time study data was completed for each vaccination clinic (Table 2). The median length of time in the vaccination clinic for patients during the pilot was 23:34, including the mandatory 15 minutes for observation. The median time for patients during the first-dose vaccination clinics was 28:01 and 22:41, respectively, including a mandatory 15-minute observation period. The median times during the subsequent second-dose vaccination clinics were 14:52 and 14:25, respectively, with an optional 10-minute observation period.

Table 2.

Median Durations by Checkpoint (minutes)

Clinic date	Greeter to registration	Registration	Registration to vaccine	Vaccination	Observation	Total clinic time
1/24/2021[a]	0:01:54	0:01:14	0:00:55	0:01:29	0:16:41	0:23:24
1/30/2021	0:07:37	0:00:45	0:01:51	0:02:11	0:14:02	0:28:01
1/31/2021	0:02:50	0:00:36	0:01:15	0:02:18	0:14:38	0:22:41
2/20/2021	0:02:59	0:01:53[b]			0:09:21[c]	0:14:52
2/21/2021	0:06:40	0:01:55[b]			0:04:55[c]	0:14:25

Pilot clinic.

Combined registration + vaccination in revised layout.

Observation was recommended, but not required.

A total of 127 health profession students were used as volunteer observers for data input over 5 days of vaccination clinics with an average of 26 volunteer observers per clinic day. These volunteers were stationed at each checkpoint throughout the process and, through Google Sheets, used the data collection tool simultaneously to input data using the arrival and departure checkboxes.

Limitations

Although we estimated that time study data were collected for approximately 18% of vehicles over the five clinic days, we did not record how many patients carpooled. Therefore, it is possible the actual number of vehicles traveling through the clinic was less. There were also significant technological and operational improvements made between the first and second-dose clinics that likely affected the success and time study data. Therefore, we are unable to statistically compare the time study data collected from each clinic.

The primary technological barrier to implementing this data collection model is the need for multiple smart devices connected to a wireless network. During the mass vaccination clinic, each volunteer observer used a smartphone to operate the Google Sheets mobile application. Furthermore, to view the incoming data in real time, each of these devices was connected to the wireless network at the clinic site or to their provider's network. In a low-resource setting, obtaining the required number of connected devices and a reliable network connection could make real-time data collection difficult.

The limiting resource, from our clinic, was the number of volunteer observers required for data input, which is dependent on the location and process being measured. At the drive-through clinic, an average of 26 volunteers per clinic day was used for data input. For the first two clinic days, we used a workflow that had patients registered and vaccinated at two different physical locations. This resulted in a need for more volunteers because of the clinic layout. However, with technological advancements and process improvement initiatives—largely informed by the data collected—the second weekend had registration and vaccination take place at the same large tent. This required fewer volunteer observers, resulting in more reliable data collection and allowed the remaining volunteers to assist in other areas of the mass vaccination clinic.

Looking to the future, we anticipate integrating more advanced technologies such as Wi-Fi sniffing, Bluetooth tags, or automated license plate readers to automate the data collection process and greatly minimize the required number of volunteer observers. This improvement would allow more complete data to be reliably captured in real time but also represents substantial financial investment in technology.

Discussion

This open-source, novel time study data collection tool was developed and successfully implemented at the mass drive-through vaccination events in our state. The data collection process occurred simultaneously with vaccination and did not affect the overall patient experience. Of note, data points were more often recorded at the registration and vaccination steps. This is likely due to allocating a higher proportion of volunteers and the ease of observing vehicles at these checkpoints. This tool worked to improve clinic efficiency, mitigate the risk of unused vaccine doses, and vaccinate as many individuals as possible. We believe that this tool could be used to streamline additional mass vaccination clinics for future immunization campaigns.

There have been previous data collection efforts in mass vaccination events, specifically the 2009 H1N1 Influenza Vaccination Campaign; however, these were not able to provide data for analysis in real time and did not evaluate a drive-through clinic.[9] In addition, compared with previous data collection tools, our patients did not have to engage in the collection process in any way.[9]

This technology was found to be adaptable and successful. Owing to extremely adverse weather events in our state, patients from the initial pilot clinic were moved to receive their second dose at an indoor vaccination clinic at our hospital. With less than 24-hour notice, the data collection tool was modified to reflect the layout and process of this specific brick-and-mortar vaccination clinic. With only minor alterations to the model, it was successfully adapted to a confined, indoor space to collect reliable time study data on approximately 200 patients.

The innovative use of this data collection tool allowed the operations team to capture data in real time and make on-the-fly improvements to the process contemporaneously. The major bottleneck in our process was in the travel time from the end of vaccination to the observation parking lots. Initially, the number of registration and vaccination tents was split equally into the two observation lots. From looking at the time study data in real time, there was an obvious delay in vehicles moving to one observation lot compared with the other. The traffic patterns were altered in the middle of the vaccine clinic to redirect additional vehicles to the more efficient observation lot, thus reducing the vehicular load on the less-efficient lot. This real-time alteration in the process allowed for improved traffic flow and increased efficiency through the clinic.

Conclusions

As the rate of vaccine doses available increases daily, there is an even greater need to incorporate technology to operate vaccine clinics more efficiently. Data-driven improvements can improve clinic efficiency, mitigate the risk of unused vaccine doses, and continue toward the common goal of getting as many doses in the arms of patients as possible. In addition, real-time evaluation and iterative process improvement will be essential in any additional pandemic settings in the future.

Implications

In addition to mass vaccination clinics, this novel time study data collection tool can be adapted to conduct workflow and efficiency assessments of any process involving the movement of people or vehicles. With the ultimate goal to improve the overall quality of patient care, this tool can be implemented to provide both real-time and post hoc data analysis to improve efficiency in various clinical settings.

Authors' Biographies

Michael D. Skaggs, MS, is a second year medical student at the University of Colorado School of Medicine in Aurora, CO. Prior to medical school, he worked as a civil engineer on mass transit and water resource projects and is interested in emergency medicine.

Sarah K. Wendel, MD, graduated from Georgetown School of Medicine and completed her residency in emergency medicine at Wake Forest Baptist Medical Center in Winston-Salem, NC. Currently, she is in the second year of an Administration, Operations, and Quality fellowship and is a practicing emergency medicine physician at the University of Colorado School of Medicine in Aurora, CO.

Richard D. Zane, MD, graduated from medical school at Temple University in Philadelphia, PA, followed by residency training in emergency medicine at the Johns Hopkins School of Medicine where he joined the faculty as Assistant Chief of Service. He then joined the faculty at Harvard Medical School and Brigham and Women's Hospital in Boston where he developed tools for preparedness and response to mass casualty care and disaster that have been used domestically and across the globe. Currently, Dr. Zane serves as the George B. Boedecker Professor and Chair of the Department of Emergency Medicine at the University of Colorado School of Medicine, Professor of Health Administration at the University of Colorado Business School, and Chief Innovation Officer at UCHealth in Aurora, CO.

Daniel Resnick-Ault, MD, MBA, graduated from The Warren Alpert Medical School of Brown University, completed his residency in emergency medicine at Boston University Medical Center in Boston, MA, and fellowship in Administration, Operations, and Quality at the University of Colorado. Currently, he is the assistant medical director at the UCHealth University of Colorado emergency department in Aurora, CO, and an assistant professor of emergency medicine at the University of Colorado School of Medicine.