Impact of COVID-19 on initial emergency medical services certification in the United States.

Objective: As the COVID-19 pandemic began, there were significant concerns for the strength and stability of the emergency medical services (EMS) workforce. These concerns were heightened with the closure of examination centers and the cessation of certification examinations. The impact of this interruption on the EMS workforce is unclear. Our objective was to evaluate the impact of COVID-19 on initial EMS certification in the United States. In addition, we evaluated mitigation measures taken to address these interruptions.
Methods: This study was a cross-sectional evaluation of the National Certification Cognitive Examination administration and results for emergency medical technician (EMT) and paramedic candidates. We compared the number of examinations administered and first-attempt pass rates in 2020 (pandemic) to 2019 (control). Descriptive statistics and 2 one-sided tests of equivalence were used to assess if there was a relevant difference of ±5 percentage points.
Results: Total number of examinations administered decreased by 15% (EMT, 14%; paramedic, 7%). Without the addition of EMT remote proctoring, the EMT reduction would have been 35%. First-time pass rates were similar in both EMT (-0.9%) and paramedic (-1.9%) candidates, which did not meet our threshold of a relevant difference.
Conclusion: COVID-19 has had a measurable impact on examination administration for both levels of certification. First-time pass rates remained unaffected. EMT remote proctoring mitigated some of the impact of COVID-19 on examination administration, although a comparison with mitigation was not assessed. These reductions indicate a potential decrease in the newly certified workforce, but future evaluations will be necessary to assess the presence and magnitude of this impact.

INTRODUCTION

Background

Emergency medical services (EMS) professionals are an essential link in the continuum of emergency medical care in the United States. Before attaining state‐level licensure to practice, EMS professionals are required to demonstrate entry‐level competency to assure safe and effective patient care. This is a process that involves both a required completion of a formal educational program and certification examination, most commonly the National EMS Certification Cognitive Examination provided by the National Registry of Emergency Medical Technicians (National Registry). This examination is administered by the National Registry with results stored in the National EMS Certification Database. , The National EMS Certification Cognitive Examination is used in all states, territories, and federal agencies in the United States to verify minimum competency for authorization to practice. The examination is traditionally given at proctored sites, where candidates have a specific time allotment to complete the examination.

Completion of this process—education and certification—represents a point of entry for EMS practitioners into the workforce.

Importance

For a number of years, there have been concerns about the strength and stability of the EMS workforce. This is defined by the number of EMS professionals who enter the workforce as well as those who leave. During the COVID‐19 pandemic, major interruptions to the delivery of EMS education led to changes at each certification level, including suspension and/or flexibility of some graduation requirements. In addition, the National EMS Certification process itself was also affected, with the administration of cognitive examinations temporarily halted completely because of the risk of infection at cognitive examination centers nationwide through local stay‐at‐home orders and directives. Furthermore, when cognitive examination centers reopened, they functioned at a reduced capacity to satisfy local COVID‐19 restrictions. Although the psychomotor examination, a separate examination that pairs with the cognitive for national EMS certification, was equally affected, this was mitigated by the creation of a provisional certification that temporarily suspended that requirement. The time, extent, and local regulations around these cognitive examination center closures are unclear and were likely variable throughout the United States. The impact of both the educational requirement changes and the disruption of certification examination operations on the available number of practitioners to join the EMS workforce is unknown.

This study used data from the National Registry database to show that there was a decrease in the total number of examinations administered during the pandemic in 2020 when compared with 2019. The total number of examinations administered decreased by 14% for emergency medical technicians (EMTs) and 7% for paramedics in 2020. Had remote proctoring of the examination not been established in 2020, the EMT reduction would have been 35%. Nevertheless, the pandemic and the addition of remote proctoring did not result in any significant change in the pass rates for EMTs or paramedics.

Goals of this investigation

Our primary objective was to evaluate the impact of COVID‐19–related educational interruptions on the availability of potential new members to the EMS workforce. To achieve this, we analyzed the number of examinations administered and first‐attempt success on the national EMS certification examination for EMT and paramedic levels. In addition, we evaluated the mitigation measures taken to address these interruptions.

METHODS

Study design, setting, and population

This study is a cross‐sectional evaluation of examination administration and first‐attempt pass rates on the National EMS Certification Cognitive Examination provided by the National Registry for EMT and paramedic candidates from January 1, 2019, to December 31, 2020. The rationale for selecting this time frame was to compare EMT and paramedic initial certification examination administration before COVID‐19 (ie, a control) in 2019 with the COVID‐19–impacted year of 2020. Included in this analysis are all initial certification cognitive examinations administered at the EMT and paramedic certification levels, representing the entire initial testing population for each respective year.

Also included in this evaluation are the online proctored examinations for the EMT cognitive examination that began on May 12, 2020, to help mitigate the effects of shutdowns on initial certification numbers. , These examinations were offered by secure internet connections in which a candidate's examination was observed by a live test administrator/proctor through audio and video monitoring. This was only available for EMT examinations; no paramedic examinations were online proctored. The test blueprints describing the test content by key domains and passing standards were the same for online proctored examinations as they were for onsite testing. This study was deemed exempt and approved by the American Institutes for Research Institutional Review Board.

Measurements

Data were collected from the National EMS Certification Database from the National Registry. Demographics of the testing candidates were collected including age, sex (male, female), and National Association of State EMS Officials (NASEMSO) region where the examination was administered. Examination administration was the total number of examinations taken by either EMT or paramedic candidates during the study period reported on a weekly basis. First‐attempt pass rate was the percentage of candidates who successfully passed their certification examination on the first attempt. Examination administration type (ie, online proctored or onsite) was collected for EMT candidates to evaluate the impact of online proctored testing.

A total of 2 certification levels (EMT and paramedic) and 2 time periods (2019 and 2020) were evaluated in this study. The 2019 testing period includes examinations delivered and first‐time pass rates during the calendar year of 2019 examined weekly for 52 weeks. Similarly, the 2020 testing period includes examinations delivered and first‐time pass rates during the calendar year of 2020 examined weekly for 52 weeks. For the EMT certification level, test administration for the 2020 sample was divided into 2 categories: all tests (online proctored and onsite) and onsite testing only. Calendar year was chosen as a unit for analysis to show how pre‐pandemic 2020 was trending with its matched 2019 time before the interruptions of 2020 as well as capture common seasonal variations throughout the analysis window.

Analysis

The primary outcome for this evaluation was the difference in cumulative number of examinations administered between 2019 and 2020. We considered the year 2019 as a historical control for year 2020 when the COVID‐19 pandemic caused shutdowns in both education and testing. To evaluate this difference, descriptive statistics were calculated and presented as mean (standard deviation [SD]) for continuous measures and frequency (percentage) for categorical measures. In addition, time series plots were created to present both EMT and paramedic cumulative test administrations for 2019 and 2020.

Because of the large sample size, evaluating the comparison between 2019 and 2020 would have likely resulted in statistically significant, but not practically relevant, findings. To address this, comparisons of EMT and paramedic certification levels between 2019 and 2020 demographics were made using 2 one‐sided tests (TOSTs) to better describe the significance of differences in characteristics between the 2 groups. TOST is used to describe the relevance of observed differences between groups as falling into 1 of 3 categories: (1) statistically significant and greater ±5 percentage points (a relevant difference), (2) statistically significant and <5 percentage points (a trivial difference), and (3) not statistically significant (equivalent). We set the threshold of equivalence at a ±5 percentage point difference in the variables based on previous evaluations. For continuous variables (eg, age), we used rank‐sum TOST comparisons to describe relevant differences between the 2 groups. For categorical variables with >2 levels (eg, NASEMSO region), we used pairwise z test TOST comparisons to categorize the relevance of the difference between the 2 groups. For these pairwise comparisons, we used a Bonferroni adjustment of α = 0.01 to control for an inflated type I error. Otherwise, α was set at 0.05. All analyses were performed using STATA IC version 15.1 (StataCorp LP, College Station, TX).

RESULTS

A total number of 141,352 examinations were administered in 2019 at the EMT and paramedic levels of certification. In 2020, 122,598 examinations were administered, a decrease of 13% in test volume. Similarly, the total number of candidates these examinations were administered to decreased from 105,656 in 2019 to 89,959 candidatesin 2020 (15% decrease). Note that candidates may have taken the examination more than once, thus the number of candidates is lower than the number of examinations administered.

EMT certification testing

At the EMT level of certification specifically, 120,132 and 102,824 examinations were administered in 2019 and 2020, respectively (14% decrease). These examinations were administered to 91,071 candidates in 2019 and 76,858 candidates in 2020(16% decrease). Candidate demographics are noted in Table 1. Trivial demographic differences (based on TOSTs) were noted in age and sex. A trivial difference was also noted in the number of examinations administered in the Great Lakes NASEMSO region, the only region to not be considered equivalent. The first‐time examination pass rate was 67.9% and 67.0% for 2019 and 2020, respectively, with a difference of 0.9% (also a trivial difference by TOST).

TABLE 1

Demographics of candidates who sat for the National Registry EMT Cognitive Examination in 2019 and 2020

Characteristic	2019 (n = 91,071)	2020 (n = 76,858)	Difference	TOST P a	TOST result
Age in years, mean (SD)	25.6 (8.3)	25.7 (8.4)	0.04	<0.001	Trivial difference
Sex, n (%)
Female	32,645 (38.1)	28,602 (39.6)	1.45	<0.001	Trivial difference
Male	52,937 (61.9)	43,642 (60.4)
Missing	5489	4614
NASEMSO regions, n (%)
East	18,784 (20.6)	15,550 (20.2)	0.40	0.046	Equivalence
South	22,646 (24.9)	18,768 (24.4)	0.45	0.03	Equivalence
Great Lakes	10,965 (12.0)	9906 (12.9)	0.85	<0.001	Trivial difference
Western Plains	9361 (10.3)	7908 (10.3)	0.01	0.95	Equivalence
West	21,292 (23.4)	17,658 (23.0)	0.41	0.05	Equivalence
Missing	8023	7068
First‐pass success, n (%)	61,862 (67.9)	51,515 (67.0)	0.90	<0.001	Trivial difference

Abbreviations: NASEMSO, National Association of State Emergency Medical Services Officials; SD, standard deviation; TOST, 2 one‐sided test.

TOSTs of equivalence to determine if the difference observed between groups is statistically significant and ±5 percentage points (relevant difference), statistically significant and <5 percentage points (trivial difference), or not statistically significant (equivalence).

Figure 1 shows the weekly cumulative examination administrations for the EMT certification level for 2019 and 2020 at cognitive examination centers. To assist in interpretation, included in the figure are lines denoting the week of the COVID‐19 Emergency Declaration from the Federal Emergency Management Agency (FEMA) on March 14, 2020 (week 11) and the ending of the last mandatory state‐level, stay‐at‐home orders in the United States in the first week of June 2020 (week 23). , Examination administration in 2019 continued to increase throughout the 52‐week period, whereas in 2020, there was a decrease in the cumulative number of examinations administered starting at week 11, consistent with the Emergency Declaration by FEMA. Overall onsite examination administration in 2020 decreased by 35% compared with 2019.

FIGURE 1

Emergency medical technician weekly cumulative examination administrations for 2019 and 2020. Compared with 2019 (dark blue), there was a total cumulative 35% decrease in on‐site only (gray) examination administrations and a 14% decrease in all examination administrations (light blue) in 2020. Vertical lines denote the following 2 events: week 11, the COVID‐19 Emergency Declaration from the Federal Emergency Management Agency; and week 23, the ending of the last mandatory state‐level, stay‐at‐home orders in the United States.

To mitigate the potential impact of the COVID shutdown, the National Registry began online proctoring examinations for the EMT certification in March 2020. The combined examination administration for online proctored and onsite examinations in 2020, compared with 2019, are also shown in Figure 1. The total number of examinations administered, even with online proctoring combined with onsite tests in 2020, was 14% lower than in 2019.

Paramedic certification testing

For the paramedic level of certification, 21,220 and 19,774 examinations were administered in 2019 and 2020, respectively (7% decrease). These examinations were administered to 14,585 candidates in 2019 and 13,101 candidates in 2020 (10% decrease). As shown in Table 2, there were trivial differences noted in the mean age of testing candidates (difference of 0.27 years) and for test administration by NASEMSO region in the Great Lakes (2.4% difference) and West regions (2.8% difference). First‐time examination pass rate was 71.8% and 69.9% for 2019 and 2020, respectively, with a mean difference of 1.9% (trivial difference by TOST).

TABLE 2

Demographics of candidates who sat for the National Registry Paramedic Cognitive Examinations in 2019 and 2020

Characteristic	2019 (n = 14,585)	2020 (n = 13,101)	Difference	TOST P a	TOST result
Age in years, mean (SD)	28.9 (7.5)	29.2 (7.8)	0.27	0.01	Trivial difference
Sex, n (%)
Female	3542 (25.2)	3184 (25.2)	0.03	0.95	Equivalence
Male	10,512 (74.8)	9466 (74.8)
Missing	531	451
NASEMSO regions, n (%)
East	2064 (14.2)	1804 (13.8)	0.38	0.36	Equivalence
South	6301 (43.2)	5556 (42.4)	0.79	0.18	Equivalence
Great Lakes	2213 (15.2)	2298 (17.5)	2.37	<0.001	Trivial difference
Western Plains	1229 (8.4)	1201 (9.2)	0.74	0.03	Equivalence
West	2339 (16.0)	1740 (13.3)	2.76	<0.001	Trivial difference
Missing	439	502
First‐pass success, n (%)	10,470 (71.8)	9156 (69.9)	1.90	<0.001	Trivial difference

Abbreviations: NASEMSO, National Association of State Emergency Medical Services Officials; SD, standard deviation; TOST, 2 one‐sided test.

TOSTs of equivalence to determine if the difference observed between groups is statistically significant and ±5 percentage points (relevant difference), statistically significant and <5 percentage points (trivial difference), or not statistically significant (equivalence).

The paramedic weekly cumulative examination administration for years 2019 and 2020 are shown in Figure 2. The examination administration in 2019 increased throughout the 52‐week period, whereas in 2020, similar to EMT, there was a decrease in the cumulative number of examinations administered starting at week 11 (start of shutdown). After this decrease, testing recovered, ending the year with only a 7% decrease in total examinations administered.

FIGURE 2

Paramedic weekly cumulative examination administrations for 2019 and 2020. There was a 7% difference between 2019 (dark blue) and 2020 (light blue) examination administrations. Vertical lines denote the following 2 events: week 11, the COVID‐19 Emergency Declaration from the Federal Emergency Management Agency; and week 23, the ending of the last mandatory state‐level, stay‐at‐home orders in the United States.

LIMITATIONS

There are several limitations to our study. First, because a concurrent study population was not possible, we used historical controls to define the impact of the COVID‐19 pandemic on test administration. In addition, because the dates of service shutdown and reopening by states were variable, we chose dates that were thought to be representative of the dynamics of the pandemic in the United States. Furthermore, we were unable to evaluate the impact of the pandemic at the program level in this study and chose to evaluate at the national level. Evaluating the impact of the pandemic at the program level would be challenging because of variations in the type and structure of programs as well as the heterogenous restrictions placed on programs due to local pandemic regulations. Finally, we recognize that this analysis examines cumulative yearly totals of examination administration for comparison rather than rates. These cumulative totals allow for additional historical comparisons to past annual examination administration as well as providing a comprehensive view of testing in 2020 with the onset of the pandemic and its effect on testing readily apparent. Further research should look at the impact of the pandemic on both EMT and paramedic programs with a rigorous study design to better identify at‐risk areas of the EMS educational infrastructure with the aforementioned confounding associations properly controlled.

DISCUSSION

COVID‐19 had a significant impact on the potential EMS workforce, with closures of educational programs across the United States. These closures were thought to be contributors to shortages in available practitioners for both pandemic and non‐pandemic emergency response. In this evaluation, COVID‐19–related closures and the interruption of normal operating procedures had a sizeable impact on the administration of the National EMS Certification Cognitive Examination provided by the National Registry at both the EMT and paramedic levels. Although this effect was apparent for both certification levels, the impact appeared to be more pronounced for EMTs (Figure 1). COVID‐19–related effects on EMT examinations were mitigated by the implementation of online proctoring, which served as an effective supplementary option in the presence of limited examination site availability.

There is no historical context for the impact of a large, ongoing emergency event, such as the pandemic, on education and workforce dynamics. Furthermore, there are no clear standards or guidelines for how to manage the impact of a widespread prolonged shutdown of high‐stake examination infrastructure. Other fields of study were similarly impacted including the legal Bar examination, Graduate Record Examination, and medical school admissions test. At the onset of the shutdown, the extent of the impact on the EMS certification examination was unclear, although the threat posed to this critical workforce pipeline was immediately recognized. In this analysis, we present counts of change in examination administration and passing success; however, this granularity may obfuscate the overall concern of stakeholders and educators—workforce stability and confirmation of entry‐level competency.

As predicted, examination administration volumes for EMT and paramedic levels of certification decreased in this evaluation. Furthermore, we noted a difference in how EMT and paramedic certification level examination administration volume changed during the pandemic. This may be multifactorial, including, for example, student engagement and personal investment in the field, the availability of examination seats, and the cost of examination administration or students choosing not to start educational programs due to the pandemic. Furthermore, the individual impact on student motivation for taking the examination immediately after completing an education program versus waiting for pandemic conditions to improve may play a strong role in testing volumes. Some students may have started programs pre‐pandemic and decided not to test, were sick and unable to test, or wanted to delay until it was safer to test. Thus, the lower examination volume may be because of other effects of the pandemic external to examination availability. Nonetheless, the overall impact of the pandemic shutdown decreasing the flow of EMS professionals into the workforce pipeline is significant. Future work will be necessary to evaluate if the effects on examination administration were also evident in job placements.

There was a positive impact from the use of online proctored examinations for the EMT cognitive examination that may have helped mitigate the effects of pandemic‐related shutdowns (Figure 1). As this was a novel method to respond to the pandemic, we are unable to speculate on the potential impact the pandemic could have had without EMT online remote testing. However, 1 important consideration in the use of online proctored examinations is the potential impact on examination security. It is also important to acknowledge the potential lessons learned from the rapid deployment of high‐stakes remote testing, as in fact, the National Registry has decided to extend the use of live remote proctored examinations to emergency medical responder and EMT examinations for the foreseeable future. After the extensive use of remote learning throughout the US educational infrastructure during the pandemic, educators should better define the utility and limitations of remote learning for the future. The possibility of an increase in the use of these methods for education in the future may lead to changes in the efficiency and effectiveness of remote‐learning curricula.

A significant concern for educators during the pandemic was the impact of the changes in the educational infrastructure on pass rates. As the pandemic began, many of the face‐to‐face interactions that are critical for the development of EMS professionals were cancelled, including clinical and field experiences. This led to the need to innovate and develop other methods to deliver educational materials to students, replacing face‐to‐face training. As students graduated with this major change in their curriculum, many questioned whether they would be equally successful on the certification examination. In this evaluation, we examined the first‐attempt pass rate between historical controls (2019) in comparison with those of students trained in the pandemic (2020). For both certification levels, there was a statistical, but not operationally relevant, difference in first‐attempt pass rates driven by the large sample size (Tables 1 and 2). This suggests that the pandemic may not have had any real unmitigated impact on certification pass rates and that although educational interruptions did occur, students were still well prepared for the certification exam and were ultimately eligible to join the EMS workforce.

The use of TOST equivalence testing allowed us to describe the nature of the magnitudes of differences in our analyses, going beyond a dichotomous determination of differences reaching statistical significance. This method has been used in prior EMS research and is particularly useful for analyses of very large data sets. This methodology combines the strengths of traditional statistical equivalence testing and subjective determination of practical significance, in this case, set at a 5% difference in our study. Future work is needed to determine guidelines for practical significance thresholds in EMS that are accepted by educators and stakeholders as having operational validity.

In summary, COVID‐19 has had a measurable impact on examination administration volume for both the EMT and paramedic levels of certification. First‐time pass rates remained unaffected. Remote proctoring mitigated some of the impact on the number of EMT examinations administered, yet still resulted in reduced examination administration compared with 2019. Paramedic examinations administered were reduced, but far less so than EMTs. These reductions indicate a potential decrease in the newly certified EMS workforce to respond to the pandemic and beyond, but future evaluations will be necessary to assess the magnitude of this impact.

CONFLICT OF INTEREST

These data were presented as a poster presentation at the 2020 National Association of EMS Physicians Annual Meeting (January 10, 2020; San Diego, CA). The authors report no additional conflict of interest.

AUTHOR CONTRIBUTIONS

Jonathan R. Powell, Jennifer Cotto, Rebecca E. Cash, Mihaiela R. Gugiu, and Ashish R. Panchal conceived and designed the study. Jennifer Cotto, Mihaiela R. Gugiu, and Ashish R. Panchal collected the data. All authors analyzed and interpreted the data and drafted the manuscript. All authors contributed substantially to the revision of the manuscript. Jonathan R. Powell takes responsibility for the article as a whole.