Respiratory Protection in a Time of Crisis: NIOSH Testing of International Respiratory Protective Devices for Emergency Use.

National Institute for Occupational Safety and Health (NIOSH)-approved respirators are required by the Occupational Safety and Health Administration (OSHA) when personal respiratory protection is used in US occupational settings. During the COVID-19 pandemic, the demand for NIOSH-approved N95 filtering facepiece respirators overwhelmed the available supply. To supplement the national inventory of N95 respirators, contingency and crisis capacity strategies were implemented and incorporated a component that endorsed the use of non-NIOSH-approved respiratory protective devices that conformed to select international standards. The development and execution of this strategy required the collaborative effort of numerous agencies. The Food and Drug Administration temporarily authorized non-NIOSH-approved international respiratory protective devices through an emergency use authorization, OSHA relaxed their enforcement guidance concerning their use in US workplaces, and NIOSH initiated a supplemental performance assessment process to verify the quality of international devices. NIOSH testing revealed that many of the non-NIOSH-approved respiratory protective devices had filtration efficiencies below 95% and substantial inconsistencies in filtration performance. This article reports the results of the NIOSH testing to date and discusses how it has contributed to continuous improvement of the crisis strategy of temporarily permitting the use of non-NIOSH-approved respirators in US occupational settings during the COVID-19 pandemic.

Introduction

The national Institute for Occupational Safety and Health (NIOSH) – National Personal Protective Technology Laboratory (NPPTL) of the Centers for Disease Control and Prevention (CDC) is responsible for the conformity assessment of respirators used in occupational settings. Conformity assessment encompasses preapproval testing and quality assurance evaluations and includes postapproval product and site audits. The stringency of this comprehensive approval process assures that once a respirator has earned NIOSH approval the user can be confident that the device will provide the expected level of protection, as long as it fits properly and is selected, maintained, and worn correctly. The Occupational Safety and Health Administration (OSHA) requires that only NIOSH-approved respirators be used in US workplaces.[1]

It had been projected that during a public health emergency the supply of N95 respirators may become strained,[2,3] and, indeed, a global shortage of NIOSH-approved N95 filtering facepiece respirators was experienced during the coronavirus disease 2019 (COVID-19) pandemic.[4] Given the high demand for respiratory protective devices (RPDs) by healthcare personnel, the shortage prompted several governmental and nongovernmental organizations to implement strategies to conserve the limited supply of NIOSH-approved respirators. Such strategies included control banding, a technique used to guide the assessment and management of workplace risks[5]; extended use and limited reuse of N95 respirators[6,7,8]; the use of respirators beyond their shelf life stored in strategic stockpiles[9]; decontamination of respirators[10]; and the use of NIOSH-approved respirators designed to be reused, such as elastomeric and powered air-purifying respirators.[11]

These strategies have not been enough to meet the demand for respiratory protection during the COVID-19 pandemic due to the increased use of RPDs beyond acute hospital care settings, such as nursing homes and ambulatory care clinics. Additional crisis strategies were, therefore, needed. To increase the availability of RPDs, the CDC[12] and the US Food and Drug Administration (FDA)[13,14] implemented a crisis strategy to permit the occupational use of non-NIOSH-approved RPDs designed to meet international standards that incorporate requirements similar to NIOSH-approved N95 respirators.[15] To inform this process, NPPTL recommended the use of performance standards that specified protection factors of at least 10* from 7 international organizations associated with Australia, Brazil, China, Europe, Japan, Korea, and Mexico.[12] It further recommended that non-NIOSH-approved RPDs developed by manufacturers that were not NIOSH approval holders should be used only in crisis situations when no other NIOSH-approved N95 respirator or respirator produced by a NIOSH approval holder is available. At the same time, because of the shortages experienced during the COVID-19 pandemic, OSHA relaxed its enforcement guidelines around RPDs used in occupational settings.[16]

The primary mechanism used to implement these crisis strategies during the COVID-19 pandemic was a series of emergency use authorizations (EUAs) issued by the FDA.[13] An initial EUA was issued on March 24, 2020, for specified disposable RPDs. A separate EUA for certain disposable RPDs manufactured in China was issued on April 3, 2020.[13] To be eligible, respirators had to meet specified criteria.[†] Authorized filtering facepiece respirators meeting the specified criteria were included in Appendix A of the EUA. Shortly after the April 3, 2020, EUA was issued, nearly 90 Chinese manufacturers were included in Appendix A.[17]

NPPTL Supplemental Filtration Performance Assessment Process

In response to public inquiries and concerns regarding the quality of international RPDs temporarily authorized for occupational use,[‡] NIOSH developed a supplemental filtration performance assessment process.[18] The NIOSH testing incorporated a modified version of NIOSH Standard Test Procedure TEB-APR-STP-0059.[§] The abridged test included a focused assessment of the particulate filtration (air-purifying) efficiency of RPDs. Inhalation and exhalation resistance were not included.

The test was developed to rapidly evaluate the filtration efficiency of international RPDs temporarily authorized for emergency use in occupational settings for the health and safety benefit of US workers and to transparently communicate the test results. NPPTL received requests directly from federal, state, and local government agencies, healthcare providers, employers in nonhealthcare industries, public safety and first responder organizations, and universities.[**] Information such as the manufacturer name, model designation, and performance standard under which the RPDs were reportedly manufactured were required components of the request.

For each valid request received, a minimum of 10 RPDs (of the same model and from the same manufacturer) were tested. The sampling protocol used to select the 10 RPDs was at the discretion of the group making the request; however, requestors reported using both a convenience, nonprobability sampling technique (in which samples were drawn from the population based on their availability) or a random sampling technique (in which each unit in the defined population has an equal probability of being selected for testing).

After NIOSH testing was complete, results were communicated to the requesting group through a formal report and were publicly released online.[19] The reports included basic information about the RPD evaluated (eg, manufacturer name, model information, international standard claim), photographs, and the results for the sample RPDs tested. The results were also openly and rapidly shared with the FDA to aid the collaborative process of continually improving the EUAs.

The results included the maximum and minimum filtration efficiencies observed among the RPDs tested.[††] By reporting the maximum and minimum efficiencies, the observed range in filtration efficiency could be computed. The range indicates the consistency of filtration efficiency between the units evaluated. Understanding the consistency in filtration efficiency is important because users must be able to trust that individual units, which are labeled and packaged identically, provide a consistent level of performance when respiratory hazards are encountered in the workplace. Further, as required in 42 CFR 84,[20] NIOSH-approved N95 filtering facepiece respirators consistently filter out at least 95% of particulate matter, and the strategy to supplement the national supply with international RPDs includes an expectation of a similar level and consistency in filtration efficiency.

NPPTL received valid requests for testing nearly immediately after initiating the supplemental performance assessment process. The first NIOSH test was completed on April 10, 2020, 1 week after the EUA was issued. By the end of April 2020, NIOSH tests for approximately 80 distinct sets of international RPDs were completed. The test results revealed quality concerns related to both the level and consistency of observed filtration efficiency. In part due to these findings, on May 7, 2020, the FDA revised the EUA for RPDs manufactured in China by removing many of the manufacturers and models initially included in Appendix A.[17] In addition, the FDA and NPPTL further partnered to heighten the role of NIOSH testing and increase the surveillance of all RPDs imported from China.[13] The testing procedure was revised to increase the sample size and statistical power and to emphasize random sampling, thereby enhancing the ability of each test to uncover distinct pockets of filtration performance within consistently labeled RPDs.

After the April 3, 2020 EUA was revised on May 7, 2020, NIOSH testing results were categorized in 2 phases: Phase 1 included tests completed up to May 6, 2020, and Phase 2 included tests completed between May 7 and August 17, 2020.

Phase 1

By May 6, 2020, NIOSH testing had been completed for 105 international RPDs at the request of state governments (29%), healthcare providers (24%), employers in nonhealthcare industries (21%), public safety and first responder organizations (10%), individuals and organizations not categorized (6%), federal government agencies (5%), universities (3%), and local governments (2%). Of the RPDs tested, approximately 90% used an ear loop design to secure the mask to the wearer’s face, and the others used a head strap design.

Through the packaging and labeling, NPPTL verified the manufacturer, model, and performance standard to which the samples claimed conformance. NIOSH testing was completed for international RPDs that were largely distinct in terms of the samples assessed: there were 87 manufacturers, 102 models, and a variety of international standards represented. Some samples received did not readily indicate the international performance standard to which the product conformed. In these cases, NPPTL classified the performance standard as “unknown.” Table 1 shows the number of tests conducted relative to the reported international standard. Most of the tests (n = 59, 56%) were conducted on samples of RPDs that were reported to conform to the Chinese standard, GB2626-2006.

Table 1.

NIOSH Supplemental Assessments Conducted, by International Standard Reported

International StandardReported	Phase 1(n = 105)	Phase 2(n = 251)	Overall(n = 356)
Brazil ABNT/NBR 13698: 2011	1	0	1
China GB/T 32610: 2016	1	1	2
China GB19083: 2010	6	10	16
China GB2626: 2006	59	175	234
Europe EN149: 2001	12	17	29
Europe/China EN149: 2001 & GB2626: 2006	22	39	61
Korea KMOEL: 2017-64	1	2	3
Unknown	3	7	10

For each of the 105 tests conducted in Phase 1, NPPTL evaluated the filtration efficiency for the individual units, recorded the maximum and minimum filtration efficiency observed, and then determined whether (1) all units within the test were observed as above 95% efficiency, (2) all units within the test were below 95% efficiency, or (3) there was a mixture of some units testing above and some testing below 95% efficiency (Table 2). In 35 (33%) of the tests, all units tested above 95% efficiency. In 42 (40%) of the tests, all units tested below 95% efficiency. In the remaining 28 (27%) tests, there was a mixture of units that tested above and below 95% efficiency.

Table 2.

Results of NIOSH Testing for International Respiratory Protective Devices

Phase	Results	Tests Completedn (%)	Average Range in Filtration Efficiency% (SD)
Phase 1 (n = 105 tests)	All units above 95% filtration efficiency	35 (33.33)	1.06 (0.86)
	All units below 95% filtration efficiency	42 (40.00)	15.31 (17.00)
	Some above and some below 95%	28 (26.67)	24.50 (23.06)
Phase 2 (n = 251 tests)	All units above 95% filtration efficiency	104 (41.43)	1.34 (1.12)
	All units below 95% filtration efficiency	82 (32.67)	25.84 (21.66)
	Some above and some below 95%	65 (25.90)	20.01 (23.06)
Phase 1 and 2 (N = 356 tests)	All units above 95% filtration efficiency	139 (39.04)	1.27 (1.06)
	All units below 95% filtration efficiency	124 (34.83)	22.27 (20.74)
	Some above and some below 95%	93 (26.12)	21.36 (23.03)

Abbreviations: NIOSH, National Institute for Occupational Safety and Health; SD, standard deviation.

Note: NIOSH calculated the average range in filtration efficiency by subtracting the lowest observed filtration efficiency from the highest observed filtration efficiency, for each of the 105 tests analyzing at least 10 consistently packaged and labeled respiratory protective device units. This provides an indication of how consistent the unit-to-unit filtration properties were within each test. For example, if the highest filtration efficiency is 95% and the lowest is 60%, the average range in filtration efficiency is 35% (indicating low confidence in the level of protection).

The average range in filtration efficiency was calculated by subtracting the lowest observed filtration efficiency from the highest observed filtration efficiency, for each of the 105 tests analyzing at least 10 consistently packaged and labeled RPD units. The average and standard deviation of the ranges is shown in Table 2. There was large variability in filtration efficiency across the tests in which all units were observed below 95% efficiency and those with a mixture of units that tested above and below 95% efficiency. The greatest variability in the range of filtration efficiency results was about 78% (maximum 91%, minimum 14%), for a sample of 10 units that claimed to conform with the Chinese standard GB2626-2006.

Phase 2

NPPTL completed an additional 251 tests of non-NIOSH-approved international RPDs following the FDA May 7, 2020 revision of its April 3, 2020 EUA. During Phase 2, 170 distinct manufacturers and 177 distinct models were tested at the request of federal government agencies (47%), employers in nonhealthcare industries (27%), state governments (12%), healthcare providers (9%), local governments (3%), individuals and organizations not classified (1%), and public safety and first responder organizations (1%). Of the RPDs tested, approximately 95% used an ear loop design to secure the mask to the wearer’s face and the remainder used a head strap design (5%). Similar to Phase 1, a majority (n = 175, 70%) of the tests were conducted on samples of RPDs that were reported to conform to the Chinese GB2626-2016 standard (Table 1).

As shown in Table 2, 104 (41%) of the units tested above 95% particulate filtration efficiency, 82 (33%) of the units tested below 95%, and 65 (26%) of the units were a mixture that tested above and below 95%. As in Phase 1, large variability in filtration efficiency was found across the tests in which all units were below 95% efficiency and those in which there was a mixture of units testing above and below 95% efficiency. The highest range in filtration results was 87% (maximum 92%, minimum 5%) for a sample of 10 RPD units that claimed to conform with both the European EN149-2001 and Chinese GB2626-2006 standards.

Figure 1 shows a comparison of the results between Phase 1 and Phase 2. In Phase 2, we observed an 8% increase in tests in which all units tested above 95% filtration efficiency and a 7% decrease in tests in which all units tested below 95% filtration efficiency. These changes may be partly due to an increased awareness of the NPPTL testing results being shared with the FDA in a transparent and timely manner.

Figure 1.

Comparison of Phase 1 and Phase 2 assessment results.

Table 2 also shows the combined results from the Phase 1 and Phase 2 assessments (N = 356). In 139 (39%) of the tests, all units tested above 95% filtration efficiency; in 124 (35%) of the tests, all units tested below 95% filtration efficiency; and in 93 (26%) of the tests, some units tested above and some tested below 95% filtration efficiency. These statistics suggest that the level and consistency of filtration efficiency of the international RPD units tested remain concerning.

Discussion

This case study represents an interagency collaboration designed to support the health and safety of the US workforce throughout the COVID-19 pandemic. The NIOSH supplemental testing of international RPDs filled a critical gap during the COVID-19 pandemic by communicating evidence-based test results within 1 week (usually within 48 hours of the units being received) to the public and other agencies involved with the response.[21] These results directly contributed to FDA continuously updating the EUAs to add and remove RPDs authorized for use in healthcare settings. A third phase of NIOSH testing is underway, representing a continuation of the supplemental testing of non-NIOSH-approved RPDs that continue to be listed on the FDA’s EUA.

Although NPPTL has conducted an extensive number of tests of international RPDs, other entities have conducted similar tests. Across the country, state governments have partnered with universities and third-party laboratories to evaluate the performance of RPDs conforming to international standards to inform purchasing decisions or remove substandard products from the market—all for the benefit of the US workforce in need of RPDs during the pandemic.[22] The Massachusetts Institute of Technology and Harvard University, for example, found that many of the filtering facepiece respirators available for procurement during the COVID-19 pandemic do not provide levels of fit and filtration similar to those of NIOSH-approved N95 respirators.[23,24] The NPPTL procedure provides a consistent protocol for evaluating these devices and enables third-party laboratories and other organizations to conduct the testing on their own.[25] As the COVID-19 pandemic continues, testing will remain important to the health and safety of the US workforce.

Several limitations should be noted. First, given that a random sampling technique could not be guaranteed, the test results are generalized to the sample of international RPDs tested and may not be representative of a larger sample. Second, the tests were designed to provide a rapid evaluation of the filtration efficiency only. Given that a standard NIOSH approval evaluation consists of additional performance tests, as well as a comprehensive quality assurance review of the quality process and manufacturing site, any type of comparison between the international devices assessed and NIOSH-approved N95 respirators is not possible. In addition, certificates of approval or validation of conformance with an international standard were not provided with the samples. The authenticity of a claim that a product met or did not meet a particular international standard could, therefore, not be made.

Given these limitations, the results suggest that a substantial proportion of RPDs tested did not provide consistent filtration protection above 95% when evaluated using an adapted version of NIOSH’s test method. While the international RPDs are not a precise equivalent, the standards considered incorporate filtration efficiency specifications that are nearly equivalent to NIOSH’s requirement for N95s to have at least 95% filtration efficiency. There are several possible reasons why the products assessed did not perform to these expectations. First, companies newly established to meet the global demand for RPDs during the COVID-19 pandemic may not have had the time or opportunity to establish quality control systems needed to consistently produce high-performing RPDs. Second, shortages or an inconsistent supply of the raw materials needed to produce high-performing RPDs may have forced manufacturers to procure lower-performing filtration material. Third, several counterfeit reports were received by NPPTL. In such cases, documents may have been altered to make RPD models appear to comply with a standard when they do not, or manufacturers’ names, logos, and model numbers may have been misrepresented or counterfeited. For questions regarding product authenticity and potential counterfeiting, consumers are urged to directly contact the manufacturers and others involved in the supply chain.

Conclusion

Potential purchasers of international RPDs should refer to the CDC and NIOSH websites for guidance on how to evaluate respirators from other countries to determine if they provide adequate protection before making purchasing decisions. Consumers are urged to consult the FDA’s current EUAs and the results of NPPTL supplemental performance assessments.[20]