Assessment of Masks Used by Healthcare Workers: Development and Validation of a Mask Qualitative Assessment Tool (MQAT).

Background: Respiratory masks can provide healthcare workers with protection from biological hazards when they have good performance. There is a direct relationship between the visual specifications of a mask and its efficacy; thus, the aim of this study was to develop tools for qualitative assessment of the performance of masks used by healthcare workers.
Methods: A mixed-methods design was used to develop a qualitative assessment tool for medical face masks (MFM) and particle filtering half masks (PFHM). The development of domains and items was undertaken using observation and interviews, the opinions of an expert panel, and a review of texts and international standards. The second phase evaluated the psychometric properties of tools. Finally, the validated Mask Qualitative Assessment Tools (MQAT) were used to assess six samples from 10 brands of the two types of masks.
Results: MQAT-MFM and MQAT-PHFM shared 42 items across seven domains: "cleanliness," "design," "marking, labeling and packaging," "mask layers," "mask strap," "materials and construction," and "nose clip." MQAT-MFM included one additional item. MQAT-PHFM included another nine items associated with an eighth "Practical Performance" domain, and the valve version had another additional "Exhalation Valve" domain and six items. The evaluation indicated 80% compliance for MFM and 71% compliance for PFHM. "Marking, labeling and packaging" and "Layers" were associated with the least compliance in both types of masks and should be checked carefully for defining mask quality.
Conclusion: MQAT can be used for immediate screening and initial assessment of MFM and PHFM through appearance, simple tools, and visual inspection.

Introduction

The high rate of transmission of the coronavirus SARS-COV-2, and the significant mortality rate of COVID-19, the disease caused by this contagion, led the World Health Organization (WHO) to announce COVID-19 as a pandemic on March 11, 2020 [1]. The US Occupational Safety and Health Administration identified healthcare workers (HCW) as an occupational group subject to very high levels of exposure to this biological hazard [2]. Similarly, WHO noted that although HCW represents less than 3% of the population, even in most developed countries, 14%–35% of COVID-19 cases (depending on the country) are HCW [3].

To respond to the COVID-19 pandemic, the use of Particle Filtering Half Masks (PFHM), which are also known as Filtering Facepiece Respirators, and Medical Face Masks (MFM), also known as Surgical Masks, was introduced as an effective, nonpharmacological intervention to reduce the prevalence of COVID-19. For HCW, SARS-COV-2 is a biological hazard, and these two types of masks are used as personal protection equipment to allow them to work safely [4]. Both are used on the basis of providing protection for HCW from COVID-19 in all its variants [5]. MFM and PFHM have a sophisticated manufacturing process to provide effective properties to filter the microscopic SARS-COV-2 virus (60–140 nanometers) [5,6]. It is important, however, to ensure that masks fully comply with published standards if they are to protect HCWs properly. Masks that are substandard or used inappropriately put HCW at risk of infection [7].

Assessment of the protective performance of a mask can be undertaken qualitatively or quantitatively. A qualitative assessment is based on the visual specifications of the mask. Essentially, this is a visual examination of the layers in terms of number, material, structure, quality, weight, integrity, stitching, meltblown layer, straps, stitching, clamps on nose clips, and the size using simple tools [8]. A quantitative assessment of the performance of masks—which includes bacterial filtration efficiency, particle filtration efficiency, total inward leakage, and a drop in level of pressure [9]—requires special laboratory equipment and facilities. Despite the advantages of being able to compare quantitative test results with standard indicators, quantitative methods require significantly more time for the assessment, technical knowledge, and expensive laboratory equipment. Critically, occupational healthcare providers generally lack these resources, especially in developing countries [8,10]. In contrast, a qualitative assessment of the performance of masks can be undertaken quickly and efficiently due to the direct relationship between the visual specifications and the performance of a mask [9,11]. To respond to the lack of standard tools for immediate screening and initial assessment, the aim of this study was to design and validate a comprehensive set of tools for the qualitative assessment of the performance of the two types of masks used by HCWs.

Materials and methods

Research design

For designing and developing mask quality assessment tools (MQAT), a mixed-methods sequential exploratory design was performed in three phases. The first phase included determining the domains and design of MQAT items using a review of the literature and international standards, HCW interviews, and an Experts Panel. The second phase was an evaluation of the psychometric properties of the developed MQAT. In the third phase, a qualitative assessment of ten widely used brands of masks used by HCWs was made using the validated MQAT.

Determining the domains and designing the items

Literature review

For finding important domains for a qualitative assessment of the performance of masks, articles were retrieved from Scopus, PubMed, Web of Science, Embase, and Science Direct databases. The search used keywords including: “appearance check,” “quality approval,” “qualitative test,” “appearance,” “medical face masks,” “particle filtering half masks,” “filtering facepiece respirators,” “surgical mask,” both separately and in combination. In order to increase the sensitivity, a hand search of the reference section of all the selected articles was also undertaken. After this search, the list was reviewed by two colleagues, and repeated titles were removed. The titles and abstracts of the remaining articles were then carefully studied, and that included important factors for a qualitative assessment of the performance of the masks were selected. Finally, the full text of the remaining articles was analyzed, and factors relevant to this study were extracted [12,13] (See Fig. 1.).

Fig. 1

Flow chart for the selection of studies in the literature review.

Standards requirements

International standards for the two types of masks used by HCW were consulted (see Table 1), and then a list of indicators for a qualitative assessment of the performance of masks was extracted and prepared. This included the number of samples to be evaluated with the QAT to provide a reliable evaluation of mask performance. For example, EN 14683:2019 [14] states that six samples from each randomly sampled batch are suitable and sufficient to assess the quality of MFM. Thus, in this study, six samples of each mask from each brand were examined, and the sampling of every mask box used was random.

Table 1

Standards for assessment of the performance of masks according to region and type

Standard domain	Standard for MFM	Standard for PFHM
Australia/New Zealand	AS/NZS 4381:2015	AS/NZS 1716:2012
Brazil	ABNT/NBR 15052:2004	ABNT/NBR 13698:2011
China	YY 0469:2011	GB2626-2006GB2626-2019GB19083-2010
Europe	EN 14683:2019	EN 149:2001EN 149:2009
Iran	INSO 6138-2020	INSO 22833-2020
Japan	ASTM F2100-19	JMHLW-2000
South Korea	EN 14683:2019	KMOEL -2017-64
USA	ASTM F2100-19	NIOSH - 42 CFR 84

HCW interviews

Fifteen HCWs participated in the interview stage. Informed consent was given. The interviews were audio recorded. The masks that the HCW used were observed; then, information about quality and problems of respiratory masks were collected via two general questions:

What are the characteristics of a mask that can provide adequate protection against bioaerosols?

What problems and concerns do you have about the respiratory masks?

For confirming the accuracy of the data, both during and at the end of each interview, a summary of the answers was reported back to participants, and corrected wherever necessary, to eliminate any ambiguity. The duration of the interviews was 10–15 minutes. After each interview, on the same day, the interview notes were carefully reviewed to correct any possible mistakes. Then, the collected information was subjected to content analysis [15], and important codes and domains in terms of mask quality, according to HCWs, were identified. The coding and extraction of the domains was done separately by two members of the research team to ensure rigor in this process, and the agreement of themes was assessed using Holsti’s formula (Reliability = 2m/N1 + N2, where m is the number of coding decisions where coders agree, and N1 and N2 are the numbers of decisions made by the coders) [16].

Experts Panel

An Experts Panel was used to summarize the information collected from the literature and standards review and the HCW interviews toward specifying the final criteria needed for a qualitative assessment of masks. The Experts Panel included 25 people with at least 5 years of experience in the production, research, distribution, or use of respiratory masks. The panel met remotely, using email and virtual meetings in groups using an online platform. Initial meetings were in topic groups, and then the whole group met together to discuss the required domains of mask qualitative assessment tools for the two types of masks. The domains were used as a conceptual framework then appropriate items were developed for each domain.

Psychometric properties of the MQAT

Validity

The tool was given to 15 specialists in the field of masks to check the grammar, sentence structure, and appropriate placement of phrases for each item. If these principles were not met, they suggested an alternative to improve the item. Then, content validity ratio (CVR), content validity index (CVI), and item impact score were evaluated [17,18]. To determine CVR, we followed Lawshe’s method. The minimum acceptable CVR value was set to 0.49 for 15 panel members [18].

To examine CVI, we followed the proportion agreement method of Waltz and Bausell [19]. The 15 experts scored each item from 1 (not relevant) to 4 (very relevant) in terms of its appropriateness, clarity and simplicity. Then, the percentage for each item was calculated. According to the guideline, CVI >0.79 was considered appropriate, between 0.7 and 0.79 in need of review, and less than 0.7 unacceptable [17].

Item impact scores were used to assess the importance of the qualitative indicators. For this purpose, the experts were asked to specify the impact of each item using a 5-point Likert scale from 1 (not at all important) to 5 (absolutely important) [20]. Impact Score = Frequency (%) × Importance. Accordingly, only items with an impact score higher than 1.5 were retained.

Reliability

Reliability was assessed using the agreement coefficient between the evaluators. Six experts evaluated samples from 10 mask models using the two developed MQATs. After 2 weeks, the same experts evaluated new samples from the same mask models. An intraclass correlation coefficient with a 95% confidence level was used to examine the agreement coefficient between experts. To examine the correlation between the score of domains and the total score in the two stages of assessment, Spearman’s correlation coefficient was used [21].

Qualitative assessment of masks used by HCW in Iran

Six samples from 10 brands widely used brands of MFM and PFHM used by HCW in Iran were evaluated (See Fig. 2). Because of a ban on the use of masks with valves in the healthcare centers in Iran, this type was not included in the test phase. Masks with valves protect the wearer by filtering inhaled air, but because they do not filter exhaled breath, the valve can allow pathogens to circulate. Thus, they are unacceptable for COVID-19 control, although preferred by some HCWs as more tolerable for long-term use when working in hospitals [22]. Nevertheless, PFHM masks with valves are used by HCW in many countries; hence, we included this type in the development of MQAT, although this could not be a part of the validation procedures.

Fig. 2

Illustration of MFM and PFHM masks assessed in this study.

The items for each domain were assessed according to compliance with the quality characteristics of the mask. Each item was scored “pass” or “fail.” Finally, the frequency and percentage of items and domains that had achieved a “pass” were calculated.

Results

Extracting domains and items

Agreement of codes and themes from the HCW interviews was confirmed by using Holsti’s formula [16]. Reliability was good (>80%). Disagreements were resolved by considering associated field notes. Altogether the findings from the qualitative phase indicated that the MQAT-MFM required seven domains and 53 items. The MQAT-PFHM required eight domains and 64 items. The two MQATs had seven domains in common: “cleanliness,” “design,” “marking, labeling, and packaging,” “mask layers,” “mask strap,” “materials and construction,” and “nose clip.” The practical performance test domain was specific to the PHFM type, and if the mask had a valve, there was an additional domain and six items associated with assessing the valve efficacy.

Psychometric properties of tools

Based on the CVI and CVR results, items that did not reach an acceptable CVI and/or CVR were not included any further in the development of the MQAT. This included 10 items related to MFM and 13 items connected with PFHM. Thus, the developed MQAT-MFM was 43 items, the MQAT-PFHM was 51 items, and by dint, the MQAT-PFHM + Valve had 57 items. The mean CVI scores were 0.86 and 0.81 for MFM and PFHM, respectively, and CVR were 0.75 and 0.62 for MFM and PFHM, respectively, indicating appropriate content validity. The results showed that the mean impact score of all final items was greater than 1.5, thus acceptable [20].

The results showed excellent agreement among experts. The interclass correlation coefficient in all domains of the MQAT was higher than 0.9 (for MFM, 0.98 and 0.96; and PFHM, 0.97 and 0.95 for the first and second rounds, respectively). There was a very high correlation coefficient between all domains of the tool in the first and second stages of the test: for MFM 0.93 (p < 0.001) and for PFHM 0.91 (p < 0.001), which indicated very good reliability of the MQAT. (The full tool (Appendix A), which can be divided into two separate tools, and information on how to complete the eight parts of the MQAT (Appendix B) are available as e-components of the paper.)

Six samples from 10 brands of MFM and 10 brands of PFHM were evaluated using the MQAT. There was consistency across the six samples used for each brand in the study. The results are reported in Table 2 and Table 3, respectively, with illustrations in Fig. 3. None of the masks complied with the item “Reference to the national standard, which approves the mask is given on the box.”

Table 2

Assessment results of MQAT-MFM used by HCW

Brand codeDomains (number of items)	1	2	3	4	5	6	7	8	9	10	Average compliance
	Compliance (%)
Cleanliness (2)	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%
Materials and construction (2)	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%
Design (3)	100%	100%	100%	67%	100%	100%	100%	100%	100%	100%	97%
Mask strap (2)	100%	50%	100%	50%	50%	100%	50%	50%	50%	100%	70%
Mask layers (13)	100%	92%	100%	85%	92%	85%	46%	38%	92%	100%	83%
Nose clip (7)	100%	86%	100%	86%	86%	86%	86%	86%	86%	86%	89%
Marking, labelling and packaging (14)	64%	64%	64%	64%	64%	64%	64%	64%	64%	64%	64%
Average compliance	88%	81%	88%	77%	81%	81%	67%	65%	81%	86%	80%

Table 3

Assessment results of MQAT-PFHM-NV used by HCW

Brand codeDomains (number of items)	1	2	3	4	5	6	7	8	9	10	Average compliance
	Compliance (%)
Cleanliness (2)	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%
Materials and construction (2)	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%
Design (5)	80%	100%	100%	80%	100%	100%	100%	100%	80%	100%	94%
Mask strap (3)	100%	100%	100%	67%	100%	100%	100%	100%	67%	100%	93%
Mask layers (13)	100%	92%	8%	100%	46%	46%	92%	8%	100%	46%	64%
Nose clip (7)	100%	57%	57%	57%	29%	29%	57%	57%	57%	29%	53%
Marking, labelling and packaging (14)	86%	64%	57%	64%	36%	36%	64%	57%	64%	36%	56%
Practical performance test: Walking test (5)	100%	80%	100%	100%	100%	100%	80%	100%	100%	100%	96%
Average compliance	94.1%	80.4%	58.8%	80.4%	58.8%	58.8%	80.4%	58.8%	80.4%	58.8%	71%

Fig. 3

Examples of compliance and noncompliance of masks.

Discussion

Occupational health and safety legislation around the world requires that HCW have PPE that will protect them, as far as is reasonably practical, from biological hazards. This has come to prominence during the COVID-19 pandemic. The efficacy of masks is an important part of providing protection. In this research, for the first time, a comprehensive and valid tool was developed to allow a qualitative assessment of masks commonly used as PPE by healthcare workers. The MQAT can be used to evaluate the performance of MFM and PFHM by using a visual examination and simple tools.

The examination of the masks with MQAT confirmed that all tested masks of both types fully complied with cleanliness requirements. Similarly, compliance with design domain items for MFM and PFHM used by HCW was very high, providing general confidence that the MFM and PFHM are designed to provide sufficient coverage of users’ mouths and noses and avoid secondary hazards resulting from factors such as sharp edges or burrs. Nevertheless, there was evidence from our assessment using the MQAT that there is a shortfall in the efficacy of masks. About one-fifth of MFM, and nearly one-third of PFHM did not meet the required quality standards.

There are two vital aspects when looking at the efficacy of masks in terms of protection of the wearer: the ability to filtrate hazardous particles and the fit to the wearer [23]. Regarding the fix on the face, we found room for improvement with the mask straps. While the PFHM, which uses a self-adjustable head strap, met the requirements of this domain very well, we found room for improvement in the MFM, which relies on one size of elastic ear loops. The examination of nose clip in PFHM, however, provided the lowest level of compliance among the domains, barely half met the standard. The clamping of the mask around the nose was an issue in nearly half of PFHM, despite the premise that this type of mask is leak-proof [5]. The clamping of the mask around the nose was much more acceptable for the MFM. The two types of masks rely on different aspects in their design to determine a good fit on the face. Ultimately both need to maximize protection, and more examination of the fit of masks for protection is required. Our findings are in line with various studies that have shown the effect of the mask strap, and also the nose clip, on the proper fit on the face—and thus personal protection [24,25].

Regarding the ability of a mask to filter hazardous particles, there were significant failings in relation to the mask layers domain in both types of masks. The lack of a meltblown layer or a low-quality meltblown layer was the principal problem. The main function of the meltblown layer is to serve as a barrier of particles and fluids [26]; thus, it is a key component of an effective mask. The meltblown process is a manufacturing system that converts polymer into continuous thread-like filaments, which are then randomly integrated into a nonwoven fabric with diameters of 1–10μm and pores of 1–3μm [27]. The process electrostatically charges the filaments and also gives the nonwoven fabric excellent wicking properties [28]. Meltblown webs are soft, and if crumpled, they should not become brittle but easily tear as they have no elastic properties. With this knowledge, it becomes possible to easily confirm the existence of a meltblown layer [29]. As protection for the meltblown layer, it will be insulated top and bottom by layers of spunbond polypropylene to provide a trilaminate nonwoven fabric. The top and bottom layers of spunbond stick to the meltblown, and act as a prefilter for the meltblown layer [30], and unlike meltblown, spunbond is elastic and does not tear easily. Thus, using visual inspection and simple tools, the condition of the mask layers can be examined and evaluated qualitatively [26], and hence, a very important component of the MQAT.

Polypropylene is used in meltblown and spunbond technology due to its low density compared to other polymers. It has the highest efficiency and coverage power (weight to volume ratio), and the maximum meltblown density is reported to be 200 kg/m3 [31]. Therefore, it floats on water, which has a density of about 1,000 kg/m3 at room temperature. As water particles are larger than the porosity of the meltblown surface, it does not absorb water, it does not get wet, and its state does not change. This feature can also be used as a confirmatory test for the presence of a meltblown layer in masks. Another characteristic of the meltblown layer is that if exposed to fire, it melts and leaves white smoke and clear ash [32,33]. All of this is important because the higher the density and weight of the meltblown layer, the better its quality and protection against foreign particles. One of the ways to determine the density status of several different meltblown samples is to check the transparency of the surface of each meltblown; that is, when a piece of each meltblown sample layer is placed on a table, if the bottom surface of the sample is not visible (opaque), it is a better quality sample. The minimum weight of meltblown and spunbond should be 17 g/m2 and 25 g/m2 [34,35], respectively.

Various studies have shown that in nonwoven textiles, there is a significant indirect relationship between pore diameter and its efficiency [27]; therefore, the surface structure of meltblown and spunbond tissue should be integrated and should not have any damage or pores visible to the naked eye [34,36]. This aspect is also incorporated into the MQAT. To maintain the efficacy of the different layers of the mask and also according to the emphasis of different standards [14,37], it is necessary to connect them ultrasonically. Connecting the layers to each other, or connecting the straps to the body of the mask, in the form of normal stitching causes damage to the body of the mask, which definitely reduces its efficacy [38]. The stitching is also a part of the assessment tools.

There were compliance issues related to marking, labeling, and packaging, which is included in the MQAT as one of the important clauses in the standards of requirements of masks that manufacturers must comply with (e.g. EN149 [37] and EN14683 [14]). The ability to quickly and easily identify the brand, type, and class of the mask, providing the necessary information to the user, and also protection against mechanical damage and contamination before use are among the objectives of this clause of standards, which can be easily examined and evaluated. Our assessments of masks showed that the compliance with requirements in this domain for MFM and PFHM was low. More than one-third of the masks were not appropriately labeled. It is difficult to provide a reason for this level of noncompliance when used for HCW. There is a need for fundamental modifications and revisions in the marking, labeling, and packaging of masks produced as PPE.

According to Standard EN149 [37], the practical performance test serves as a measure of performance, specifically for evaluating PFHM in realistic conditions. Using the walking test as part of the MQAT-PFHM, it was found that almost all of the PFHM practical performance test items complied with the requirement. This gives confidence in this aspect of PFHM.

One of the limitations of this study was that the MQAT development did not include masks made of nanomaterials, cartridge masks equipped with particle filters such as P100, and fabric masks – although the latter, in particular, are not recommended as PPE for HCW. Similarly, we were not able to evaluate the PFHM + valve. Nevertheless, it remains that these masks are not suitable for mitigating the spread of COVID-19 [22]. In due course, further research should examine the relationship between the developed MQAT and the quantitative performance tests of the masks.

There is also a need to consider the sufficiency of our sampling following the publication of two recent papers that raise the specter of unacceptable intersample and intrasample variability of masks in the same production batches [39,40]; that is, these papers indicate the need for assurance that our sampling strategy was sufficient. If we did not sample enough masks, this would impact the reliability of MQAT. In their study of bacterial filtration efficiency (BFE) and differential pressure (DP), Tessarolo and colleagues examined three types of MFM with respect to standard EN 14683:2019 [14] using complex microbiological and engineering methods using a classic round-robin design of samples nominally from the same production batch. They found significant BFE and DP measurement variability. Two points are important for our paper: First, the authors acknowledge that it was hard to thoroughly distinguish between nonconformities in the samples and nonconformities in the methods used in the nonaccredited laboratories that did the testing; and second, the destructive nature of the tests used, and the use of different equipment at different locations does not throw any light on whether the variability found was real or an artifact. In this study, we referred to and followed standard EN 14683:2019 [14], which recommends testing at least five masks of each type. Moreover, we proved that the MQAT is less susceptible to the measurement errors discussed by Tessarolo et al. [39]. Taborri et al. [40] refer to the same standard EN 14683:2019 [14] to assess the breathability of MFM. They found just 37.5% of masks met the standard for breathability, which is not good enough to assure reliable protection for HCW. The authors report issues of measurement relating to the repeatability of results, and even wearthe mask to reliably test for breathability. The EN 14683:2019 indicates that five samples from each batch are suitable and sufficient to assess the quality of MFM. In this study, as indicated in the methods section, six samples of each mask from each brand were examined. The sampling of every mask box used was random. The results were the same every six times. Again, we suggest that the MQAT provides reliability for assessing quality, that our sampling did not provide such variability, and critically, our sampling met the requirements of standard EN 14683:2019, as well as the other standards referred to in this study. This strongly suggests the methodology of testing masks is important, and the MQAT provides a valid and reliable method. Nevertheless, it is important that further research should endeavor to explore this issue, given the importance of masks that serve as PPE for HCW. In the meantime, we will assert that an advantage of MQAT is that they can be used as a screening index and provide inclusion criterion for quantitative tests, which would reduce the costs of the more expensive quantitative tests that seem to be no more reliable from the two papers discussed [39,40].

Conclusion

The results of this study showed that MQAT, developed as a valid and applicable tool for specialists in health centers, industries, and organizations, can be used for immediate screening and initial assessment of the qualitative performance of both MFM and PFHM, through visual specifications using simple tools. MQAT can be used for the initial identification of defects and poor performance of the masks. Our findings using the developed MQAT indicate that the quality of the layers used for masks used by HCW should be considered and improved to fully assure of fulfilling their role as PPE. Similarly, there are concerns around marking, labeling, and packaging of masks in terms of complying with many international standards.

Conflict of interest

None to report.