Quality management system implementation in human and animal laboratories.

BACKGROUND: The ability to rapidly detect emerging and re-emerging threats relies on a strong network of laboratories providing high quality testing services. Improving laboratory quality systems to ensure that these laboratories effectively play their critical role using a tailored stepwise approach can assist them to comply with the World Health Organization's (WHO) International Health Regulations (IHRs) and the World Organization for Animal Health's (OIE) guidelines.
METHODS: Fifteen (15) laboratories in Armenia's human and veterinary laboratory networks were enrolled into a quality management system strengthening programme from 2017 to 2020. Training was provided for key staff, resulting in an implementation plan developed to address gaps. Routine mentorship visits were conducted. Audits were undertaken at baseline and post-implementation using standardised checklists to assess laboratory improvements.
RESULTS: Baseline audit general indicator scores ranged from 21% to 46% for human laboratories and 37% to 60% for the veterinary laboratories. Following implementation scores improved ranging from 7 to 39% for human laboratories and 12% to 19% for veterinary laboratories.
CONCLUSION: In general, there has been improvement for both human and veterinary laboratories in the areas of QMS implementation, particularly in organizational structure, human resources, equipment management, supply chain and data management. Central facilities developed systems that are ready for international accreditation. This One Health strengthening project ensured simultaneous strengthening of both human and veterinary laboratories which is not a common approach.

Introduction

A core component within global health security initiatives including the International Health Regulations (IHRs) and Global Health Security Agenda (GHSA) is the need for responsive and technically competent laboratories [[1], [2], [3], [4]]. These laboratories play a frontline role in disease detection, surveillance and response efforts especially crucial in light of threats from emerging and re-emerging infections of pandemic potential [5], like the current COVID-19 response. Laboratory results and data generated from these entities are useful if reliable and reproducible eliciting trust and confidence in end users [6]. However, in many low- and middle-income countries (LMICs) laboratory quality standards are hampered by a myriad of factors including lack of regulations, scarce resources and expertise to set up such systems including high cost of international accreditation programmes [1].

Laboratory strengthening efforts that incorporate stepwise implementation of quality management systems (QMS) have been promoted globally since the WHO 2008 Lyon meeting on quality and subsequent key global calls to action [3,7]. An adequate nationwide laboratory system that is able to reliably support outbreak and surveillance activities consists of human health laboratories among other sectors including animal, food, water and environmental health depending on the context, as most of the emerging and re-emerging disease threats are proving to be zoonotic [[8], [9], [10]]. Therefore, a quality management system implementation strategy that seeks to not just capacitate human health laboratories but leverages a One Health approach is encouraged.

Quality assured diagnostics for both human and veterinary services are key in enhancing of efficiencies in the government of Armenia (GoA)’s laboratory testing capacities to detect select agents at a minimal number of safe and secure facilities as well as in enhancing safe, secure, and sustainable infectious disease surveillance and reporting [11]. In 2017 the GoA and the United States of America Defense Threat Reduction Agency (DTRA) collaborated in the strengthening of human and animal laboratories. They fall under the National Center for Disease Control (NCDCP) for human health and the Food Safety Inspectorate (FSI) for animal health. Through the International Science and Technology Center, Astana Kazakhstan, the Integrated Quality Laboratory Services (IQLS) -providing technical assistance- worked with the GoA and DTRA to address key gaps related to laboratory QMS. Selected laboratories were composed of central facilities located in the capital city and satellite branches in regional locales (locally Marzes).

IQLS conducted laboratory assessments of both the human and animal laboratory networks. These assessments highlighted that a majority of laboratories were challenged in quality management systems. In this article, we present three years of laboratory level QMS implementation (2017–2020). We describe a QMS strengthening approach that uses evidence-based results from laboratory system and on-site laboratory facility assessments to guide implementation including training and on-site mentorship . We used adapted international tools allowing for a phased approach as recommended for such settings.

Methods

Laboratory quality strengthening process

The baseline state of QMS was determined through an inception period, of system and site assessment of nine laboratories (human (n = 6) and animal health (n = 3) in the fourth quarter of 2017 and early 2018. Information from these assessments was subsequently used for further site selection. Pursuant the inception period, Fifteen (15) key laboratories that would form the backbone of the laboratory system were purposively selected by a joint GoA/DTRA working group and were composed of the following (see Fig. 1):

Fig. 1

Map of the Republic of Armenia showing laboratories in the quality management system strengthening initiative.

7 human health laboratories: Central Reference level (Yerevan) and its 6 Marz level branches

2 infectious disease hospital laboratories

6 Veterinary laboratories: Central Reference level and 5 Marz level laboratories

Two trainings were organized (level 1 and 2), to pass knowledge and skills on QMS implementation. The level one workshop was designed as a two weeks train the trainer model to assist laboratory quality managers and their deputies to develop competencies that are transferrable to other staff and also for QMS implementation at their laboratories. A teach back approach was used for the second week to develop knowledge sharing skills among the participants. The second training was conducted over one week, bringing together human and veterinary specialists for experience sharing. A local IQLS team composed of an international laboratory specialist and a national laboratory specialist provided direct on-site mentorship alone or in conjunction with visiting international experts - either veterinary or human health. IQLS teams worked in conjunction with the national quality focal persons on these visits. Onsite visit activities included review of previously developed action plans for progress with the laboratory management and quality focal person. Each laboratory was encouraged to form a quality team which would meet weekly to deliberate on QMS activities. Further follow-ups utilized an adapted checklist from World Health Organization (WHO) Laboratory Quality Stepwise Improvement process (LQSI) [12] customized by IQLS into a user-friendly MS Excel workbook and validated in other countries. This tool is based on the ISO15189:2012 standard, broken down into 4 implementation phases enabling laboratories to move towards accreditation in a step-by-step manner as follows:

Phase 1: Ensuring that the primary process of the laboratory operates correctly and safely

Phase 2: Controlling and assuring quality and creating traceability

Phase 3: Ensuring proper management, leadership and organization

Phase 4: Create continuous improvement and prepare for accreditation

It was chosen by the NCDCP top management as their Central facility laboratory was targeting accreditation for several analytes. A practical tool developed by IQLS to assess veterinary laboratories (animal health and/or food safety) previously used in Mali and Mauritania (2016) and Pakistan (2017–2018) was selected for the animal sector and customized for Armenia. These tools assisted facilities with tracking implementation of activities. Additional activities included supporting facilities with development and review of standard operating procedures (SOPs) and guidelines as well as their adaptation and adoption at lower-level facilities.

Assessment process, data capture and analysis

Both baseline and final assessments for the human and veterinary laboratory networks were conducted by IQLS laboratory specialists. Each laboratory was assessed by three individuals composed of local IQLS team members and international laboratory specialists. Human health laboratory specialists assessed only the human laboratories whilst veterinary laboratory specialists focused on the animal laboratory network. National quality and biosafety managers accompanied the assessors for skills transfer in assessing the Marz facilities. The assessment process included an initial opening meeting with laboratory top management. This was followed by a walk-through visit of laboratory facilities following sample path from collection to results reporting. Assessors looked for evidence of implementation of quality practices in the different areas visited, including availability of documentation in terms of SOPs, guidelines, and forms or records where applicable. After the laboratory visit the assessment tool was completed, in conjunction with main stakeholders (quality manager/officer (if any), laboratory management and other lead staff) ensuring that data was verified and reconciled, with additional document verification for availability of corroborative evidence. Finally, feedback was given to the laboratory through a closing meeting with head of the facility and laboratory management.

Assessment tools

Human laboratories were assessed using the adapted WHO-LQSI checklist. All 4 phases were assessed at the Central Facility encompassing the complete 12 quality system essentials (QSEs) [12]. Nevertheless, this tool was not appropriate for other facilities (NCDCP branches and hospitals) due to its strict demands as it assumes in-depth knowledge of QMS and its specific vocabulary. We therefore only assessed phases 1 and 2 (meaning basic but robust QMS in place) for 10 QSEs (excludes QSE 11 and 12). The veterinary laboratories tool assesses different aspects of the laboratory with 14 elements. Modules of both tools are outlined in Table 1.

Table 1

Lists of elements in assessment tools.

LQSI checklist elements	Veterinary tool elements
1. Information about the laboratory, list of documents available	1. Information about the laboratory, list of documents available
2. Facility and Safety	2. Premises quality
3. Organization	3. Specimen collection, recording and handling
4. Personnel	4. Biosafety
5. Equipment	5. Quality management
6. Purchase	6. Supply and equipment management
7. Process	7. Equipment availability
8. Information	8. Budget and finances
9. Documents	9. Data management
10. Customers	10. Diagnosis capacities (clinical, general microbiology, antimicrobial susceptibility tests - AST)
11. Assessment	11. Staff available in the laboratory, Staff management
12. Non-conformity	12. Training and supervision
13. Continual Improvement	13. Information technologies (IT)
14. Research (Not assessed)	14. Communication
15. BSL 2 + 3 (not assessed)	15. Bottle neck analysis
16. Bottle neck analysis

All tools (provided as supplementary documents) are MS Excel files with similar functionalities,

Composed of different tabs corresponding to the different aspects/quality system essential assessed and include easy export features for data aggregation

All tools/files are multilingual, including English and Armenian

The tools only include closed questions with drop down list answers, automatically generating a score in percentages. For each question, a specific field allows for comments and information

A summary tab displays a comprehensive overview of the assessment

Descriptive statistics were used to classify laboratory implementation status using the general indicator score (which is the average of all module indicator scores).

The judgement criteria of the performances of the laboratories and modules were as follows: excellent (>90%); very good (>70% but ≤90%); good (>60% but ≤70%); fair (≥50% but ≤60%); weak (≥35% but <50%); very weak (< 35%). The same tools were used for baseline and final assessments.

Results

Training and mentorship

In total, 54 laboratory staff were trained in quality management systems.

For the QMS Level 1 Human laboratories (train the trainer) 22 laboratory quality managers, and microbiologists were trained. For the QMS Level 1 Veterinary (train the trainer) 12 veterinarians were trained. Twenty-one (21) specialists were trained for the QMS Level 2 composed of 13 human health and 9 veterinary health specialists. Trainings exhibited improved scores from pre, and post tests results with all three sessions recording positive gains (+21%, +15%, +3.33% for QMS level 1 human laboratory specialists, QMS level 1 veterinary laboratory specialists, and QMS level 2 respectively). Following the training, each participant conducted at least one step down training for staff at their laboratories which was verified during mentorship visits.

Assessment results

Assessment results were available for 9 human health laboratories (n = 6 with baseline and final results while n = 3 had final assessment results only as enrolled post inception). Of the laboratories with both results, the central laboratory (human 1) had a general indicator score of 46% across all four phases (see Fig. 2). However, as previously stated the rest of the laboratories were not suitable to assess all four levels and were graded based on the two phases. All of the 5 Marz laboratories with both results scored below 50% meaning they classified weak to very weak (range 21% to 47%). After implementation the central facility had a very good average score (73%), whilst the Marz laboratories graded as follows: one had very good score (74%); three had good scores (60%, 65%, 66% respectively); one had a fair score (54%); two had weak scores (40, 47% respectively) and one scored very weak at 31%. (see Fig. 2).

Fig. 2

Comparative general indicator scores human health- baseline compared to final assessment.

Average scores of the different module indicators for baseline assessment showed that out of the 12 QSEs only two scored above the weak grading: Facility and Safety 56% (with low scores for facility and safety assessments, biosafety manual and information, infection prevention and risk group indicators) and Non-conforming events 93% also the highest (see Table. 2). The lowest scoring QSE was the continual improvement (0%) which also remained a weak area even after final assessment (one of the last steps prior to formal accreditation). Following implementation all QSEs scored fair and above. The highest scoring QSE was Documents and records 89% (63% increase) and the lowest was Customers (50%) (low scores in service manual and management indicator). Improvement difference in QSE scores from baseline to final assessment ranged from 0 to 63% with the highest improved score recorded for Documents and records.

Table 2

Comparative QSE indicator scores human health- baseline compared to final assessment.

	Baseline	Final	% Difference
General Indicator	37%	57%	19%
1- Facility and safety	56%	76%	19%
2- Organization	38%	69%	31%
3- Personnel	28%	59%	31%
4- Equipment	44%	76%	32%
5- Purchase	46%	62%	16%
6- Process	43%	79%	36%
7- Information	37%	68%	30%
8- Documents	26%	89%	63%
9- Customers	32%	50%	18%
10- Assessment	37%	54%	17%
11- Non-conformity	93%	100%	7%
12- Continual improvement	0%	0%	0%

Three animal health laboratories were assessed during the baseline assessment. The Central facility (Vet 1) scored highest with a grading of good whilst the other two laboratories had weak grading score (37 and 49% respectively) see Fig. 3. Following implementation at final assessment all laboratories except for one scored fair and above, the highest score was for the Central facility which scored excellent at 79%. The highest Marz veterinary laboratory had a good score at 61% with lowest scoring 46%.

Fig. 3

Comparative general indicator scores animal health- baseline compared to final assessment.

As shown in Table 3, the General indicator average score for the veterinary laboratories improved 8% from weak to fair mainly due to improvements in Analytical quality management (32%), Equipment management and supply (16%), and Data management (21%). The highest score was obtained for Sampling and sampling transportation (75%) and Diagnostic capacity (83%). The lowest scoring was for Budget and finances (44%), Training and supervision (37%), Information technologies (38%), and Communication (42). Quality of installations were the only indicator that regressed, decreasing by 7%.

Table 3

Comparative indicator scores animal health- baseline compared to final assessment.

	Baseline	Final	%Difference
General indicator	49%	57%	8%
1- Quality of installations	71%	64%	-7%
2 - Sampling and sampling transportation	67%	75%	8%
3 - Biosafety	57%	57%	0%
4 - Analytical quality management	19%	51%	32%
5 - Equipment management and supply	53%	69%	16%
6 - Budget and finances	40%	44%	4%
7 - Data management	31%	52%	21%
8 - Diagnostic capacity	80%	83%	3%
9 - Training and supervision	37%	37%	0%
10 - Information technologies	38%	38%	0%
11 - Communication	41%	42%	1%

Discussion

Implementing quality management systems in laboratories is expected to enhance diagnostic testing processes ensuring high levels of competence for reproducible and accurate results [1,3,4]. IQLS implemented a One Health laboratory QMS strengthening project in Armenia focusing on both human and veterinary laboratories which eengaged high level officers through an intersectoral approach with the Ministries of Health and Agriculture facilitating collaboration. This is not always easy to achieve due to the vertical nature of how these sectors operate [13,14]. A comparative analysis of the final assessment against baseline scores shows increased scores per the different laboratory QSEs.

Improvement for both networks were in areas of organizational structure, human resources, equipment management, supply chain and data management. These advances were due to practical implementation of training and the numerous SOPs, procedures and guidelines that were developed and adopted by all laboratories. Central facilities implemented QMS to the level of considering accreditation for key analytes. Central facilities showed stronger results since they are reference laboratories, linked to international networks and received prior support from other entities which raised awareness of QMS concepts earlier.

For the human Marz laboratories all except 4 laboratories moved from weak to good and or very good. Of the remaining 4, one of the laboratories moved from weak to fair, this was mainly due to staff motivation challenges emanating from employment uncertainties for lead staff. The other three laboratories that remained in the weak zone lacked appointed quality managers for a large part of the implementation period. Two of the lowest results (average 39%) were from the hospital network which was not part of the NCDCP network and thus had little to no prior exposure to documentation of QMS activities in addition to lacking key leading staff during implementation emphasizing the need for a system wide focus to lab strengthening.

Per the QSEs general indicator scores there is an average 19% increase for all human laboratories. The non-conforming events QSE showed excellent results. The lowest scoring QSE was on continual improvement which can be explained by a new QMS system still focused on developing many documents with limited implementation especially at Marz level. Personnel, customers and assessments had fair scores. Personnel nonconformities were a result of gaps including lack of staff appraisal systems, job descriptions and continuous professional development which are outside the immediate purview of laboratory operations requiring long intervention time to change. Customer satisfaction was also an area in progress with laboratory user manual which constituted a major requirement still in draft format. From the results the laboratories scored above average on all the modules except continual improvement and customers, not currently applicable for these laboratories recently implementing QMS, thus focusing mostly on document development and process management.

For the veterinary network the central facility results showed substantial progress during the last two years evidenced by the general indicator percentage difference of 19%. Of note is the improvement in the critical areas such as analytical quality management, equipment management and supplies, data management, and AST performance. However, diagnostic capacity (testing of bacterial, viral and parasitic infections) requires more focus to increase functionality of the central facility for disease surveillance and testing of bacterial and viral infections by PCR and ELISA.

The Marz veterinary laboratories, also recorded some improvement (9%). The little progress is due to lack of a dedicated quality and biosafety manager as FSI have not appointed this person for the veterinary laboratory network as this requires some policy changes from top management. Quality of installations regressed due to limited funding to laboratories as well as lack of general maintenance that could be only provided by qualified engineers within the network. Areas of progress related to analytical quality management and data management. Other areas showed limited improvement indicating further attention is required as well as supervision from central level.

There were limitations on generalizing progress as not all laboratories had baseline results, due to a limited number of facilities being assessed during the inception period. However, considering the state of the rest of the Marz level facilities there is room to conclude improvements were brought about through the implementation of this project. The LQSI tool used to assess human laboratories was appropriate for NCDCP (targeting accreditation) but not for branches. It would have been easier and more result-driven to use an adapted tool which would have provided much more concrete ideas for improvement than LQSI, which asks for significant knowledge in QMS even for lower-level laboratories which may never be targeted for international accreditation. We however were able to use the same tool to guide implementation with laboratories following expected outputs from each phase. Similarly, the veterinary laboratory assessment tool was appropriate for the central facility, but not for small rural laboratories which only perform few tests.

The improvements demonstrated within the project can be credited to collaboration between local and international specialists working to provide direct mentorship to facilities. Other QMS projects also report the usefulness of mentorship on improving laboratory QMS processes [15,16]. Mentorship provided a platform for routine interaction with Marz facilities enabling the sharing of documentation as well as assisting with other implementation challenges thus enabling the laboratories to move forward. The mentorship visits also highlighted the need for better communication within the network and continuation of supervision programs post project implementation (especially where central level staff were aware of availability of policies that were not known at Marz level).

Laboratories showing poor results post implementation lacked quality managers for long periods of time, demonstrating importance of a full time or substantive quality manager. Without a quality manager, progress is slow. The combined level two QMS training with veterinary and human specialists was useful with staff from both sides learning about and from each other's implementation processes and means to solve challenges. Additional activities which were conducted in a One Health aspect included a joint external quality assessment program and equipment purchased for both networks to enhance quality testing.

Conclusion

Comparative analysis of final assessment results against baseline scores showed improvement in Armenian laboratory operations. A one health approach to laboratory strengthening can be implemented with multisectoral stakeholder involvement.

Funding

The development of this publication was supported by the and the United States project implemented in Armenia under the grant/agreement HDTRA117F0048P00003 Laboratory Strengthening Project in Armenia.

Author contribution

The original manuscript was developed and drafted by O.K., A.O., J.M., Z.D, and A.P. provided considerable input into structure and reviewed the manuscript. J.C, P.T, A.P, N·H, reviewed the manuscript.

Declaration of Competing Interest

The authors declare no conflict of interest with respect to the research, authorship, and/or publication of this article.