Development and implementation of National External Quality Assurance Programs in a One Health approach: The Armenian experience.

INTRODUCTION: Early warning and objective evidence of systematic errors in laboratory diagnosis ensures evidence based corrective and preventive actions that instill patient safety and confidence. External quality assessment contributes significantly to the above as an essential component of laboratory quality assurance. However, implementation of External Quality Assessment in resource-limited settings is challenged by high costs of enrolling in international schemes. To ensure sustainability, a National External Quality Assessment Program in Armenia was developed using a One Health approach.
METHODS: Through engagement of stakeholders from Ministry of Health and Department of Agriculture under Ministry of Economy the government of Armenia started the implementation of the Armenia Laboratory External Quality Assessment (ALEQA) program. Policies and procedures were defined, a web interface for return of results and feedback reporting was created. A training was offered for characterization of simulated samples for bacterial pathogens. Following a pilot survey, the program was successfully scaled up, with later addition of a Brucella serology discipline.
RESULTS: The return rate of results was 100% for all surveys. There was an improvement in the performance of the laboratories from the 2015 to the 2019 surveys. The bacterial pathogens EQA survey's, was interrupted between 2017 and 2019. The Brucella Serology survey showed 77% of the 26 participating laboratories had satisfactory performance.
CONCLUSION: This is one of the few National EQA Programs that have embraced the One Health approach to improve reach of EQA Programs in resource-limited settings in both human and veterinary laboratories.

Introduction

More than 60% of human infectious diseases worldwide are caused by pathogens of a zoonotic nature mostly originating from wildlife and crossing over to livestock [1]. The ability to stop the spread of these pathogens relies on the capacity of both human and animal health systems to accurately detect these events early and rapidly implement control measures. Quality laboratory testing is therefore a key component in the detection and control of infections in both humans and animals.

Participation in External Quality Assessment (EQA) schemes to monitor the performance of diagnostics, prognostic, or screening testing in routine testing is fundamental in any laboratory quality management system [2]. It is mandatory for international laboratory accreditation [3,4]. EQA enables laboratories to regularly check the performance of routine tests and compare themselves against other laboratories [5]. EQA can be accessed through accredited international EQA providers worldwide covering the whole spectrum of laboratory tests. However, participation in these EQA schemes comes at a cost and in most resource-limited settings like Armenia, this can become an obstacle for enrollment. Additionally, customs clearance of the samples can also be a challenge particularly in Armenia. National EQA programs can minimize the cost and provide a sustainable solution by localizing the production, packaging, transportation, results analysis, and feedback of EQA schemes [6]. National EQA schemes also have the potential to increase scope and coverage of tests in their schemes in response to national needs at a lower cost. They also provide for in-country capacity building in terms of data analysis, specimen characterization and other EQA related activities [7]. They also improve samples availability and integrity as everything is within the same country.

Setting up National EQA schemes require establishing policies, procedures, results interfaces, training to EQA providers, specimen packaging supplies, and transportation modalities [7]. Setting up the management structures to govern the programs needs careful consideration to maintain confidentiality and objectivity of the programs. If the schemes are to cover different laboratory networks representation from all the networks will be key so that the laboratory testing needs from all networks are fully represented. Getting approval from the relevant authorities and making the schemes mandatory to all laboratories will make the program more effective in detecting testing gaps in the different networks thereby strengthening ability to detect any unusual disease events [8].

One Health approaches enable collaboration between human, veterinary and food safety sectors to collaborate in detecting zoonotic diseases and other pathogens that affect humans and animals. External Quality Assessment (EQA) in a One Health approach enables the assessment of human, veterinary and food safety laboratories with the same schemes thereby helping preparedness of both laboratory networks for diseases that affect humans and animals. Having a single EQA Program Management Committees (PMC) for human, veterinary and food safety laboratories helps with identification of common areas of concern. This is critical in preventing spillover of infections from animals to humans and vice versa [9].

This manuscript documents the process of development, implementation, challenges, and lessons learnt from the Armenian National Laboratory EQA (ALEQA) Scheme that was initially established in 2015. The program was launched in a One Health approach with the human veterinary, and food safety laboratories participating.

Materials and methods

The development of ALEQA scheme was initiated in 2015, facilitated by IQLS under the Pennsylvania State Veterinary University/U.S. Defense Threat Reduction Agency (DTRA) supported project. Initially, two workshops were held with participation from Armenian National Center Disease Control and Prevention (NCDCP) and the food safety laboratory of the Republican Veterinary Sanitary and Phyto-Sanitary Center for Laboratory Services (RVSPSCL). The workshops were to set the goals of the EQA program, determine structure of a web interface and develop program policies and procedures. The workshops were held in March 2015.

Bench level practical training on preparation of simulated stool samples

Following the two workshops, a bench-level training was organized in Armenia in June 2015. Laboratory specialists from the Armenian NCDCP and RVSPSCL participated. They were taught how to create simulated stool samples using overcooked lentils which were spiked with reference strains representing enteric pathogens (Shigella, Salmonella and Enterococcus) and/or commensals [10].

Training on use of the web interface

A web interface was created following the definitions from the workshop participants. The workshop participants were chosen with the potential of becoming members of the Technical Implementation Group (TIG) that was to be constituted later. The training on the web interface was done concurrently with the practical bench-level training. The participating laboratories and the proposed TIG were trained on using the interface to create surveys, enter results and grading them.

Bacterial pathogens surveys

Immediately following the practical bench-level training, a pilot survey on bacterial pathogens was sent out in October 2015. The participants comprised NCDCP Marz Branches and the food safety laboratory. The survey consisted of 6 blinded samples for enteric pathogens. Of the 6 samples one did not have any pathogens. The samples assessed the ability of the laboratories in performing microscopy, bacterial identification, antibiotic selection and antibiotic susceptibility testing. Participating laboratories were given 2 weeks to send back results via the web interface.

Following the pilot survey in 2015, the ALEQA program had 2 more surveys in the backdrop of organizational difficulties and problems with the web interface prompting the need for a relaunch. Through a new DTRA/ISTC project that started in late 2017 a workshop was held in October 2018 to refresh the members of the TIG on use of the web interface and preparation of simulated stool samples. The reasons for suspension of the program between 2016 and 2018 were also discussed.

To expand reach of the relaunched EQA program, a government order was granted. The order allowed the EQA to be sent to all public hospital laboratories. In October of 2019, another round of bacteriological EQA panels was sent to all NCDCP laboratories, public hospital laboratories, and the Veterinary Reference Laboratory.

Addition of Brucella serology discipline 2019

The Brucella serology discipline was added to ALEQA in November 2019, giving the option to assure quality of Rose Bengal test, Wright Huddlestone test for both human and veterinary laboratories, and ELISA and Florescence Polarization Assay only for veterinary laboratories. A team consisting of human and veterinary laboratory specialists was trained on this discipline with the Veterinary Reference Laboratory taking lead in the production of the EQA panels, distribution, and analysis of results. Since this serology survey was a pilot, formalization of the program is planned for the second round of the PT to be sent out in 2020.

Scoring of responses

Bacterial pathogens

The bacterial pathogens survey evaluated laboratories for 4 different aspects of bacterial analysis. The grading of the different aspects was as follows:

Microscopy

Correct gram reactivity of the organism was awarded a score of 4 and incorrect gram reactivity was awarded 0 for each of the 6 specimens sent out. The minimum passing threshold for the module was arbitrarily set at 83,3%. The score is calculated by dividing the mean of the scores of individual samples by the maximum score and expressing it as a percentage.

Identification

Individual specimens sent were graded as follows:

The minimum passing threshold for this module is set at 80%. The score is calculated by dividing the mean of the scores of individual samples by the maximum score and expressing it as a percentage.

Antibiotic selection

The participants choose 6 antibiotics based on the pathogen and site of the pathogen. The grading of the individual samples is as follows:

The minimum passing threshold for the module was set at 70% and later revised to 75% in subsequent surveys. The upward revision of the passing score was meant to initially ease participants into the program before setting the intended score of 75%. The score is calculated by dividing the mean of individual scores by the maximum score and expressing it as a percentage.

Antimicrobial susceptibility testing (AST)

The minimum passing threshold for the module was set at 70% in the pilot survey and was revised to 75% in subsequent surveys. The score is calculated by dividing the mean individual samples scores by the maximum score and expressing it as a percentage.

Brucella serology

The individual samples were graded as followed:

The minimum passing threshold for the survey was set at 100%.

Ethics statement

For this study no ethical approval was required. The samples for intestinal pathogens were simulated using overcooked lentils and spiked with reference strains. Brucella serology samples were created using routinely collected samples from cattle for diagnostic purposes that were made available for pooling and creation of serum aliquots.

Results

ALEQA was launched in 2015 with a pilot survey that covered human health, veterinary and food safety laboratories for bacterial pathogens emphasizing a One Health approach. The pilot survey focused only on enteric pathogens. The structure of the program with its different committees and groups were formed as shown in Fig. 1. Participating laboratories able to use a customized web interface for results reporting, and feedback was given to participants within a month before survey closure. (See Table 1, Table 2, Table 3, Table 4.)

Fig. 1

Organization and interaction between different groups in the operation of the ALEQA program.

Table 1

Grading of organisms' identification.

Score	Definition	Performance assessment
4	Correct genus and species	Correct
3	Correct genus only	Acceptable
0	False Positive/False negative	Unacceptable

Table 2

Grading of antibiotic selection.

Score	Definition	Performance assessment
4	6 correct antibiotics	Correct
3	5 or 4 correct antibiotics	Acceptable
1	1,2 or 3 correct antibiotics	Unacceptable
0	Zero correct antibiotics	Unacceptable

Table 3

Grading of AST.

Score	Definition	Performance assessment
4	Completely Correct result	Correct
1	An antibiotic that is supposed to be susceptible is resulted as intermediate or an antibiotic that is supposed to be resistant is graded as intermediate	Unacceptable
0	Incorrect result	Unacceptable

Table 4

Grading of Brucella serology.

Score	Definition	Performance assessment
4	Completely Correct result	Correct
0	Incorrect result	Unacceptable

Creation of the different groups for the operation of ALEQA

As depicted in Fig. 1, the following different groups were created to monitor and implement the survey. They successfully implemented all the policies and procedures set out for the ALEQA program.

PMC

The functions of the PMC were to coordinate the operational activities of the ALEQA program through the technical implementation group. At this time the PMC was not yet fully constituted but instead its functions were taken over by the Ministry of Health (MoH). The MoH was responsible for the review and approval of technical operations, ensuring surveys were conducted with planned frequency and ensuring confidentiality of results. They were also responsible for providing recommendations for program improvement.

TIG

The TIG was responsible for providing recommendations to the PMC. It was constituted by Laboratory Quality Officers (from the MoH and MoE), Biosafety Managers (from MoH and MoE), microbiologists, and serologists. The TIG is responsible for preparation and management of the survey, preparation of reports, and sending back reports to participating laboratories and PMC. They also developed program policies and procedures and standard operating procedures for all implementation operations. The TIG is responsible for the creation of surveys (number of samples, strains to be sent, grading), coordination with the referee laboratory and identification of the factors affecting performance.

Technical Partner

The technical partner was chosen by the TIG with approval of the PMC and tasked with the responsibility to prepare the simulated samples under the guidance of the TIG. The technical partner for intestinal pathogens was the Bacteriology unit of the NCDCP Reference Laboratory, while the technical partner for Brucella serology was the Veterinary Reference Laboratory. The two technical partners produced the EQA samples, packaged and sent them to all participating laboratories.

Referee Laboratory

A specialized laboratory with recognized national expertise was used as a referee laboratory. The criteria for selection of the referee laboratory was successful participation in international EQA. In this respect the especially dangerous pathogen (EDP) department of the Reference Laboratory for the NCDCP was chosen for the Bacteriology Survey. Its main role was to validate the surveys by analyzing the survey samples. The Veterinary Reference Laboratory was the Referee laboratory for the Brucella serology EQA.

Participating Laboratories

The ALEQA program is open and free of charge to all diagnostic laboratories in the country.

Development and use of the web interface

All participants successfully returned results via the web interface for both the bacterial pathogens and Brucella serology surveys. Antibiotic selection grading for bacterial pathogens proved to be challenging for the organizers. The best choice of antibiotics for the country was not standardized resulting in many possible combinations of antibiotics being used. This posed a challenge with grading as the interface assumed participants were supposed to select all the available antibiotics as expected antibiotics. Ideally, participants were supposed to choose only 6 antibiotics from the available list. A solution was sought via minor changes to the interface. There were no challenges for the organizers of the Brucella serology survey.

Summary of performance of labs in bacterial pathogens surveys

The pilot survey started on the 21 September and ended on 2 October 2015. Fifteen laboratories participated. The 2016 survey ran from 30 March to 12 April and the 2019 survey ran from 21 October to 8 November with 16 laboratories participating. There was a 100% response rate in all surveys. The surveys assessed 4 areas of bacteriological analysis, namely microscopy, identification, choice of antibiotics, and AST. The summary of performances is shown in Fig. 2, Fig. 3, Fig. 4.

Fig. 2

Average scores per module per survey.

Fig. 3

Number of laboratories above passing threshold per module.

Fig. 4

Number of modules passed per survey.

In the 2015 survey, 6 of the 15 laboratories managed to get the minimum passing threshold for all 4 areas of the assessment. There was a general weakness for laboratories in the country in the choosing of antibiotics and the performance of AST. Two of the 15 laboratories did not manage to pass any of the areas. There was progressive improvement in the laboratories in 2016 and in 2019. In the 2019 survey, 14 out of the 16 laboratories managed to score the minimum passing threshold for all the 4 areas of assessment, with the remaining 2 laboratories performing satisfactorily in 3 areas.

Non-continuation of the bacterial pathogens surveys

Following the pilot survey in 2015, one additional survey followed in 2016. The Program however suffered from drawbacks that were mainly organizational in nature. The developed web interface was only used for entry of results and return of results in the 2016 survey and had bugs that made it difficult to use. The 2017 survey did not use the interface and the results could not be graded. The reasons leading to non-continuation can be summarized as follows:

Lack of a PMC and TIG not fully defined from the beginning.

Lack of trust from the participating laboratories because the program was not compliant with ISO17043.

The Policies and Procedures have not been officially approved by the authorities and were conflicting with regulations already in place.

The design of the program was not appropriate for all the participants: in the Veterinary and Food Safety laboratories. AST is not performed routinely in by the veterinary and food safety laboratories.

Some participants complained from the fact that no certificates were issued following their participation.

The web interface experienced bugs and translation issues leading to difficulties in entering and extracting results from the interface.

Summary of performance of labs in Brucella serology pilot survey

This survey had 26 laboratories participating from human and vet laboratories. It ran from the 4th to the 17th of November 2019 and included 7 samples. The survey had a 100% response rate and the passing score for this survey was set at 100%. In this survey, 20 out of the 26 laboratories passed. Summary of the performance of the laboratories per sample is shown in Fig. 5. Sample A was diluted to create a weak positive sample. The rest of the samples were clear cut positive and negative samples.

Fig. 5

Summary of performance of laboratories in the Pilot Brucella Serology Survey per sample.

The samples were identified as A, B, C, D, E, F and G.

Discussion

The ALEQA program improved reach of EQA in Armenia for both human and animal laboratories with some of the peripheral laboratories receiving EQA for the first time via the pilot surveys. The pilot survey of the bacterial pathogens indicated an overall poor performance with only 6 of the 15 participating laboratories passing all 4 areas of assessment. There was great improvement in the performance in the 2016 and 2019 surveys with 13 and 14 of the 16 participating laboratories managing to pass all 4 areas of assessment respectively.

The National EQA for bacteriology helped to recognize the need for a national policy for antibiotic use due to poor performance in antibiotic selection. Additionally, some laboratories had no international standards to guide antibiotic susceptibility testing. All laboratories were provided with the international standard (EUCAST) to guide antibiotic selection and threshold zones for the different antibiotics. Meanwhile discussions on the development of a national antibiotic use guideline have been initiated.

The approval of the National EQA program by the GoM under order N1587-A of the MoH within the framework of the ALEQA allowed for the program to be mandatory for all government run medical laboratories. This is an important step in improving the quality of testing in medical laboratories in Armenia. However, the implementation of the order is not yet complete as some of the laboratories did not participate in the last survey. The cost of international EQA subscription is prohibitive for all laboratories in the country to be enrolled due to resource constraints. Customs clearance of the samples can also be a challenge particularly in Armenia. International EQA subscription was therefore reserved for the reference laboratories that are used to benchmark the National EQA program. Since EQA is organized nationally the TIG plans to hold meetings, workshops, and trainings with the laboratories to address common areas of concern. Ability to address the needs of the country in a timely manner is one of the advantages of having National EQA programs as meetings were organized immediately after the official release of the EQA results to discuss observed shortcomings. Formal trainings were also planned.

The expansion of the National EQA program focused on Brucellosis, which remains one of the major zoonotic diseases in Armenia that requires a One Health approach. In 2012 alone, there were a total of 5063 cases in livestock [11]. In humans, brucellosis was responsible for 265 hospitalizations at Nork Infectious Diseases Hospitals in 2016 [12]. Reference laboratories and Marz (first-level administrative entity in Armenia) laboratories for human and veterinary, all carry out brucellosis testing using serological techniques. The Brucella serology EQA assured the quality of tests used in human and veterinary laboratories. The established Inter-ministerial Committee (IMC) allowed for cooperation between human, veterinary and food safety laboratories. The cooperation was in the form of joint planning and implementation of EQA activities. The EQA activities included selection of pathogens and composition of panels. The overall performance of the laboratories was fair with 20 out of the 26 laboratories obtaining the passing score of 100%, whereas 6 laboratories did not correctly evaluate the weak positive sample and one laboratory failed all positive samples. This data indicated poor sensitivity of the test systems used in failed laboratories (an important factor for initial diagnostic tests) and respective corrective action needs to be implemented. The need to have a post market surveillance program for all laboratory tests was recognized to ensure consistent quality of diagnostics used in the laboratory and possibly to ban poor quality reagents.

Implementation of the National EQA faced several challenges that needed to be overcome to keep the program running. The proposed structure of the program took time to be fully functional resulting in only one other EQA round for bacterial pathogens after the initial pilot in 2015. Some of the participants did not have full trust of the program since it was not accredited by the relevant international standard (ISO 17043). The policies and procedures of the National EQA program were only officially approved in 2019 and before that the participating laboratories did not feel compelled to participate as it was informal. The official approval made participation in ALEQA mandatory. These issues emphasize the need for national regulations to ensure national EQA programs take root and receive full support from participating laboratories. The program did not offer certificates of participation. Except for the accreditation, all these challenges were resolved before the relaunch in 2019. However, the approval of the bacterial pathogens only made it mandatory for the human health laboratories. This leaves out the veterinary and food safety laboratories who have been participating informally. The formation of a joint PMC under the IMC comprising of human, veterinary and food safety laboratories is being sought as the One Health approach is being emphasized. The involvement of regulation in making sure that this is sustainable cannot be over emphasized [8].

Armenia has engaged in Laboratory Quality Stepwise Implementation (LQSI) process to improve quality management systems implementation towards accreditation. EQA is one of the 12 quality system essentials (QSE) making participation in programs like ALEQA an indispensable part of its implementation.

Conclusion

From our knowledge, this is one of the few National EQA programs that has embraced the One Health approach and has allowed both human and veterinary laboratories to be enrolled in regular EQA programs. Laboratory specialists were trained on preparation of simulated samples, data management, and managing an EQA program according to ISO:17043. The web interface in Armenian is ready to take up more EQA program and expand the program.

Author contribution

The original idea of the perspective manuscript was developed by all authors who has equal contribution to the development of this manuscript.

Availability of supporting data

All supporting data for manuscript is available as supporting material as attached.

Consent for publication

All authors have consented to the publication of this manuscript in the current format.

Funding

The funding for the work done as part of this manuscript was supported by the and the project implemented in Armenia.

Ethical approval and consent to participate

For this study no ethical approval was required. The samples for intestinal pathogens were simulated using overcooked lentils and spiked with reference strains. Brucella serology samples were created using routinely collected samples from cattle for diagnostic purposes that were made available for pooling and creation of serum aliquots.

All the laboratories participated as part of routine EQA.

Declaration of Competing Interest

The authors declare no conflict of interest with respect to the research, authorship, and/or publication of this article.