The challenge of producing an EQA for the COVID-19 pandemic.

The COVID-19 pandemic has changed the clinical medicine landscape. The importance of pathology testing has come to the forefront. Patients or potential patients are dealing directly with laboratories as they line up in carparks or testing staff come to the front doors to obtain samples. Laboratories have had to increase capacity to deal with the high volumes of testing driven by the need to identify and quarantine cases. Supporting this effort, External Quality Assurance scheme providers have also needed to produce COVID-19 Proficiency Testing (PT) programs which are fit for purpose. COVID-19 Point of Care testing has become critical frontline testing and has required the PT programs to be simple to use, readily accessible and robust. We describe a COVID-19 PoCT Serology PT program supported by a mobile phone App. The App is described, and the advantages made explicit. This App suggests that the way that PoCT EQA/PT programs may be deployed in the future.

Introduction

Producing a new EQA by a proficiency testing provider usually requires compliance to ISO/IEC 17043 [1] and careful consideration of sourcing reference samples with a suitable range of measurand, the range of methods and units that may be in the market, ensuring stability and homogeneity and stability of the material, determining the appropriate analytics to report and logistics issues. As the COVID-19 pandemic hit Australia the Royal College of Pathologists of Australasia Quality Assurance Programs (RCPAQAP) [2] was asked to create a new EQA for the serological lateral flow PoCT devices that were being evaluated as suitable devices to assist in the protection of high risk populations, for example aged care facilities and hospitals. The normal process described above may take 12 months but this new EQA was required in a few weeks. POCT is not widely used in Australia outside of laboratory networks, but these new PoCT devices would be deployed into areas without laboratory oversight. They would be introduced into an environment without the normal quality framework in which other EQAs are used. The 10 most frequently cited deficiencies in POCT [3] are: 1) Failure to perform quality control testing, 2) document QC activities, 3) follow manufacturer’s instructions explicitly, 4) document personnel training and competency, 5) document and take appropriate corrective action for control outliers, 6) failure to perform proficiency testing or external quality assessment schemes, 7) have a procedure manual for testing and result reporting, 8) perform and document calibration verification at least every 6 months, 9) verify accuracy for analytes not included in a PT program, and 10) provide for continuing education for testing personnel.

To address these potential errors Martin et al. [4] have described a quality framework consisting of device validation/verification, operator training, inventory management, QC and EQA procedures, competence assessment, record maintenance and governance. However, this framework has at its heart a high level of governance and training of the PoCT operator. The requirements for the EQA were that suitable material to identify poor performance was used, the process of performing and recording the EQA results was simple and intuitive, there was rapid turnaround of the results and support was available.

What is described in this paper is the rapid development and deployment of an EQA program for serological testing for SARS-CoV-2 specifically designed for staff with low levels of training and direct supervision. The aim of the EQA was to be as simple and error free as possible in the areas of registration, data entry, assessment, feedback, and supervisor monitoring.

Materials and methods

Application design & development

The requirement to deliver an EQA to an entirely new cohort of users within such a short timeframe created unique demands on both the delivery process and the application design. The time required to extend RCPAQAP’s existing EQA infrastructure to support this program would have been prohibitive, and the existing platform may have posed challenges for unfamiliar users outside of the laboratory context. For these reasons, a separate system was designed and delivered independently from RCPAQAP’s existing EQA platform. This system was built on the Heroku cloud platform as a service (PaaS), allowing rapid prototyping, deployment, and scaling to many customers, without specialised infrastructure expertise.

In addition to faster application delivery, the lack of existing constraints in a new system also allowed the design to be tailored specifically to point-of-care users. Recognising the additional stress these users would be under during exceptional conditions, the application was designed with as few prompts as possible. Reducing the required user interactions reduces the chances of data entry errors and decision fatigue. The desire to allow users to run the application on their personal devices needed a lightweight and responsive technical design so that the user-experience would be fast and reliable on both desktop platforms and low-power mobile devices using low-speed network connections.

To make the process of recording the EQA results intuitive and straightforward, some affordances were made. To avoid unnecessary complexity identifying participants, no credentials are required from users. Instead, users are pre-registered and begin by scanning a QR code on the sample instructions using the camera on their mobile device. In case the user is unable to scan the QR code, an alphanumeric code — carefully designed to avoid ambiguous characters — can be manually entered to identify the sample. The participant then selects from a list of pre-registered test kits using large and clear touch targets to reduce the chance of mis-selection (Fig. 1). By selecting the test kit in the first step, the app can prompt the participant only for measurands available on the device they have. The results of these measurands are then entered by the participant using large touch targets, along with lot number and operator details. Where possible, operator details are auto filled by the device to minimise the required data entry. Along with this data, the user is prompted to upload a photo of the test kit showing the test result. On supported devices, the user can access their camera directly from the application.

Fig. 1

Simplified 4-step result entry on a mobile device, showing real-time feedback.

To expedite the turnaround of the results, multiple stages of feedback are provided (Fig. 1).

Immediately after submission, the participant is given a provisional assessment to provide initial real-time feedback on their performance. Submitted results including the uploaded photo are then presented to RCPAQAP staff as part of a rapid review workflow, to verify the user-entered result and check for transposed results or other recording errors (Fig. 2). As a suitably large corpus of result photos is collected, this review workflow can be enhanced by machine learning algorithms for automatic verification. If the interpretation is assessed as valid, a performance certificate is issued, and the participant is notified by email (Fig. 3).

Fig. 2

Rapid review workflow showing participant results.

Fig. 3

Certificate of Completion delivered by email after review.

The application has been designed to allow RCPAQAP to scale the customer enrolment process, sample management, and logistics to allow for the higher EQA sample volumes potentially required for COVID-19. Devices and users can be enrolled in bulk using a csv file supplied by the participating organisation or enrolled by RCPAQAP staff. The application allows samples to be registered along with expected results for the measuring platform and the sample and customer are linked at the time of enrolment with the QR code. The platform needed to account for rapid changes in the availability of testing platforms, so new testing platforms can be added and are immediately available to customers, even if a sample has already been dispatched.

Support

Simplified, consumer-oriented user guides were designed to supply the unique QR code and lead users through the testing process. Text was minimised to essential handling and testing procedures, and a 24-h support line was implemented to provide support for testing and follow-up actions in the event of a failure. Telephone support in the use of the device, phone App and interpretation of the assessment is available.

EQA sample

Samples were acquired from patients who had been infected with SARS-CoV-2 as confirmed by an PCR assay. Age, sex, clinical history, and SARS-CoV-2 symptomology were noted. The sample was collected 3–8 weeks post recovery, as defined as two negative SARS-CoV-2 PCR, tested 24 ​h apart. The samples were obtained by informed patient consent as per Ethical Approval (Protocol No X18-0306 & HREC/18/RPAH/425 Sydney Local Health District, RPAH Zone).

All samples were screened using an accredited (laboratory based) method for SARS-CoV-2 Antibody (IgG/IgM) testing and two lateral flow point of care devices, Onsite Rapid Test (CTK Biotech Inc, CA, USA) and COVID-19 IgG/IgM Rapid Test Cassette (Healgen Scientific, Houston, TX, USA). In addition, samples are also screened for HIV and Hepatitis and excluded if positive.

Homogeneity and stability testing were performed in accordance with ISO/IEC 17043:2010 [1]. Stability testing was undertaken at 37° Celsius for 14 days, which showed no deterioration in sample result quality. Homogeneity testing using 15 samples was also performed with 100% correlation for IgG SARS-CoV-2 antibodies.

Frequency of samples

Each survey consists of a single sample sent to the user within one week of formal enrolment. Submission of results.

Discussion

There have been many consequences of the COVID pandemic, one has been the rapid education of the public on pathology testing and PoCT. Obviously with a pandemic there is a need to have rapid reliable testing available at places where decisions will be made instantly about someone being admitted to a workplace or high-risk site. The aim being the reduction of spread of the virus to susceptible people in hospitals or aged care facilities.

However, the need for rapid deployment of the PoCT did not allow a quality framework to be developed and was susceptible to error because of the attitude of the people performing the testing, the nature of the PoCT device used, the testing environment itself, access to training, and the traceability of both documentation and results, including patient results, and quality control results. Untrained staff, even health care workers, can have high confidence in the result of any testing device leading to an underestimation of risk by the user and a false perception of infallibility [5]. Combined with the pressures of the urgency of the testing, the large number of operators performing the testing, and the concerns of the client being tested, there is a need to identify rapidly operators or devices that are not performing appropriately.

As many errors around POCT have been associated with the extra-analytical stages of the process, such as poor documentation and data management, having the ability to automatically capture data electronically can ensure a much greater degree of test accuracy [6,7]. Rapid feedback to the operator of success or failure of the PoCT device is critical, as is simple intuitive instructions on how to perform the EQA exercise.

The PoCT App will also be used for other EQAs. The strengths of this approach are the simple nature of the concept, easy and permanent recording of the result, real time turnaround of the result to the operator, the analytics available to supervisors and the support. This allowed operators with minimal training to quickly understand and use the EQA program with confidence.

Conclusion

This type of testing illustrates the key role that laboratory data play in many clinical decisions; and that raises the possibility of whether more testing will have to move closer to the healthcare consumer of the future [8].

The reliability of serological testing for COVID-19 in the early phases of the pandemic became a critical issue when government were considering the use of these tests [[9], [10], [11]] and subsequently the POC devices have not used to date in Australia. However, they have been used in other countries where the EQA is being used.

Funding

There were no external funding sources.

CRediT authorship contribution statement

Tony Badrick: Conceptualization, Methodology, Writing - original draft, Writing - review & editing. Louise Wienholt: Writing - review & editing. Daniel Fone: Writing - review & editing. Derek Holzhauser: Writing - review & editing.