Automation and standardisation of clinical molecular testing using PCR.Ai - A comparative performance study.

BACKGROUND: We undertook a prospective clinical study to evaluate PCR.Ai's (www.pcr.ai) accuracy and impact when automating the manual data-analysis and quality control steps associated with routine clinical pathogen testing using real-time PCR (qPCR).
OBJECTIVES: We evaluated the impact of PCR.Ai when used as the final interpretation/verification step for routine in-house qPCR tests for respiratory pathogens and for norovirus for a total of 22,200 interpretations. STUDY
DESIGN: We compared PCR.Ai to our existing manual interpretation, to determine accuracy and hands-on time savings. PCR.Ai was accurate. RESULTS AND
CONCLUSIONS: There was 100% concurrence between validated respiratory virus and norovirus detection by our manual routine analysis method and PCR.Ai. Furthermore, there were significant routine savings with PCR.Ai of 45 min/respiratory run and 32 min/norovirus run. Our conclusion is that PCR.Ai is a highly accurate time-saving tool that reduces complexity of qPCR analysis and hence the need for specialists and hands-on time. It demonstrated capabilities to enable us to get results out more quickly with lower costs and less risk of errors.

Background

qPCR has now become a core component of most diagnostic virology laboratories’ testing repertoires. It remains the most popular method for molecular detection of disease and is used globally for pathogen detection. It is considered the gold standard for molecular detection due to its high assay specificity and sensitivity and in our lab comprises 75% of sample testing. However, qPCR methods are still largely in-house developed and maintained. Almost all of this testing method is now automated by off-the-shelf machinery and consumables. However data-analysis and associated quality control still need to be done manually. Manual interpretation is also required for most commercial qPCR assays (e.g. Focus Diagnostics, CDC kits, etc.). This final interpretation process is complex requiring expertise and rigorous oversight, so is a burden on laboratory staffing. The complex interpretation process can be off-putting to laboratories considering implementing such assays.

Automated solutions for this stage have been attempted, however most have been unsuccessful. For example, some qPCR platforms will set a threshold and then any traces (output curves) that cross this threshold will be deemed positive. However, such systems are in danger of either missing low level positives or can result in false positives, consequently, these automated systems still need to be checked by a user. Furthermore, these systems do not offer whole system interpretation of results (i.e. interpret runs in relation to the control results) and QC (e.g. interpretation of runs in relation to laboratory-based rules such as Westgard).

As a result, most laboratories using qPCR methods and commercial kits still interpret and validate results manually. The methods used by laboratories are likely to differ. In our laboratory this is a multi-stage process that incorporates multiple members of staff, and systems for every run, following specific per-test standard operating procedures (SOPs) - (detailed description in Materials and Methods).

Due to its complex nature, mistakes can be made and there may be a lack of standardisation between users and across labs. It frequently requires the additional input of expert, often senior, staff. Large amounts of paperwork are required to document the processes. In the era of pressures and “doing more with less” anything that simplifies or automates this aspect of qPCR methodology would be welcomed. Clinical qPCR labs still use significant resources to ensure consistent manual analysis and reporting. Processes remain similar to those described some ten years ago by the Mayo Clinic [1]:

Well-written training materials including checklists and detailed SOPs are required for each qPCR test;

Identifying a technical expert to provide one-on-one training for qPCR is critical;

Ensuring proficiency in analysis and use of qPCR instrument software, including for samples with unusual results.

In addition, most tests require review by two (or more) scientists. Regular proficiency testing, including ensuring accurate manual analysis of data, is also necessary.

PCR.Ai (www.pcr.ai ) is a novel system created by diagnostics.ai (59a Brent Street, London, UK; www.diagnostics.ai) that is designed to automatically and rapidly interpret qPCR curves. It is carried out directly on the qPCR platform after the run is complete, using either an on-site or internet server, and interfaces with the Abbott Laboratory Information Management System (LIMS) to ensure results are downloaded. The system is described in further detail below (Study Design).

Objectives

The aim of this prospective clinical study was to compare PCR.Ai to the West of Scotland Specialist Virology Centre (WOSSVC) existing interpretation/verification practice, to determine its accuracy and hands-on time from initial qPCR output to result download. Two in-house-developed, multiplexed, qualitative qPCR tests were used in the study: one for the detection of respiratory pathogens (run on the ABI7500) and the other for detection of norovirus (run on the Rotor-Gene Q). These tests were chosen for this study as they are commonly used, high-throughput assays, which are run on two commonly used qPCR platforms and in one month produced 22,200 test results for comparison purposes.

Study design

Description of the laboratory methods assessed

Data analysis methods

Fig. 1 below compares and contrasts the data analysis methods for WOSSVC and the PCR.Ai system. For each task in the figure it is indicated whether the task was completed by the technician (ie manually) or done by the application (ie automated). Both systems were interfaced with the Telepath LIMS system.

Fig. 1

Comparison of PCR.Ai to WOSSVC current analysis on ABI 7500.

Description of the PCR.Ai analysis process

PCR.Ai is a novel tool that automatically interprets qPCR amplification curves providing rapid and standardised results. PCR.Ai identifies positive and negative amplification curves and highlights ambiguous results requiring further analysis, by analysing each curve shape both individually and in the context of run/assay history. The system automatically validates qPCR runs by assessing positive controls using user-defined rules such as Westgard Medlab QC 3 St Dev rule, interprets the IC values and examines negative controls. It can also verify qPCR runs by assessing quantitation standards and will quantify positive traces by comparing them to standard curves. After analysis is complete PCR.Ai interfaces with the LIMS to enable result download. PCR.Ai was accessed via a physical server connected directly to the qPCR machine (in this case the ABI7500 and Rotor-Gene Q), although the company also offers a cloud-based product, and automatically interpreted data directly on the qPCR platform following run completion. Conveniently, PCR.Ai is able to interpret multiple tests on one 96 well plate.

For interpretation of each assay, PCR.Ai is calibrated to determine positive, negative or indeterminate samples by analysing endpoint dilution series of the targets in the multiplex to be used. It is further calibrated by using a large number of qPCR amplification curves (typically containing at least 200 positive and 200 negative results) from previous runs that have been assessed on the PCR platform to be used. If necessary, this is then further refined by analysing a number of prospective runs, and comparing outcomes to the manual method. On completion of this calibration, PCR.Ai can then be used as often as required for the specific test to provide automated analysis.

Current qPCR interpretation and validation steps for inhouse qPCR methods (ABI7500 and Rotor-Gene Q)

Conventional interpretation of qPCR output using qPCR instrument software (we used both the ABI7500 and Rotor-Gene Q in this study) is a complex manual process. It requires analysis by at least two scientists for each run - a junior for initial interpretation and a senior to analyse ambiguous results and validate the run.

Fig. 1 outlines the multiple manual steps and software packages involved in conventional analysis of qPCR instrument output data for clinical reporting. Assays used

The WOSSVC respiratory assay screens for 16 pathogens plus an internal control in 5 multiplex qPCR assays using the ABI7500 instrument. For each multiplex assay, a one-step qPCR reaction was performed as described previously [2].

Norovirus is screened using a single qPCR multiplex assay [3].

These multiplexes and number of runs/samples for each are described in Table 1, Table 2 .

Table 1

Respiratory Assay.

Test	Target	Target (abbr)	No . runs	No. tests
RUT1	Influenza A	FluA	20	1200
	Parainfluenza 1	PF1	20	1200
	Human metapneumovirus	HUMP	20	1200
RUT2	Parainfluenza 2	PF2	20	1200
	Parainfluenza 3	PF3	20	1200
	Parainfluenza 4	PF4	20	1200
	Internal Control (Equine Arteritis Virus)	IC	20	1200
RUT3	Adenovirus	Adeno	20	1200
	Influenza B	FluB	20	1200
	Respiratory syncytial virus (RSV) A+B	RSV	20	1200
RUT4	Coronaviruses 229E	229E	20	1200
	NL63	NL63	20	1200
	OC43	OC43	20	1200
	Rhinovirus	Rhino	20	1200
RUT5	Mycoplasma pneumoniae	Mpn	20	1200
	Bordetella pertussis IS110	Pertussis	20	1200
	Bordetella pertussis toxin	Pertussis toxin	20	1200
----	Total	-----	340	20,400

Table 2

Norovirus assay.

Test	Target	Target (abbr)	No. runs	No. tests
Noro	Norovirus G1	G1	20	600
	Norovirus G2	G2	20	600
	Internal Control (Equine Arteritis Virus)	IC	20	600

Currently, these assays are performed and manually analysed/interpreted using the qPCR instrument software (and with other QC packages such as MedLab QC).

Comparison of the WoSSVC and PCR.Ai qPCR evaluation methodology

Over a period of 1 month (mid April-mid May 2017), 20 respiratory and 20 norovirus runs were analysed by the WOSSVC biomedical scientists by both the normal manual methods outlined above and by PCR.Ai. For both systems, results were analysed in terms of positive controls passing Westgard (1:3S rules), negative controls showing no contamination, internal control failures and the positive/negative status of each sample. Once this was carried out the results generated by both methods were compared for accuracy. The same biomedical scientist (in total seven to minimise bias) carried out both analyses on each run; carrying out the manual analyses first as these were the results downloaded into LIMS, followed immediately by PCR.Ai. Each result was analysed only once. For the comparison of both methods all the final results were reviewed by one experienced clinical scientist to check the results of the biomedical scientist. Here the manual and PCR.Ai results were analysed on separate days to prevent bias. Where there were discrepant results, a second experienced clinical scientist reanalysed the run and the two clinical scientists decided the final status of these discrepant results and which method had correctly analysed these discrepant results.

Time auditing was carried out by recording the time taken for various stages involved in both manual and PCR.Ai analysis. To obtain this timing data, the following data was recorded by the biomedical scientist/laboratory technician carrying out the test analysis:

System (ABI/Rotor-Gene/PCR.Ai).

Type of Run (Respiratory/Norovirus).

Date.

Time analysis started.

Time analysis finished.

Time controls were assessed by Westgard rules.

Time spent checking results and downloading into LIMS.

Time results were downloaded into LIMS.

Results

Table 3, Table 4 details results of the comparison (described above) between WoSSVC and PCR.Ai qPCR evaluation methodology for the 22,200 amplification curves (20,400 respiratory assay and 1800 norovirus) that were assessed by both methods. The figures refer to the number of positive traces in the ABI7500 or rotorgene analysis with the discrepancy referring to the different number of positives detected by PCR.Ai.

Table 3

Resp Assay results.

Test	Target (abbr)	No. samples	PC that were positive	Neg controls positive	No. positive samples	No. Discrepant	Discrepancy %
RUT1	FluA	1200	20		151		0.00%
	PF1	1200	20		7		0.00%
	HUMP	1200	20		23		0.00%
RUT2	PF2	1200	20		9		0.00%
	PF3	1200	20		7		0.00%
	PF4	1200	20		3		0.00%
	IC	1200			1170	15	1.28%
RUT3	Adeno virus	1200	20		25		0.00%
	FluB	1200	20		49		0.00%
	RSV	1200	20	2	45		0.00%
RUT4	229E	1200	20		5		0.00%
	NL63	1200	20		3		0.00%
	OC43	1200	20		7		0.00%
	Rhino	1200	20		70		0.00%
RUT5	Mpn	1200	20		19		0.00%
	Bordetella pertussis	1200	20		5		0.00%
	Bordetella pertussis toxin	1200	20		2		0.00%
Sub Total			320	2	1600
Total		20400	1922			15

Table 4

Noro assay results.

Test	Target (abbr)	No. samples	PC that were positive	NC that were positive	No positive samples	No. Discrepant	Discrepancy %
Noro	G1	600	20		5	0	0.00%
	G2	600	20	4	83	0	0.00%
	IC	600			520	5	0.96%
Sub Total			40	4	608
Total		1800	652			5

In no cases was the PCR.Ai analysis incorrect. There were a small number (5) of results where the original manual analysis was wrong and was corrected by PCR.Ai. The small number of control failures were correctly identified by both methods.

There were a small number of differences in results when comparing Westgard checks (0.5% for respiratory and norovirus) and internal control failures (5 more for PCR.Ai for norovirus and 15 more for respiratory), which reflects positively for PCR.Ai (see discussion for details).

Over the 20 runs there were time savings of 45 min per run for the respiratory screen and 32 min per run for norovirus (see Table 5, Table 6 ).

Table 5

PCR.Ai Timesaving for Respiratory Runs.

	ABI		PCR.Ai
	Analysis time	Check controls/ Westgard/LIMS entry	Analysis time	Check controls/ Westgard/LIMS entry	Timesaving
Time taken (mins)	10	45	5	5
Total (mins)		55		10	45
Total per week (1 run/day) (mins)		275		50	225
Total per year (mins)		14,300		2,600	11,700
Total per year (hrs)		238		43	195

Table 6

PCR.Ai Timesaving for Norovirus Runs.

	Rotor Gene Q		PCR.Ai
	Analysis time	Check controls/ Westgard/LIMS entry	Analysis time	Check controls/ Westgard/LIMS entry	Timesaving
Time taken (mins)	5	35	3	5
Total (mins)		40		8	32
Total per week (1 run/day) (mins)		200		40	160
Total per year (mins)		10,400		2,080	8,320
Total per year (hrs)		173		35	139

Discussion

qPCR methods in clinical virology are mostly in-house developed. Various adaptations have aided automation, but result interpretation and verification are still largely performed manually. Manual interpretation is also required for most commercial qPCR assays (e.g. Focus Diagnostics, CDC kits, etc.). Consequently, the complex interpretation process can be off-putting to laboratories considering implementing such assays.

qPCR interpretation can take time and requires expert staff, and is therefore a cost burden for the laboratory. This can also impact the test turnaround-time (TRT) and throughput, which in turn can affect the clinical management of a patient. Therefore, any successful developments that simplify/automate this process would be welcome.

PCR.Ai is a new system that aims to automate qPCR interpretation. We compared it to our current manual interpretation process (outlined above) for the interpretation of two typical in-house qPCR assays, carried out on different qPCR platforms. The main findings are discussed below.

Accurate interpretation of samples and controls

Our data shows that PCR.Ai is as accurate as our manual-based system and is unaffected by the qPCR platform used. We assessed PCR.Ai prospectively on a total of 22,200 clinical results over a one-month time period, with 100% concordance between the PCR.Ai and conventional manual interpretation. In general, interpretation of the positive and negative controls was also concordant as was the validation/verification decision/outcome. Slight differences in control interpretation arose, with PCR.Ai being more accurate as it used current data (from the start of the PCR.Ai study) to set Westgard ranges, whereas manual analysis used historical data from the introduction of the control to set ranges.

Slight differences were noted in IC interpretation between PCR.Ai and the manual method, with PCR.Ai detecting slightly more samples as inhibited in comparison with the manual method. It is likely that this is a reflection of the different methods used to assess sample inhibition. The manual method compares the Ct value of the internal control to a retrospective rule based on the first 20 runs with that batch of internal control. In our opinion the PCR.Ai method is more likely to be accurate, as unlike the manual method it automatically calculates the average based on samples in that run, giving a tighter range. In addition PCR.Ai analyses IC trace quality as well as Ct. As a result, PCR.Ai is likely to pick up inhibited samples that could be missed by the rule-based system. As this was a study the results of PCR.Ai were not allowed to influence clinical release of samples results.

Although the study showed similar accuracy between the two methods where the experienced biomedical scientists were performing optimally, in practice, PCR.Ai is likely to be more accurate than lab staff since it will be unaffected by operator fatigue or time pressure (e.g., end of the day or on-call interpretation) and level of operator expertise. Furthermore, it should be noted that the WoSSVC has strict qPCR run interpretation guidelines with in-built double checks that may be over and above that in place in other laboratories (perhaps no double checks etc).

PCR.Ai was also able to interpret large mixed qPCR plates (i.e. plates containing >1 multiplex assay with multiple controls), as encountered when assessing the respiratory qPCR. When questioned, staff found it to be easy to use and required minimal training (data not shown).

PCR.Ai offers more rapid qPCR run interpretation and verification than the current manual interpretation system

Our data also showed the PCR.Ai system to be quicker than the established manual system. For the respiratory qPCR, there was an average time-saving of 45 min per run. A saving of 5 min per run was achieved for the initial interpretation process (i.e. setting the threshold and identification of problem curves). A far larger saving of 40 min per run was achieved for the verification/validation steps (e.g. second check and entering control data into Westgard). This probably reflects the fact that PCR.Ai does this automatically, whereas the manual system is reliant on the second checker reviewing the whole run. During our review, further delays were observed when the second checker was unavailable to carry out checks and Westgard entry (e.g. breaks, other runs to interpret, meetings, end of shift, absence etc). Applying these savings to a whole year of respiratory service would result in a time saving of 160 h per year based on one run per day over a 5 day week. In reality, we tend to run more than one respiratory qPCR per day and offer our service 7 days a week. Similar savings of 32 min per run were achieved for the norovirus qPCR (the lower time savings likely reflect that this is a more straightforward assay to interpret due to it having less targets and associated controls). If we were to apply PCR.Ai to the remainder of our qPCR tests (>20) this would significantly increase the overall savings to the laboratory. We would expect that in laboratories with less qPCR expertise and efficiency, time savings would be greater.

PCR.Ai is straightforward to learn and use. Expertise is only required for the final check of each run (i.e. are ambiguous traces negative, positive or indeterminate). Currently in the WOSSVC we use highly experienced biomedical scientists for the entire process. The ease of use of PCR.Ai may allow us to use less skilled staff to do the initial qPCR process download. We would then only require more skilled staff for the second check. This could lead to further staff savings and would allow us to redistribute skilled staff within the department.

It would also have a clinical impact as tests will have better TRT and a higher throughput, as more samples can be processed per day. This will lead to further cost benefits relating to improved patient care. It should increase the number of tests which can be performed on our out-of-hours service. It should also enable labs that do not have qPCR specialists to run qPCR assays if ambiguous results are repeated (or reviewed by an off-site expert).

Additional performance reviews

In our study, PCR.Ai calibration required the user to send a large database of historical qPCR run data to PCR.Ai for analysis. This includes clinical samples and/or dilution series. PCR.Ai analyses the run data to learn to identify positive, negative and ambiguous samples. Depending on the assay complexity, this can take time (weeks to months). PCR.Ai have already begun to introduce methods that use less clinical data and a small number of dilution series to calibrate the learning machine and we will evaluate the performance of this in due course.

The assays assessed here are qualitative assays. We wish to assess PCR.Ai on quantitative assays to determine how well it works on these more complex assays.

PCR.Ai works with most qPCR machines. It would be useful to see how well PCR.Ai works with other platforms (e.g. Roche LightCycler or Bio-Rad CFX).

Data examined here all were produced by one primer/probe lot. We plan to investigate PCR.Ai’s performance with lot-to-lot reproducibility and this may need to be included in our validation of new primer/probe lots.

Other benefits of PCR.Ai for laboratories

Although not often discussed in the literature, we find that in practice different in-house qPCR tests vary in their ease of interpretation and therefore different tests may be more burdensome for staff than others. This could be due to assays being “messy” and having many ambiguous traces to interpret. Furthermore, in our laboratory, quantitative assays are often considered by staff to be more complex than qualitative assays and take longer to interpret. We have also encountered differences between the software used by different qPCR platforms, which can in turn affect the ease of result and run interpretation. Consequently, we find that some staff attempt to avoid particular assays, whereas others can become stuck on one particular assay as the resident expert. Taken together such issues can result in variable expertise between the “expert” staff members.

Use of PCR.Ai, should standardise the interpretation of all in-house qPCR assays within the laboratory irrespective of test complexity or platform used. Tests will no longer be only under the auspices of particular staff. Furthermore, its ease of use should allow laboratories to use less skilled and experienced staff to interpret such assays. This is beneficial to laboratories already with such assays and could lead to restructuring of staff within a department. It is also beneficial to laboratories without such assays or staff expertise, as now they will be easier to implement and interpret, and lead to better access to qPCR technology (either in-house or commercial kits). If PCR.Ai was to become widely adopted, there would be better service and standardisation by and between labs - improving tests and patient care by region, nationally and internationally.

PCR.Ai offers numerous other benefits to the laboratory. It can link directly with laboratory LIMs, reducing manual steps still further. This should lead to a reduction in transcription errors and improved test TRT. PCR.Ai stores all the qPCR run data on hardware/in the cloud, thus providing long-term records of each qPCR run as well as all related decisions. For example, for each run it is possible to see the user who carried out the run, any decisions relating to samples or controls (and who made these decisions), as well as the outcome. Presently it would be difficult for most laboratories to keep such information. This aspect is excellent for various accreditation systems (e.g. ISO15189 as now mandated by UKAS). Furthermore, as all the PCR.Ai data is stored in a single database, it is simple to automatically produce reports that currently are done manually - for example validation reports for new assays, controls, EQA panels or other qPCR reagents. In future, it could possibly be used to highlight unusual result patterns relating to particular users (which could aid targeted training), contamination or unexpected outbreaks.

Use of PCR.Ai will result in reduced paper use within laboratories. For example, we perform most interpretation using paper. Thus many pages are printed per run, and run data is stored for 12 months until storage is full. Keeping such data in the related PCR.Ai cloud also provides excellent security should qPCR platforms break down, reduces storage requirements, is eco-friendly and reduces costs.

We have determined that PCR.Ai is a highly accurate qPCR interpretation tool. It is a valuable time-saving tool, by reducing the complexity of qPCR interpretation and validation and as such the need for specialists and hands-on time. In practice it should enable us to get results out more quickly, at a lower cost and with improved standardisation.

Author contribution

Dr Alasdair MacLean carried out the majority of the laboratory work.

Dr Rory Gunson initiated the study and was extensively involved in data analysis.

Both authors work for NHS Greater Glasgow and Clyde.

Declaration of Competing Interest

None.