System performance evaluation of the cobas t 711 and cobas t 511 coagulation analyzers in routine laboratory settings.

: Utility of coagulation analyzers in real-world settings depends on characteristics that are often not studied comprehensively. This study aimed to investigate the analytical performance, system functionality, practicability, consistency and throughput of two new automated coagulation analyzers in routine laboratory practice. Real-world settings were simulated in three major European hemostasis laboratories and multiple assays were performed in anonymized plasma samples in parallel with routine clinical practice on the cobas t 711 (high-throughput) and cobas t 511 (mid-throughput) analyzers using activated partial thromboplastin time (aPTT), aPTT Lupus, aPTT Screen, Antithrombin (AT), D-Dimer, Fibrinogen, Prothrombin Time (PT)-derived Fibrinogen, PT Owren, PT Rec (recombinant human thromboplastin reagent) and Thrombin Time assays. Precision was tested in a 21-day experiment and accuracy was compared with reference methods of the same laboratory. A number of experiments simulated challenging real-life situations. Pearson's correlation coefficient was more than 0.98 in all assays. Across assays, coefficients of variation ranged from 0.0 to 1.5% for intermediate precision; 0.2 to 3.0% for repeatability and 0.4 to 3.7% for total precision. Good between-run comparability was seen when testing samples under random conditions. Calculated maximum throughput was 197 and 387-402 tests/h for the cobas t 511 and 711 analyzers, respectively. Practicability met or exceeded user expectations in 98% of cases. In a simulated real-life setting of three major laboratories, the new cobas t 511 and cobas t 711 coagulation analyzers demonstrated a good functionality, practicability and performance and the throughput was high.

Introduction

Coagulopathies comprise a broad group of hemostatic disorders with various causes ranging from congenital to trauma and pharmacotherapy related. Coagulopathies share a common risk of serious and life-threatening episodes of bleeding or thrombosis contributing to significant morbidity and mortality [1-7]. Coagulation tests are used widely in clinical practice for the screening, diagnosis and assessment of coagulopathies, and also for the therapeutic drug monitoring of anticoagulant therapies [8-12]. Key parameters include prothrombin time (PT), activated partial thromboplastin time (aPTT), thrombin time (TT), fibrinogen and d-Dimer [13-23].

Indicators of the hemostatic function need to be accurately, reliably and quickly measured to assist clinical diagnosis and treatment of coagulation abnormalities [24,25]. The availability of new-generation, fully automated, high-throughput coagulation analyzers offers potential benefits toward meeting clinical need and increasing accuracy while reducing errors [26].

The cobas t 711 and cobas t 511 analyzers (Roche Diagnostics GmbH, Switzerland) are new-generation, fully automated coagulation analyzers. Both analyzers utilize reagent cassettes offering automatic reconstitution of lyophilized reagents, which is designed to maximize automation and programmability. The potential advantages include increased precision and accuracy compared with pipetting when manually reconstituting reagents, increased walkaway time for operators and efficiencies in scheduling sample runs [26]. Assays used for routine coagulation testing with the cobas t 711 and 511 analyzers include three different aPTT reagents (aPTT, aPTT Lupus, aPTT Screen), AT, d-Dimer, Fibrinogen, PT-derived Fibrinogen, PT Owren and PT Rec (recombinant human thromboplastin reagent); their analytical performance has been described in detail previously [26-29]. The cobas t 711 and 511 analyzers are built on identical technology and operating methodology, and differ only in capacity and throughput, with the cobas t 711 a high-throughput analyzer capable of running a maximum of 390 tests/h, and the cobas t 511 a mid-throughput analyzer capable of running 195 tests/h. In addition, the cobas t 711 analyzer can be directly connected to laboratory automation and information technology systems, forming part of integrated laboratory solutions.

While the analytical performance of the cobas t 711 and cobas t 511 analyzers and the associated assays have been demonstrated in controlled conditions, the applicability of the system in real-world settings has not yet been described. Determining the practicability, functionality, throughput and consistency of analytic instruments is vital to guide laboratory managers on their use. This multicenter study aimed to evaluate the analytical performance, functionality and reliability of the cobas t 711 and cobas t 511 analyzers under simulated routine-like ‘intended-use’ laboratory conditions.

Materials and methods

Study design

The study was conducted between January 2017 and April 2017 at three European teaching hospitals (Inselspital University Hospital, Bern, Switzerland; Erasmus University Medical Center, Rotterdam, The Netherlands; Sheffield Haemostasis and Thrombosis Centre, UK). Anonymized residual sodium citrate (3.2%/0.109 mol/l) plasma samples were evaluated on the cobas t 711 (high-throughput; max. 390 tests/h: all three centers) and cobas t 511 (mid-throughput; max. 195 tests/h: UK center only) analyzers, using a number of coagulation assays (aPTT, aPTT Lupus, aPTT Screen, AT, d-Dimer, Fibrinogen, PT-derived Fibrinogen, PT Owren, PT Rec and TT; Roche Diagnostics, Switzerland) described in detail previously [27,29]. All assays and analyzers were used according to manufacturers’ instructions.

Independent ethics committee approval was obtained before study initiation and the study was performed according to the principles of the Declaration of Helsinki and International Council for Harmonisation Good Clinical Practice guidelines.

Reproducibility and quality control

A 5-day interlaboratory reduced scope reproducibility survey was conducted using control material to confirm comparable recovery levels between the three sites. Quality control was performed daily throughout the study prior to running experimental samples to ensure completeness, correctness, plausibility and validation of readings. Dependent on the assay, two to three quality control levels were used with an acceptability cutoff of ±2 SDs of defined analyte recovery target range, as per routine laboratory practice.

Intermediate precision

Intermediate precision – within-laboratory, day-to-day precision – was assessed as analytic performance over 21 days according to Clinical Laboratory Standards Institute (CLSI) EP05-A3 guidelines. Quality control target value recovery was assessed in selected control material. Two runs daily of each quality control material were conducted, with at least two quality control samples per test using the coagulation assays described above (84 measurements per material-test combination). Over the 21 days, each control sample measurement was performed with all parameters from the same sample, to better simulate routine conditions. Acceptance criteria included system behavior as expected (i.e. processing, loading and unloading without problems); sample handling as expected (i.e. loading and unloading of sample racks, handling of sample containers, all results available on graphical user interface and uploaded to WinCAEv); and coefficient of variances according to industry-standard test-specific criteria for intermediate precision (Table S1; Supplementary Materials).

System precision

Precision – agreement between-run and within-laboratory series – was also assessed in several runs of a routine simulation precision (RSP) experiment that tested for systematic or random errors that may occur during ‘real-life’ routine use by comparing recovery and imprecision during randomized processing. This test was designed to partially reflect sampling and testing sequences in each laboratory, with that produced during batch analysis [30]. Reference runs using all 10 assays and the same request panel from each sample (15 measurements) were followed by a random part where the same samples were tested, but both the sample order and the test requests were randomized and spiked with provocations (e.g. insufficient sample or reagent) to challenge the functionality of the analyzer (7–47 measurements depending on test, to represent routine testing patterns). Acceptability criteria included no random or systematic errors detected; reproducibility during random part comparable with reference part; and system handling provoked variability being within specification.

System consistency

Consistency – agreement in changing and challenging situations – was assessed over several runs of a routine simulation series (RSS) experiment that confirmed the absence of random errors when running each analyzer under simulated routine conditions [31]. The experiment consisted of the routine simulation download (RSD) test described below, followed by immediately running the same samples in the same sequence representing various sample concentrations over the analytical range. Results ±5% were considered acceptable with no further analysis required, while findings more than ±5% required careful assessment to identify route of deviation.

System throughput

Throughput was assessed in the RSD experiment that replicated routine laboratory workflows and determined the throughput of machines under ‘real-world’ usage. This experiment evaluated the same test request pattern and workload under seven different scenarios. The detailed scenarios and outcomes for the RSD are shown in Table S2 (Supplementary Materials). Briefly scenarios A1–3 and B determined the time required for all sample measurements performed on the cobas t 711 or cobas t 511 analyzers, according to various special conditions (e.g. quality control status time out on every application, optimization of rack release times etc.); scenario C added automatic hemolysis, icterus, lipemia (HIL) testing; scenario D included provocations (e.g. foamy, clotted or closed sample tubes with pressure errors) performed to enable reporting on machine performance under stress; and scenario E determined walkaway time (i.e. time the analyzer can run unattended) with the maximum number of samples.

System practicability

Practicability and usability were assessed via a user questionnaire. Operators were asked to rate both the cobas t 711 analyzer and their routine laboratory analyzer (score 1–10) for the following domains: general, general aspects of software, processing of samples, test reagents, calibration, quality control and maintenance. The features of each domain are shown in Table S3 (Supplementary Materials).

Data analysis

All assay output was directly captured, statistically analyzed and archived by WinCAEv, a Code of Federal Regulations Title 21 Part 11-compliant electronic data capture and statistical analysis software developed and validated for Roche-sponsored studies [30]. Where possible, discordant values and unexpected hardware or software behavior were assessed to determine if samples or experiments should be rerun. Outliers that could not be omitted due to clearly identified and documented errors (e.g. transcription, calculation errors etc.) were included in statistical analyses. System-related outliers were always included in the statistical analysis set, with elucidating information on the root cause added.

The coefficient of variance was calculated for repeatability and intermediate/total precision and compared against prespecified acceptance ranges (Table S1; Supplementary Materials). For AT only, SD was calculated at less than 80% activity as it was defined at a level that was still accurate at the medical decision point of 70–80%, where coefficient of variance would be too great. Coefficient of variances and analyte recoveries were calculated for RSP experiments, with an acceptance criterion of CVrandom ≤ 1.5 × CVbatch. Potential system malfunction was defined by an analyte recovery per aliquot during random part more than 10% of the batch mean. In RSD experiments, measured average throughput, calculated maximum throughput and median sample processing time were calculated. For RSS, slope and intercept were calculated according to Passing–Bablok regression analyses and Pearson's r correlation coefficient was estimated; proportion of recoveries in the following ranges were reported: 5, +5–10, −5–10, +10–15, −10–15%, more than +15%, more than −15%. Measures of practicability derived from operator questionnaire results were expressed as average score weighted based on responder-reported importance (weighted averages).

Results

Analytical performance of the cobas t 711 and cobas t 511 analyzers for the 10 coagulation assays studied in the 21-day precision analysis are presented in Table 1. All values were within the prespecified acceptance criteria (Table 1). Across assays, coefficient of variances for intermediate precision ranged from 0.0 to 1.5% on the cobas t 711 analyzer, and from 0.0 to 1.3% on the cobas t 511 analyzer; coefficient of variances for repeatability ranged from 0.2 to 3.0% on the cobas t 711 analyzer, and from 0.3 to 2.6% on the cobas t 511 analyzer; and coefficient of variances for total precision ranged from 0.4 to 3.6% on the cobas t 711 analyzer, and from 0.4 to 3.7% on the cobas t 511 analyzer.

Table 1

Analytical performance of the cobas t 711 and cobas t 511 analyzers for each coagulation assay (21-day precision)

Up to six RSP runs were performed in each laboratory in the RSP experiment: site 1 cobas t 711 analyzer, 3 runs/∼1500 samples/∼5000 measurements; site 2 cobas t 711 analyzer, 3 runs/∼700 samples/∼3500 measurements; site 3 cobas t 511 analyzer, 2 runs/∼330 samples/∼1500 measurements; site 3 cobas t 711 analyzer, 4 runs/∼1500 samples/∼5000 measurements.

Across centers, imprecision was less than 1% for the majority of assays, with all batch and random part results within the allowed ranges (within-run precision and intermediate precision limits, respectively). Overall performance for random part coefficient of variances never exceeded the intermediate precision specifications (Fig. 1). Minor variation between sites, with more frequently seen imprecision in site 3, was attributed by investigator to the use of older samples (up to 48 h). Where CVrandom exceeded CVbatch by greater than 1.5 times, coefficient of variances were always within the predefined specification for intermediate precision. All recovery levels during the random part of the analysis (simulating systematic or random errors occurring during routine use) were within a ±10% range (Fig. 2 a–d).

Fig. 1

Precision of assays run on cobas t 711 and cobas t 511 analyzers, presented by study site and analyzer. Variation in measurements for the 10 assays observed during the routine simulation precision experiment, when sample order and test requests were randomized and spiked with provocations to simulate random or systematic errors. CV, coefficient of variation.

Fig. 2

Analytical recovery of assays run on cobas t 711 and cobas t 511 analyzers at: (a) site 1 (cobas t 711), (b) site 2 (cobas t 711), (c) site 3 (cobas t 711) and (d) site 3 (cobas t 511). Representative data from a single run showing mean analytical recovery for the 10 assays observed during the routine simulation precision experiment. Relative recovery of each assay in the batch mode (same test request pattern of each sample, defined order of samples) is shown to the left of the black dividing line, relative recovery is shown to the right of the dividing line, when sample order and test requests were randomized and spiked with provocations to highlight any possible random or systematic errors. aPTT, activated partial thromboplastin time; AT, antithrombin; d-DI, d-dimer; FIBR, fibrinogen; HSP, human sample pool; PT, prothrombin time; Rec, recombinant human thromboplastin reagent.

In ‘routine-like’ conditions, the measured average throughput for the cobas t 511 analyzer was 124 tests/h, and for the cobas t 711 analyzer, 303–340 tests/h. The calculated maximum throughput was 197 tests/h for the cobas t 511 analyzer compared with 387–402 tests/h for the cobas t 711 analyzer.

In the RSS experiment, good between-run comparability was seen when testing single samples under random mode conditions (N = 1370 samples: site 1 cobas t 711 analyzer, n = 650; site 2 cobas t 711 analyzer, n = 280; site 3 cobas t 511 analyzer, n = 240; site 3 cobas t 711 analyzer, n = 200). Results from the Passing–Bablok regression analyses are shown in Table 2, with site-specific correlation graphs in Figs. S1–S4 (Supplementary Materials).

Table 2

Comparison of technical accuracy assessed by correlation and regression coefficients for samples run on the cobas t 711 (three study sites) and cobas t 511 (one study site) analyzers for each assay

The vast majority (95%) of the 3180 measurements conducted across the three sites met the acceptance criteria of ±5% of expected result. The distribution of scores was narrow in this analysis, with 3.2 and 1.1% of samples falling into less than –5% and more than 5% categories, respectively, and less than 1% of samples falling into other categories. Investigator-reported explanations for results in the more than ±5% categories included age of samples (>24–48 h) and precision of assays. No system malfunctions were observed.

In the RSD experiment, each laboratory used their own daily routine's workload from a typical day and tested the same test request pattern and workload under different conditions for each scenario. The results were broadly comparable between centers (Table S4; Supplementary Materials). Of note, the addition of HIL testing in scenario C impacted both throughput and the time the racks spent on the instrument (from around 10 min without HIL vs. up to 38 min with HIL). Similar outcomes were observed in scenario B when replacing PT Rec with PT Owren (around 10 vs. 38 min). The maximum walkaway time measured with continuous-feed rack on the cobas t 711 analyzer was 4 h 42 min compared with 1 h 1 min on the cobas t 511 analyzer with manual rack loading. The measured throughput of the cobas t 711 instruments varied between 90 and 273 tests/h, with a calculated throughput of around 200–400 tests/h. For the cobas t 511 instrument the measured throughput was 120–150 tests/h and the calculated throughput was 168–189 tests/h. All provocations were handled by the operators without difficulty and did not cause disruption to the routine. All the erroneous samples were correctly identified via alarm messages.

Weighted average ratings by domain from the operator-completed practicability questionnaire are presented in Fig. 3. Based on the responses, 98% (898/915) of all questions were graded as met or exceeded expectation. Among areas where operators were less satisfied, these could be explained by the age of samples used in the study (e.g. >24 h causing closed sample tube pipetting issues), which would not be seen in standard laboratory practice, or easily addressable through planned software updates (e.g. prominence of alarms and notifications).

Fig. 2 (Continued)

Analytical recovery of assays run on cobas t 711 and cobas t 511 analyzers at: (a) site 1 (cobas t 711), (b) site 2 (cobas t 711), (c) site 3 (cobas t 711) and (d) site 3 (cobas t 511). Representative data from a single run showing mean analytical recovery for the 10 assays observed during the routine simulation precision experiment. Relative recovery of each assay in the batch mode (same test request pattern of each sample, defined order of samples) is shown to the left of the black dividing line, relative recovery is shown to the right of the dividing line, when sample order and test requests were randomized and spiked with provocations to highlight any possible random or systematic errors. aPTT, activated partial thromboplastin time; AT, antithrombin; d-DI, d-dimer; FIBR, fibrinogen; HSP, human sample pool; PT, prothrombin time; Rec, recombinant human thromboplastin reagent.

Practicability of the cobas t 711 analyzer vs. a routine analyzer. Weighted average scores for the seven domains rated by operators on a scale of 1–10, where average responses were weighted against responder-reported importance of each domain. Scores of 0.00 represent not applicable responses.

Discussion

The current study assessed analytical performance, functionality and reliability of the cobas t 711 and cobas t 511 coagulation analyzers under simulated routine laboratory conditions. Observed analytical performance was excellent, with coefficient of variances for repeatability of 3.0% or less and coefficient of variances for intermediate precision and total precision of 1.5 and 3.7% or less, respectively. Despite their increasing use, sigma scores were not calculated in the current study. Instead, we used reference specifications from the cobas t 711 assays’ multicenter evaluation studies, where precision and accuracy assessments follow CLSI guidelines, and do not prescribe the use of sigma metrics. Furthermore, recent evidence questions the utility of the six-sigma method for assessing hemostatic assay performance [32]. For this reason we followed statistical methods outlined in the respective CLSI guidelines and have presented descriptive parameters of variation. The acceptance criterion used in this analysis was an upper percentage limit, and observed values were consistent with investigator expectations and data from similar analyzers. No systematic or random errors were detected in the system precision (RSP experiment) analysis. Imprecision was less than 1% for the majority of assays used and recovery levels were within acceptance criteria. In the RSD test, both analyzers demonstrated correct system functionality and an acceptable number of hardware issues. There was good comparability between runs using single samples under random mode conditions in the RSS test, with near-perfect correlation for all comparisons and few deviations which were attributable to precision of assays. Finally, system practicability determined by user survey met or exceeded operator expectations. Previous studies have described the excellent analytical precision of assays used on the cobas t 711 and cobas t 511 analyzers compared with existing commercially available methods [26,27,29]. Our results complement and build on these previous data by further demonstrating consistency in analytical performance with the two analyzers and user satisfaction.

Our results demonstrated equivalence between the cobas t 711 and cobas t 511 platforms. This was to be expected, as both systems are built from functionally identical components and implement identical assay processes using the same reagents and disposables. The platforms differ only with respect to throughput. The cobas t 711 – a high-throughput analyzer capable of running a maximum of 390 tests/h according to manufacturer specifications – had a calculated maximum throughput for the equivalent scenario in our study (PT/aPTT only) of 396 tests/h at one site, and an average throughput of 303–340 tests/h in routine-like conditions across assays and sites. In comparison, the cobas t 511 – a mid-throughput analyzer capable of running a maximum 195 tests/h – had an average throughput of 124 tests/h in routine-like conditions. These results compare favorably with data describing other high-throughput automated analyzers [33-35], and offer potential benefits in terms of increased efficiency and capacity in laboratories.

Of particular note in this study, the high reagent loading capacity with the cobas t 711 analyzer resulting in less frequent reagent top-up required, coupled with the automatic reagent reconstitution, means a reduced technician daily workload and increased laboratory efficiency. In addition, the potential walkaway time of several hours seen with the cobas t 711 analyzer will be of benefit for all laboratories with regards to resource management and increasing efficiency.

The current analyses were conducted in a simulated real-world, routine setting to test instrument performance under stress, beyond the clinical conditions for precision testing in CLSI guidelines. In addition, the sites selected for this study represent high-workload core laboratories for large teaching hospitals. These settings are high case-load with broad patient populations across emergency medicine and chronic care settings, offering good external validity to these data in the real world. One limitation of this study that should be considered when interpreting the results is the lack of comprehensive standardized method comparison with another device. As this study was only conducted to determine system behavior of the two analyzers in a real-world setting, no conclusions can be made regarding superiority or inferiority to other available laboratory methods; further studies may be warranted in this instance. In addition, while not a limitation of this study design, current data describe only the analytic performance of the analyzers and not their clinical performance (e.g. ability to detect coagulation factor abnormalities).

Conclusion

In conclusion, this multicenter study confirmed the analytical performance, functionality and reliability of the cobas t 711 and cobas t 511 analyzers when used in simulated routine laboratory conditions. Both analyzers are suitable for the accurate and reliable measurement of coagulation parameters in routine clinical practice and offer high-workload core laboratories options and advantages over existing methodologies.

Acknowledgements

The authors thank Angela Catanese, Robbie de Jong, Coby Wegman and Debby Priem-Visser (Erasmus MC, University Medical Center Rotterdam, The Netherlands) and all technicians involved from the Sheffield Haemostasis and Thrombosis Centre and Inselspital University Hospital laboratory for technical support. This study was sponsored by Roche Diagnostics International Ltd (Rotkreuz, Switzerland). Third-party medical writing and editorial support, under the direction of the authors, was provided by Chloe Fletcher, MSc (Gardiner-Caldwell Communications, Macclesfield, UK), and Jesse Quigley Jones [independent on behalf of MIMS (Hong Kong) Ltd] and funded by Roche Diagnostics International Ltd (Rotkreuz, Switzerland). COBAS, COBAS T and ELECSYS are trademarks of Roche.

J.B., K.G., G.R.: Study design and concept development, analysis and interpretation of data. G.R.: Authorship of protocol and study report. S.K., M.d.M., M.N.: Collection and interpretation of data. All authors drafted the article and approved the final version for submission.

Conflicts of interest

S.K. has received travel support or speaker fees from Roche Diagnostics, Sysmex, Siemens, Werfen and Diagnostic Grifols. M.N. has received research support from Roche Diagnostics, Stago, Technoclone, Pentapharm and Siemens Healthcare. M.d.M. has received travel support or speaker fees from Roche Diagnostics, Sysmex, Siemens and Werfen. K.G., J.B. and G.R. are employees of Roche Diagnostics.