Concerns with conclusions in the article by Sherwood et al 'Key differences between 13 KRAS mutation detection technologies and their relevance for clinical practice'.

Dear Editor,

We wish to raise some questions and concerns about the design, execution and evaluation of results in Sherwood et al.1

Concerning the study design, it is not clear that the same amount of sample material was analysed by all of the technologies tested. For example, while it appears that most platforms processed 1 μL of each sample in table 2 (ie, 50 or 100 mutant copies per test reaction), it seems that for Oncomine duplicate libraries were made and their results averaged in table 3, potentially enhancing the assessed sensitivity of this platform. Was data for all other platforms derived from a single experimental run of the sample test plate? For droplet digital PCR (ddPCR) tests, was 9 μL of each of these undiluted samples used per reaction (as written)? That would suggest that 9×50 and 9×100 mutant copies were not detectable by the QX200 platform, whereas it is documented that as few as a couple of mutant copies can be detected among as many as 100 000 wild-type copies (eg, refs 2–4). The Methods section would benefit from greater clarity.

Data analysis and the authors’ conclusions raise a number of questions. It is surprising that several of the best performing platforms (eg, Idylla and Oncomine) appeared to perform ~10X more sensitively than stated by the manufacturers (compare tables 3 and 4), whereas the ddPCR platform, widely recognised to be among the most sensitive and specific (reviewed in5), grossly underperformed (by several orders of magnitude). These discrepancies were not addressed.

For those technologies performing best in this study (ie, Idylla, Oncomine and UltraSEEK), could these outcomes have been influenced by relying on the vendors themselves to have run and analysed their own systems? In contrast, there is no discussion of the data quality behind the apparently flawed ddPCR testing run by a party different from the manufacturer. The fact that the QX200 results were only reported qualitatively (table 3) is very surprising as the inherent benefit of digital PCR is to provide absolute quantitation and this should raise a red flag. Were appropriate ‘no template controls’ run to assess possible contamination in this run? Were recommended positive and negative controls present in the run to permit proper manual thresholding of droplet clusters and data analysis? (Or was only autothresholding relied on?) These data raise clear concerns and need to be reconciled with the extensive peer-reviewed literature demonstrating markedly different results with the QX200 ddPCR technology, including a recent international interlab study across 21 European and North American labs showing reproducible KRAS mutant detection down to 0.17% MAF.6 Additional analytical validation data for all five KRAS PrimePCR assays used in the Sherwood et al study can be found in the URLs listed in the legend to figure 1. This figure also illustrates high specificity of the three KRAS G12 assays (C, D & V) in contrast to the authors’ findings.

Figure 1

Specificity of PrimePCR KRAS G12C, G12D and G12V Rare Mutation Detection Assays. The mutant allele is present at ~0.5% in ~40,000 genome copies/well. Additional analytical validation data showing detection limits <0.1%mutant fraction for all five assays reported in Sherwood et al (2017) are available in the ’Validation Data' tabs at: G12C c.34G>T: http://www.bio-rad.com/en-us/prime-pcr-assays/assay/dhsacp2500584-primepcr-ddpcr-mutation-assay-kras-p-g12c-human; G12V c.35G>T: http://www.bio-rad.com/en-us/prime-pcr-assays/assay/dhsacp2500592-primepcr-ddpcr-mutation-assay-kras-p-g12v-human; G12D c.35G>A: http://www.bio-rad.com/en-us/prime-pcr-assays/assay/dhsacp2500596-primepcr-ddpcr-mutation-assay-kras-p-g12d-human; G13D c.38G>A: http://www.bio-rad.com/en-us/prime-pcr-assays/assay/dhsacp2500598-primepcr-ddpcr-mutation-assay-kras-p-g13d-human; Q61H c.183A>T: http://www.bio-rad.com/en-us/prime-pcr-assays/assay/dhsacp2000131-primepcr-ddpcr-mutation-assay-kras-p-q61h-human.

We look forward to the authors’ clarification of these issues.

Sincerely,

George Karlin-Neumann, PhD

Director of Scientific Affairs

The Digital Biology Center, Bio-Rad