Comparing two assays for clinical genomic profiling: the devil is in the data.

Dear editor

We read with concern the paper “Evaluation and comparison of two commercially available targeted next-generation sequencing platforms to assist oncology decision-making”.1 The study directly compared results for the Paradigm Cancer Diagnostic test to the FoundationOne test for formalin-fixed, paraffin-embedded specimen pairs from 21 advanced cancer cases. We believe this study is fundamentally flawed, misleading, and potentially dangerous for patient care, for the reasons outlined herein.

The paper neglected to address innumerable discordances between a rigorously analytically validated test (FoundationOne) and the experimental assay. It erroneously ascribes categorization of many genomic alterations detected on FoundationOne as “none”, when in fact available drugs have demonstrated activity or mechanism-based clinical trials exist, and it claims high levels of actionability based on the results of RNA-expression profiling of a single gene – TOPO2A.

This study is notable for a remarkable lack of concordance between the genomic alterations detected on each platform, even in genes common to both assays. For the majority of specimens, there were no genomic alterations in common, and only six of 21 samples shared a single alteration, matching at the gene level. Paradigm DNA-based testing found only two cases wherein a result was labeled by them as “commercially available” or “clinical trial”, an EGFR mutation in a lung adenocarcinoma, and a KIT mutation in a colon adenocarcinoma. This KIT mutation was not detected by FoundationOne. Similarly, lack of concordance is noted for mutations in ARID1A (case 1), PDGFRA and PI3K (case 2), KRAS, PI3K and PTEN (case 5), PTEN and ARID1A (case 6), EGFR amplification (case 9), ERBB2 (cases 12, 14), and PI3K (case 19). Given the significant discordance in the genomic alterations detected and reported, it must reasonably be concluded that at least one assay has extremely poor mutation-detection performance. FoundationOne underwent an extensive 2-year analytic validation study. This study rigorously demonstrated the test’s high-performance characteristics.2 The Paradigm assay has no published analytic validation studies. The lack of concordance between the two platforms calls into serious question the performance of the Paradigm assay and the validity of the data.

Clinical activity and/or mechanism-directed trials exist for genomic (DNA-based) alterations in PTEN,3 RICTOR,4 CDK6 (NCT02187783), FGFR4 (NCT02325739), ERRFI1,5 FBXW7,6 and PI3KR1 (NCT01971515). The “actionable target” determination, as displayed in Table 2, also reveals a failure to acknowledge KRAS as a gene for which therapeutic approaches exist in clinical trials, and for which prior genomically driven trials have shown selected activity.7 Although an acknowledged therapeutic challenge, KRAS-mutated tumors have been the object of no fewer than 26 clinical trials with targeted agents requiring a KRAS mutation as molecular eligibility entry criteria (www.ClinicalTrials.gov). Despite this, all of these genes are scored as “none” as shown in Table 2, column two.1

The authors report RNA alterations linked to potential therapeutic intervention in 16 of 21 cases: eleven with TOPO2A overexpression alone, four with ERBB2 over-expression alone, and one case with both; however, in only two of five cases was ERBB2 amplification detected by FoundationOne, an assay that has undergone rigorous concordance testing with the gold-standard assays of fluorescence in situ hybridization and immunohistochemistry. TOPO2A overexpression was present in six of six lung adenocarcinomas. Unfortunately, this result is not surprising or helpful, as anthracyclines have no reproducible activity in NSCLC, ie, the results provide no useful clinical information. Furthermore, it is well known that TOPO2A expression is regulated by the cell cycle8 and is independent of TOPO2A gene amplification.9 It is TOPO2A gene amplification, not TOPO2A expression, that has been linked in some but not all studies as a biomarker of anthracycline efficacy in breast cancer.10

In order to permit a fully informed comparison between commercial cancer genome-profiling assays, we believe it necessary for all parties to make public most aspects of their test methodology and data. In this case, we believe the authors need to publish all aspects of the testing methodology for the Paradigm Cancer Diagnostic test. For example, which exons of which genes are targeted in this test? Is the test achieving high, even coverage of target regions? Are reported sequence coverage metrics from unique input DNA molecules and not PCR duplicates? Are appropriate computational methods employed? None of these critical questions can be evaluated with the information provided.

Oncologists increasingly rely upon results of comprehensive genomic profiling in making important treatment decisions. For this reason, rigorous peer-reviewed validation of genomic tests offered for use in clinical applications is essential. We believe the Paradigm study does not meet this standard and is sufficiently flawed such that we encourage the authors to consider retracting the paper.

We appreciate the opportunity to respond to the letter regarding our evaluation and comparison of targeted next-generation sequencing (NGS) platforms.1

The Paradigm Cancer Diagnostics (PCDx) test is a high-quality clinical grade NGS test with impeccable regulatory compliance and rigorous validation, and is run in a Clinical Laboratory Improvement Amendments (CLIA)-certified laboratory. The test was thoroughly validated against multiple orthogonal platforms, and includes a large number of clinically relevant actionable markers associated with therapies based on published patient research information and also integrates additional clinical trial information related to the biomarkers identified. The depth of coverage for DNA copy-number variation and mutation is sequenced to a mean depth of 56,085× and 13,656×, respectively, and RNA to a mean depth of 21,562×. This mutation depth compares to a previously reported DNA mean depth of 250× and a median of 500× on Foundation Medicine’s FoundationOne (F1) test.

First, we would like to recap that the purpose of the study was to perform a pragmatic real-world clinical comparative assessment of two NGS-based solid tumor-profiling offerings: the F1 and PCDx tests. The purpose was not to perform an assessment of laboratory or informatics methods, nor was this a review article primarily focused on prior publications.

Paradigm Diagnostics will continue to publish additional academic validation data in the future that will be presented in a similar manner as Foundation Medicine has done with their NGS platform following their CLIA launch. Furthermore, since the comparison between PCDx and F1 was done during the first months of PCDx’s CLIA launch, the turnaround time (TAT) for PCDx has since been reduced by about half, to 4–5 business days, and the panel has been broadened to also include immunohistochemistry (IHC) when clinically appropriate.

As stated, the purpose of the analysis was to perform a simple and real-world clinical comparison of two commercially available NGS-based solid tumor-profiling assays. The two main parameters assessed were 1) TAT and 2) clinically actionable biomarkers. The TAT required for obtaining test results for patients suffering from aggressive cancers is of significant importance to the treating clinician. The steps associated with obtaining the block from pathology and the associated analysis can take a patient out several weeks to months prior to enacting a new therapy. The ability of a service provider to deliver high-quality results that fit well into the office workflow, reduce a patient’s time to new therapy, and enable timely scheduling is of significant clinical importance. We and others have published data that show that delays in providing a late-stage patient NGS-associated drug information can result in up to 25% of the patients either not obtaining clinical benefit2 or deteriorating during the interval awaiting for results to the extent where additional systemic therapy is no longer inappropriate.3

Furthermore, while there may be disagreement on the definition of “clinically actionable biomarkers” as we have defined in our paper, we (GJW, DML, and VK) would contend that in our experience as both oncologists and clinical trialists, we can comfortably present and defend our previously published definition. We (GJW, RJP, DWM, SMM, EJT, DML, and VK) do understand and appreciate the importance of the “n of 1” experience. Unfortunately, the broad claims by F1 on a clinical report to community physicians that a particular low-frequency genomic alteration has actionability based on cell-line mechanistic data, meeting abstracts, germ-line tumor data, or a single case report currently does not rise to the level required of comfort. Furthermore, this practice, in and of itself, may be perceived as “experimental”, since physicians are not fully informed of the clinical impact of many variants of unknown significance (VUS) or whether an alteration is germ-line or somatic. This was further elaborated on in our discussion, and the low genomic confidence among adult oncologists was also highlighted.4 On its website, Foundation Medicine posts that VUS “should not be used for treatment decisions because the overwhelming majority will prove to be non-pathogenic and could indeed expose the patient to costly and ineffective therapies”.5 We agree with this statement, and find it concerning that in several instances, which we point out in the following section, the F1 report cited a VUS as a genomic alteration identified and was matched with a suggestion for a targeted therapy.

To address some of the specifics noted:

PTEN mutation (L25fs*1): The data cited by F1 for the clinical actionability of this genomic alteration is from a single case study investigating a germ-line pediatric tumor.6 We believe that making the leap to suggest this alteration is a putative targetable somatic alteration for adult tumors, in a different cancer type (colorectal cancer), would require additional clinical support. In light of this information, we would change the designation for F1 for case 5 from “clinical trial” (CT) to “none”. This does not alter the highest category assigned to case 5 nor alter the conclusion.

RICTOR amplification: The data cited by F1 for the clinical actionability of this genomic alteration is based on a single lung cancer patient treated with an investigational drug in a case series reporting the prevalence of this amplification in a conference abstract.7 We do not believe this alteration has the peer-reviewed data supporting broad clinical actionability.

CDK6 amplification: This genomic alteration does associate with an available clinical trial, and we do agree that it should be reclassified as CT. However, for the purposes of this analysis, this change does not impact the final results, as case 12, which harbored this genomic alteration, had already been assigned a classification of “commercially available drug” as per other alterations present. Therefore, the conclusions of our analysis are not altered by this observation.

FGFR amplification: This genomic alteration in case 3 is associated with a clinical trial that was initiated after this analysis was performed. Again, this reclassification would not alter the conclusion of our analysis.

ERRFI1 R245fs*: The data cited by F1 for this genomic alteration are a description of a single patient experience with no associated clinical trial.8 Once again, we do not believe the data suggest or support classification as a putative clinically actionable target.

FBXW7 R465C: The data cited for this alteration in case 20 are in a single case-report study with no associated clinical trial. We do not believe the data suggest or support classification as a putative clinically actionable target.

PIK3R1 E451_Y4: The clinical trial cited for this target in case 20 does not directly mention PIK3R1 mutations as being analyzed for therapeutic response or as an inclusion criteria (NCT01971515). Therefore, we would not reclassify this observation.

KRAS mutations: As oncologists and clinical trialists (GJW, DML, and VK), we find it misleading to assert that a KRAS mutation is an actionable target with a single-agent MEK inhibitor (on or off protocol). F1’s assay reports list KRAS in a table with the heading of therapeutic implications, and include trametinib, a MEK inhibitor, as a US Food and Drug Administration-approved therapy (in another tumor type). Additional text states that KRAS mutations may be sensitive to inhibitors of the Raf/MEK/ERK pathway. To our knowledge, there are no compelling published clinical data demonstrating single-agent MEK inhibitors as a reasonable choice of therapy and outperforming existing cytotoxic therapies in cancers with KRAS mutations, particularly in the cancers in our analysis identified to possess a KRAS mutation (colorectal, lung adenocarcinoma, and small-intestine adenocarcinoma). Blocking one downstream target of KRAS will be unlikely be sufficient and may require multiple targeted therapies, and these are currently being explored in numerous trials but over the years have remained largely unsuccessful.9,10

We acknowledge that the F1 report makes recommendations for combination-therapy clinical trials for KRAS. Unfortunately, the one F1 report (case 5) that listed trials with combination cytotoxic and targeted therapy for a KRAS mutation was in the setting where the cytotoxic chemotherapy offered would be part of standard-of-care therapy for colorectal cancer. This patient would not yet have undergone NGS based on the parameters for NGS testing discussed in the “Methods” section of our paper to have allowed enrollment on such a trial.

We believe that alternate approaches, such as cytotoxic therapy or even immunotherapy,11 on clinical trials may be more reasonable therapeutic options to consider for KRAS-mutant cancer than combination targeted therapy and certainly single-agent MEK inhibition. Since KRAS mutation does not identify patients that may benefit from these alternate approaches and is for the most part the dominant driver trumping other targeted therapy approaches, we felt that it would be fair to assign “none” to all cases with KRAS alteration identified.

Herein, we address the letter’s questioning about ranking and discordances on a case-by-case basis:

Case 1’s ARID1A mutation: As a targeted assay, the PCDx does not currently analyze for ARID1A mutations. The clinical utility of this alteration points to a singular clinical trial that is suspended and for a tumor type different from that of the patient (NCT02059265).

Case 2’s PDGFRA A210V mutation: The clinical utility of this genomic alteration is not clear, as this is a VUS. There was a PIK3CA mutation detected by the F1 assay and not picked up by the PCDx assay that was noted in the PCDx clinical report due to low gene coverage for the sample analyzed. Paradigm Diagnostics has since redesigned the portion of the assay covering this region for more consistent coverage. However, the result was nonetheless accurate, as it stated low coverage for this gene.

Case 5: The KRAS mutation in exon 4 was not tested by PCDx during the time of this analysis, and is discussed on page 965. This alteration is currently available on the PCDx platform. There was a PIK3CA mutation detected by the F1 assay that was not picked up by the PCDx assay. As stated previously, Paradigm Diagnostics accurately reported this target in the clinical report as low coverage, and has redesigned the assay since. The PTEN mutation detected by the F1 assay at the time of the analysis would be considered a VUS, as the data supporting this is in a single case report.6 This alteration is currently not part of the PCDx commercial platform.

Case 6: The PTEN mutation reported by F1 is based on a single case report.6 We do not believe the data suggest or support classification as a putative clinically actionable target. As noted earlier, the PIK3R1 mutation that was identified points to a clinical trial that is not clearly investigating a therapeutic response for this particular alteration. This mutation is not currently analyzed by the PCDx test. As noted earlier, the ARID1A alteration is not analyzed by the PCDx assay, as this alteration is of unknown significance.

Case 8: We believe the letter is mistakenly referring to an EGFR amplification in case 9 (not case 8).

Case 9’s EGFR amplification: This was not reported by the PCDx assay. For this case, the PCDx report stated that no results for copy number were available because the tissue received by Paradigm Diagnostics was highly degraded.

Case 12: We presume the letter meant to state an ERBB2 mutation rather than a gene amplification as reported in Table 2 on page 962. The ERBB2 mutation was not tested by PCDx during the time of this analysis, and is discussed on pages 965–966.1 This alteration is currently available on the PCDx platform.

Case 14’s ERBB2 mutation: As noted earlier, ERBB2-mutation analysis was not part of the PCDx assay at the time of the analysis, and is currently available on the platform.

Case 19’s PIK3CA H1074R: Upon further review, this alteration should be considered a VUS at this time and thus changed from CT to “none” for the F1 categorization.

For all cases discussed above, the highest categorization was not altered and the conclusions of our analysis remain as published.

While the letter did appropriately scrutinize certain find by F1 that were not found by PCDx, it is important to draw attention to the large number of RNA-expression findings not measured by the F1 assay. We have previously demonstrated that relying on panels of RNA-expression markers can lead to improved clinical outcomes.2 While PCDx accurately reported and has redesigned regions with low coverage and expanded the targets offered, including IHC for protein expression, F1 does not offer gene expression or protein expression.

To address the issue raised on TOP2A mRNA expression, we refer the reader to Brase et al12 who clearly demonstrated in breast cancer that TOP2A gene expression but not TOP2A gene amplification is correlated with protein by IHC (P<0.001 and P<0.283, respectively). Furthermore, TOP2A gene expression is associated with favorable response to anthracycline-based therapy.12 These data are also supported by earlier work performed on brain tumors showing strong correlation of TOP2A mRNA levels with IHC (P<0.0001).13

PCDx is actively being employed for clinical treatment decision-making in two separate prospective clinical trials in adult oncology (NCT01919749 and NCT02101385). We agree that a more robust clinical comparison of the two platforms is warranted. Foundation Medicine is invited to participate in a larger, more robust head-to-head clinical comparison to evaluate further the clinical utility and actionability of both commercial platforms.

As we reported in our paper and continues to be the case, the PCDx test has better utility for rapidly identifying a larger number of clinically actionable biomarkers in a shorter amount of time.