Evaluation and comparison of two commercially available targeted next-generation sequencing platforms to assist oncology decision making.

BACKGROUND: It is widely acknowledged that there is value in examining cancers for genomic aberrations via next-generation sequencing (NGS). How commercially available NGS platforms compare with each other, and the clinical utility of the reported actionable results, are not well known. During the course of the current study, the Foundation One (F1) test generated data on a combination of somatic mutations, insertion and deletion polymorphisms, chromosomal abnormalities, and deoxyribonucleic acid (DNA) copy number changes at ~250× coverage, while the Paradigm Cancer Diagnostic (PCDx) test generated the same type of data at >5,000× coverage, plus provided messenger RNA (mRNA) expression levels. We sought to compare and evaluate paired formalin-fixed paraffin-embedded tumor tissue using these two platforms.
METHODS: Samples from patients with advanced solid tumors were submitted to both the F1 and PCDx vendors for NGS analysis. Turnaround time (TAT) was calculated. Biomarkers were considered clinically actionable if they had a published association with treatment response in humans and were assigned to the following categories: commercially available drug (CA), clinical trial drug (CT), or neither option (hereafter referred to as "None").
RESULTS: The demographics of the 21 unique patient tumor samples included ten men and eleven women, with a median age of 56 years. Due to insufficient archival tissue from the same collection period, in one case, we used samples from different collections. PCDx reported first results faster than F1 in 20 cases. When received at both vendors on the same day, PCDx reported first results for 14 of 15 cases, with a median TAT of 9 days earlier than F1 (P<0.0001). Categorization of CA compared to CT and none significantly favored PCDx (P=0.012).
CONCLUSION: In the current analysis, commercially available NGS platforms provided clinically relevant actionable targets (CA or CT) in 47%-67% of diverse cancer types. In the samples analyzed, PCDx significantly outperformed F1 in TAT, and had statistically significant higher clinically relevant actionable targets categorized as CA.

Introduction

During the course of their lifetime, advanced cancer patients will often receive several lines of treatment. With each subsequent line of therapy, there may be diminishing rates of return on durable response.1 At a minimum, the identification of the Achilles’ heel of a tumor to invoke disease stabilization and durable tumor control is what many clinicians are seeking to help their patients. The era of precision medicine and consequent genomic profiling a patient’s tumor to identify a unique treatment option is upon us.2 We and others have reported on use of next-generation sequencing (NGS) technologies in prospective studies.3–7 The field of sequencing technologies has rapidly evolved over the last decade, and there are now benchtop sequencers that allow for high-depth coverage of several megabases in a matter of hours with relatively lost cost and reasonable turnaround time (TAT) for clinical applications.7

There are now a variety of commercially available assays that can be ordered by clinicians. How commercially available NGS platforms compare with each other and the clinical utility of the reported actionable results is not well known. One of the most popular and prevalent is the Foundation One (F1) test (Foundation Medicine, Cambridge, MA, USA), launched in 2011. Paradigm Cancer Diagnostic (PCDx) is a newer commercially available NGS platform (Paradigm, Ann Arbor, MI, USA) launched in 2014. During the time frame of the current analysis, the F1 test generated data on a combination of somatic mutations, insertion and deletion polymorphisms (indels), chromosomal abnormalities, and deoxyribonucleic acid (DNA) copy number changes at ~250× coverage, while the PCDx test generated the same type of data at >5,000× coverage, plus provided messenger ribonucleic acid (mRNA) expression levels. To replicate the experience of real clinical practice, additional validation of actionable targets reported by commercial laboratory tests was not performed by another independent laboratory. We sought to compare and evaluate paired formalin-fixed, paraffin-embedded (FFPE) tumors using these two platforms.

Materials and methods

Ethics statement

After obtaining ethics approval from the Western Institutional Review Board (IRB) under Exemption 4 of Title 45 Code of the US Federal Regulations (CFR) concerning retrospective study of existing data, qualifying NGS reports were collected. Under exemption 4, patient consent is not required and was not obtained for this study, as Title 45 CFR Part 46 does not apply. NGS results were de-identified and the clinical information associated with these specimens is not individually identifiable, such that subjects cannot be identified either directly or indirectly through identifiers linked to the subjects.

Tumor specimens and clinical annotation

FFPE tumor tissues were ordered by the treating physician when deemed clinically appropriate for F1 testing and were submitted for commercial testing according to Foundation Medicine’s commercial laboratory standards.8 Clinically appropriate testing was defined as (i) patient having a stage 4 solid tumor by AJCC staging and (ii) tumor progressed on at least one line of therapy when there exists a standard of care (SOC) treatment or (iii) there is no SOC for that cancer diagnosis. Where feasible, an additional tumor tissue block from the same collection date or with the same pathologic diagnosis from a different collection time point was also sent for PCDx testing.

Methods for PCDx testing

FFPE specimens were received for PCDx testing, and the diagnosis of each case was confirmed on a freshly cut hematoxylin and eosin (HE)-stained slide by a board-certified pathologist (author RJP). Cases were micro/macro dissected when less than 60% tumor cells were present in order to enrich for tumor cells. DNA was extracted from all specimens, and where feasible, ribonucleic acid (RNA) was also extracted. Complementary DNA was created from RNA. A proprietary polymerase chain reaction (PCR)-based method was used to create libraries. All libraries from a given case were simultaneously sequenced on an Ion 318™ chip on the Ion PGM sequencer (Thermo Fisher Scientific, Waltham, MA, USA). mRNA was analyzed for elevated expression at P≤0.001. Copy number variants and alterations were reported. Mutations were compared to a database of mutations, and only those contained within the database were reported. The test was optimized to detect base substitutions with 4% frequency at 99.9% sensitivity and indels with 7% frequency at 99.4% sensitivity. The specificity of mutation assays was optimized to be >99.99% at the patient level, meaning that <0.01% of patient reports will contain a false positive result. A report was generated, reviewed by a board-certified oncologist (author DML) and pathologist (author RJP), signed out, and transmitted to the patient’s physician.

Statistical analysis

Clinical annotation related to the tumor samples analyzed, such as age at the time of NGS testing, sex, histology, and tumor stage at the time of NGS testing were collected and summarized. TAT was calculated as the difference in calendar days between the date the sample was received at the vendor to the date of the first NGS report for that sample. A biomarker was considered clinically actionable if it had a published association with treatment response in humans. The NGS report was assigned the highest ranked category of the following categories based on the list of actionable biomarker(s): commercially available drug (CA) (highest category) > clinical trial drug (CT) > neither option (abbreviated as “None” hereafter) (lowest category). In other words, if there was more than one actionable biomarker in the F1 or PCDx report, the biomarker with the highest category was used in the comparison between the commercial NGS reports.

To be considered CA, one of the DNA actionable biomarkers or mRNA had to have demonstrated clinical efficacy in human cancer prospective trials using the biomarker and a CA that can target that biomarker.9 For PCDx, both ERBB2 and TOP2A mRNA have high (>80%) concordance with protein expression, and were the only mRNA biomarkers included for the CA categorization. For example, ERBB2 is a known biomarker candidate for trastuzumab or lapatinib, and EGFR is a known biomarker candidate for erlotinib or afatinib.

To be considered CT, one of the DNA actionable biomarkers was required to have a drug in clinical development using the biomarker. For example, MET is being explored as a biomarker for MET inhibitors, and PIK3CA is a biomarker being explored for PI3 kinase inhibitors. To be considered None, either KRAS mutation was reported or none of the remaining DNA or mRNA biomarkers was reported to have a drug candidate that had demonstrated clinical efficacy in human cancer prospective trials. A Student’s t-test was applied to TAT results and chi-square and Fisher’s exact tests were used to analyze the categorization of actionable biomarkers. A two-tailed P-value of <0.05 was considered to indicate a significant difference.

Results

Twenty-one cases were identified as meeting the inclusion criteria of paired F1 and PCDx testing. There were ten men and eleven women associated with the tumors analyzed, and these patients had a median age of 56 years (range 35–65 years). The most common cancer types were thoracic (n=7), gastrointestinal (n=4), and genitourinary (n=3) (Table 1). Paired F1 and PCDx results for the 21 unique patient tumors were submitted between March 2014 and September 2014. The results for the samples are provided in Table 2, and the list of targets for these assays is provided in Figures 1 and 2. Due to insufficient archival tissue from the same collection period, for one case, we used tumor tissue samples from different collection periods. All other samples included in this analysis underwent DNA analysis. For PCDx testing, RNA was extracted from all specimens, with the exception of one case for which sufficient tissue was not available. RNA was sequenced to a mean depth of coverage of 21,562×, while DNA copy number and mutation testing were sequenced to a mean depth of 56,085× and 13,656×, respectively.

Table 1

clinical annotation and TAT results

Case number	Age, years	Sex	Cancer type	Cancer stage by AJCC	Was tumor sent to vendor the same day?	Was tumor from the same collection date?	F1 TAT in days	PCDx TAT in days	Difference in TAT in days
1	59	M	Bladder cancer	4	No	Yes	30	14	16
2	65	F	Breast cancer	4	Yes	Yes	15	6	9
3	52	F	Breast carcinoma	4	Yes	Yes	13	7	6
4	56	F	Colon adenocarcinoma	4	No	Yes	15	8	7
5	63	M	Colon adenocarcinoma	4	Yes	Yes	12	5	7
6	44	F	Endometrial carcinoma	4	Yes	Yes	16	6	10
7	64	M	Gastroesophageal adenocarcinoma	4	No	No	19	7	12
8	59	M	Kidney cancer	4	Yes	Yes	17	6	11
9	61	F	Kidney cancer	4	Yes	Yes	15	11	4
10	53	F	Leiomyosarcoma	4	Yes	Yes	20	6	14
11	62	F	Lung adenocarcinoma	4	No	Yes	20	10	10
12	64	M	Lung adenocarcinoma	4	No	Yes	15	6	9
13	59	M	Lung adenocarcinoma	4	Yes	Yes	15	4	11
14	56	F	Lung adenocarcinoma	4	Yes	Yes	15	8	7
15	33	F	Lung adenocarcinoma	4	Yes	Yes	14	7	7
16	63	M	Lung adenocarcinoma	4	Yes	Yes	16	6	10
17	36	M	Mesothelioma	4	Yes	Yes	7	14	−7
18	51	F	Ovarian serous carcinoma	4	Yes	Yes	14	8	6
19	55	F	salivary gland carcinoma	4	No	Yes	19	12	7
20	55	M	Small intestine adenocarcinoma	4	Yes	Yes	15	3	12
21	35	M	Synovial soft tissue sarcoma	4	Yes	Yes	20	10	10

Abbreviations: AJCC, American Joint Commission on Cancer; M, male; F, female; TAT, turnaround time; F1, Foundation One test; PCDx, Paradigm Cancer Diagnostic test.

Table 2

Summary of categorization results

Case number	F1 reported actionable aberrations	PCDx reported actionable aberrations	Best clinical assessment of F1 actionable targets	Best clinical assessment of PCDx actionable targets	Categorization favors (F1, PCDx, or neither)
1	TET2, None; TP53, None; ARID 1A, None; KDM6A, None	(BRCA1, None; TUBB3, None; BIRC5, None; TYMS, None; BAX, None; TYMP, None; EZH2, None)	None	None	Neither
2	PDGFRA, CT; PIK3CA, CT; KDR, CT; TP53, None; APC, None; MAG12, None; MAP3K1, None; MLL3, None; RARA, None; SMAD4, None; SPTA1, None	(AR, None; TUBB3, None; ERBB2, CA; PARPI, None; TYMP, None; BIRC5, None)	Clinical trial	Commercially available drug	PCDx
3	CCND1, None; FGF19, None; FGF3, None; FGF4, None	(PR, None; SSTR2, None; ER, None; AR, None; TUBB3, None; ERBB2, CA; PARPI, None; ERBB3, None)	None	Commercially available drug	PCDx
4	SRC, None; TP53, None; APC, None	KIT, CA; (CDA, None; BRCA1, None; TUBB3, None; NFKB1, None; RelA, None; ERBB2, CA; MET, None; IGF1R, None; PDGFRB, None; CDH1, None)	None	Commercially available drug	PCDx
5	KRAS, None; P1K3CA, CT*; PTEN, CT*; APC, None; MAP3K1, None	(BRCA1, CAIX, NFKB1, TYMS, SLC29AI, RelA, MET, IGF1R, PDGFRB)	None	None	Neither
6	P1K3R1, None; PTEN, None; CTNNB1, None; TP53, None; ARID1A, None; ARID1B, None	(SSTR2, None; AREG, None)	None	None	Neither
7	ERBB2 amplification, CA; CDKN2A/B, None; TP53, None; PPP2R/A, None; SMAD4, None	(CDA, None; TUBB3, None; TOP2A, CA; ERBB2, CA; BIRC5, None)	Commercially available drug	Commercially available drug	Neither
8	TSC1, CT; KDM5C, None; PBRM1, None; SETD2, None	(EREG, None; CAIX, None; BIRC5, None; TOP2A, CA; TYMS, None; BAX, None; TYMP, None; VEGFA, None)	Clinical trial	Commercially available drug	PCDx
9	EGFR amplification, CA; ERBB2 amplification, CA; HGF amplification, None; RAF1 amplification, None; VHL, None; TP53, None; EPHA3, None; KDM5A, None; KDM6A, None	(SSTR2, None; TUBB3, None; BIRC5, None; TYMS, None; ERBB2, CA; BAX, None; TYMP, None; PDGFRB, None; EZH2, None)	Commercially available drug	Commercially available drug	Neither
10	TP53, None; C17orf39, None; RBI, None	(mTOR, None;BRCA1, None; MET, None; ERCC1, None; FGFR1, None; PARP1, None; BIRC5, None; BAX, None; TUBB3, None; CAIX, None; AREG, None; TYMP, None; TYMS, None; EZH2, None; TOP2A, CA; CDHI, None)	None	Commercially available drug	PCDx
11	HGF, None; MET, CT; CDKN2A/B, None; RICTOR, None; TP53, None; FGF10, None	(AREG, None; EREG, None; CDA, None; TUBB3, None; BIRC5, None; TOP2A, CA; TYMS, None; MET, None; CDHI, None; EZH2, None)	Clinical trial	Commercially available drug	PCDx
12	ERBB2-CA, HGF amplification-None, CD274-None, JAK2-CA, PDCD1LG2-None, CDK6-None, CDKN2A/B-None, TP53-None, LRP1B-None	(TUBB3-None, BIRC5-None, TOP2A-CA)	Commercially available drug	Commercially available drug	Neither
13	KRAS, None; MCL1, None; MYC, None; TP53, None; MYST3, None; SRC, None; ZNF703, None	KRAS, None; (AREG, None; EREG, None; TUBB3, None; BIRC5, None; TOP2A, CA*; TYMS, None; BAX, None; MET, None; VEGFA, None; EZH2, None)	None	None	Neither
14	ERBB2, CA; TP53, None; RAD50, None	(TOP2A, CA; TYMS, None)	Commercially available drug	Commercially available drug	Neither
15	EGFR, CA; FGFR4, None; CDKN2A/B, None; TP53, None; APC, None; CSF1R, CT	EGFR, CA; (TUBB3, None; BIRC5, None; TOP2A, CA; TYMS, None)	Commercially available drug	Commercially available drug	Neither
16	ERRFI1, None; IDHI, CT*; KRAS, None; PIK3C2B, None	KRAS, None; (TUBB3, None; BIRC5, None; TOP2A, CA*; TYMS, None; VEGFA, None)	None	None	Neither
17	None	(TUBB3, None; CAIX, None; TOP2A, CA; TYMS, None; BIRC5, None)	None	Commercially available drug	PCDx
18	GNAS amplification, CT; ATM, None; MCLI, None; TP53, None; ATRX, None	(PARP1, None; FABP5, None; TYMP, None; EREG, None; TYMS, None; EPHA2, None; BIRC5, None; ERBB3, None; EZH2, None; TOP2A, CA; CDHI, None; TUBB3, None)	Clinical trial	Commercially available drug	PCDx
19	HRAS, CT; PIK3CA, CT; TP53, None; APC, None	(TUBB3, None)	Clinical trial	None	F1
20	FBXW7, None; KRAS, None; PIK3R1, None; SOX9, None	KRAS, None	None	None	Neither
21	SS18, None	(ADRA2C, None; TOP2A, CA)	None	Commercially available drug	PCDx

Notes: Genes listed are mutations unless indicated as an amplification;

these results were ultimately categorized as None because of the presence of a KRAS mutation in the report; mRNA results are in parentheses.

Abbreviations: CT, clinical trial drug; CA, commercially available drug; None, neither CT nor CA option; F1, Foundation One test; PCDx, Paradigm Cancer Diagnostic test

Figure 1

Distribution of F1 gene targets and categorization.

Notes: *Both gene and rearrangement; **rearrangement only. The F1 list is listed as current as of august 4, 2014 (http://www.foundationone.com/genelist1.php).

Abbreviations: F1, Foundation One test; CA, commercially available drug; CT, clinical trial drug; None, neither a CA nor a CT option.

Figure 2

Distribution of PCDx gene targets and categorization.

Notes: *Both gene and rearrangement. The PCDx list provided by the vendor between June and October 2014.

Abbreviations: PCDx, Paradigm Cancer Diagnostic test; CA, commercially available drug; CT, clinical trial drug; none, neither a CA nor a CT option.

PCDx reported first results faster than F1 in 20 of 21 cases (95.2%), and these TAT were statistically significant (P<0.0001). PCDx had a median TAT of 7 days (range 3–14 days) to report first results compared to F1’s median TAT of 15 days (range 7–30 days). When both vendors received tumor samples on the same day, PCDx reported NGS results first for 14 of 15 cases (93.3%), with a median TAT of 9 days earlier than F1 (range −7 to 14 days). These TAT were also statistically significant (P<0.0001).

PCDx and F1 reported a CA 14 and five times, CT zero and five times, and None seven and eleven times, respectively (P=0.004). Categorization of CA compared to CT and None significantly favored PCDx (P=0.012). PCDx provided higher ranking actionable targets for nine cases vs one case for F1, while the remainder of cases had equivalent ranking for actionable targets. Six paired cases were categorized as None for both NGS platforms. F1 and PCDx platforms yielded eleven and seven cases, respectively, that were categorized as None, resulting in a 47%–67% rate of clinically relevant actionable targets (CA or CT) in diverse cancer types using a commercially available NGS platform.

Discussion

In recent years, there has been a surge of new molecular targeted agents that are available to treating medical oncologists. In parallel, multiplexed molecular tests have arrived on the market, with varying degrees of evidence or summary reports that pose challenges to treating medical oncologists. A key issue in clinical oncology practice is the ability to accurately interpret what is truly clinically meaningful and actionable on a report from a commercially available NGS assay. Recently, 22% of adult cancer physicians from a tertiary-care, National Cancer Institute-designated comprehensive cancer center reported low confidence in their genomic knowledge.10 The majority of cancer care is not delivered at tertiary comprehensive cancer centers, but rather in busy community oncology practices. Treating physicians will have time constraints to read and synthesize NGS assay reports, so there is a need for clear, concise reporting of clinically relevant targets.

In this study, we sought to compare the F1 and PCDx NGS platforms to determine if we could discriminate differences in TAT and clinically relevant actionable biomarkers in tumor tissue from patients with advanced cancers. It should be noted that these comparisons are based on the information provided on the commercial vendors’ reports, the same reports that a treating medical oncologist would be provided. Furthermore, since the purpose of the current study was to compare and evaluate these two NGS platforms, outcome data and prescribed therapeutic agents that may have been based on these reports are not available. As in real clinical practice, when there was more than one tissue block collected at the same time point, multiple blocks were not sent to a commercial vendor to independently run each sample as a unique test with its own unique NGS report. We expected to observe a large degree of overlap on actionable targets from the same individual’s cancer, particularly when tissue collected at the same time point and from the same source was sent to the vendors for analysis.11,12

In the current study, we found that PCDx significantly outperformed F1 in TAT in nearly every paired case by a median time of 9 calendar days. For patients with advanced cancers, particularly those who have recently progressed, the treating physician is trying to make a determination on the next appropriate treatment as soon as possible. The longer the delay in initiating a new treatment for patients with progressing metastatic cancer, the shorter the window of opportunity to alter the disease course is. Recent guidelines for genomic testing to determine first-line therapy in advanced non-small cell lung cancer (NSCLC) called for a 5–10 working day TAT.13 Therefore, in the real world scenario where the use of a NGS testing platform is likely to occur, an even shorter TAT may be more clinically relevant. The results from our present study show the median TAT for F1 is 15 calendar days, while for PCDx, it is 7 calendar days. Considering that shipping the tumor sample to the commercial vendor adds at least 1 calendar day, PCDx’s TAT would fall well within the 5–10 working day TAT to meet the published guidelines for first-line therapy decision making in NSCLC.13

There are two notable discrepancies in the categorization of actionable biomarkers between F1 and PCDx that warrant further elaboration. Case number 5 did not have a KRAS mutation identified on PCDx. During the time frame these analyses were conducted, KRAS exon 4 was not available on the PCDx platform; however, none of the mRNA targets were CA or CT, so the resulting categorization for PCDx was the same as F1. It is important to note that the other three cases with KRAS mutations and the one case with an EGFR mutation were identified on both platforms (cases 13, 16, and 20, and case 15, respectively). In case number 12, while F1 reported a mutation in ERBB2, PCDx did not identify an ERBB2 mutation, as this was not yet available on the PCDx platform at the time of these analyses; it has been added by the vendors since. However, ERBB2 mRNA was not elevated, suggesting that perhaps F1 was likely upstaged with the CA categorization, when in fact there was unlikely protein expression of the ERBB2 biomarker. In this same case, TOP2A mRNA was elevated, so the PCDx result was categorized as CA.

We find in the current study that some commercially available NGS platforms are misleading customers with exceedingly high rates of “actionable targets” that can be offered to their patients based on test results. Barring the drug safety concerns, treatment coverage, and reimbursement issues that arise when prescribing recommended agent(s) off-label either alone or in combination, our study demonstrates that for diverse cancer types, the rate of clinically relevant targets is 47% for F1 and 67% for PCDx. This rate includes both CA and CT targets. The addition of protein expression measurement to an NGS platform would assist in determining clinically relevant targets. Indeed, at least two other vendors are offering immunohistochemistry (IHC) with NGS testing.3,14 Since the analysis of this study has been conducted, PCDx now offers IHC for several actionable biomarkers that would be categorized as CA or CT, including: AR, ER, HER2, MET, MGMT, PTEN, PR, TOP1, TYMS, PD-L1, and PD-1.

At the time of the study, the PCDx test total charge to insurance was US $4,800 per sample vs US $5,800 for the F1 test. The average depth of coverage for PCDx was >5,000× compared with >250× for the F1 test. PCDx reports results based on published clinical literature on drug response. The F1 results are tied most directly to cancer drivers found in the TCGA (The Cancer Genome Atlas), with some of these results also associated with the clinical literature on drug responses. During the study, PCDx interrogated 114 genes (DNA and mRNA combined), whereas the F1 platform tested 315 genes. The fact that PCDx identified clinically relevant actionable alterations more frequently while querying approximately one-third as many genes speaks to the relative utility of the gene sets tested. The interrogation of cancer driver genes does not necessarily have current clinical utility, although some subset of these driver genes is likely to become druggable in the future.

While a potential drawback to our study is that the sample size is potentially too small to draw definite conclusions for a wide range and large number of cancer types and specimens, it does call attention to disparate results on TAT, and to reporting of clinically meaningful targets in commercial vendor NGS assay reports. The results of this study underscore the need to differentiate clinically relevant and meaningful targets with a likelihood of benefit. It is important to distinguish actionable targets associated with CAs vs suggested options that should be tried in the context of a clinical trial vs lack of a reasonable or plausible treatment suggested by NGS analysis. With the integration of additional levels of testing such as proteomics and/or epigenetics, NGS reports can hopefully provide more context and substance that will lead to better outcomes for patients and toward the goal of precision medicine, which is to create better outcomes for patients.

Conclusion

In this analysis, commercially available NGS platforms provided clinically relevant actionable targets (CA or CT) in 47%–67% of diverse cancer types. In the samples analyzed, PCDx significantly outperformed F1 in TAT, and had statistically significant, higher clinically relevant actionable targets categorized as CA.