Clinical performance evaluation of the Idylla NRAS-BRAF mutation test on retrospectively collected formalin-fixed paraffin-embedded colorectal cancer tissue.

AIMS: Understanding the molecular mechanisms of underlying disease has led to a movement away from the one-drug-fits-all paradigm towards treatment tailored to the genetic profile of the patient. The Biocartis Idylla platform is a novel fully automated, real-time PCR-based in vitro diagnostic system. The Idylla NRAS-BRAF mutation test has been developed for the qualitative detection of mutations in NRAS and BRAF oncogenes, facilitating genetic profiling of patients with cancer. The aim of this study was to carry out a formal clinical performance evaluation.
METHODS: Two-hundred and forty-two formalin-fixed paraffin-embedded (FFPE) human malignant colorectal cancer (CRC) tissue samples were identified in departmental archives and tested with both the Idylla NRAS-BRAF mutation test and the Agena Bioscience MassARRAY test.
RESULTS: The overall concordance between the Idylla NRAS-BRAF mutation test and the MassARRAY comparator reference test result was 241/242 (99.59%, lower bound of one-sided 95% CI=98.1%) for NRAS and 242/242 (lower bound of 95% one-sided 95% CI=98.89%) for BRAF. The Idylla NRAS-BRAF test detected one NRAS mutation that had not been reported by the MassARRAY comparator reference test. Reanalysis of this sample by droplet digital PCR confirmed that the mutation was present, but at an allelic frequency below the stated sensitivity level of the MassARRAY system.
CONCLUSION: These results confirm that the Idylla NRAS-BRAF mutation test has high concordance with a widely used NRAS-BRAF test, and is therefore suitable for use as an in vitro diagnostic device for this application. © Article author(s) (or their employer(s) unless otherwise stated in the text of the article) 2018. All rights reserved. No commercial use is permitted unless otherwise expressly granted.

Introduction

The epidermal growth factor receptor (EGFR) and its downstream signalling pathways are involved in the development and progression of several human tumours including colorectal cancer (CRC).1–3 Activated signalling pathways including the RAS-RAF-BRAF-MAPK and the phosphatidylinositol 3-kinase (PI3K)-Akt pathways are key drivers of tumour growth and progression.4 5 Patients with CRC can be successfully treated with anti-EGFR monoclonal antibody therapies, such as panitumumab or cetuximab, which work by disrupting EGFR signals6–10—provided the tumours do not have mutations in EGFR signalling pathways.11–14 Testing patients for specific mutations that are linked to lack of response to EGFR blockade prior to commencing therapy ensures they receive the most appropriate treatment, and that treatment benefits, adverse effects and costs will be optimised.

Reduced anti-EGFR treatment efficacy was initially thought to be the result of mutations within KRAS exon 2 (codons 12 and 13). Therefore, KRAS genotyping was routinely used to stratify patients with CRC into those likely to respond to anti-EGFR therapy and those unlikely to respond. Further mutations causing resistance to anti-EGFR therapy were subsequently discovered through two large clinical studies (PRIME and CRYSTAL) which showed that 14%–17% wild-type (WT) KRAS exon 2 individuals actually responded poorly to anti-EGFR therapy.9 15 It is now known that resistance to therapy has a range of genetic causes: mutations in KRAS (exons 2, 3 and 4) and NRAS (exons 2, 3 and 4) are found in 40%–45% and 5%–10% of patients, respectively.5 16 17 BRAF V600E mutation is detected in 4%–18% of CRC cases and is prognostic of outcomes, but, in contrast to KRAS which is predictive of response is not used to guide anti-EGFR treatment.16–19

The Idylla System (Biocartis, Mechelen, Belgium) is a fully automated, real-time PCR-based molecular diagnostics system.20–26 The Idylla NRAS-BRAF mutation test is a novel in vitro diagnostic medical device (IVD) for the qualitative detection of mutations in codons 12, 13, 59, 61, 117 and 146 of the NRAS oncogene and codon 600 in the BRAF oncogene in formalin-fixed paraffin-embedded (FFPE) human malignant CRC tissue. This non-randomised study, was a clinical performance evaluation of the Idylla NRAS-BRAF mutation test with retrospectively collected samples and prospectively planned data generation.

Materials and methods

The Idylla platform

The Idylla molecular diagnostic platform is a fully automated, allele-specific real-time PCR-based molecular diagnostic system.20 It combines sample preparation with PCR thermocycling to detect molecular targets from a variety of solid and liquid samples including FFPE slices. Briefly, FFPE tissue sections are placed directly into the Idylla system cartridge, with no preprocessing. The cartridge is then inserted into the Idylla platform, where high-intensity focused ultrasound (HiFu) technology, buffers, reagents and heat induces deparaffinisation, tissue disruption, cell lysis and subsequent release of nucleic acid. Allele-specific primers and fluorescent probes then perform real-time PCR amplification and mutation detection. All required consumables are provided in the cartridge, and the Idylla Console and instruments are CE marked.

The Idylla NRAS-BRAF mutation test

The Idylla NRAS-BRAF mutation test uses allele-specific multiplex PCR reactions to amplify 23 mutations in codons 12, 13, 59, 61, 117, 146 of the NRAS oncogene and codon 600 of the BRAF oncogene. A set of parameters describing the generated PCR curves are determined by the Idylla software, for example, ΔCq value (calculated as the difference between the quantification cycle value (Cq) of the gene control signal and the Cq of the mutant signal). A sample is classified as mutation positive if the parameters of the PCR curve generated are within the validated range. Otherwise the sample is reported as being mutation negative, that is, WT. Table 1 lists the mutations detected by the test.

Table 1

NRAS and BRAF mutations detected by the Idylla NRAS-BRAF mutation test

Gene	Exon	Codon	Mutation	Protein	Nucleotide change	Genetic call
NRAS	2	12	G12D	p.(Gly12Asp)	c.35G>A	G12D
			G12C	p.(Gly12Cys)	c.34G>T	G12C
			G12S	p.(Gly12Ser)	c.34G>A	G12S
			G12A	p.(Gly12Ala)	c.35G>C	G12A/V
			G12V	p.(Gly12Val)	c.35G>T
		13	G13D	p.(Gly13Asp)	c.38G>A	G13D
			G13R	p.(Gly12Arg)	c.37G>C	G12R/V
			G12V	p.(Gly13Va)	c.38G>T
	3	59	A59T	p.(Ala59Thr)	c.175G>A	A59T
		61	Q61K	p.(Gln61Lys)	c.181A>G	Q61K
			Q61R	p.(Gln61Arg)	c.182A>G	Q61R
			Q61L	p.(Gln61Leu)	c.182A>T	Q61L
			Q61H	p.(Gln61His)	c.183A>T	Q61H
			Q61H	p.(Gln61His)	c.183A>T
	4	117	K117N	p.(Lys117Asn)	c.351G>C	K11N
			K117N	p.(Lys117Asn)	c.351G>T
			A146T	p.(Ala146Thr)	c.426G>A	A146T/V
			A146V	p.(Ala146Val)	c.437C>T
BRAF	15	600	V600E	p.(Val600Glu)	c.1799T>A	V600E/D
			V600E	p.(Val600Glu)	c.1799_1800delinsAA
			V600D	p.(Val600Asp)	c.1799_1800delinsAC
		600	V600K	p.(Val600Lys)	c.1798_1799delinsAA	V600K/R
			V600R	p.(Val600Arg)	c.1798_1799delinsAA

The results obtained from the Idylla NRAS-BRAF mutation test during this study were not used for clinical or diagnostic purposes and were not made available to patients or their clinicians.

The comparator test

The NRAS and BRAF mutation status for each FFPE tissue sample was assessed using the MassARRAY System (Agena Bioscience). Existing test results were used if the comparator test had been performed within 2 years of the start of this study and no further slides or slices had been taken from the source FFPE block. H&E slides were taken at the start and end of block slicing to confirm sample homogeneity. If the sample block had been cut since the initial comparator test analysis, newly prepared alternating slides/slices were taken to rerun the comparator test. To reduce the risk of bias, different operators performed testing on the MassARRAY System and the Idylla NRAS-BRAF mutation test.

The Agena MassARRAY mutation test

The Agena MassARRAY mutation test used in this study is a Laboratory Developed Test which is within the scope of the ISO 15189 accreditation held by NewGene Ltd, and is routinely used clinically. Samples with known mutation profiles, as used during clinical validation of the assay for ISO 15189 accreditation, were used as positive controls. ‘No template’ reactions, prepared with deionised H2O in place of template DNA were used as controls for non-specific amplification.

MassARRAY utilises the primer extension analytical method to determine the presence or absence of single nucleotide polymorphisms (SNPs) which cause specific mutations in the protein encoded by a gene. First, target region-specific PCR with short amplicons is used to amplify the DNA encoding the target sequence. These PCR products are then probed for the presence of the mutation of interest using an oligonucleotide which is complementary to the region immediately adjacent to the SNP location. This is then extended by only one base, using 2',3' dideoxynucleotides (ddNTPs); the ddNTP which terminates the oligonucleotide chain corresponds to the base code at the SNP location. A matrix-assisted laser desorption/ionization time-of-flight (MALDI-ToF) mass spectrometry system is then used to differentiate between oligonucleotide species on the basis of their differing masses, identifying the various SNP bases present at that location and also quantifying the relative allelic frequency of the mutant sequence versus the WT.

Study size estimation

The number of samples required for a diagnostic agreement study with 80% power and expected agreement rate of 95% is 188. To allow for invalid runs and samples, the target number of FFPE tumour tissue blocks to be tested was set at 250. At least 30 valid FFPE samples with known NRAS mutations, at least 30 samples with known BRAF mutations, and up to 190 samples with WT KRAS were planned to be included.

Tissue selection criteria and process

The inclusion criteria for samples can be found in box 1. Samples were obtained from two centres: The Royal Victoria Infirmary, Newcastle upon Tyne Hospitals NHS Foundation Trust (UK; n=113), and the Institute of Pathology, Aarhus University Hospital (Denmark; n=156). All available samples were entered into the study. Figure 1 shows how FFPE tissue samples were prepared for use in the Idylla NRAS-BRAF mutation test.

Figure 1

Formalin-fixed paraffin-embedded (FFPE) sample preparation algorithm prior to Idylla NRAS-BRAF mutation test. Histologically confirmed, retrospectively collected FFPE colorectal cancer tissue samples were identified and tissue sections 5 or 10 µm thick were sampled as close as possible (within the same FFPE block) to the sections used for reference testing. Prior to analysis, the tumour content and area were determined on a H&E slide by a consultant histopathologist at both sites. If required, macrodissections were performed to ensure tumour nuclei content of more than 10% in the sample for analysis. The volume of material was specified to ensure that there was sufficient for the assay while not overloading the device.

Formalin-fixed paraffin-embedded (FFPE) colorectal cancer (CRC) samples were included if they were:

Obtained from male or female patients≥18 years of age

Approved for investigational purposes according to the applicable laws

Histologically confirmed to be primary or metastatic malignant CRC

Fixed in formalin for a maximum time of 48 hours

Less than 10 years old and had been stored at ambient conditions

Stored at room temperature for not more than 60 days prior to Idylla testing

CRC samples were excluded if:

The patient declined to provide their sample to third parties for research purposes

The protocol specified that samples should have at least 10% of their content identified as tumorous cells because the clinical importance of results (concordant or discrepant) from samples with very low proportions of tumorous cells could be debatable.

Baseline testing

No discrepancies were observed between the original results and the repeat results when retesting with the MassARRAY was performed on specimens that had originally been tested with other devices.

Percentage agreement

As it is not feasible to set-up a study with large numbers of each individual mutant, the performance of the Idylla NRAS-BRAF mutation test was scored based on total agreement. Overall agreement (% total agreement), and negative and positive agreements were determined together with their 95% one-sided CIs. Overall agreement was defined as the proportion of concordant results in all results. Positive agreement was defined as the proportion of concordant results in positive Idylla results. Negative agreement was defined as the proportion of concordant results in negative Idylla results.

Discordant by design results

The Idylla NRAS-BRAF panel detects 18 NRAS mutations and 5 BRAF mutations (table 1), while the MassARRAY NRAS-BRAF panel detects 25 NRAS mutations and provides a BRAF V600E result as a generic call rather than distinguishing between V600E/D or V600K/R. The seven additional NRAS mutations detected by the MassARRAY but not by the Idylla NRAS-BRAF mutation test panel include G12R, G13C, G13S, G13A, Q61P and Q61E. Samples that were anticipated to produce discordant results because of design differences in the Idylla and MassARRAY systems were classed as ‘discordant by design’ and were excluded from analysis.

Investigation of discordant results

Discordant results were further investigated by using droplet digital PCR as a second comparator test to evaluate the accuracy of the Idylla result. Droplet digital PCR was performed on a QX100TM system (Bio-Rad Laboratories, Hercules, California, USA) according to the instructions of the manufacturer.

Results

Samples included

Two hundred and sixty-nine FFPE CRC samples were initially identified for the study (figure 2), and 242 samples included in the performance evaluation. Twenty-seven FFPE samples were excluded because they failed to meet the inclusion criteria. The most common reason for exclusion was insufficient sample quantity (n=11). One FFPE sample was excluded because it was classified as discordant by design. Seven samples resulted in a failed run, three of which were due to insufficient sample (assay failure) and four of which were due to instrument errors (system failure). This equates to assay and system failure rates of 1.24% and 1.74%, respectively. All invalid and error runs were successfully repeated and included in the agreement analysis as specified in the protocol. Samples included in the study ranged from blocks of tumour from resections (198) to small diagnostic core biopsies (44). Macrodissection was required in 126 samples. The proportions of tumour ranged from 20% to 60% in sections without macrodissection, and from 10% to 75% in sections requiring macrodissection.

Figure 2

Process of enrolling samples into the study.

The photomicrographs in figure 3 are from samples which show extensive mucinosis (A) extensive necrosis (B) and a small core biopsy (C). The Idylla NRAS-BRAF mutation test was able to detect mutations in all such samples.

Figure 3

Photomicrograph of tumour samples used for performance evaluation study. (A) sample with extensive mucinosis; (B) sample with extensive necrosis; (C) sample from small core biopsy of metastatic colon adenocarcinoma in the liver. The core was divided lengthways before embedding and the total section area was 25 mm2, the minimum specified in the protocol. The sample tested positive in both platforms without macrodissection and included solid adenocarcinoma (top left) normal liver (top right) and mucinous stroma (lower core).

Baseline demographic and clinical characteristics

The final study population demographics and clinical characteristics are presented in table 2. Gender representation was balanced between males and females (54% and 46%, respectively). The mean age at tumour collection was 67.6 years (range: 24–96 years). The majority of samples (91%) were of primary tumour tissues originating from the colon. The remaining samples were from metastatic deposits originated in the liver (3%) or another (6%) location. All samples were histologically confirmed as malignant CRC; 85% were KRAS WT, 15% had an NRAS mutation and 21% had a BRAF mutation.

Table 2

Baseline demographic and clinical characteristics for participants with valid test results

Gender
Male	131 (54%)
Female	111 (46%)
Mean age (years) at tumour tissue collection date	67.6 (SD 10.96)
Histologically confirmed malignant colorectal cancer	242 (100%)
Tissue location
Primary	220 (91%)
Metastatic (liver)	7 (3%)
Metastatic (other)	15 (6%)

Clinical performance of the Idylla NRAS-BRAF mutation test

The NRAS and BRAF mutational status of 242 retrospectively collected FFPE CRC samples was tested with the NRAS-BRAF mutation test. Results were compared with assessments made by the MassARRAY comparator reference test. Of the 242 FFPE samples analysed by the NRAS-BRAF Mutation Test, 36 tested positive for a mutation in NRAS, and in 205 cases, no NRAS mutation was found (table 3). As presented in table 4, results obtained by the NRAS-BRAF mutation test for NRAS testing were in agreement with the results obtained from the MassARRAY comparator reference test for 241 of 242 samples, resulting in an overall concordance of 99.59%, (lower bound of one-sided 95% CI=98.17%) with positive agreement of 100% (lower bound of one-sided 95% CI=93.0%) and a negative agreement of 99.51% (lower bound of one-sided 95% CI=97.85%).

Table 3

Mutation concordance analysis for the Idylla NRAS-BRAF mutation test for NRAS compared with MassARRAY. Table entries on the diagonal with orange shading are concordant results. The one discordant result, G12D with Idylla and WT with MassARRAY, is shaded light blue and discussed in the text

Idylla	MassARRAY
	G12A	G12V	G12C	G12D	G12S	G13D	Q61H	Q61K	Q61L	G61R	A146T/V	WT
G12A/V	1	4	0	0	0	0	0	0	0	0	0	0
G12C	0	0	2	0	0	0	0	0	0	0	0	0
G12D	0	0	0	11	0	0	0	0	0	0	0	1*
G12S	0	0	0	0	0	0	0	0	0	0	0	0
G13D	0	0	0	0	0	1	0	0	0	0	0	0
Q61H	0	0	0	0	0	0	1	0	0	0	0	0
Q61K	0	0	0	0	0	0	0	7	0	0	0	0
Q61L	0	0	0	0	0	0	0	0	3	0	0	0
G61R	0	0	0	0	0	0	0	0	0	6	0	0
A146T/V	0	0	0	0	0	0	0	0	0	0	0	0
WT	0	0	0	0	0	0	0	0	0	0		205

The orange entries on the diagonal highlight the correspondence between the MassARRAY and Idylla results. The one discrepant result has a light blue background.

*ddPCR confirmed the G12D result (<5% allelic frequency).

WT, wild type.

Table 4

Idylla NRAS-BRAF mutation test results compared with the reference test (MassARRAY) results

(a) NRAS testing
Idylla		MassARRAY
		Mutation	No mutation	Total
	Mutation	36	1	37
	No mutation	0	205	205
	Total	36	206	242
Idylla performance	Overall agreement	241/242 (99.59%, lower bound of one-sided 95% CI=98.17%)
	Positive agreement	36/36 (100%, lower bound of one-sided 95% CI=93.0%)
	Negative agreement	205/206 (99.51%, lower bound of one-sided 95% CI=97.85%)
(b) BRAF testing
Idylla		MassARRAY
		Mutation BRAF c.1799T>A p.(Val600Glu)	No mutation	Total
	Mutation BRAF c.1799T>A p.(Val600Glu)	52	0	52
	No mutation	0	190	190
	Total	52	190	242
Idylla performance	Overall agreement	242/242 (100%, lower bound of one-sided 95% CI=98.89%)
	Positive agreement	52/52 (100%, lower bound of one-sided 95% CI=95.05%)
	Negative agreement	190/190 (100%, lower bound of one-sided 95% CI=98.60%)

Of the 242 FFPE samples, 52 tested positive for BRAF V600E/D mutation; no mutation was found in 190 cases. Results obtained by the NRAS-BRAF mutation test for BRAF testing were fully in agreement with the results obtained from the MassARRAY comparator reference test resulting in an overall concordance of 100% (lower bound of one-sided 95% CI=98.89%) (table 4).

Discordant result

In one case, a G12D NRAS mutation was detected by the NRAS-BRAF mutation test but not by the MassARRAY comparator reference test (tables 3 and 4). The NRAS G12D result was confirmed to be a true positive by ddPCR analysis with an allelic frequency of around 0.5%, which is well below the 5% limit of detection of the MassARRAY system (table 5).

Table 5

Discordant result between Idylla NRAS-BRAF mutation test and MassARRAY reference method

Sample no	Tissue	Age of block	No of FFPE tissue sections	Total tissue area (mm2)	Tumour cells (%)	Macrodissection	Idylla NRAS result	Further analysis method	Further analysis and final classification
A050	Primary	<1 year	1	225	30	No	G12D	ddPCR	G12D (TP)

ddPCR, droplet digital PCR; NMD, no mutation detected; FFPE, formalin-fixed paraffin-embedded; FP, false positive; TP, true positive.

Discussion

The importance of testing for NRAS and BRAF mutations in tumours from patients with CRC is reflected in recent clinical practice recommendations of the American Society of Clinical Oncology27 and the European Society for Medical Oncology.28 NRAS mutations in colorectal tumours are associated with poor response to anti-EGFR therapy while BRAF mutations are associated with reduced survival.29 Evidence-based guidelines therefore recommend NRAS (predictive) testing to guide treatment decisions and BRAF (prognostic) testing to provide information on outcomes.27 28

The performance of the Idylla platform has been previously reported in various validation studies.23 24 30–33 In the present study, results from the Idylla NRAS-BRAF mutation test, which can detect 18 NRAS and five BRAF clinically relevant and actionable mutation targets, were compared with results obtained from NRAS and BRAF mutation testing by the MassARRAY.

There was greater than 99.5% concordance between the Idylla platform results and MassArray findings for NRAS and BRAF testing. Droplet digital PCR testing of the one discordant result confirmed the Idylla identification. One plausible explanation for this finding is that Idylla may have a lower limit of detection than the MassARRAY reference test; other possible explanations include test error and sample error.

Both the Idylla platform and MassARRAY methods were highly sensitive, and mutations could be reliably detected in small core biopsies and in samples with extensive mucinosis or necrosis.

As with other products based on similar technologies that identify specific mutations, mutations that are not intended to be detected will not be detected. However, these technologies in general, and the Idylla system, in particular, have the advantage of being able to readily update their assays to include mutations newly recognised as being clinically important, although revalidation for CE marking may be required.

In conclusion, the Idylla NRAS-BRAF mutation test is a fully automated and integrated real-time PCR-based test with high sensitivity and rapid turnaround time, providing results that are readily interpreted by clinicians. The Idylla platform proved straightforward to use with or without macrodissection. The method of handling the FFPE sections is distinct to Idylla and contrasts with the more usual method of placing the FFPE sections into a container. FFPE samples are loaded directly into single use cartridges with minimal sample preparation: histological sections (curls) for analysis are easily ‘sandwiched’ between the filter papers in the cassette. Potential drawbacks include the limited throughput (four samples per unit per day) and inability to retrieve samples and rescue an analysis if the system malfunctions mid-run. The Idylla platform would ideally be set up in a clean room close to where the paraffin sections are being prepared in a pathology laboratory and employed in clinical settings where the time saved by analysis on demand (compared with routine batch processing) would provide clinical and/or efficiency benefits. The Agena MassARRAY system can test from 1 to 384 samples on a chip (depending on the panels of tests and number of wells) and can process around 120 colorectal mutation tests per day.

Identification of mutations in NRAS and BRAF is important for treatment and prognosis in patients with colorectal cancer.

The Idylla NRAS-BRAF mutation test is as accurate as the widely used Agena MassARRAY.

The Idylla platform and Agena MassARRAY system are both highly configurable for low to high throughputs, but address different clinical and processing needs. The Idylla platform provides rapid analysis on demand, thus allowing pathology reports to integrate histology and mutation analysis, and treatment decisions to be made with minimum delay. The Agena MassARRAY system can process around 120 colorectal mutation tests per day.