Comparative analysis of targeted next-generation sequencing panels for the detection of gene mutations in chronic lymphocytic leukemia: an ERIC multi-center study.

Next-generation sequencing (NGS) has transitioned from research to clinical routine, yet the comparability of different technologies for mutation profiling remains an open question. We performed a European multicenter (n=6) evaluation of three amplicon-based NGS assays targeting 11 genes recurrently mutated in chronic lymphocytic leukemia. Each assay was assessed by two centers using 48 pre-characterized chronic lymphocytic leukemia samples; libraries were sequenced on the Illumina MiSeq instrument and bioinformatics analyses were centralized. Across all centers the median percentage of target reads ≥100x ranged from 94.2- 99.8%. In order to rule out assay-specific technical variability, we first assessed variant calling at the individual assay level i.e., pairwise analysis of variants detected amongst partner centers. After filtering for variants present in the paired normal sample and removal of PCR/sequencing artefacts, the panels achieved 96.2% (Multiplicom), 97.7% (TruSeq) and 90% (HaloPlex) concordance at a variant allele frequency (VAF) >0.5%. Reproducibility was assessed by looking at the inter-laboratory variation in detecting mutations and 107 of 115 (93% concordance) mutations were detected by all six centers, while the remaining eight variants (7%) were undetected by a single center. Notably, 6 of 8 of these variants concerned minor subclonal mutations (VAF <5%). We sought to investigate low-frequency mutations further by using a high-sensitivity assay containing unique molecular identifiers, which confirmed the presence of several minor subclonal mutations. Thus, while amplicon-based approaches can be adopted for somatic mutation detection with VAF >5%, after rigorous validation, the use of unique molecular identifiers may be necessary to reach a higher sensitivity and ensure consistent and accurate detection of low-frequency variants.

Introduction

Chronic lymphocytic leukemia (CLL), a clinically and biologically heterogeneous B-cell malignancy, was one of the first cancers for which the genomic landscape was uncovered using next-generation sequencing (NGS) technologies, in particular whole-exome/genome sequencing (WES/WGS).[1-4] These initial studies led to the discovery of a large number of recurrently mutated genes affecting diverse cellular pathways and processes contributing to the pathobiology of CLL. That said, in a diagnostic or untreated cohort, only a limited number of genes carry mutations at a frequency >5% (i.e., TP53, ATM, SF3B1 and NOTCH1), with the majority of gene mutations detected in only a minor proportion of CLL patients.[5,6] To date, more than 2,000 genes have been reported as mutated in CLL, with mutations occurring in classic tumor suppressor genes (e.g., TP53 and ATM), signaling pathways (e.g., the Toll-like receptor [MYD88], NF-κB [BIRC3, NFKBIE], and NOTCH [NOTCH1, FBXW7] pathways) as well as genes involved in essential cellular processes such as RNA processing (e.g., SF3B1, RPS15, XPO1).[5-10]

In addition to mutations or defects in TP53, which have long been associated with a poor prognosis and progressive disease, mutations in several of these recently analyzed genes (with the exception of MYD88) have been linked to an aggressive clinical course with a significantly shorter time to treatment and a poor outcome when treated with chemo(immuno)therapy (e.g., ATM, BIRC3, EGR2, NFKBIE, NOTCH1, RPS15, SF3B1, XPO1).[8-20] In recent years it has also been reported that patients carrying minor clones harboring TP53 mutations (i.e., <10% variant allele frequency [VAF]) may have an outcome equally as poor as patients with TP53 mutations with VAF >10%, the approximate detection limit of Sanger sequencing. [21-24] Employing ultra-deep sequencing technologies these minor clones have been detected at frequencies as low as 0.1% and may expand as the disease progresses and/or at treatment relapse. Minor clones carrying other gene mutations may also affect outcome, however there are few studies published to date, hence precluding firm conclusions from being drawn.[23] That said, based on currently available data it appears that both the mutational complexity and subclonal diversity strongly influence the evolution of CLL.[6,25-29]

Owing to the increasing number of clinically relevant gene mutations identified in CLL, the shift from Sanger sequencing to high-throughput technologies is essential and targeted amplicon-based gene panels are a promising option.[30,31] These assays have numerous attractive features including the ability to custom design and screen a large number of genes (complete coding sequence or hotspots) and samples simultaneously, ultimately leading to a reduced cost per sample and a higher throughput. Another appealing facet of targeted gene panels is that the sequencing capacity is efficiently utilized, resulting in higher coverage of the regions of interest (ROI), and hence allowing for more sensitive detection of low-frequency variants; thus, they are ideally suited for routine diagnostics and monitoring procedures. However, despite the necessity for mutation detection using NGS assays, with numerous technologies available, including both commercial and laboratory developed tests (LDT), important issues that commonly arise concern the specific technique to choose, and the sensitivity, specificity and reproducibility of individual methodologies. These concerns are magnified within the diagnostic setting as the test results may impact on clinical decision-making and the therapeutic stratification of patients.

The European Research Initiative on CLL (ERIC) conducted this multi-center study to better understand the comparability of several gene panel assays by assessing various analytical parameters such as coverage, sensitivity and reproducibility; with the overall aim of highlighting the critical parameters users should take into consideration when introducing targeted NGS into the laboratory. In brief, we selected three amplicon-based assays (HaloPlex, TruSeq and Multiplicom), each protocol was tested by two centers, and all sequencing was performed on the Illumina MiSeq system. Our cohort comprised 48 well-characterized CLL cases and all centers sequenced the same samples. We observed high concordance between the various technologies and test centers when considering gene mutations with VAF >5%. Although low-frequency variants were detected by all techniques, greater diversity was observed for mutations with a VAF between 1-5%.

Methods

Patient material

Genomic DNA (gDNA), prepared from tumor and germline samples (buccal swabs or CD19-depleted peripheral blood mononuclear cells), was used as the analytical material. Samples were sourced from archival material that had previously been analyzed using established molecular techniques to interrogate either a single gene or numerous targets. Cases were selected based on the available mutational data such that 45 of 48 cases contained a previously identified somatic variant in at least one of the genes included in the panel designs (Online Supplementary Figure S1). Quality control was performed centrally before distribution of the samples to the six participating institutes. With the exception of the coordinating center, details regarding pre-characterized mutations were not disclosed to participating laboratories. All cases were diagnosed according to the International Workshop on Chronic Lymphocytic Leukemia (iwCLL) guidelines and displayed a typical CLL immunophenotyped.[32] Informed consent was obtained in accordance with the Declaration of Helsinki and ethical approval was granted by local review committees.

Target enrichment and library construction

Three amplicon-based targeted NGS assays were used in this study. Two assays, the HaloPlex Target Enrichment System (Agilent Technologies, Santa Clara, CA) and the Illumina TruSeq Custom Amplicon (TSCA) (Illumina, San Diego, CA), were customized and targeted 11 genes: full coding sequence (ATM, BIRC3, EGR2, FBXW7, MYD88, NFKBIE, POT1 and TP53) or hotspot regions (NOTCH1 [exon 34], SF3B1 [exons 14-16 and 18] and XPO1 [exons 15-16]). The third assay, the Multiplicom CLL Multiplex MASTR Plus (Agilent Technologies, Santa Clara, CA), is a commercially designed panel targeting the full coding sequence of nine of the genes listed above; NFKBIE and EGR2 are not included in the design. Intron-exon boundaries were covered by all assays to enable the detection of splice-site mutations. A HaloPlexHS capture-based custom-design assay incorporating unique molecular identifiers (UMI) was used to validate and more accurately quantify variants present at low frequencies within the 11 genes detailed above. The specifics of each assay are detailed in the Online Supplementary Appendix.

Next-generation sequencing and data analysis

Cluster generation and paired-end sequencing was performed on the MiSeq instrument (Illumina, San Diego, CA). All sequencing data were centrally analyzed using a custom bioinformatics pipeline. Illumina sequencing adapters were removed using TrimGalore v.0.6.0 and trimmed reads were aligned to the human reference genome hg19/NCBI GRCh37 using BWA mem v.0.7.12 with standard parameters. Variants were identified using VarScan2 v.2.3.7 in mpileup2cns mode with a minimum average quality of 30 and annotated with SnpEff and SnpSift. Bioinformatic analyses of the NGS dataset containing UMI was performed as follows: SurecallTrimmer v4.0.1 (Agilent Technologies, Santa Clara, CA, USA) was used to remove adaptor sequences, mask HaloPlex enzyme footprints and trim low-quality bases. Processed reads were aligned to the hg19 human reference genome with BWA v.0.7.12 and duplicate reads were marked using the Agilent LocatIt tool (v4.0.1). Variants were called with Pisces v.5.2.7.47 using a VAF cut-off of 0.5% and annotated with SnpEff and SnpSift. Variants had to meet the following conditions to be included in downstream analyses: (i) located within an exonic or splicing region; (ii) be non-synonymous; and (iii) not listed in the European 1,000 genomes variant database.

Results

Gene Panel Coverage

The gene panel designs for Illumina TSCA, HaloPlex and Multiplicom covered 100%, 99.9% and 100% of the targeted coding regions, respectively. Sequencing 48 CLL samples on the MiSeq platform yielded a median coverage of 3,834x and 1,540x (range: 2,991-7,761 and 161-3,068, respectively) for the two centers utilizing the Illumina TSCA panel while a median coverage of 1,062x and 1,953x (range: 334-2,427 and 338-7,496, respectively) was achieved in the centers performing the analysis with the HaloPlex enrichment system (three samples failed to generate any reads during the sequencing run in center 5) (Figure 1; Online Supplementary Table S2-3). For the Multiplicom CLL MASTR kit, 24 samples were sequenced over two runs and this reduction in samples/run combined with the smaller size of the panel (NFKBIE and EGR2 were not included in this predesigned panel) was reflected in the increase in median coverage obtained; 5,469x and 3,426x (range: 1,432-13,589 and 2,244-4,194, respectively) (Figure 1; Online Supplementary Table S4).

Variability was observed in the depth of coverage generated for individual samples analyzed within the same center and also between centers, however the percentage of bases covered at 100x was comparable between samples prepared using the same technology at partner sites (Figure 1; Online Supplementary Table S2-4). More specifically, when considering the percentage of target bases obtaining at least 100x coverage, the two centers utilizing the Multiplicom technology achieved a median of 99.8% and 99.6% (range: 99.4-99.8% and 99.5-99.7%), the Illumina TSCA test centers obtained median values of 99.7% and 96.5% (range, 99.6-100% and 70.5-98%) while the centers evaluating the HaloPlex system recorded median values of 94.2% and 97.6% (range: 64.8-97.2% and 90.2-99.4%) (Figure 1; Online Supplementary Table S2-4). Although overall high sequencing coverage was obtained, a few samples in individual centers had less than 90% of bases within the targeted ROI achieving at least 100x coverage (Figure 1, Online Supplementary Table S2-5). Detailed information on the coverage per target region for the samples not fulfilling a 90% coverage threshold are provided in the Online Supplemental Table S6-7.

Figure 1.

Gene panel coverage. Boxplots illustrating the coverage obtained for all samples and the percent of bases covered at 100x across all six test centers. As illustrated, high coverage was obtained for the majority of samples across all centers, however a few samples had less than 90% of bases within the targeted regions of interest (ROI) obtaining >100x coverage. Notably, the partner center utilizing the same technology obtained high coverage for these particular samples across all ROI indicating that the low coverage arose from a local issue rather than an inherent technological or design issue. For the Multiplicom CLL MASTR kit, 24 samples were sequenced over two runs and this reduction in samples/run combined with the smaller size of the panel was reflected in the increase in median coverage obtained compared to the other assays whereby all 48 samples were sequenced in a single run. C1-C6 represent the six participating centers: C1 & C2 (Multiplicom), C3 & C4 (Illumina TSCA) and C5 & C6 (HaloPlex).

Pairwise analysis of variant calls

Although the gene content of the panels was comparable, with the noted exception of the absence of the NFKBIE and EGR2 gene hotspots from the Multiplicom assay, the number and precise location of probes varied between the assays. In order to rule out any inherent assay-specific technical variability or probe bias we first assessed variant calling at the individual assay level i.e., pairwise analysis of variants detected amongst centers utilizing the same assay. We initially limited this analysis to the nine genes analyzed by all techniques (NFKBIE and EGR2 are discussed separately below). After filtering for variants present in the 1,000 genomes project, the paired normal sample and synonymous variants, and removal of PCR/sequencing artefacts, variants with a VAF >0.5% were compared. Application of a strict VAF cut-off at this stage of the analysis could be misleading as variants borderline of the threshold would not be taken into consideration and be deemed as discordant. Instead, concordance was defined as the ability to detect a variant irrespective of the VAF, while accuracy was defined as the degree of agreement between the VAF detected by the same panel. In order to aid in this analysis, variants were assigned to 1 of 5 groups: group 1, variant detected in both centers with a VAF >5%; group 2, variant was detected in one center with a VAF >5% and in the partner center with a VAF <5%; group 3, variant was detected in both centers with a VAF <5%; group 4, variant was detected in only one of the partner centers and with a VAF >5%; and, group 5, variant was detected in only one of the partner centers and with a VAF <5%.

Looking first at the mutational data generated by the Multiplicom assay, a total of 133 variants passed the quality filters and these could be grouped as follows: group 1 (n=104), group 2 (n=8), group 3 (n=16), group 4 (n=1) and group 5 (n=4) (Figure 2; Online Supplementary Table S8). Hence, variant calling between the two centers utilizing the Multiplicom assay reached a concordance of 128 of 133 (96.2%) with five variants detected by only one center (4 of these 5 variants had a VAF <5%; the remaining variant had a VAF of 22.8%) (Figure 2; Online Supplementary Table S8-10). This latter higher frequency variant concerned a p.L265P substitution within MYD88 and likely constitutes a false-positive stemming from the wet-lab sample handling (further detailed in the Online Supplementary Appendix). Overall, none of the four variants with a VAF <5% reported by a single institute using the Multiplicom assay were found by any other center utilizing the Illumina or HaloPlex methodologies; further supporting the notion that they are false positive results (Online Supplementary Table S10).

Pairwise comparison of the data generated from the centers utilizing the Illumina assay revealed that 128 variants were detected and could subsequently be assigned to the groups defined above as follows i.e., group 1 (n=100), group 2 (n=6), group 3 (n=19), group 4 (n=0) and group 5 (n=3) (Figure 2; Online Supplementary Table S9, S11). The concordance rate was 97.7% (125 of 128) with three variants detected by only one center; all three variants had a VAF <5% (Online Supplementary Table S9-10). The finding of 2 of 3 of these variants in the centers utilizing the other technologies portends to them being true subclonal mutations present in a minor proportion of leukemic cells (Online Supplementary Table S9).

A total of 127 variants were found using the HaloPlex assay; however, as three samples failed the sequencing run in center 5 (T28, T41 and T48) which collectively harbored eight variants, these samples (and their variants) were excluded from the pairwise analysis. The remaining 119 variants were distributed amongst the various groups as follows: group 1 (n=89), group 2 (n=5), group 3 (n=13), group 4 (n=7) and group 5 (n=5) (Figure 2; Online Supplementary Tables S9, S12). The concordance rate was 90% (107 of 119) with 12 variants detected by only one center; 7 of these 12 variants had a VAF >5% with the remaining 5 variants having a VAF <5%. Investigating the variants with a VAF > 5% (n=7) revealed that (i) 3 of these variants were found in all other centers indicating that they were false-negatives for center 6; (ii) two variants in T5 could be false-positives for center 5 (discussed in more detail in the Online Supplementary Appendix); and, (iii) for the remaining two variants (VAF 10% and 6.7%), one was not found by any other center while the latter was found at a very low frequency of 0.56% (Online Supplementary Table S9-10). Notably, the five variants found with a VAF <5% were all detected by the other methodologies.

Overall, the VAF reported were in agreement between partner centers and this similarity was evidenced by the narrow range of VAF for a particular variant, thereby indicating the accuracy of each assay in reporting VAF (Figure 2).

Figure 2.

Pairwise comparison of mutations detected by centers utilizing the Multiplicom, Illumina TSCA and HaloPlex assays. Mutation plots depicting the concordance and accuracy (narrow range) of the variant allele frequency (VAF) of variants found in this study. Variants are listed on the y-axis and variants detected by each center are denoted by different shapes. The VAF scale is on the x-axis and a VAF difference of less than 10% between partner centers is denoted by black shapes while red shapes indicate a VAF difference of greater than 10% between partner centers.

NFKBIE and EGR2 were only included in the Illumina and HaloPlex gene panel designs. Within the HaloPlex dataset a total of six mutations were reported in the NFKBIE gene, five of which were found in both institutes at a VAF >5%, with the remaining variant found either above or below 5% in the partner center (Online Supplementary Table S13). Thus, the concordance rate for NFKBIE mutations detected using the HaloPlex assay was 100%. A total of seven NFKBIE mutations were found by the Illumina assay. However, while four were concordant, being found in both centers and at a VAF >5%, the remaining three variants were only found by a single center (two with a VAF >5% and the remainder with a VAF <5%) (Online Supplementary Table S13). Two of the three variants were found by the centers performing the analysis with the HaloPlex technology, indicating that these mutations are unlikely to be false-positives in the single Illumina center and false-negatives in the second center is the more probable explanation.

Finally, the HaloPlex assay detected a total of five variants within the EGR2 gene; three variants were found with a VAF >5% by both centers while two variants were found by both centers with a low VAF (<5%) (Online Supplementary Table S13). Taken collectively, the concordance for EGR2 mutation detection reached 100% for the HaloPlex assay. In contrast, EGR2 proved to be the most difficult gene in this study to sequence when using the Illumina assay. Four variants were detected using the Illumina panel, however, only one was found by both centers, which, surprisingly, concerned a low frequency variant (VAF 2.3% and 2.2%) (Online Supplementary Table S13). Notably, two of the variants found by Illumina were not present in the HaloPlex datasets. This dropout of amplicons for EGR2 is supported by the coverage data (Online Supplementary Table S6). While reaching a definitive conclusion as to why this dropout occurred is not possible, drawing from our previous experience, reagents based on the older TSCA chemistry had a relatively short period within which they performed optimally i.e., while well covered regions maintained a high coverage when using reagents nearing or older than 3 months, the coverage for problematic regions such as the NOTCH1 PEST domain, NFKBIE, EGR2 and certain regions within the TP53 gene decreased dramatically.

Inter-laboratory variation in detecting mutations

We next assessed the reproducibility of targeted NGS by looking at the inter-laboratory variation in detecting mutations. As these assays were not specifically designed to detect variants with a very low VAF i.e., <5%, filtering was set such that a VAF of 5% had to be reached in at least one center for the variant to be included in the concordance counts. We found that 107 of 115 (93% concordance) mutations were consistently detected by all six participating centers (Online Supplementary Table S9). These 107 variants could be segregated into two groups: (i) 87 variants (81%) were found at a frequency >5% in all six centers; and, (ii) 20 variants (18.7%) were bordering the 5% cut-off, and, with the exception of sample T5, their VAF spanned a narrow range (1-12%) (Online Supplementary Table S9). This is illustrated in Figure 3 for variants detected within TP53, SF3B1, and NOTCH1. The remaining 8 of 115 (7%) variants were not detected by a single center i.e., the variants were successfully found in the sequencing data from 5 of 6 of the participating centers. Two of these variants went undetected in centers utilizing the HaloPlex technology and concerned frameshift insertions within the ATM gene (p.L1327fs [center 5] and p.V2201fs [center 6]) (Online Supplementary Table S9). Analysis of the raw sequencing data revealed that amplicon dropout leading to a gap in coverage resulted in these undetected ATMframeshift mutations. The final six mutations were present at a low frequency and were not found within the following centers sequencing datasets, center 4 (n=1), center 5 (n=2) and center 6 (n=3). These minor subclonal mutations were detected by the partner institute utilizing the same methodology, thus demonstrating the ability of the assay to detect and amplify in this genomic region.

Figure 3.

Heatmaps and boxplots illustrating mutations within TP53, SF3B1, and NOTCH1. Columns represent the six test centers (C1-C6) and rows correspond to a mutation found in a specific sample. Color coding indicates the type of mutation whereas the shading depicts the variant allele frequency (VAF) (lighter to darker tones correspond to an increasing VAF). Boxes marked with a hash symbol (#) indicate that the variant was not detected by that particular center. TP53 splice site mutations resulted in the following nucleotide changes: c.96+1G>T, c.673-2A>T, c.783-2A>C and c.920-2A>G). C1 & C2 (Multiplicom), C3 & C4 (Illumina TSCA) and C5 & C6 (HaloPlex).

Finally, a variant was deemed as a false-positive if it was only found in a single test center. Overall, four false-positives with a VAF ≥5% were detected in two centers and analysis of these variants led to the conclusion that it is highly probable that 3 of these 4 false-positive calls arose independent of the assay design or sequencing protocol. A full description of these variants is provided in the ‘Interpretation of false-positive findings’ section in the Online Supplementary Appendix and in the Online Supplementary Table S14.

High-sensitivity assay to confirm variant calling

In order to gain a better understanding of whether the low-frequency mutations were indeed true variants or false-positives arising from the library preparation or sequencing steps, the entire experiment was repeated using a custom-design HaloPlex high-sensitivity assay and gDNA from 38 of the CLL patients originally sequenced (additional material was unavailable for 10 of the original 48 samples). The HaloPlexHS system follows a similar workflow to the standard HaloPlex assay (described in the Online Supplementary Materials and Methods), however a cardinal feature is the attachment of a UMI to individual captured DNA molecules. During downstream analyses the UMI is used to collapse reads originating from the same molecule, thereby improving base calling accuracy and permitting accurate quantification of the mutant allele fraction by excluding PCR amplification bias and improving discrimination of variant nucleotides from background sequencing errors.

A median of 95.3% of bases within the targeted ROI achieved at least 100x coverage (Online Supplementary Table S15). Of the 128 somatic variants identified during the comparative analysis, excluding 18 mutations in samples not re-sequenced, 103 variants were confirmed (VAF range: 0.5-100%) (Online Supplementary Table S9). No additional variants were identified in the repeat NGS data that were not identified previously, while a few low-frequency variants (VAF <5%) could not be verified. The high-sensitivity assay data also enabled us to investigate the variants found in only a single center (described above) and provided further evidence that variants detected by only a single center were indeed false-positives (Online Supplementary Table 10).

Discussion

As the number of genes with diagnostic, prognostic, or predictive significance increases, there is a need for robust assays that detect multiple alterations simultaneously. Whether for research, in order to provide a better understanding of the molecular mechanisms driving disease pathogenesis and evolution, or within clinical diagnostics, targeted gene panels in combination with NGS have evolved as a pragmatic cost-effective approach. These assays are also uniquely positioned to yield a volume of data that is more manageable and easier to interpret than the complex datasets generated by WES or WGS. As a consequence, targeted gene panels are becoming increasingly popular in the cancer molecular diagnostics arena and systems for implementing NGS within routine clinical practice need to be established. This is essential since, despite its numerous attractive attributes, mutational profiling by NGS can be challenging and involves workflows comprising several distinct parts, i.e., wet-bench components, bioinformatics analyses, and clinical interpretation of the variant calls. In addition, with the abundance of assays and sequencing platforms on the market, all with their own technical and methodological specifications, ensuring test-to-test reproducibility and inter-laboratory reliability is fundamental.

Several regulatory and professional organizations have published guidelines and best-practice recommendations to assist laboratories in the transition from the previous gold-standard methodology of Sanger sequencing for mutation detection to NGS.[33-37] Whilst these reports focus on critical aspects of clinical gene sequencing such as documentation of the protocols and reporting the specificity and sensitivity of an assay, comparative analyses of different enrichment techniques and workflows are rare.[38,39] Additional inter-laboratory studies focusing on variant calling comparisons between different methodologies are therefore required. To aid in this endeavor, the present study compared the ability of different NGS technologies to accurately detect a spectrum of mutations with varying VAF within genes of prognostic relevance in CLL, specifically focusing on the sensitivity and reproducibility of targeted gene panels. Aside from the particular gene panel utilized in the various test centers, we kept parameters as uniform as possible including use of the same sequencing system, same patient samples etc. In order to avoid confounding results due to individual customization of bioinformatic pipelines at participating institutes i.e., combinations of aligners, variant callers, differences in filtering parameters and quality control, bioinformatics was performed centrally.

A first critical step in our analysis was to compare the metrics of sequence coverage and depth. Variability in depth of coverage between centers using different technologies could be attributed to the differing size of the panel designs i.e., despite targeting the same regions, the HaloPlex design includes redundancy to ensure targets are covered even if an amplicon drops out, whereas the Multiplicom technology sequenced fewer samples per run. While incomplete coverage was not a consistent problem for any of the methods utilized, considerable differences in the coverage of EGR2 and NFKBIE were observed, particularly when using the TSCA gene panel. Certain regions within these genes failed to amplify and had lower read depth in comparison to other genes targeted indicating that they were intrinsically more difficult to amplify. This may stem from the varying ability of the different probe and primer sets to anneal in these high GC regions, thus leading to a reduction in the efficiency of target capture or amplification. These difficulties may have been exacerbated by the decline in the TSCA chemistry over time. As a final note on coverage, direct comparison of the sequencing depth obtained from panels designed with/without UMI is not meaningful since the number of reads does not reflect the actual number of unique template gDNA molecules as many reads will be duplicates generated during PCR. While molecular barcodes do not prevent PCR duplication from occurring they do facilitate their removal and hence the overall coverage can appear lower albeit more accurate.

Next, the ability of disparate NGS methods to detect variants with varying allelic frequencies was compared and we observed a high degree of concordance and accuracy when performing pairwise analysis. Imposing an arbitrary VAF cut-off i.e., 10% or 5%, at the initial stage of the analysis could provide an inaccurate view of concordance as variants borderline of a threshold may appear to be discrepant, hence we first looked at the agreement between variants irrespective of their VAF and yielded concordance rates of 96.2% (Multiplicom), 97.7% (Illumina) and 90% (HaloPlex). We next assessed the reproducibility of targeted NGS by looking at the interlaboratory variation and found that 107of 115 mutations (93% concordance) were consistently detected by all six participating centers. Another encouraging finding was the accuracy or similarity i.e., narrow range, amongst the VAF recorded from all centers. Notable exceptions to the narrow range of VAF pertained to EGR2 and NFKBIE, which provided less homogeneous results, especially between different methodologies and, as mentioned above, these differences likely originate from the suboptimal coverage obtained in certain regions of these genes.

There is currently no data that definitively determines what VAF cut-off is clinically significant, indeed this cutoff may vary depending on the particular gene in question, and the clinical relevance of any gene mutation can only be deemed by a clinical trial demonstrating a significant association with outcome. That said, a VAF cut-off of 5% or 10% is currently widely used in clinical routine. In this study, several variants were observed at VAF’s bordering the 5-10% threshold and hence would have been removed had a hard filtering step been applied. From a technical perspective, a plausible strategy to prevent clinically significant calls from being removed due to stringent cut-offs is to flag relevant variants as hotspots within the bioinformatic pipeline thus increasing the sensitivity by reducing the VAF threshold for that specific variant. Noted exceptions to this work-around would concern genes without defined hotspots i.e., tumor suppressor genes, within which, in theory, any mutation could be detrimental e.g., TP53 and ATM. In order to circumnavigate the VAF/cut-off filtering issue in these instances, one solution would be to lower the technical cut-off of the method and avoid setting a hard cut-off for reporting and implementing a review and visualization step, thus removing the possibility of filtering out a potentially relevant mutation.

Owing to the heterogeneous subclonal composition of tumors, together with ongoing clonal evolution, clinical samples are often fraught with somatic mutations that exist at low VAF.[26,28,29,40] As evidence accumulates it is becoming apparent that these low-level variants, at least within certain genes, may contribute to disease pathogenesis or serve as early indicators of resistance to specific therapies.[21-24,41-44] Despite data supporting the importance of somatic mutations with low VAF, their detection by conventional NGS is challenging. Even at a high read depth, NGS shows a marked decrease in the detection of somatic mutations with low VAF and accurate separation of erroneous variants from true low-level variants cannot always be achieved by stringent filtering alone. We performed an additional round of testing using a high sensitivity assay containing UMI as a comparator method. This decision was made for two reasons: (i) to obtain the most accurate estimate of VAF; and, (ii) to confidently detect low-frequency variants. When exploring the validity of minor variants using the dataset generated by the high sensitivity assay as a validator we found that although low frequency variants were detected by all techniques, greater diversity was observed for mutations with VAF <5%. While these results advocate the use of ampliconbased approaches for mutation detection within a clinical or research setting, the need for stringent validation should not be underestimated and is essential prior to the implementation of assays into routine. In order to further help with this transition, ERIC has recently initiated a multi-center project specifically focusing on low-frequency TP53 mutations with the aim to reach a consensus on whether the current cut-off of 10% for reporting TP53 mutations should be lowered.[45]

Our study is not without limitations, one of which relates to the fact that the mutational status of all genes to be analyzed was not known prior to their inclusion in the study. While desirable, this was not possible due to the difficulty in trying to obtain samples that had sufficient material to be distributed to all centers. However, this limitation was mitigated by performing an additional sequencing round using a high-sensitivity assay incorporating UMI that was independent of the three gene panels being assessed. Another limitation related to the determination of the sensitivity level for each assay included in the study. Although our study was focused on the comparability and reproducibility of results obtained in different centers using various amplicon-based assays, commercial controls harboring variants and allelic frequencies that have been well-characterized, would have been beneficial in determining the reliable limits of detection (which can differ depending on the specific variant). In addition, read counts from such controls can aid in the assessment of run variability. For the validation and use of NGS within a clinical laboratory service, sensitivity control samples with a range of VAF for target regions should always be included to ensure that the validated lower limit of detection is maintained while controls harboring specific variants and types of variants verify assay performance.

In conclusion, the data herein provides strong evidence that distinct gene panel designs and workflows have high analytical specificity and sensitivity and are capable of making consistent variant calls. Importantly, this assessment of interlaboratory reliability is not only paramount in the molecular diagnostics setting but also impacts on research where combining data generated from centers globally is crucial to provide homogeneous, reliable results and ultimately improve our understanding of rarer gene mutations, thus facilitating stratification of patients based on their molecular profiles. As NGS technologies continue to advance and assays extend beyond variant calling to include copy-number analysis and allow for the robust identification of clinically relevant low-frequency variants, diagnostic and research laboratories need to remain flexible so as to adapt in this dynamic era of precision medicine.