Artificial Intelligence Approach for Variant Reporting.

Purpose: Next-generation sequencing technologies are actively applied in clinical oncology. Bioinformatics pipeline analysis is an integral part of this process; however, humans cannot yet realize the full potential of the highly complex pipeline output. As a result, the decision to include a variant in the final report during routine clinical sign-out remains challenging.
Methods: We used an artificial intelligence approach to capture the collective clinical sign-out experience of six board-certified molecular pathologists to build and validate a decision support tool for variant reporting. We extracted all reviewed and reported variants from our clinical database and tested several machine learning models. We used 10-fold cross-validation for our variant call prediction model, which derives a contiguous prediction score from 0 to 1 (no to yes) for clinical reporting.
Results: For each of the 19,594 initial training variants, our pipeline generates approximately 500 features, which results in a matrix of > 9 million data points. From a comparison of naive Bayes, decision trees, random forests, and logistic regression models, we selected models that allow human interpretability of the prediction score. The logistic regression model demonstrated 1% false negativity and 2% false positivity. The final models' Youden indices were 0.87 and 0.77 for screening and confirmatory cutoffs, respectively. Retraining on a new assay and performance assessment in 16,123 independent variants validated our approach (Youden index, 0.93). We also derived individual pathologist-centric models (virtual consensus conference function), and a visual drill-down functionality allows assessment of how underlying features contributed to a particular score or decision branch for clinical implementation.
Conclusion: Our decision support tool for variant reporting is a practically relevant artificial intelligence approach to harness the next-generation sequencing bioinformatics pipeline output when the complexity of data interpretation exceeds human capabilities.

INTRODUCTION

Over the past decade, several areas in clinical oncology have been revolutionized by cancer genotyping.[1-3] And next-generation sequencing (NGS) technologies will continue to play an integral part of precision medicine.[3-6] From an analytic perspective, the amount and complexity of raw NGS data require sophisticated pipelines (Fig 1) for read alignment, variant calling, and annotation, each of which may be associated with error and noise; this is also true for the sequencing instrumentation itself. Ultimately, these pipelines procure a text file containing a list of sequence variants, called a variant call format (VCF) file, wherein each variant is listed with a set of features (ie, annotations). These annotations span hundreds of columns and include functional predictions, variant calling metrics, frequencies in public databases, and/or clinical implications. Molecular pathologists/geneticists interpret these VCF files and integrate selected data points when issuing a clinical report (Fig 1).

Fig 1.

The complexity of variant reporting in clinical practice. (A) The amount and complexity of raw next-generation sequencing (NGS) data requires NGS pipelines for read alignment, variant calling, and variant annotation to provide a (filtered) variant call format (VCF) file for manual review by a pathologist/geneticist. The reporting decision is a complex process that requires experience, involves management of the VCF file and various resources, and ultimately results in a reporting decision. (B) Distribution of tumor types included in the variant training data set (V1). Variants are represented in 37 principal tumor types that combine 383 histologic subtypes. (C) After manual review of 19,594 variants, only 24% (n = 4,787) are reported, and 76% of the review effort is not captured in the final report. (D) The reporting fraction by site (left) and gene (right) shows considerable variation (range, 0% to 100%). (E) The effect of the variant reporting decisions illustrated on a variant frequency matrix; green bars represent the number of variants within each disease site or gene. We used the formula "all" minus "no" equals "yes." Specifically, the filtered pipeline output represents "all" reviewed variants and after subtraction of the variants that received "no" calls (ie, are vetted not to be included in the report), the resulting matrix shows the variant frequencies by gene and site in the final report (ie, "yes" calls). The resulting "yes" matrix is similar to that in recent publications[7]; however, in clinical practice, pathologists/geneticists are confronted with all data ("all" matrix on the left). The portrayed distribution of variants by gene and site represents only two of approximately 500 pipeline features attached to each variant. The full pipeline output and the dimensionality of interrelations exceed the human ability to handle all available data efficiently. AD, adenocarcinoma; BAM, binary alignment map; CRC, colorectal cancer; CUP, carcinoma of unknown primary; EGC, esophagogastric cancer; GIST, GI stromal tumor; Heme, hematologic malignancies; LCNEC, large-cell neuroendocrine carcinoma; NE, neuroendocrine carcinoma; Non-Ca, nonepithelial malignancy; NSCLC, non–small-cell lung cancer; PDAC, pancreatic cancer; QC, quality control; SAM, sequence alignment map; SCLC, small-cell lung cancer; SQ, squamous cell carcinoma.

Knowledge derived from the literature and public databases provides comprehensive disease-variant associations.[7] By using clinical practice data, we can achieve similar disease-variant associations as exemplified by a two-feature representation (disease sites and reported variants by gene; Fig 1). The final condensed matrix does not, however, fully represent the complex reporting process and does not aid the molecular pathologist/geneticist with the day-to-day, case-by-case, and variant-by-variant reporting decision.

Numerous approaches[2,8-10] and guidelines for variant calling,[11-13] annotation,[12,14,15] and interpretation[8,9,16-18] have been proposed. Ultimately, these strategies share the common end goal of the pathologist/geneticist rendering a seemingly basic decision: whether to report a variant. To our knowledge, the binary reporting decision has not been captured or used systematically.[8-10,16,19] In particular, the decision to not report a variant (Figs 1C and 1E; Data Supplement) is equally valuable because the same variant may be found in subsequent cases. In times of escalating costs, building on prior knowledge may save time and effort. Comprehensive pipeline results may help, but the reporting decision remains difficult relative to the context of the called variant (eg, presence of a mutation at low levels, variants at sites of private polymorphisms, unknown functional consequences of variants, splice site variants). To render the reporting decision, a board-certified pathologist/geneticist uses experience with numerous features distributed over many columns to render a reporting decision (called"vetting"; Fig 2A). Some full pipeline outputs consist of > 500 dimensions (or features) for each variant. In other words, for a clinical NGS laboratory, the integration and processing of the full potential of the entire pipeline output as it relates to the reporting decision exceed human capabilities.[8,19,20]

Fig 2.

Performance assessment of the artificial intelligence model for variant reporting. (A) Concept of a decision support tool for variant reporting. Current practice (top) is shown with the tested implementation (bottom). The artificial intelligence/machine learning model was built on the basis of prior human reporting decisions. Note that the implemented model provides a reporting decision for each variant on a scale from 0 (no) to 1 (yes) without regard for potential clinical actionability; contextual or clinical consequences (eg, oncology knowledge database) have been excluded intentionally, and we have addressed the topic in prior studies.[16] (B) The number of calls in the aggregate model (by using a naive threshold of 0.5) as well as distribution of no and yes calls per pathologist (A to F). (C) Distribution of 19,954 model scores in the reported and not reported variants. Two call thresholds illustrate two use cases: (1) a more-sensitive 0.25 threshold with fewer false-negative (FN) results (n = 150) and (2) a more-specific 0.75 threshold with fewer false-positive (FP) results (n = 323). (D) Receiver operating characteristic curves with selected performance metrics for model threshold scores of > 0.01 (screening test) and > 0.9 (confirmatory test). (E) The FN rate decreases with increasing prevalence; however, for several genes, the model performance is excellent despite low prevalence (eg, ALK). (F) Specificity over sensitivity (red) and precision over recall (blue) for the aggregate and individual models (A to E). The outline of all individual models can be viewed as the model-based performance spectrum of the examined group practice composed of six pathologists (A to E). TN, true negative; TP, true positive; Sens., sensitivity; VCF, variant call format.

Artificial intelligence is one approach to mine big data and derive models for decision making.[21-23] Common repetitive tasks in medicine are amenable to modeling by artificial intelligence[21-23]; however, computers are notoriously poor at understanding unstructured data (eg, medical notes).[21,23-25] In contrast, artificial intelligence tools for structured data are available and have already surpassed human performance in many areas.[25-31] Of note, bioinformatics pipelines generate mostly structured, discrete data. Thus, we consider the inability of humans to gain access to the full potential of the pipeline output, coupled with the discrete nature of the data and the final binary reporting decision, as an ideal setting[32] to assess the performance of an artificial intelligence–based decision support system for variant reporting.

On the basis of our experience with implementing clinical genotyping[33,34] that uses NGS technologies,[35] we report on how we aligned big clinical sequencing data with human decisions in our group. As a result, we established an artificial intelligence approach for variant reporting and describe its performance in two independent data sets from routine clinical practice.

METHODS

Design, Regulatory Approval, and Clinical Setting

The project was undertaken as a retrospective analysis of existing data obtained as part of routine cancer care in a clinical molecular genetics laboratory. All patients provided written informed consent for molecular genotyping. Institutional review board approval was obtained (protocol 2014P000940). Details about the project site, case volume, and assays can be found in Zheng et al[35] and the Data Supplement.

NGS Assays and Bioinformatics

For sequence analysis, we used our laboratory-developed, Clinical Laboratory Improvement Amendments–validated NGS bioinformatics pipeline. We used Illumina (San Diego, CA) MiSeq (V1) or NextSeq (V2) instruments. We performed tumor-only sequencing, and the 2 × 151–base paired end sequencing results were aligned to the hg19 human genome reference by using Burrows-Wheeler Aligner MEM.[36] Variant calling for V1 (Data Supplement) was performed with MuTect version 1.1.7 for single-nucleotide variants,[37] and Oncotator (version 1.2.10.0)[38] was used for variant annotation. For V2 (Data Supplement), we used Novoalign (www.novocraft.com) for read alignment and an ensemble variant calling approach, including MuTect1, LoFreq, Genome Analysis Toolkit, and a laboratory-developed hotspot caller, for variant detection. The variant effect predictor tool was used for variant annotation.[39] A detailed standard operating procedure is available upon request; a description of the variant scoring and features used for analysis in V1 and V2 are provided in the Data Supplement.

Machine Learning and Classifier Selection

Python version 2.7.11, Pandas version 0.18.1, NumPy version 1.11.0, and scikit-learn version 0.17.1 libraries (Python Software Foundation, Wilmington, DE) were applied for model building. By following recommendations by Kohavi[40] for real-world data sets, the best method to use for model selection is 10-fold stratified[40-44] cross-validation.[42,43] For selection of the most accurate classifier,[42] we compared area under the curve (AUC) values for naive Bayes, logistic regression, decision trees, random forests (depth of trees, 10, 15, 50, and 100), and support vector machines (scikit-learn library).[45] We refer to artificial intelligence model or model as the selected predictor/classifier for implementation (Data Supplement).

Aggregate and Individual Models

In our clinical practice, each case is signed out by one pathologist (detailed call frequency by gene and disease site across all pathologists are provided in the Data Supplement). For the aggregate model, we trained and tested by using all reporting call decisions as labels (ie, not included as a feature in the model). For the individual models, we filtered the training sets to include only calls made by one pathologist (Fig 2B); the six resulting models (A to F) can be regarded as representative of each individual pathologist’s reporting practice. In addition, we assessed transferability of the approach to a new assay (T1/T2 experiments [ie, trained model on variants V1 and tested on V2]) and validated the performance in an independent data set (V2; Data Supplement).

Statistical Analysis

We defined reporting rates of pathologists as the number of yes decisions over the total number of decisions. To account for different use cases, we describe the model performance by using different score cutoffs (naive v outliers v screening v confirmatory; Data Supplement). To assign P values to individual features, we used univariable analysis of variance testing (Data Supplement). We assessed the performance of the model by comparing the model-based predictions versus the original clinically reported variant calls and provide precision, recall, sensitivity, specificity, F1 score, and Youden index. Statistical significance was defined as P < .05.

RESULTS

Data Extraction Results

An overview of the data set V1 and the validation data set V2 is listed in Table 1. In V1, our pipeline assigns 507 features to each variant (Data Supplement). The reporting pathologist/geneticist could be regarded as a feature; however, this feature was not included because it is not available when confronted with a new variant. We provide however, the number of cases, variants, and yes call rates by pathologist (Table 2). The number of reviewed variants (approximately five per case) and the number of calls per case (fewer than two plus or minus two per case) did not differ significantly (Table 2). In other words, despite some variation in case and/or variant exposure of every pathologist (Table 2; Data Supplement), we considered the call rates to be internally consistent and the aggregate total variant number as representative of our sign-out practice.

Table 1.

Overview of the Data Sets

Table 2.

Number of Cases, Variants, and Calls by Pathologist in V1

Selection of the Machine Learning Classifier

We compared the AUCs for several different predictor methods (Data Supplement). In V1, we selected a logistic regression model (over the equally predictive random forest classifier), and in V2, we selected random forests as the top performing model (Data Supplement). Specifically, although each model derived a single number, we also considered human interpretability; we regard the individual coefficients or decision branches for each feature as an additional benefit for manual review (transparency of the model).

Results From Implementing the Model

The prediction model assigns a score from 0 to 1 that represents a continuous scale from the ground truth reporting calls (no to yes, respectively). Review of the frequency distribution of calls for all 19,594 variants by model score showed that the model assigned the majority of scores near 0 and 1 (Fig 2B, top). To allow comparison of how each pathologist contributes to the aggregate model, we plotted individual model scores (Fig 2B; Table 3) and assessed several call thresholds (Figs 2B and C; Data Supplement). When plotting the true-positive rate over the false-positive rate for all existing model score cutoffs as a receiver operating characteristic curve (Fig 2D), the AUC can be interpreted as the probability that a randomly selected positive variant in the test data receives a higher score from the model than a randomly selected negative variant. The resulting receiver operating characteristic curve shows an AUC of 0.990, which indicates an almost perfect prediction model (Table 3, aggregate model V1), and we provide the specific test characteristics and case numbers of a screening cutoff at 0.01 and a confirmatory cutoff at 0.9 (Fig 2D).

Table 3.

Performance of the Aggregate and Individualized Models

Clinical Implementation

We implemented the model for screening (cutoff > 0.01) and observed that the false-negative rate decreases with the overall variant prevalence by gene (Fig 2E), which is related to our call practice (Fig 2B; Data Supplement). For example, SMO variants in our setting are overall rare (n = 7) yet frequently reported (n = 5; Data Supplement). As a consequence, the one false-negative call by the model results in a relatively high false-negative rate (15%). However, despite these limitations, the overall error and false-negative rate is only approximately 3%. We used the same approach to derive individual models for each pathologist (Table 3; Figs 2D and F), which illustrates three things. First, the aggregate model outperforms most individual models. Second, the performance among all individual models differs only by 0.8% (AUCs, 0.982 to 0.990). Third, our approach to derive a model for every pathologist suggests that there are only small differences in sign-out practice for variant reporting that are reflected in the six models (akin to a consensus difference). Figure 2F shows the model-based landscape of our variant reporting practice.

Validation Experiments

The launch of a new assay in July 2016 provided an opportunity to transfer our artificial intelligence model from the V1 assay to the new V2 assay, and in doing so, to validate our approach. We used V2 as an independent data set that contained 16,123 consecutive variants from clinical practice (V2; Table 1). The performance measures (Table 3; Data Supplement) and the Youden index of 0.927 validate the machine learning model as robust in capturing our sign-out experience with respect to reporting decisions.

Transferability Experiments

The launch of a new assay also triggered the following question: What would happen to the model performance if we intentionally transferred the wrong model? We consider this transferability question to be highly relevant because any new assay (or laboratory test) will have slightly different frequencies of cancer types, genes, and so forth. As an analogy, when a pathologist/geneticist starts to work with a new assay, he or she would need to retrain and gain familiarity with the assay to confidently sign out cases. We designed transferability experiments to assess the importance of retraining and the risk imposed by wrongful artificial intelligence model transfer (Data Supplement). The lower performance (up to −35%) is no surprise and related to substantially different panels; however, some performance measures are preserved (precision, approximately 0.9; Table 3). These findings indicate that some variants are shared between the assays and that transfer of artificial intelligence models requires careful multiparametric performance evaluation. The validation experiments (V1 v V2) confirmed transferability of the machine learning approach, whereas the transferability experiments (T1/T2) assessed the transferability of the model. These experiments emphasize that pathologists’ calls are necessary to retrain a model and that the approach can be accurately transferred to capture reporting decisions on an entirely different assay.

Implementation in Clinical Practice

To illustrate how we implemented the models in clinical practice, we provide screenshots of two variant review popup tools. First, a consensus tool (Fig 3A) allows for the review of the reporting prediction scores to augment variant reporting decisions and instantly access the diversity of reporting suggestions along with a drill-down functionality that allows review of the features that contributed to a particular score. These functionalities capture two key aspects of a traditional consensus conference: the decision and the reasoning. Second, we implemented a tree map visualization of the decision branches and the underlying reasoning (Fig 3B). The added level of transparency was a core design component that empowers the reviewing pathologist/geneticist to understand the underlying reasoning; in other words, the model provides the user with a justification for a given decision.

Fig 3.

Model decision exploration in clinical practice. (A) Screenshot shows our variant review and graphic user interface used to select variants for inclusion in the report (background; old assay, V1). The inset shows individual pathologists' model scores (P1 to P6) and the aggregate. When hovering over one model, the drill-down option shows the top five predictors derived from the logistic regression pathologist's model that contributed to the report recommendation (report). (B) Screenshot shows our variant review and graphic user interface used to select variants for inclusion in the report (background; new assay, V2). The machine learning (ML) score links out to the ML tree module, which allows for exploration of 15 random forest decision branches. Each branch contains the order of contributing features and findings that resulted in the decision (green argues for reporting, red against). Each circle represents one feature, and the drill-down option (inset) shows the feature (eg, a quality control [QC] metric of a caller), the finding in this variant (eg, 1), and the cutoff used by the model (here > 0.5). The added level of transparency that allows review of the features that underlie a model-derived decision is an important design component of our implementation in clinical practice, and we propose the term next-generation decision support. CADD, combined annotation dependent depletion; LOFREQ, low frequency; SNV, single-nucleotide variant.

DISCUSSION

We present an artificial intelligence approach for genetic variant reporting in cancer. The core design idea is to use artificial intelligence to fully exploit the entire bioinformatics pipeline output for modeling the reporting decision. We combined the structured variant annotations with the collective, multiyear reporting experience of six pathologists in a clinical molecular diagnostic laboratory. By using > 19,000 variants, we derived substantial performance metrics with AUC > 99%, and we validated the approach in > 16,000 independent variants in another assay with similarly high performance metrics. We also share two clinically relevant visualization tools (a consensus and a tree map visualization module) that allow human exploration of the underlying (supporting) reasons by the artificial intelligence model. This approach is a practical example of how to apply artificial intelligence meaningfully when the complexity of data exceeds human capabilities.

Artificial intelligence and machine learning as tools have been established,[40,42-45] and in our model design,[32] we aimed for two things: to generate a single score output as a contiguous score from 0 to 1 as a simple way to recognize the decision and to give the reviewing pathologist/geneticist the opportunity not only to review the decision but also to explore the underlying reasoning by the model (Fig 3). We offer this additional layer of transparency as a drill-down function and propose the term next-generation decision support tool. Given that the model takes the full set of variant calling pipeline output into consideration, we foresee review of these features that underlie a reporting decision as valuable (eg, machine-based education of the reviewing pathologist). Unfortunately, when machines or programs mimic components of the cognitive functions that humans attribute to learning, the disparity between intended use and public perception generates unnecessary confusion. For example, in science fiction, the term artificial intelligence typically is applied to excite ethical and emotional reactions. Thus, we want to re-emphasize the critical importance of thoughtful design and that we limited our approach to the binary reporting decision. Thus, the intended use of our model is not to substitute human decision making. Comparison of human versus model decisions enables the pathologist to decide whether the variant should be re-reviewed by an independent pathologist or presented at a consensus conference. Thus, the tool may increase efficiency and reinforce human interaction. In other words, we built, implemented, and validated a model that continues to rely on unbiased pathologist calls; this small integral design part literally means: no pathologist, no model.

Future adoptions may extend to incorporate other data types (eg, histopathologic image features),[46] other artificial intelligence approaches (ie, neural networks, deep learning),[47,48] or the transfer to other laboratories or assays (as demonstrated in our transferability experiments). Despite the overall promising nature of our findings, numerous limitations apply. The model currently does not distinguish somatic from germline variants for prediction, and we did not include other forms of variants (ie, indels, copy number variation). With regard to implementing our approach in other clinical laboratories, limitations exist in terms of different frequencies of diseases or insufficient clinical volumes to drive model building. Our model is restricted to the collective experience of our group (eg, we do not sequence matched normals). Confounders and biases may exist in our model with regard to such nuances as the ethnicity or ancestry of our testing population compared with the reference genome used for variant calling. Furthermore, the launch of a new assay shows evolution of performance over time, and our transferability experiments demonstrate that this also applies to an artificial intelligence model because the model is, by definition, assay, instrument, and pipeline specific. Although these limitations can be perceived as disadvantageous, they are not specific to our own practice and are well acknowledged in the artificial intelligence and machine learning community.[20,23,25,49] Despite these limitations, we have shown that although models may be assay and setting specific (V1 v V2; Table 3), the application of artificial intelligence for decision support is indeed a practically feasible and clinically relevant strategy.

To the best of our knowledge, our artificial intelligence model as a decision support tool for variant reporting is new. Given the performance metrics of the model, appropriate model cutoffs (< 0.01) significantly reduce the number of variants to be reviewed (Fig 2D). Thus, we view the deployment of an artificial intelligence model in a clinical environment as a tool to filter out nonreportable variants and to enable exploration of the underlying reasoning as highly relevant because it makes pipeline result vetting reasonable timewise and scalable in terms of content and for increased numbers of clinical samples.

The implicit understanding that big data efforts will ultimately improve decision making in health care entails optimization of the care coordination process. We consider the complexity of NGS bioinformatics pipeline outputs as an opportunity to apply artificial intelligence because the complexity exceeds human capabilities. In times of increasing health care costs, tools for increased efficacy are in demand, and we hope that by sharing our approach to leverage the power of artificial intelligence and apply it to genetic variant reporting in cancer, other groups will be motivated to explore this emerging field and improve patient care.