Criteria for the use of omics-based predictors in clinical trials.

The US National Cancer Institute (NCI), in collaboration with scientists representing multiple areas of expertise relevant to 'omics'-based test development, has developed a checklist of criteria that can be used to determine the readiness of omics-based tests for guiding patient care in clinical trials. The checklist criteria cover issues relating to specimens, assays, mathematical modelling, clinical trial design, and ethical, legal and regulatory aspects. Funding bodies and journals are encouraged to consider the checklist, which they may find useful for assessing study quality and evidence strength. The checklist will be used to evaluate proposals for NCI-sponsored clinical trials in which omics tests will be used to guide therapy.

Main

High-throughput ‘omics’ technologies hold great promise to provide detailed characterization of diseases to more effectively predict a patient’s clinical course or to select the most beneficial therapies (see Box 1). These technologies have been embraced enthusiastically in oncology, as the heterogeneous character of malignant diseases presents substantial challenges for cancer detection, prognosis and optimal selection of therapy. Many preclinical studies using these technologies to elucidate biological features and mechanisms have been published, and retrospective studies applying omics assays to stored human biospecimens have been conducted to develop mathematical models to predict clinical endpoints such as survival or response to therapy.

Despite numerous publications, however, few omics-based predictors have been translated successfully into clinically useful tests. A factor that contributes to the slow pace of clinical translation is the challenge of assessing whether the body of evidence for an omics-based test is sufficiently comprehensive and reliable that the test is ready for definitive evaluation in a clinical trial in which it could be used to direct patient care. Translation from research-grade omics assays to clinical-grade omics-based tests[1] requires a rigorous development and validation process with attention to the complexities of omics assays and their application to clinical specimens, specialized expertise required to appropriately develop and evaluate mathematical predictor models built from high-dimensional data, and multiple ethical, legal and regulatory issues.

Recently there have been some widely publicized cases of premature advancement of omics-based tests to use in trials in which they were used to guide patient treatment decisions. These cases led to calls for examination of the field of translational omics. The Institute of Medicine (IOM) conducted a study[1] to review the field and formed the Committee on the Review of Omics-Based Tests for Predicting Patient Outcomes in Clinical Trials. The group’s task statement included recommending an evaluation process for determining when omics tests are fit for use in clinical trials and applying it to several specific cases of premature use of omics-based tests[1]. The resulting report laid out a three-phase process for the development and evaluation of omics-based tests for use in clinical trials: the discovery phase, the test validation phase, and the evaluation for clinical utility and use stage.

During the IOM committee deliberations, the NCI convened a workshop to bring together scientists and stakeholders who had an interest in this area of research to stimulate community dialogue. Subsequently, a working group was formed to develop a checklist that would operationalize the principles set forth in the IOM report and the NCI workshop discussions.

The results of those efforts are presented in Table 1, which lists 30 criteria that should be addressed to determine the readiness of an omics test for use in a prospective clinical trial. These criteria apply to any clinical trial involving the investigational use of an omics test that will influence the clinical management of patients in the trial; for example, the selection of therapy. These criteria cover not only the strength of evidence in support of an omics test but also the practical issues that must be considered before the test is used in a clinical setting. The criteria can also be helpful in assessing the reliability and credibility of an omics predictor to justify its use on valuable non-renewable archived specimens collected from patients who were prospectively enrolled in previous clinical studies. This paper presents the criteria in checklist form with brief background. Readers are referred to a recently published companion paper[2] for a more complete explanation and elaboration of the rationale for each criterion.

Table 1

Criteria for the use of omics-based predictors in NCI-supported clinical trials

Domain	Criteria
Specimen issues	1. Establish methods for specimen collection and processing and appropriate storage conditions to ensure the suitability of specimens for use with the omics test.
	2. Establish criteria for screening out inadequate or poor-quality specimens or analytes isolated from those specimens before performing assays.
	3. Specify the minimum amount of specimen required.
	4. Determine the feasibility of obtaining specimens that will yield the quantity and quality of isolated cells or analytes needed for successful assay performance in clinical settings.
Assay issues	5. Review all available information about the standard operating procedures (SOPs) used by the laboratories that performed the omics assays in the developmental studies, including information on technical protocol, reagents, analytical platform, assay scoring, and reporting method, to evaluate the comparability of the current assay to earlier versions and to establish the point at which all aspects of the omics test were definitively locked down for final validation.
	6. Establish a detailed SOP to conduct the assay, including technical protocol, instrumentation, reagents, scoring and reporting methods, calibrators and analytical standards, and controls.
	7. Establish acceptability criteria for the quality of assay batches and for results from individual specimens.
	8. Validate assay performance by using established analytical metrics such as accuracy, precision, coefficient of variation, sensitivity, specificity, linear range, limit of detection, and limit of quantification, as applicable.
	9. Establish acceptable reproducibility among technicians and participating laboratories and develop a quality assurance plan to ensure adherence to a detailed SOP and maintain reproducibility of test results during the clinical trial.
	10. Establish a turnaround time for test results that is within acceptable limits for use in real-time clinical settings.
Model development, specification, and preliminary performance evaluation	11. Evaluate data used in developing and validating the predictor model to check for accuracy, completeness, and outliers. Perform retrospective verification of the data quality if necessary.
	12. Assess the developmental data sets for technical artefacts (for example, effects of assay batch, specimen handling, assay instrument or platform, reagent, or operator), focusing particular attention on whether any artefacts could potentially influence the observed association between the omics profiles and clinical outcomes.
	13. Evaluate the appropriateness of the statistical methods used to build the predictor model and to assess its performance.
	14. Establish that the predictor algorithm, including all data pre-processing steps, cutpoints applied to continuous variables (if any), and methods for assigning confidence measures for predictions, are completely locked down (that is, fully specified) and identical to prior versions for which performance claims were made.
	15. Document sources of variation that affect the reproducibility of the final predictions, and provide an estimate of the overall variability along with verification that the prediction algorithm can be applied to one case at a time.
	16. Summarize the expected distribution of predictions in the patient population to which the predictor will be applied, including the distribution of any confidence metrics associated with the predictions.
	17. Review any studies reporting evaluations of the predictor’s performance to determine their relevance for the setting in which the predictor is being proposed for clinical use.
	18. Evaluate whether clinical validations of the predictor were analytically and statistically rigorous and unequivocally blinded.
	19. Search public sources, including literature and citation databases, journal correspondence, and retraction notices, to determine whether any questions have been raised about the data or methods used to develop the predictor or assess its performance, and ensure that all questions have been adequately addressed.
Clinical trial design	20. Provide a clear statement of the target patient population and intended clinical use of the predictor and ensure that the expected clinical benefit is sufficiently large to support its clinical utility.
	21. Determine whether the clinical utility of the omics test can be evaluated by using stored specimens from a completed clinical trial (that is, a prospective–retrospective study).
	22. If a new prospective clinical trial will be required, evaluate which aspects of the proposed predictor have undergone sufficiently rigorous validation to allow treatment decisions to be influenced by predictor results; where treatment assignments are randomized, provide justification for equipoise.
	23. Develop a clinical trial protocol that contains clearly stated objectives and methods and an analysis plan that includes justification of sample size; lock down and fully document all aspects of the omics test and establish analytical validation of the predictor.
	24. Establish a secure clinical database so that links among clinical data, omics data, and predictor results remain appropriately blinded, under the control of the study statistician.
	25. Include in the protocol the names of the primary individuals who are responsible for each aspect of the study.
Ethical, legal and regulatory issues	26. Establish communication with the individuals, offices, and agencies that will oversee the ethical, legal, and regulatory issues that are relevant to the conduct of the trial.
	27. Ensure that the informed consent documents to be signed by study participants accurately describe the risks and potential benefits associated with use of the omics test and include provisions for banking of specimens, particularly to allow for ‘bridging studies’ to validate new or improved assays.
	28. Address any intellectual property issues regarding the use of the specimens, biomarkers, assays, and computer software used for calculation of the predictor.
	29. Ensure that the omics test is performed in a Clinical Laboratory Improvement Amendments-certified laboratory if the results will be used to determine treatment or will be reported to the patient or the patient’s physician at any time, even after the trial has ended or the patient is no longer participating in the study.
	30. Ensure that appropriate regulatory approvals have been obtained for investigational use of the omics test. If a prospective trial is planned in which the test will guide treatment, consider a pre-submission consultation with the US Food and Drug Administration.

In its report, Evolution of Translational Omics: Lessons Learned and the Path Forward, the Institute of Medicine Committee on the Review of Omics-Based Tests for Predicting Patient Outcomes in Clinical Trials defines ‘omics’ as the study of related sets of biological molecules in a comprehensive fashion. Examples of omics disciplines include genomics, transcriptomics, proteomics, metabolomics and epigenomics. An omics-based test is defined as “an assay composed of or derived from multiple molecular measurements and interpreted by a fully specified computational model to produce a clinically actionable result”[1].

Specimen issues

Molecular profiles generated by the use of omics technologies can be sensitive to specimen collection, processing and storage conditions[3]. Investigators should consider the conditions under which specimens used in developmental studies were collected and handled to assess the robustness of an omics test to various specimen conditions. It may be necessary to conduct additional feasibility studies to document that the omics test will perform satisfactorily under the range of conditions in which the specimens will be obtained and stored in typical clinical settings; alternatively, more restrictive requirements for specimen collection, processing and storage should be clearly specified before the test is used in a clinical trial or other clinical validation study.

Criteria for specimen quality, amount (mass or volume), and composition should be clearly specified in order to qualify a specimen or its isolated analytes as suitable for assay by the omics test. Appropriate criteria will depend on the specimen type and the particular omics assay platform to be used. Details of the specification might include per cent purity of the target cells or intact analyte of interest and specific mass or volume of the specimen or analytes isolated from the specimen. It should be established that it is feasible to achieve these criteria in clinical settings.

Assay issues

Variations in assay procedures due to differences in technical protocols, reagents, and scoring and reporting methods can have a substantial impact on the analytical performance of an omics assay and its comparability among laboratories[4,5]. Many omics tests are developed using data from retrospective studies in which these aspects of the assay were not standardized. This can lead to uncertainties in how the test will perform when based on assay data from a new laboratory, including the laboratory or laboratories that will generate the assay data for a prospective trial. It is important to develop detailed standard operating procedures (SOPs) for the assay underlying the omics test and to establish that studies conducted previously to clinically validate the omics test were based on data expected to be comparable to new data generated under the specified SOPs.

Analytical performance of the omics assay under the proposed SOPs must be documented and found to be acceptable in terms of metrics such as accuracy, precision, coefficient of variation, sensitivity, specificity, linear range, limit of detection, and limit of quantification, as applicable. Calibrators, analytical standards, and controls are essential components of the SOPs and should be described clearly. Quality assurance procedures should include criteria for acceptance or rejection of assay batches and results from individual specimens. When multiple technicians or laboratories will conduct the assays, monitoring procedures should be in place to ensure comparability across technicians and laboratories. Methods for assay scoring and reporting should be clearly specified. Turnaround times for return of test results should be within acceptable limits that will be dependent on the particular clinical situation and should be sufficiently rapid to not impede clinical management timelines. Feasibility studies to assess assay analytical performance, reproducibility and turnaround times may be required in advance of initiating a clinical trial to firmly establish the suitability of the omics test for use in a real-time clinical setting.

Model development and evaluation

Many omics tests are developed using existing omics, clinical and pathology data or using data generated from retrospective specimen collections. These data may be incomplete or unreliable and should be examined for errors, inconsistencies or bias. Omics assays can be sensitive to a variety of ancillary technical influences that result in artefacts in the generated data. Of particular concern is the potential for such artefacts to be confounded with clinical variables or endpoints. Efforts should be made to identify potential confounders, including source of specimens (for example, clinical sites processing specimens differently), laboratory performing the omics assay, and assay batches[6].

Examples of flawed applications of statistical approaches for development of omics predictors and for evaluation of their performance are abundant in the literature[7,8,9]. Model overfitting, which occurs when a statistical model describes random noise instead of capturing the true association between predictor variables and a clinical endpoint, is a common problem in omics research projects, in which the number of analytes measured per specimen exceeds the number of specimens studied. Overfitting can be reduced by the use of model ‘regularization’ approaches that constrain the complexity of the model, but these approaches do not completely eliminate overfitting risk. It is common for researchers without the appropriate expertise to misunderstand and misapply modelling techniques. In addition, if flawed methods for model performance assessment are used, then overfitting may escape detection. A common mistake is failure to maintain strict separation between data used to build a model (‘training set’) and data used to assess model performance (‘testing set’). Numerous published papers have inappropriately reported model performance estimates based on resubstitution of data used to build a model back into that same model. These so-called ‘resubstitution estimates’ are severely (optimistically) biased. Assessment of model performance on the combined training and testing data sets is similarly problematic. Re-use of training data is acceptable only if performed properly using data resampling methods[10] that iteratively split the training data to hold out subsets of the data that are not used for model building and can therefore be used to check model performance.

Development of an omics predictor can be an iterative process involving several adjustments to improve performance. With regard to the three phases of the development and evaluation process in the IOM report[1] on omics tests, it is noted in the report that preliminary validations may occur in the test validation phase, and the definitive evaluation of clinical utility takes place in the final phase. It is important to be able to discern the point at which the omics test is ‘locked down’, or finalized, in all aspects, including specimen requirements, technical protocol for assay, data preprocessing, the form of mathematical predictor model, and interpretation of the test result. The test is then ready to enter the final evaluation for clinical utility and use stage, at which there are three basic options for clinical utility evaluation: first, a prospective evaluation of the omics test on a retrospective specimen collection from a clinical trial or prospective cohort study; second, a prospective clinical trial in which the test does not direct patient management; and third, a prospective clinical trial in which the omics test is used to direct patient management. Ideally, there should have been a blinded and rigorous preliminary validation of performance of the locked-down model on an external independent specimen set during the test validation phase. If an independent external validation set is not possible because adequate specimen collections do not exist, then existing performance evaluations based on internal validations should be carefully reviewed to ensure that they were rigorous and used appropriate methods. In this situation, it may be necessary to use a clinical trial design that does not allow the test to influence patient care.

When further adjustments are made to the omics test or data after the final validation data have been unblinded, there is a risk of compromising the validation. If the omics test is adjusted, either a new validation must be performed or additional evidence must be obtained; for example, by conducting an assay-bridging study to ensure that the adjustments to the test have not adversely affected its performance.

Investigators should be prepared to supply data and computer code as part of the review process for proposals to use omics tests in clinical trials. It is highly recommended that investigators follow reproducible research practices so that they will be able to supply the needed information quickly and easily for verification of the validation of the test and its locked-down form. Readers are referred to the companion publication[2] for further discussion of recommended reproducible research practices.

Clinical trial design

A clinical trial for definitive evaluation of an omics test should be conducted using the same rigorous standards expected for clinical trials evaluating experimental therapies. In some circumstances, high-level evidence can be obtained by use of specimens from an already-completed clinical trial[11]. Accepted standards for good clinical practice must be followed[12,13], including development of a formal protocol with clearly stated objectives and eligibility criteria, an informatics plan for management of clinical and omics data, a pre-specified study design[14] and statistical analysis plan, complete specification of the omics test, and justification for equipoise for any treatment randomizations (if the trial is conducted prospectively). The study team must include individuals with appropriate expertise to assume responsibility for the clinical, laboratory, pathology, bioinformatics, data management and statistical aspects of the study.

Ethical, legal and regulatory issues

Numerous ethical, legal and regulatory issues must be addressed in the course of developing an omics test for clinical use. Research involving human subjects, which includes retrospective use of specimens from living subjects, requires that adequate protection is in place to ensure the safety of patients and the privacy and confidentiality of patient information[15]. Ensuring appropriate protections has become more challenging as omics technologies make it possible to provide detailed genetic characterizations of individuals and much research data are made publicly available. Informed consent documents for a clinical trial using an omics test to guide patient management must accurately describe any potential risks from participation in a study and all potential conflicts of interest on the part of study investigators or sponsoring institutions. Laboratory tests must be conducted in environments that meet Clinical Laboratory Improvement Amendments certification requirements if the results will be reported to the patient or the patient’s physician[16]. Responsible parties at participating institutions (for example, institutional review boards, protocol review committees), trial sponsors (for example, the NCI, universities, companies), and the US Food and Drug Administration (FDA) (for example, for Investigational Device Exemption (IDE)[17] or Investigational New Drug[18] applications) must be fully informed of study details and approve the study before it proceeds.

If the omics assay to be used in a clinical trial could be considered a significant-risk assay, including—but not limited to—one used to choose among treatments, investigators must consult with the FDA to determine whether an IDE from the Center for Devices and Radiologic Health, or a similar evaluation carried out through the Investigational New Drug process, is required. The complexities of omics-based tests, together with the FDA’s evolving view of regulatory enforcement discretion for these tests, make it important to have early communications with the FDA. Investigators may find it helpful to discuss the trial formally with the FDA in a pre-submission process if they are not familiar with IDE requirements[19].

Intellectual property issues may apply to the use of the specimens, biomarkers, assays, and computer software used for calculation of the predictor. Intellectual property rights should be documented and respected by all parties involved. Potential conflicts of interest of study investigators must be disclosed and managed.

Summary

Evaluation of the readiness of an omics test to be used for clinical care requires careful consideration of the body of evidence supporting the test’s analytical and clinical validity and potential clinical utility, as well as an understanding of ethical, legal and regulatory issues. Funding bodies and journals are encouraged to consider using the checklist as an evaluation guide in their review processes. The NCI plans to use the checklist presented here to evaluate proposals for the use of omics tests in clinical trials where the test will be used to guide patient care. Although it is not expected that exploratory studies using omics assays or studies aiming to develop omics tests will meet all of the checklist criteria, the checklist does provide a convenient framework by which to assess the stage of development of an omics test and the strength and quality of the accumulated evidence. Several of the checklist criteria (those that are not specific to the development of models from high-dimensional data) also apply to studies of single biomarkers, or limited panels of biomarkers, measured by a variety of conventional assay methods. The checklist may, therefore, serve as a useful reference in a variety of review settings.

It is hoped that this 30-point checklist will guide investigators towards the use of best practices in omics test development, help them to more reliably evaluate the quality of evidence in support of omics tests, and assist them in planning appropriately for the clinical use of omics predictors. The ultimate goal is to develop a more efficient, reliable and transparent process to move omics assays from promising research results to clinically useful tests that improve patient care and outcome.