Positioning whole exome sequencing in the diagnostic pathway for rare disease to optimise utility: a protocol for an observational cohort study and an economic evaluation.

INTRODUCTION: Despite the superior diagnostic performance of exome and genome sequencing compared with conventional genetic tests, evidence gaps related to clinical utility and cost effectiveness have limited their availability in routine clinical practice in many jurisdictions. To inform adoption and reimbursement policy, this protocol provides a chain of evidence approach to determining the diagnostic utility, clinical utility and cost-effectiveness of whole exome sequencing (WES) from seven medical genetic centres in two Canadian provinces. METHODS AND ANALYSIS: Using a multicentre observational cohort design, we will extract data specific to the pre-WES diagnostic pathway and 1-year post-WES medical management from electronic medical records for 650 patients with rare disease of suspected genetic aetiology who receive WES. The date from the clinical record will be linked to provincial administrative health database to capture healthcare resource use and estimate costs. Our analysis will: (1) define and describe diagnostic testing pathways that occur prior to WES among patients with rare disease, (2) determine the diagnostic utility of WES, characterised as the proportion of patients for whom causative DNA variants are identified, (3) determine the clinical utility of WES, characterised as a change in medical management triggered by WES results, (4) determine the pattern and cost of health service utilisation prior and 1 year following WES among patients who receive a diagnosis, do not receive a diagnosis, or receive an uncertain diagnosis and (5) estimate the cost-effectiveness of WES compared with conventional diagnostic testing pathways, measured by the incremental cost per additional patient diagnosed by WES using simulation modelling. ETHICS AND DISSEMINATION: This protocol was approved by Clinical Trials Ontario (CTO-1577) and research ethics boards at the University of Calgary (REB18-0744 and REB20-1449) and University of Alberta (Pro0009156). Findings will be disseminated through academic publications and policy reports. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

This represents the largest Canadian clinical genetics cohort for which multiple dimensions of the value of whole exome sequencing (WES) (ie, diagnostic utility, clinical utility and cost-effectiveness) are assessed concurrently.

A real-world evidence and expert-informed framework was developed to support the economic evaluation described herein and represents a strength of this approach. However, this framework is limited by its specificity to a largely paediatric rare disease population and current practice patterns in Canada.

A non-WES comparator group could not be included in the study design. As such, counterfactuals required for modelling purposes rely on estimates from the literature and expert opinion.

Introduction

The journey to diagnosis for a patient with a suspected rare disease can be long, expensive and often unsuccessful, adversely impacting patient care.1–4 Until recently, the identification of disease-causing DNA mutations in a person’s genome was a laborious process. The introduction of next-generation sequencing technologies has created a paradigm shift in our approach to the diagnosis of genetic disease. Hypothesis-free strategies, such as whole exome sequencing (WES) which interrogates the 2% of the genome that encodes proteins, and whole genome sequencing (WGS) which interrogates both coding and non-coding regions, have demonstrated significant diagnostic utility for rare disease.5 Compared with conventional genetic tests such as chromosome microarray and targeted gene panels, the diagnostic yield of WES and WGS have been reported to be 2–3 fold higher.6

Evidence gaps with respect to the clinical utility and cost-effectiveness of WES and WGS have limited their availability in routine clinical practice in many jurisdictions.7–9 Unlike prospective clinical research where the effectiveness of an intervention can be tied to a specific health outcome, the concept of clinical utility in genetic medicine is rarely uniformly defined nor directly tied to a specific health outcome. As such, generating and adjudicating evidence of clinical utility and cost-effectiveness is complex.10–13 Applying Fryback and Thornbury’s hierarchical model of efficacy to genomics,14 the constructs of utility and cost-effectiveness can be characterised as a chain of evidence, rather than placing emphasis on diagnostic yield alone.15 For example, genetic testing provides information that guides prognostication and medical management, which in turn can influence patient-related health and non-health outcomes. While a recent review of the utility of WES and WGS extended beyond diagnostic yield to include medical management outcomes, these outcomes were reported inconsistently and only half as often as diagnostic yield.16 With respect to estimating the economic impacts of sequencing, a recent systematic review identified 36 studies.17 Estimates of test costs ranged from US$55518 to US$516919 20 for WES and from US$190621 to US$24 81022 for WGS. Most studies concluded that WES and WGS were superior to other conventional testing methods in terms of incremental cost per additional diagnosis. While informative, many of these analyses were based on small samples sizes and provided limited detail on the components included in cost estimates. The authors of this review, as well as those who have completed more recent economic evaluations, conclude that knowledge gaps remain with respect to comprehensive measures of value and value for money of WES and WGS.23–26

To address these gaps, the protocol described herein reflects on a chain of evidence approach to determining the diagnostic utility, clinical utility and cost-effectiveness of WES at various points in the diagnostic journey for patients with rare disease in Canada.

Aims

The specific aims of this study are:

To define and describe diagnostic testing pathways that occur prior to WES among patients with rare disease for whom a genetic aetiology is suspected.

To determine the diagnostic utility of WES, characterised as the proportion of patients for whom causative variants are identified.

To determine the clinical utility of WES, characterised as a change in medical management triggered by WES results.

To determine the pattern and cost of health service utilisation from birth to 1 year following WES among patients who receive a diagnosis, do not receive a diagnosis, or receive an uncertain diagnosis.

To estimate the cost-effectiveness of WES relative to conventional diagnostic testing pathways, as measured by the incremental cost per additional patient diagnosed via WES.

Methods

Design and settings

This is a multicentre observational cohort study of 650 patients who will receive clinical WES for the purpose of establishing a genetic diagnosis. For each patient enrolled between 2019 and 2022, diagnostic investigations performed prior to WES are collected retrospectively. One-year outcomes are collected prospectively (figure 1). The study settings include five genetics clinics in the Canadian province of Ontario and two genetics clinics in province of Alberta. Since four sites are children’s hospitals, the majority of participants are anticipated to be <18 years of age. Patients are referred to these clinics from specialists (eg, paediatricians, neurologists) or primary care providers.

Figure 1

Look-back window: earliest record of diagnostic investigation recorded in medical record. For administrative data, start date is date of birth.

Sample and recruitment

Eligibility criteria align with the most recent position statement from The Canadian College of Medical Geneticists on implementing clinical sequencing.27 Specifically, patients are eligible for WES and for this study if a baseline clinical genetics evaluation has been completed and a genetic aetiology for the phenotype is suspected. Clinical presentations that include >2 of: (1) moderate to severe developmental or functional impairment; (2) multisystem involvement; (3) progressive clinical course; (4) differential diagnosis that includes >2 conditions that would require evaluation by separate gene panel tests; (5) suspected severe genetic syndrome for which multiple family members are affected, or where parents are consanguineous. Patients are ineligible if their clinical presentation is limited to: (1) isolated mild intellectual disability or learning disabilities, (2) non-syndromic autism, (3) isolated neurobehavioural disabilities or (4) isolated neuropsychiatric conditions. Based on clinical volumes and the expected distribution of eligible phenotypes, we estimated that 500 Ontarians and 150 Albertans would provide sufficient sample to populate each of the phenotypic categories and be feasible to achieve within the recruitment period. This represents the largest clinical genetics cohort for which WES-related diagnostic utility, clinical utility and cost data has been ascertained in Canada.

Eligible patients or their family members are informed of the study by a clinical geneticist, subspecialist with expertise in genetics, or a genetic counsellor during or soon after their clinical consultation. All patients are informed that access to WES is not contingent on study participation. If interested in participating, patients provide informed consent for medical record review and for access to administrative health data in their respective province; Ontario’s Institute for Clinical and Evaluative Sciences (ICES) or Alberta Health Services (AHS).

Data collection

Medical records

Data entry staff at each site were trained to review records and identify relevant datapoints prior to data entry. At a given site, the same individuals complete pre-WES data entry, post-WES data entry and quarterly data auditing functions. Data from medical records specific to each patient’s pre-WES diagnostic pathway and post-WES medical management are entered into Genomics4RD, a centralised data repository for rare disease research.28 The data collection tool includes pre-WES and post-WES modules. The pre-WES variables include: (A) patient characteristics, (B) DNA-based diagnostic tests performed to date, (C) non-DNA based diagnostic investigations performed to date, (D) specialists involved in ongoing care and (E) a record of the diagnostic tests that the responsible clinician would have ordered in the absence of WES. To ascertain what this clinician would have ordered in the absence of WES, the clinician completes a checklist at the time WES is ordered and their checklist responses are entered into Genomics4RD (ie, hypothetical care pathway; table 1). We requested this information to inform our cost-effectiveness modelling analysis (aim 5) since it was not feasible to have a non-WES comparator groupincluded in the study design.

Table 1

Data collection—timeline, variables and data sources

Variables			Pre-WES*†	WES resultreported*	Post-WES*†
					One mo	Six mo	12 mo
Patient characteristics	Site		X
	Age		X
	Sex		X
	Family history		X
	Phenotype‡		X
	Genetics referral/consult dates		X
AIM 1: To define and describe diagnostic testing pathways that occur prior to WES among patients with rare disease for whom a genetic aetiology is suspected
Pre-WES diagnostic pathway
Diagnostic investigations to date	Cytogenetic/molecular, biochemistry, imaging, physiological, pathology		X
Diagnostic investigations in the absence of WES (hypothetical)			X
Specialist involvement to date	Allied health, MD subspecialists		X
Anticipated management impact of WES	Limit dx investigations, guide repro decision making, enable early identification/intervention		X
Aim 2: To determine the diagnostic utility of WES, characterised as the proportion of patients for which causative variants are identified
WES outcome
WES strategy	Singleton, duo, trio			X
WES turnaround time	Date submitted to MOH, approved, received by lab, reported, disclosed to family			X
WES results	Laboratory interpretation or primary variants (ie, pathogenic, likely pathogenic, variant of uncertain significance)§Clinical interpretation or primary variants (ie, diagnostic, partially diagnostic, potentially diagnostic, non-diagnostic)¶Presence/absence of secondary variants			X
Aim 3: To determine the clinical utility of WES, characterised as a change in medical management triggered by WES results
Post-WES Management Implications
Diagnostic investigations ordered (primary variants)	Cytogenetic/molecular, biochemistry, imaging, physiological, pathology				X
Diagnostic investigations averted (primary variants)	Cytogenetic/molecular, biochemistry, imaging, physiological, pathology				X
Management recommendations(primary variants)	Monitoring and long-term management (ie, care team, surveillance)Active treatment (ie, medication initiation/alteration, invasive procedure)Cascade genetic counselling/genetic testingResearch opportunities (ie, clinical trial, natural hx study, disease mechanism study)				X
Management recommendations pursued (primary variants)						X	X
Management recommendations (secondary variants)	Monitoring and long-term management (ie, care team, surveillance)				X
Management recommendations pursued (secondary variants)						X	X
Aim 4: To determine the pattern and cost of health service utilisation from birth to 1 year following WES among individuals who receive a diagnosis, do not receive a diagnosis, or receive an uncertain diagnosis via WES
Overall Health Service Utilisation	ICES Database	AHS Database
Demographics	Registered Persons Database	Registered Persons Database	X
Use of outpatient physician services	Physician Claims Database	Physician Claims Database	X		X	X	X
Use of laboratory testing	Ontario Laboratory Information System	Consolidated Laboratory Data Repository	X		X	X	X
Use of medical imaging	Ontario Laboratory Information System	AHS DI Shared Data Model	X		X	X	X
Use of emergency services	Discharge Abstract Database, National Ambulatory Care Reporting System	Discharge Abstract Database, National Ambulatory Care Reporting System	X		X	X	X
Inpatient admissions	Discharge Abstract Database, National Ambulatory Care Reporting System	Discharge Abstract Database, National Ambulatory Care Reporting System	X		X	X	X
Ambulatory services	National Ambulatory Care Reporting System	National Ambulatory Care Reporting System	X		X	X	X
Medication	N/A (only available for >65 years)	Pharmaceutical Information Network	X		X	X	X

Post-WES data from medical records are entered into Genomics4RD following the disclosure of WES results to the patient. The post-WES variables include: (A) laboratory interpretation of WES results; (B) clinical interpretation of WES results; (C) diagnostic testing ordered if WES result was non-diagnostic; (D) diagnostic testing averted if WES result was diagnostic and (E) medical management activities triggered by WES results (table 1). Across all sites, the laboratory interpretation of WES results aligns with the standardised classification system recommended by the American College of Medical Genetics, and includes pathogenic variants, likely pathogenic variants and variants of uncertain clinical significance. As recommended by this guideline, all WES is performed in Clinical Laboratory Improvement Amendments-approved laboratories.42 Clinical interpretation categories include diagnostic, partially diagnostic, potentially diagnostic and non-diagnostic. Where laboratory or clinical interpretations are difficult to decipher or not provided in the laboratory reports, the ordering clinician provides this assessment. The categories of medical management implications were derived from prior work by our team,35 related literature12 and input from our clinical collaborators. Recommended tests and services are ascertained from the clinical consult note written following result disclosure and contained within the medical record. One year following the initial entry of the post-test management data, medical records are rereviewed to ascertain whether and when recommended management activities were pursued.

*Electronic Medical Records.

†Administrative Data: Members of the Armed Forces and the Royal Canadian Mounted Police and federal penitentiary inmates are excluded.

‡Phenotype classification was completed by clinical geneticists, trained in the use of the Human Phenotype Ontology classification system (https://phenotips.com/). As such, the approach to capturing phenotype data was standardised across all clinical sites.

§Definition of laboratory interpretation categories align with the standards and guidelines for the interpretation of sequence variants, a recommendation of the American College of Medical Genetics and Genomics and the Association for Molecular Pathology.42

¶Definitions of clinical interpretation categories: (1) diagnostic (ie, pathogenic/likely pathogenic variant that provides a complete explanation of phenotype; (2) partially diagnostic (ie, pathogenic/likely pathogenic variant that provides a partial explanation of phenotype, (3) potentially diagnostic (ie, a variant of unknown significance that could provide a complete explanation of phenotype OR is a pathogenic/likely pathogenic variant in a recessive gene without a second hit, and (4) non-diagnostic (ie, test result that provides no explanation of phenotype).

AHS, Alberta Health Services; ICES, Institute for Clinical and Evaluative Sciences; MoH, Ministry of Health; N/A, not available; WES, whole exome sequencing; WGS, whole genome sequencing.

Administrative health databases

Both Ontario and Alberta have single-payer, government-administered healthcare systems that provide services free at the point of access for residents with a valid provincial health card. All publicly funded services accessed are recorded within health administrative datasets. In Ontario, patient-level data and episodes of care will be linked across various administrative databases using the encrypted ICES key number. In Alberta, the analogous patient level linked data will be obtained using personal health numbers (PHNs), a unique, nine-digit PHN, which is used when accessing healthcare services (table 1).

Costing

Costs for laboratory tests (basic biochemistry, small-molecule disorders, mitochondrial diseases, peroxisomal diseases) will be based on the 2020 Schedule of Benefits for Laboratory Services from the Ontario Health Insurance Plan (OHIP). For diagnostic procedures (imaging, biopsies and invasive procedures, electrical activity studies), the primary costing sources will be the 2020 Schedule of Benefits, Physician Services OHIP and the Schedule of Facility Fees. Prices for genetic tests will be obtained from the laboratories that performed these tests, as recorded in Genomics4RD (ie, local or external laboratories). For administrative health datasets, costs associated with hospitalisations, emergency department visits, procedures, tests, physician visits and medications will be included, and costs associated with genetic counselling will be excluded. For cases in Ontario, validated standard ICES costing algorithms will be applied.29 For cases in Alberta, individual emergency department visits and hospitalisation costs will be estimated using the most recent resource intensity weights.30 The Alberta Ambulatory Care Classification System Interactive Health Data Application will be used to value emergency department visits and imaging. Unit costs for prescribed medications will be from Alberta Blue Cross Interactive Drug Benefit List. All costs will be reported in 2020 CAD.

Data analysis

Aim 1: diagnostic testing pathways prior to WES

Based on diagnostic investigations captured in the pre-WES period, patients will be assigned to groups that reflect the complexity of their diagnostic pathway. To organise these pathways, we engaged in an expert-driven consensus process.31 Using professional guidelines related to diagnostic algorithms for rare disease diagnosis as a reference point,32–34 we asked three medical geneticist coinvestigators to assist with developing a framework for categorising tests recorded in our dataset. Categorising tests as indicator and non-indicators tests, we established the SOLVE Framework for organising pre-WES diagnostic pathways.35 Specifically, indicator tests are defined as those with high specificity for diagnosing rare diseases and likely to contribute specific information towards achieving a clinically valid molecular diagnosis. Non-indicator tests are defined as those performed as a routine part of a diagnostic workup for a patient referred for evaluation of a rare disorder. Indicator tests are typically higher cost, potentially invasive, less accessible and ordered/interpreted by a subspecialist and non-indicator tests are typically lower cost, non-invasive, locally accessible and ordered/interpreted by a generalist.31 Guided by this framework, we will use frequency counts and descriptive statistics to summarise the number, type and cost of diagnostic tests per person in each type of diagnostic pathway and the time to diagnosis for each pathway. Diagnostic pathways will serve as comparator groups for the economic evaluation (aim 5).

Aim 2: diagnostic utility of WES

We will also determine the proportion of cases for whom WES identifies variants that are pathogenic, likely pathogenic or of uncertain clinical significance and the proportion of cases for which WES establishes a diagnosis, a partial diagnosis, a potential diagnosis and no diagnosis. The diagnostic yield of WES (ie, proportion of cases who receive diagnostic results) will be the primary measure of diagnostic utility, a core grouping variable for the clinical utility and health service utilisation analyses (aims 3, 4), and the primary outcome for the cost-effectiveness analyses (aim 5).

Aim 3: clinical utility of WES

We will summarise data related to the medical management implications of WES using descriptive statistics. We will determine the type and volume of management activities overall and per person. Where sample size permits, point estimates for change in medical management overall and for specific types of medical management will be compared statistically among those who receive a diagnosis, a potential diagnosis and no diagnosis and among pre-WES diagnostic pathway groups established in aim 1. We will examine the relationship between clinical characteristics and management change(s) using parametric or non-parametric statistics as appropriate. If indicated, we will construct regression models to determine predictors of changes in medical management.

Aim 4: pattern and cost of healthcare utilisation pre-WES and post-WES

Rates of outpatient visits, laboratory testing, imaging services, emergency department visits, admissions and associated costs will be compared among those who receive a diagnosis and those who do not. Utilisation rates and costs will be compared pre-WES and 1-year post-WES. We will use standard methods for comparing proportions, and Poisson regression to test whether there are significant differences in the volume, type and costs of service utilisation in the presence/absence of a diagnosis. We will examine total volume of activities based on the distribution of healthcare resource use as well as mean and median number of activities per patient pre-WES and post-WES. The total costs for each pre-WES and post-WES pathway will be summed and results will be grouped according to the WES result type.

Aim 5: cost-effectiveness of WES at different timepoints in the diagnostic pathway

The cost-effectiveness analysis will assess the incremental cost associated with WES at different points in the diagnostic pathway from the healthcare payer perspective, with diagnostic yield of WES as the primary measure of effectiveness. Change in medical management will be used as a secondary measure of effectiveness. To facilitate these comparative analyses, we will develop a simulation model to reflect alternative diagnostic pathways to achieve a molecular diagnosis, as defined in aims 1 and 2.36–39 The diagnostic pathway is marked by the number of events (i.e. indicator tests other than WES as defined by the SOLVE Framework).35 Each patient’s timeline is modelled according to the time between events and the resource utilisation associated with events. The time of each event is determined by the presence of an indicator test. Informed by our expert-driven consensus process,31 patients will be grouped according to the number of events observed (ie, 1, 2, 3 or 4+ events). The number of events and the resource utilisation and costs incurred for each time period will be informed by observed data; the model will draw probabilistically from the observed distributions for each subgroup. Ultimately, the model will simulate patients’ test trajectory, evaluating the cost-effectiveness of performing WES at distinct time points within the testing sequences. To populate the diagnostic trajectory for the non-WES comparator group, the estimates for the probability of diagnosis via WES at different timepoints will be informed by our expert-driven consensus process. Deterministic and probabilistic sensitivity analysis and expert-defined scenario analyses will be conducted to define model uncertainty and the incremental cost effectiveness ratios will be assessed using cost-effectiveness acceptability curves at multiple levels of willingness-to-pay.40 41

Patient or public involvement

Given the policy relevance of our work, this protocol was designed in collaboration with key decision-maker partners at The Ontario Ministry of Health and AHS. Our findings will inform recommendations related to the clinical implementation of WES for each Canadian province. Patients were not involved in the codesign of this work.

Ethics and dissemination

The research protocol for Care for Rare Solve was approved by Clinical Trials Ontario (CTO-1577), the provincial platform responsible for approving clinical trials and observational studies involving two or more academic or healthcare institutions in Ontario, and by research ethics boards at the University of Calgary (REB18-0744 and REB20-1449) and University of Alberta (Pro00091561). Findings will be disseminated through academic publications and policy reports.

Limitations

We acknowledge several limitations. First, the distinction between indicator and non-indicator tests in the SOLVE Framework relies on expert opinion and is specific to a largely paediatric rare disease population in Canada and ordering practices that reflect currently available tests. Assessment of its face validity over time is warranted. Second, the availability of WES in Canadian clinics, through exceptional access programmes to US-based laboratories, precluded the inclusion of a non-WES comparator group. As such, counterfactuals required for modelling purposes will rely on estimates from the literature and expert opinion. Third, our sample size was informed by projected case volumes and was not hypothesis driven. Finally, a 1-year post-WES observation period presents only a short-term view of medical management impacts and resource utilisation patterns for this patient population. Longer-term impacts are not included. Limitations notwithstanding, this analysis is unique in its use of real-world data and is the largest cross-provincial analysis of the diagnostic utility, clinical utility, and cost effectiveness of clinical WES in a Canadian rare disease population. Findings may be used to inform the clinical genetics service delivery models in Canada and internationally.