Proposed outcomes measures for state public health genomic programs.

PURPOSE: To assess the implementation of evidence-based genomic medicine and its population-level impact on health outcomes and to promote public health genetics interventions, in 2015 the Roundtable on Genomics and Precision Health of the National Academies of Sciences, Engineering, and Medicine formed an action collaborative, the Genomics and Public Health Action Collaborative (GPHAC). This group engaged key stakeholders from public/population health agencies, along with experts in the fields of health disparities, health literacy, implementation science, medical genetics, and patient advocacy.
METHODS: In this paper, we present the efforts to identify performance objectives and outcome metrics. Specific attention is placed on measures related to hereditary breast ovarian cancer (HBOC) syndrome and Lynch syndrome (LS), two conditions with existing evidence-based genomic applications that can have immediate impact on morbidity and mortality.
RESULTS: Our assessment revealed few existing outcome measures. Therefore, using an implementation research framework, 38 outcome measures were crafted.
CONCLUSION: Evidence-based public health requires outcome metrics, yet few exist for genomics. Therefore, we have proposed performance objectives that states might use and provided examples of a few state-level activities already under way, which are designed to collect outcome measures for HBOC and LS.

INTRODUCTION

The National Academy of Sciences, Engineering and Medicine’s (The Academies) Health and Medicine Division (HMD), Board on Health Science Policy formed the Genomics and Population Health Action Collaborative (GPHAC) in 2015, as an ad hoc activity under the auspices of the Roundtable on Genomics and Precision Health[1]. Key stakeholders at the state and federal levels, researchers, patient advocates, and clinical practitioners with expertise in public/population health, health disparities, health literacy, implementation science, medical genetics, and patient advocacy were brought together to work toward the effective and timely integration of genomics into existing public health programs.

This publication describes one activity involving a subset of the Action Collaborative members to develop potential performance objectives and outcome measurements for successful implementation of genomic activities within public health practice. These will become part of a genomics public health tool kit available online through The Academies website, and are intended to stimulate further dialogue among public health, population health and academic researchers to determine feasibility and refine with baseline and target measures.

Organizing Framework for Genomic Implementation

The implementation of evidence-based genomic services requires identification of outcome measurements at the level of implementation, in addition to those at the system and client levels. In 2009, Enola Proctor and a number of colleagues[2], primarily working in mental health services research, published a paper laying out a framework for the emerging field of implementation research. The anchoring framework, reproduced below (Figure 1), explained the distinction between a focus on interventions at the individual level and implementation strategies employed to get those interventions incorporated into service systems.

Figure 1.

Proctor et al. Implementation Framework[2]

The framework recognizes that in order to ultimately get successful client outcomes (e.g. improved health status, functioning, and satisfaction with care) at a population level, one must first ensure that evidence-based health interventions are successfully implemented. In turn, these actions will improve the performance of the health system. Health system improvement, here summarized by pursuit of the Academies standards of care[3], cascades from the achievement of key implementation outcomes, such as feasibility, fidelity, penetration, sustainability, uptake, and costs.

If we apply this framework to genomic services, we define the “interventions” as evidence-based approaches to the delivery of genetic/genomic screening at a population level. Examples of these include: 1) genetic testing for Lynch syndrome (LS), a hereditary colon cancer syndrome that is also associated with other early onset malignancies, for all individuals with newly diagnosed colorectal cancer (CRC) recommended by the Evaluation of Genomic Applications in Practice and Prevention (EGAPP™) Working Group[4]; and 2) screening to identify family history associated with Hereditary Breast and Ovarian Cancer (HBOC) syndrome, with the majority of cases caused by mutations in BRCA1 or BRCA2 genes, as recommended by the U.S. Preventive Services Task Force (USPSTF)[5]. Furthermore, we define the implementation strategies as the efforts to embed screening within existing health care services (e.g. provider training, referral to certified genetic counseling, ordering of tests), and public health efforts to monitor and evaluate those services (e.g., monitoring the frequency of tests, provider awareness, consumer awareness, etc.). The set of outcomes in the framework could then be used to establish baseline, realistic targets and ultimately to record progress and population health impact of genomics.

METHODS

The eight GPHAC members who were actively involved in working on outcome metrics represented varied perspectives including patient advocacy, professional organizations, state health departments, and federal agencies. The members focused on genomic applications specific to HBOC and LS, recognizing that these examples may be adapted to other genetic conditions. The initial proposal was to research and summarize existing metrics (process, performance and outcomes measures), identify broad domains for outcome metrics and then to document the findings. Searching was conducted by members individually assigned to review 1) Healthy People 2020 objectives[6], 2) National Cancer Institute (NCI) funded grants 3) Center for Disease Control and Prevention (CDC) funded grants, 4) peer-reviewed literature, and 5) web-based searches using key words (i.e., outcome measures, HBOC, LS, implementation, genetic testing).

During the literature review phase, articles with potential frameworks for guidance emerged, including Proctor, et al. described above, and another specific to genetic services outcomes by Silvey, et al.[7] We developed a modified framework for state public health genomic program performance objectives, and defined outcome measures specific to public health genomics activities, with a major emphasis on HBOC and LS. In our modified framework, outcomes were organized per the three broad outcome categories in the Proctor et al. framework (i.e. implementation outcomes, system level outcomes, client outcomes), and three additional outcome categories, including ‘access to services’, ‘healthcare performance’, and ‘public health infrastructure’, based upon the Silvey et al. article, and the 10 Essential Public Health Services[8]. ‘Access to services’ identifies public health-specific measures related to efforts to embed and monitor screening and testing within health care systems. ‘Healthcare performance’ focuses on the uptake of health providers or health systems implementing evidenced-based and recommended health services. The third, ‘public health infrastructure’, spotlights processes essential to the successes of the implementing public health agency. Because our task was to focus specifically on prevention efforts concerning HBOC and LS, two additional objectives were added to the framework - expanding universal screening for LS for all colorectal cancer tumors. One focused on LS tumor screening adoption and a second focused on LS tumor screening reach.

Through an iterative process, members defined performance objectives, and identified relevant outcome measures and possible data sources. We also discussed the likelihood that states or academic researchers would readily have access to data sources such as Behavioral Risk Factor Surveillance System (BRFSS)[9], state operated Medical Claims databases, or other service utilization data. We then posited a state genomics programs’ potential ‘readiness’ or capacity for measuring each specific outcome. Through consensus, we categorized each outcome according to capacity, and identified top priority outcomes, with any disagreements resolved by discussion. The categories are as follows:

Top priority outcomes all states are encouraged to pursue.

Outcomes that states should pursue.

Outcomes states should pursue if data sources are available.

Aspirational outcomes.

As the outcome measures were formulated, discussion followed regarding health disparities. Although specific measures for health disparities were identified in the ‘equity’ outcome category, it was recognized that for other outcome measures, specific health disparity measures could be developed.

RESULTS

In the preliminary research to identify existing measures, we found that the majority were process measures rather than outcome measures. Only two measures had existing baseline data and targets, both part of Healthy People 2020, including increase in the proportion 1) of women with a family history of HBOC/LS who receive genetic counseling and 2) of persons with newly diagnosed colorectal cancer who receive genetic testing to identify LS (or other familial colorectal cancer syndromes). Therefore, using the framework described above, additional outcomes measurement included outlining multiple potential metrics and prioritizing each. These are provided in Table 1.

Table 1.

Performance Objectives, State Outcome Measures and their Data Sources for Genomic Services

Outcome Category	Performance Objective	State Outcome Measure	Data Source
I. Top priority outcomes all states are encouraged to pursue:
Access to Services
	Increase the proportion of women with a family history of HBOC/LS who receive genetic counseling (reworded version of HP2020 Objective)	Number of women with a family history of HBOC/LS who receive genetic counseling.	National Health Interview Survey by CDC/NCHS; State BRFSS; State PRAMS Cancer module; Claims data
	Increase the proportion of persons with newly diagnosed colorectal cancer who receive genetic testing to identify LS (or other familial colorectal cancer syndromes (HP2020 Objective)	Number of persons with newly diagnosed colorectal cancer who receive genetic testing to identify LS (or familial colorectal cancer syndromes)	State Cancer registries; Surveillance Epidemiology and End Results program (SEER); State BRFSS
Healthcare Performance
	Increase the number of family members (per family) tested for HBOC/LS through cascade screening	Number of family members screened following identification of HBOC/LS mutations	Claims data; Data collected from genetics providers in clinics across the state; number of single mutation tests ordered in state as reported by ~5 largest cancer genetic testing laboratories
II. Outcomes states can readily perform:
Implementation Feasibility
	Mechanisms exist for adequate billing and reimbursement of services	Number of health plans with existing reimbursement for services	CPT codes; payer policies; licensure data; hospital credentialing data
	Hospitals have the infrastructure needed to conduct universal tumor screening (i.e., pathology, tracking, genetic counseling and follow-up to ensure effectiveness)	Number of hospitals with the following infrastructure: pathology, tracking systems, counselors, follow-up procedures	Survey data
Implementation Acceptability
	Increase the number of providers who are comfortable providing HBOC/LS screening services	Number of providers for each item	Survey data
	Increase the number of providers who are willing to provide HBOC/LS screening services	Number of providers for each item	Survey data
	Increase the number of providers who appropriately refer HBOC/LS at-risk families	Number of providers for each item	Survey data
Implementation Sustainability
	Mechanisms for adequate billing and reimbursement of services are maintained over time	Description of existing mechanism for billing and reimbursement	CPT codes; payer policies; licensure data; hospital credentialing data
	Training programs continue to recruit, train and graduate genetic service providers	Number of training programs and numbers of applicants/graduates for each type of provider; number of slots being filled; types of applicants (i.e., diversity)	ABGC, ABMG, and ANCC data; training program data
Implementation Uptake
	Increase the number of hospitals/institutions that have implemented tumor screening to identify LS	Number of hospitals	LSSN membership data; Survey hospitals on current practices
Service Safety
	Increase appropriate genetic counseling linked with HBOC/LS testing	Frequency of genetic counseling; frequency of HBOC/LS Testing
	Reduce misinterpreted genetic test results	Quality control of interpreted tests
	Reduced inappropriate treatments (pharmacologic, surgical or other) due to misinterpreted HBOC/LS genetic test results.	Number of inappropriate treatments from quality control of interpreted tests	FDA guidelines for testing validity
Service Timeliness
	People can access genetic services in a timely manner	Time from referral	Contact facilities and determine “3rd to available new” appt. times
Access to Services
	Providers are available to perform genetic services including in rural and frontier areas	Number of providers across geographical areas	ABGC, ABMG, and ANCC credentialing and state licensing data
	Increase the availability of telegenetic services (telemedicine).	Number of originating sites connected to a distant site	Regional Telehealth Offices
Lynch syndrome tumor screening adoption
	Increase the number of hospitals/institutions that have implemented tumor screening to identify LS	Number of hospitals/Institutions offering tumor screening	LSSN membership data; Survey hospitals on current practices
Healthcare Performance
	Increase the number of hospitals performing tumor screening that have a tracking system in place	of hospital with tracking system for tumor screening	LSSN; survey data
Public Health Infrastructure
	Increase state’s readiness to implement public health genetics programs	Level of readiness, including willingness and capacity to implement public health genetics	Survey states
	States have access to reliable information/data to inform program planning and policy.	Number’s and types of population level data inclusive of genomics	BRFSS; cancer registry; internet access to payer policies; other state’s public health genetics program activities and information
	Increase partnerships with regional clinics, academic institutions, CDC-funded programs, state programs, non-profits, insurance groups and industry to ensure efforts are sustainable	Number of partnerships	Survey states
III. Outcomes states can readily perform if data sources are available:
Service Equity
	Decrease health inequalities (population subgroups who are more vulnerable than others due to social forces) regarding access to genetic testing/counseling	Number of genetic tests/counseling sessions by sub-group	Claims data; BRFSS
Access to Services
	Increase the proportion of individuals diagnosed with potentially heritable cancers who undergo genetic testing	Number of individuals diagnosed with potentially heritable cancers who undergo genetic testing	State cancer registries; survey data
IV. Aspirational Outcomes[a]:
Implementation Feasibility
	Health care providers receive initial training and periodic refresher training to diagnose, treat and counsel families for HBOC/LS in accordance with the most current NCCN recommendations.	Number or percentage of facilities offering initial training on NCCN guidelines for HBOC/LS; Number or percentage of providers receiving initial training on NCCN guidelines for HBOC/LS; Number of facilities offering periodic refresher training on NCCN guidelines	Training program data; reporting data indicating number of providers trained and efficacy of training on provider knowledge
	Data sources exist to measure outcomes at multiple levels	Levels need to be identified to determine outcome measures
Implementation Penetration
	Increase the proportion of providers in rural and frontier areas that screen and refer patients for HBOC/LS	Number or percentage of providers delivering HBOC/LS screening; number of patients screened for HBOC/LS in rural and frontier counties; number who screen positive; percentage of population in rural and frontier areas screened	Survey systems/providers
Service Efficiency
	Increase the proportion of clinics/hospitals/facilities using genetic laboratory utilization services to ensure the most appropriate genetic test(s) are ordered	Proportion of clinics/hospitals/facilities using genetic laboratory utilization services	Time-motion data; Survey data; policy review findings
Service Patient-Centeredness
	Cancer patient treatment plans include genetic counseling at the time of diagnosis	Number or percentage of facilities that include genetic counseling in treatment plans for new patients; number of providers with additional genetic training; number of new providers with genetic fellowship	Survey or reporting from oncology programs
Client Satisfaction
	Families receive written visit summary information, including risk assessment that can be shared with other family members	Number of facilities that have policies in place for written visit summaries; number of families who reported receiving materials; number or percentage of families who receive a visit summary and information they can share with families	Patient satisfaction surveys; site level policies
	Increase the proportion of patients who report timely appointments for genetic counseling/testing	Number or percentage of patients who report good or very good levels of satisfaction	Patient satisfaction surveys
Client Symptomatology
	Symptoms or complications from HBOC/LS are eliminated or decreased through early identification and treatment	Number of HBOC or LS associated cancers that are reported after known mutation identification	Chart review data
Access to Services
	Increase Number or percentage of women diagnosed at or below age 50 with breast cancer who undergo genetic risk assessment (per NCCN guidelines)	Number or percentage of women diagnosed at or below age 50 with breast cancer who undergo genetic risk assessment (per NCCN guidelines)	Claims data; national surveys
Lynch Syndrome Tumor Screening Reach
	Increase the number of tumors screened for LS at each institution	Number of tumors screened for LS by Institution	LSSN membership data
	All newly diagnosed patients with CRC are screened for LS	Number of CRC patients screened for LS	Proportion of patients diagnosed with CRC who have received screening
Healthcare Performance
	Decreased incidence of HBOC/LS	Incidence rate of HBOC/LS	Cancer Registries; SEER; Claims data
	Decreased morbidity and mortality of HBOC and LS	Morbidity and mortality rate of HBOC and LS	Cancer Registries; SEER; Claims data
Public Health Infrastructure
	Initiate bidirectional reporting by identifying individuals at increased risk for hereditary cancer through personal history in cancer registry	Number of state cancer registries that offer bidirectional reporting; number of investigations conducted/year; number of hospital cancer registries that have the capacity for bidirectional reporting	Cancer registries

-data sources listed here are suggested for further development.

Abbreviations: HBOC, hereditary breast and ovarian cancer; LS, Lynch syndrome; HP2020, Healthy People 2020; CDC, Centers for Disease Control and Prevention; BRFSS, Behavioral Risk Factor Surveillance System; PRAMS, Pregnancy Risk Assessment Monitoring System; SEER, Surveillance, Epidemiology and End Results; CPT, Current Procedural Terminology; ABGC, American Board of Genetic Counseling; ABMG, American Board of Medical Genetics; ANCC, American Nurses Credentialing Center; LSSN, Lynch Syndrome Screening Network; FDA, Food and Drug Administration; NCCN, National Comprehensive Cancer Network

Overall, we identified 38 outcome measures, each with a specified performance objective. Eighteen of these measures could be applicable to more general genomic applications, including multiple cancer genetic tests. There were 12 that were specific to both HBOC and LS, 7 specific only to LS, and 1 specific only to HBOC.

Three of the performance objectives and associated outcome measures were considered priority items that all states are encouraged to pursue at present. These objectives include an increase in the proportion of 1) persons with newly diagnosed colorectal cancer who receive genetic testing to identify LS, 2) women with a family history of HBOC/LS who receive genetic counseling, and 3) increased use of cascade screening for HBOC/LS. Sixteen additional outcomes were considered measures that state programs could currently capture, but were not identified as a priority. There were two service-related outcome measures that states could readily perform if data sources are available, and 17 measures that were considered aspirational, as they have no obvious data source currently available. Data sources for 10 outcome measures included survey data, including surveys of individuals, providers, health systems, hospitals, and at the state level through the Behavioral Risk Factor Surveillance System (BRFSS). Additional data sources identified included service utilization data such as the American Medical Association’s Current Procedural Terminology (CPT®) codes, payer policies, state licensure data, hospital credentialing data, professional society and/or networks data (e.g., Lynch Syndrome Screening Network), State cancer registries, or Food and Drug Administration guidelines for testing validity. All survey data sources were considered ‘potential’, meaning that validated surveys’ may or may not exist, but could be developed and initiated if resources permitted.

Two measures identified as potential sources of health equity included identifying the number of 1) genetic tests performed, and 2) genetic counseling visits for underserved sub-groups of the population. These were both categorized as measures that individual states can pursue if data sources are available. Stratifying other outcome measures for underserved individuals/populations was not performed, although it was identified as a future activity.

Costs to implement the outcome measures were also discussed including individual state costs (e.g., cancer registry maintenance, development and maintenance of state specific All Payer Claims database, survey development and data collection and analysis, and staff capacity), costs to researchers and others that develop, conduct and analyze survey data, and costs to networks such as the Lynch Syndrome Screening Network (LSSN). Costs associated with meeting performance objectives were often shared costs to individuals, families and/or payers for services, and costs to health systems and/or labs for services.

DISCUSSION

One of the major activities of the GPHAC Implementation Work Group was to develop a common understanding of appropriate metrics for health departments to assess ongoing implementation of genetic and genomic services (medical evaluation, genetic counseling and testing) within their state or local systems. The project members identified a dearth of existing metrics therefore using an overarching framework, taken largely from implementation research, crafted and categorized metrics, which fostered discussion both of objectives for genetic and genomic services and of available measures to meet those objectives. The group discussed existing data sources where available, and potential data sources that could be mined in the future. This summary lays out the rationale and context for the final set of performance objectives and outcome measures created. It describes the metrics developed for population based genomics focused on HBOC and LS, which may also be applicable for other genetic conditions. It is expected that similar, yet unique metrics can be applied to future evidence-based public health genomic activities. The group stopped short of actually proposing targets since states vary widely in their current application of public health genomics activities and actual baseline data are not currently known for many of the metrics proposed. Our goal in proposing such metrics is to stimulate further dialogue and accelerate efforts to further refine population health genomics metrics.

The three performance objectives and outcome measures that all states are encouraged to pursue consistent with Healthy People 2020, include increasing: 1) the number of women with a family history of HBOC/LS who receive genetic counseling; 2) the number of persons with newly diagnosed colorectal cancer who receive genetic testing to identify LS, or familial colorectal cancer syndromes; and 3) the number of family members screened following identification of HBOC/LS mutations (i.e. cascade screening). This suggests that the current capacity for measurement of genomic testing implementation is still at an early phase. However, the identification of these three performance measures is considered feasible for implementation, and provides strong targets for assessment of effective public health impact.

Although the full list of performance objectives and outcome measures identified is extensive, it is not complete, nor has any validity or effectiveness testing been performed. Further, most performance objective descriptions will warrant additional detail. For example, under the broad category of implementation, within the ‘Acceptability’ subgroup, a more detailed definition of what ‘comfortable’ entails under the objective ‘increase in the number of providers who are comfortable providing HBOC/LS screening services’ is required. In this same category, ‘appropriately’ needs to be defined under an ‘increase the number of providers who appropriately refer HBOC/LS at-risk families’. Additional suggestions include the possibility of sub-dividing a more general outcome into more specific outcome measures, for example, under the broad category of ‘Client Outcomes’, within the ‘satisfaction’ measures, more specific patient sub-groups (e.g. disease group) could be measured.

Other outcome measures such as costs for direct or indirect effect are also of importance because they may impact feasibility and implementation of public health genomic programs. Frequently, return on investment (ROI) analysis is performed at the ‘back-end’ of program evaluation; however, within economic conditions, the importance of conducting ROI analysis prior to program design and implementation will become even more critical to support, sustain and expand a program. In cases where ROI may be challenging to determine, social return on investment is another essential outcome measure that incorporates benefits that may not have immediate monetary value but have substantial social value.

A number of state genomics programs are already addressing some of the performance objectives identified. For example, one of the key objectives of the CDC funded cancer genomics program at the Michigan Department of Health and Human Services is to reduce the impact of hereditary cancer within the state[10]. They have collected data through the 2015 Michigan BRFSS (MiBRFSS) to assess personal and family history of breast and ovarian cancer and utilization of cancer genetic counseling[11]. Similarly, the Washington State Department of Health Screening and Genetics Unit in partnership with the CDC funded Oregon Health Authority Genomics Program, surveyed gastroenterologists to determine how often clinicians routinely screen the tumors of patients found to have colorectal cancer (CRC) to identify LS, as well as identify any resources this group of practitioners would need in order to initiate universal screening protocols among newly diagnosed cases of CRC (DL Doyle, written communication, February 2017). The results of the survey were returned to the practitioners along with an educational resource guide that included materials specific for the clinician as well as materials that can be shared with families.

State public health genomic activities are viewable through the CDC’s Public Health Genomics Knowledgebase (PHGKB) State Implementation Map[12], a clickable map identifying implemented state-specific genomics applications. These cover applications for HBOC syndrome, LS, familial hypercholesterolemia, newborn screening, and more. Results can be filtered for data sources, programs, education, policy, tools, and general information. In addition, CDC has a useful Genomics Applications Toolkit for Public Health Departments[13] that allows states that are just beginning to implement strategies, to benefit from work already conducted by the five CDC funded states.

Given the current state of genomics measurement within state public health systems, the GPHAC Implementation Work Group members suggest three ‘next steps’ for consideration, with the goal of increasing the number of measures that a plurality of states and territories could integrate within their operations.

Refinement and adoption of common measures; particularly for the outcomes that all states are encouraged to pursue as well as those that most programs would be expected to capture. The group felt that establishing common measures across states for currently captured data would improve the ability to track progress in genomic testing and enable states to learn from one another where system improvements are made.

Developing new data collection sources; given the number of measures that had no obvious existing data source, the group recommended the development of new mechanisms to collect data, particularly around key service system outcomes, like safety, efficiency and patient centeredness.

Pooling data; where common measures are already being collected, the group recommended consideration of a broader effort to pool data. This could enhance existing consortia efforts (e.g. Lynch Syndrome Screening Network) to include system-level performance outcomes, or could represent a new effort among states to collaborate.

As we enter an era of precision medicine, organizing and planning to mitigate disparities in access to genetic/genomic services is of utmost importance. The GPHAC efforts included consideration of health disparities at every juncture. The Academies six domains of health care quality includes equity, defined as ‘providing care that does not vary in quality because of personal characteristics such as gender, ethnicity, geographic location, and socioeconomic status’[14]. We have included equity as an outcome measure, however it is recognized that additional identification and measurement of predictors that give rise to disparities in access needs to be further elucidated. A substantial body of literature has documented health disparities in the utilization of genetic tests for hereditary cancers[15, 16, 17, 18], but most of this work to date has focused on individual-level factors that affect the way patients and providers make decisions about testing (e.g., a patient’s health literacy, risk comprehension, social supports, or a provider’s knowledge of clinical practice guidelines, cultural competence, or implicit bias). Research on these health inequalities, however, has tended to ignore system-level factors that give rise to disparities, even though those are viable targets for public health surveillance and policy-level action. Future work of the GPHAC will identify and incorporate these predictors in the state public health genomics toolkit.

In summary, we are proposing multiple genomics outcome measures specific to HBOC and LS, to help guide the implementation of genomic evidence-based tools, programs and policies within public health systems. This is clearly a starting point intended to engage more states and academic researchers in refining and adopting population based measures that will allow for assessing the implementation of evidence-based genomic medicine and its population-level impact on health outcomes.