Recommendations for the use of operational electronic health record data in comparative effectiveness research.

There is an increasing amount of clinical data in operational electronic health record (EHR) systems. Such data provide substantial opportunities for their re-use for many purposes, including comparative effectiveness research (CER). In a previous paper, we identified a number of caveats related to the use of such data, noting that they may be inaccurate, incomplete, transformed in ways that undermine their meaning, unrecoverable for research, of unknown provenance, of insufficient granularity, or incompatible with research protocols. In this paper, we provide recommendations for overcoming these caveats with the goal of leveraging such data to benefit CER and other health care activities. These recommendations include adaptation of "best evidence" approaches to use of data; processes to evaluate availability, completeness, quality, and transformability of data; creation of tools to manage data and their attributes; determination of metrics for assessing whether data are "research grade"; development of methods for comparative validation of data; construction of a methodology database for methods involving use of clinical data; standardized reporting methods for data and their attributes; appropriate use of informatics expertise; and a research agenda to determine biases inherent in operational data and to assess informatics approaches to their improvement.

Introduction

The last several years have seen substantial investment in the adoption of electronic health records (EHRs) in the U.S. and elsewhere, offering great potential to improve the quality, safety, and cost of healthcare.1 EHR adoption is also likely to improve our ability to advance biomedical and healthcare science and research through the reuse of clinical data.2–4

At the same time, there has been substantial U.S. investment in various aspects of clinical and translational research, including comparative effectiveness research (CER) that aims to study populations and clinical outcomes pertinent to real-world clinical practice.5 Additional federal investment in research infrastructure includes the Clinical and Translational Research Award (CTSA) program of the U.S. National Institutes of Health, leading many institutions funded by CTSA awards to develop research data warehouses derived from operational EHRs and other systems.6,7 Another source of federal investment has come from the Office of the National Coordinator for Health Information Technology (ONC) through its Strategic Health IT Advanced Research Projects (SHARP) Program, with one of the four major funded research areas focusing on reuse of clinical data.8

Our previous paper reviewed some successful efforts to use operational EHR data for research.9 One prominent success has come from the Electronic Medical Records and Genomics (eMERGE) Network, which has demonstrated the ability to validate existing research results and generate new findings, mainly in the area of genome-wide association studies (GWAS) that associate specific findings from the EHR (the “phenotype”) with the growing amount of genomic and related data (the “genotype”).10, 11 Another successful effort has come from analyses derived from the Health Maintenance Organization Research Network’s Virtual Data Warehouse (VDW) Project, where, for example, researchers were able to use data to demonstrate a link between hyperglycemia in pregnancy and childhood obesity.12, 13 Using similar methods, other researchers have been able to replicate the findings of randomized controlled trials using EHR data and appropriate statistical methods.14–18

As we noted, however, routine clinical data are collected for clinical and billing uses, not for research.9 We described and detailed several caveats for the use of such data for CER, including inaccuracy, incompleteness, transformation in ways that undermine their meaning, inaccessibility for research, unknown provenance, insufficient granularity, and incompatibility with research protocols. Informed by these caveats and motivated by the potential benefits of reusing operational clinical and administrative data for research, quality measurement and improvement, and other analytical purposes, we believe that there have been significant informatics advances in support of such reuse. In this paper, we adopt the critical appraisal approach of evidence-based medicine (EBM) to the problem, discuss some of the informatics challenges to the use of operational clinical data for CER, and then develop an informatics roadmap for moving forward.

Recommendations for Using EHR Data for CER

The following sections outline nine major recommendations for advancing the use of operational EHR for CER (see Figure 1). Table 1 provides a summary and description of each recommendation, which correspond with the sections and facets in the rest of the paper.

Figure 1.

Assessing data from operational sources for clinical research.

Table 1.

Summary of recommendations for advancing the use of operational EHR data for CER.

Recommendation	Description
Apply an Evidence-Based Approach	Ask an answerable question, find the best EHR data (“evidence”), appraise the data, apply data to the question
Evaluate and Manage Data	Assess availability, completeness, quality (validity), and transformability of data
Create Tools for Data Management	Create software (especially pipelines) for data aggregation, validation and transformation
Determine Metrics for Data Assessment	Determine whether a particular site’s data are “research grade”
Develop Methods for Comparative Validation	Develop tools that support analysis of multisite data collections
Develop a Methodology Knowledge Base	Develop a data catalogue that relates data elements to recommended transformations
Standardize Reporting Methods	Provide details of data sources, provenance and manipulation, to support data comparison
Engage Informatics Expertise	Ensure validity of findings derived from data collected from disparate sources
Include an Informatics Research Agenda	Generate systematic studies of inherent biases in EHR and data collection methods, such as data entry user interfaces

Apply an Evidence-Based Approach

The EBM process offers many analogies for that can guide the use of operational clinical data for CER and other forms of research. Some may consider EBM to be antagonistic to EHR data reuse, as EBM gives the most value to evidence from controlled experiments, especially to randomized controlled trials (RCTs), while viewing the use of real-world observational data as a lower quality form of evidence, due to its being incomplete, incorrect, or inconsistent. In reality, observational data analysis holds a role in relation to CER (or any type of clinical research) that is different from and complementary to that of traditional clinical trials. RCTs provide information on efficacy in well-defined populations under tightly regulated circumstances. Observational studies, by their nature, provide a more real-world view, cover broader populations than can be analyzed in a clinical trial setting, and can explore the associations with and effects of different levels of exposure to the intervention being studied. Therefore, both types of studies are necessary to understand how and why an intervention is working compared with expectations about it.

In contrast to clinical trials data, EHR data are voluminous and much more reflective of the real clinical world. In addition, performing RCTs on all clinical questions of interest would be prohibitively expensive. But we can look to the process of EBM to guide us in how to best consider EHR data as part of our body of evidence. EBM uses a principled approach to find and apply the best available information to make clinical decisions. In particular, EBM uses four steps that we can apply analogously to research use of EHR data, whether in single institutions or across multiple institutions and settings, analogous to the use of systematic reviews and meta-analysis:19

Ask an answerable question – In applying EBM, the first step of asking an answerable question may be the most important. For example, knowing to ask whether a test or treatment is efficacious is not enough. Rather, we need to know at a minimum the effectiveness of the intervention relative to some alternative approach in a particular patient population or setting. This same approach is obviously necessary for using operational EHR data for clinical research. In selecting EHR data for use in a study, the researcher should adopt the established population, intervention, comparison, outcomes, timing, and setting (PICOTS) framework for formulating the research question.20

Find the best evidence – For the second step, the principle from EBM is very much the same, even if the techniques of obtaining evidence are very different. The “evidence” in this case is the data in EHRs and other systems that, as noted previously, may be incomplete, incorrect, and inconsistent. We therefore need to determine if we have the proper data and, if so, whether they can be applied to answer our question.

Critically appraise the evidence – In the third step, just as with EBM, the evidence must be critically appraised. Can we trust the inferences and conclusions from the data? Are there confounding variables in those data? Have the appropriate methods been used? These may be critical with EHR data where assignment of cause and effect could be difficult, if not impossible. The solution likely comes back to asking the right question, i.e., one for which we can have confidence that we have found the correct answer.

Apply it to the patient situation – Lastly, we need to ask, can the evidence be applied in our setting? Just as some RCTs answer questions in patient populations very different from those of the clinician making decisions, we must ascertain if the results obtained from this approach can be applied to a specific clinical context.

As EBM recognizes that the single studies are limited to reflecting a single setting and often with small sample sizes, the techniques of systematic review and meta-analysis have been developed to aggregate results from multiple studies and, where possible, aggregate their data in meta-analysis. The equivalent of this approach for clinical data is combining clinical data from multiple settings or institutions.

Evaluate and Manage EHR Data for CER

Any research study, whether CER or not, requires the highest quality data available for observing the object of that study. For data coming from an operational EHR, this means we need to determine whether the data required are available, complete, and valid.

To illustrate the need for these types of data evaluation and management apparatus and tools, Figure 1 shows a prototypical workflow consisting of the major steps needed to assess such factors for each datum in a study that uses EHR data. This process is inclusive of recommendations about data management, including evaluating data, tools for data management, metrics for data assessment, and methods for comparative validation.

Once the researcher defines the question, he or she must then identify the EHR data elements corresponding to the desired data and determine whether they are actually available. For each desired datum, the researcher must next assess if that element is complete, i.e., available for all individuals whose data will be used in the study. Of course, determination of data completeness and data quality are challenging.21,22

In addition to determining whether data are available and complete, the researcher must also ascertain whether the data are valid. As was seen in the caveats above, we know there are many reasons why data are not valid. A review by Weiskopf and Weng identified past methods of assessing data validity.23 These included comparison to a gold standard, agreement of data elements among sources, comparison of data distribution across sources, validity checks, review of data capture logs, and element presence in multiple sources. If the data are not valid, then the researcher must determine if it is possible to transform them. There are a number of ways that data can be transformed, such as combining them with other data, e.g., linking claims and EHR data or adding data from other sources.24, 25 Of course, such transformation requires further validity checking.

If individual data are not available, complete, or transformable, then the researcher must either continue the study with noted limitations, or he or she must find alternative or surrogate sources of data. The researcher must also carefully consider the usefulness and relevance of each data element against a testable hypothesis and an analytic model or analytic plan in order to be efficient with the time of those maintaining and performing operations on the data. Often the scope of technical and methodological skills and tools required to address the preceding information needs is beyond the capabilities of an individual researcher, necessitating the engagement of a broader, multidisciplinary team of investigators, informaticians, and research or technical support staff. Building upon this requirement and informed by the findings and caveats introduced previously, we consider several critical facets of our recommendations, with the ultimate objective of motivating a CER-relevant informatics research and development agenda aligned with them. Figure 2 then illustrates the interrelationships between our proposed recommendations, their constituent facets, and the data-analytic workflow introduced in Figure 1. We now describe these facets.

Figure 2.

Relationships between core datum assessment operations and the facets of informatics research and development needed to support and enable comparative effectiveness research.

Facet 1: Tools and methods capable of supporting the aggregation, transformation, and validation of heterogeneous data sets

Investigators who carry out certain types of research, such as CER, but also health services research, epidemiology, pharmacogenomics, and genome-wide association studies, all need to aggregate information obtained from different health systems and their data sources. In each case, investigators must also aim to verify the accuracy of information obtained from each source and use the appropriate tools to transform and optimize the data for research use. In order to satisfy such information needs in a scalable and contextually appropriate manner, CER investigators must be provided with access to highly usable, configurable, and shareable tools and methods that can be used to build, instrument, and apply data-analytic pipelines targeting the preceding requirements. Such pipelines would ideally be composed of multiple “plug ins” or modules that are made available for community-wide reuse via public libraries or equivalent means.

CER researchers using EHR and other operational data must also apply methods to ensure their completeness and quality. Emerging methods for data completeness include explicit and predictive analysis of data in the EHR.21 Likewise, methods for data quality assessment include such techniques as checking for adherence to common data models across sites and comparing actual versus expected observations within sites.22, 26 CER researchers must also work with clinicians, health care organizations, and others to improve the data that informs their “downstream” research.

It is also possible to gain insight into potential approaches to specific data aggregation and quality-assurance best practices based on analogous efforts that have been undertaken in the context of methods used to evaluate the quality and reproducibility of clinical laboratory results. A variety of tests can be carried out to characterize internal consistency of results at each site. Results obtained by laboratories are typically compared; for example, laboratories analyze specimens produced by an accrediting body such as the College of American Pathologists (CAP), which in turn generates reports comparing the performance of participating laboratories. A comparable approach to testing EHR data could be developed to ascertain a clinical organization’s ability to produce “research-grade” data.

Facet 2 – Tools and methods that can enable the comparative validation and analysis of multisite data collections

Comparisons between clinical sites are complicated by the fact that patient populations naturally differ between health systems. However, statistical properties associated with relationships between diagnostic entities, laboratory values and medication orders can be evaluated for each site and then compared from site to site, possibly applying normalization or other transformations of the data where appropriate. The key here is likely to lie in examining interrelationships between comorbidities with given patient demographics; for instance, one would expect characteristic collections of comorbidities related to each broad class of diseases. Thus we must develop and use a compendium of reference information models (RIMs), verification- and validation algorithms, and reference data sets (e.g., for use as comparative “gold standards”) so that CER investigators can draw upon and rigorously apply a well-known, validated, and reproducible “tool box” of comparative analysis methods. For example, a rich set of “sanity check” characterizations can and should be carried out to assess internal consistency of each institution’s clinical data characterizations. For instance, one can expect characteristic constellations of diagnostic codes, laboratory test values and medication orders for a patient classified as having diabetes mellitus (DM) with renal, ophthalmic or neurological manifestations; e.g., a patient classified as having DM with renal complications would be expected to have abnormal estimated glomerular filtration rate (eGFR) and/or proteinuria.

Facet 3 – Tools and methods for data transformation and synthesis in support of CER

As noted previously, CER researchers require validated point-by-point comparative data among the study populations. The use of EHR data presents unique challenges for obtaining sufficient data of the particular comparators – whether the study compares, for example, different medications, medical devices, or types of surgical versus nonsurgical interventions. Many EHR systems are not designed to collect accurate and complete data sets that are sufficiently robust for such comparisons.

Other unique challenges and needs in CER are the lack of nationally adopted data-related research methodologies that can be utilized in person-centered comparative outcomes studies and pragmatic studies that reflect real-world settings. EHR data that are collected, archived, and made accessible must be standardized across EHR vendors and users with ongoing procedures (as described in the previous section) to ensure consistency of results within sites and comparability across sites. For example, patient-reported outcomes (PROs), patient problem lists, and functional outcome measures are now collected using different tools, and may not be uniformly or completely collected in EHRs.27 In addition, there are differences in services provided, data entered by health care entities, and information provided by patients that contribute to the inconsistency of completeness and accuracy of EHR data for CER.

The above challenges need to be addressed to advance CER through the development of widely applicable methodological and technical solutions.28 One approach is the creation, maintenance, and dissemination of collections of platform-independent tools: (1) data capture instruments; (2) algorithms for the inference or synthesis of complex and, potentially, temporally dependent phenotypes from primary data; and (3) knowledge management technologies and best practices. Such tools could support improved understanding of data provenance in EHRs and other systems such that researchers have a sufficient understanding of the sources and potential biases surrounding the data for a given research study.

Develop a Methodology Knowledge Base

In addition to devising methods and tools for improved data reconciliation and integration as introduced in the above three facets, solutions to the challenges in reusing data for research purposes require an informatics roadmap. Such a roadmap must include resources to aid CER researchers in applying complex methods arising from the use of operational data. At the simplest level, a resource that lays out the concerns raised in this article is needed to which CER researchers can refer. This may include, for example, a table relating data elements and types, and transformations recommended or not. Such a resource – a methodology knowledge base – would need to be updated as novel issues are raised, as novel solutions are offered, and as instructive examples are shared. The next level would be a more interactive resource that supplies this knowledge in a more active mode than does a passive knowledge base. CER researchers, especially novices, employing EHR data could benefit from integrating methodological decision support into their research environment. This should also include new research methodologies as well as new ways to measure and encourage research-related activities during routine practice.29–31

Another informatics challenge and opportunity is making the efficient recording of high-quality and standardized data easier for clinicians. As noted above, clinical documentation is not usually a priority for busy clinicians.32, 33 Research has shown clinicians already spend a substantial amount of time (20–103 minutes per day) entering data related to patient care.34 Where possible, clinicians must be aided by the easy incorporation of other sources of reliable data.35 We also must “bake in” data standards so that their use by clinicians and others in the health system is seamless.36 Better adherence up-front to standards will be make data interoperability easier at later stages, especially in its use for CER.

Whereas traditional observational studies usually enlist a human being to enter data into a research database, manual curation in large-scale EHR-based studies is not practical. Therefore, an even more detailed description of the exact data sources and methods to encode the data into research variables is needed to allow future researchers to repeat a study more reliably, to compare studies, and to better identify potential biases in the results. For example, when there are multiple database queries for the same intended search, e.g., “all patients with diabetes,” this may be searched as all patients “with a diabetes-related billing code,” “with a diabetes-related problem listed on their last problem list,” “with an order for insulin,” “with an elevated hemoglobin A1C level,” or any number of other methods. The performance characteristics (e.g., sensitivity and positive predictive value) of these queries should be ascertained or, at least, alternative methods should be compared and reported.

Standardize Metadata Reporting

Likewise, the readers and consumers of research using these data require reporting methods that go beyond stating the data source or a cursory explanation of how they were transformed. To this end, much more detailed methodology of data sources, provenance, and manipulation must be provided in sufficient detail to provide an understanding and the limitations of such use. This will likely require a reporting standard comparable to the Consolidated Standards of Reporting Trials (CONSORT) and STrengthening the Reporting of OBservational studies in Epidemiology (STROBE) tools.37, 38 The STROBE checklist, for example, stipulates that all outcomes, exposures, predictors, potential confounders, and effect modifiers be clearly defined, giving diagnostic criteria (if applicable), and that for each variable of interest, data sources and details of assessment methods (measurement) be given, describing comparability of assessment methods if more than one group is being studied.

Creating a common format for reporting such details would be beneficial, as researchers would be encouraged to provide complete information that make comparison of studies easier. The format can be seen as a way of presenting an explicit description of a study’s data assumptions. Multicenter observational studies may require separate descriptions for each institution if the algorithm used to generate the study variables differs among institutions. The description may require not just the researcher’s procedures, but also a description of relevant aspects of how the EHR is used in the institution. For example, if only a subset of clinicians use order entry, then using orders as a source for coding the use of a medication could bias the results.

Studies using EHR data must also embrace the diversity of disease definitions in ways that set different expectations for presentation of results than for clinical trials. Results of clinical trials, on the one hand, provide a single point estimate and a confidence interval. Observational studies, on the other hand, should present multiple results under different assumptions. Sensitivity analyses should be the norm in reports of these types of analyses. Readers should see how results compare under different assumptions regarding disease definitions. If results are robust to assumptions, the findings are strengthened. If results differ, then the trends in the difference related to the characteristics of the underlying cohorts should be clinically plausible. Adhering to a “clinical trial” standard of making an a priori decision about cohort definition, and presenting findings based on that one analysis, can be unintentionally misleading (or purposefully misleading if the investigators did test different cohorts and reported on only one result that supports their hypotheses).

Engage Informatics Expertise

Of course, as the sections above illustrate, numerous informatics issues must be addressed to enable the successful reuse of clinical data for CER. To that end, another key recommendation must be for data-based research projects to include experts in biomedical informatics, a field closely related to but distinct from computer science, IT, and biomedicine.39 Just as the inclusion of those with particular methodological expertise in key research activities (e.g., biostatisticians, subject matter experts, etc.), the inclusion of informatics experts can help address the validity of findings generated during CER studies that rely on the reuse of clinical data often collected initially for different and varied purposes and deriving from disparate systems.40

Generate an Informatics Roadmap for CER

Achieving the goals of the recommendations as outlined will require an informatics research agenda. There has been very little study, for example, of how the EHR (or really any medical record) works and what biases the health care process creates in its data. 41,42 This is an especially challenging area because whereas completeness and accuracy are somewhat apparent and measurable, bias is insidious. The data may look about right, but due to influences such as the health care process 41, the effects on the recorded data can influence the outcome .42 We must study workflows, develop and evaluate user interfaces that allow the entry of high-quality data in time-efficient ways, and characterize the limitations of all data to better assess how they can be improved. Such research is not simply a part of the methodology of a clinical study but a research endeavor in its own right.

While there are many caveats related to the reuse of operational clinical data, there are also tremendous opportunities. Some successes have already been achieved, and with attention to the techniques proposed in this paper, we believe that additional ones will likely be forthcoming.