New analytical methods for a learning healthcare system: a message from the guest editor.

Whether one reads Computerworld1 or Institute of Medicine issue briefs2, it’s clear that most health professionals now accept the idea that existing electronic clinical data (ECD) and other health records can be used to manage and improve the processes, outcomes, and quality of health care. Indeed, the increasing popularity of the term “learning health care system” signals the broad acceptance of the idea that routinely collected clinical data can—and indeed should—be used to advance knowledge and support continuous learning.

Okun et al. (2013), for instance, demonstrate how electronic health data can help to improve disease monitoring and tracking; better target medical services for improved health outcomes and cost savings; help inform both patients and clinicians to improve how they make decisions during clinical visits; avoid harm to patients and unnecessary costs associated with repeat testing and delivery of unsuccessful treatments; and accelerate and improve the use of research in routine medical care to answer medical questions more effectively and efficiently.1 Existing ECD is also essential for the Center for Medicare and Medicaid Innovation’s new, rapid-cycle approach to evaluation, which aims to deliver frequent feedback to providers in support of continuous quality improvement, while rigorously evaluating the outcomes of each model tested.3

But despite what the big data enthusiasts say, none of this is easy. Whether it’s used to manage the care of individuals, to carry out quality improvement (QI) studies, or to conduct comparative effectiveness research (CER), data in individuals’ clinical records must be accurate and complete, and the collection of records available for analysis must be reasonably representative of the population served. And, without randomization, research results can be biased by selection effects and confounding with factors not recorded in existing electronic health records (EHR). The breadth of information available, the information’s timeliness and volume, machine learning algorithms, and sophisticated visualizations cannot easily overcome problems of inaccurate or incomplete data, and cannot confidently infer cause and effect relationships from observational data. Without appropriate analytical methods, none of the benefits that Okun et al. (2013) promise is achievable.

The Electronic Data Methods (EDM) Forum was established three years ago to advance the national dialogue on the use of electronic clinical data for the conduct of CER, quality improvement (QI), and clinical decision support. One way the Forum and its sponsor, the Agency for Healthcare Research and Quality (AHRQ), chose to do this was to publish eGEMs; and this special issue was conceived to address the methodological challenges inherent in this work. The seven papers in this issue are the result of invitations to researchers who have participated in EDM Forum activities as well as responded to an open call for paper in early summer 2013. The invitation indicated a focus on methods for conducting CER, patient-centered outcomes research (PCOR), and QI using data from EHRs. We also expressed an interest in publishing researcher perspectives on using ECD for both observational and experimental study designs, especially submissions focusing on methodological challenges resulting from using these new data sources and lessons learned from facing these challenges. Papers focusing on new or emerging methods that analyze the strengths and weaknesses of these methods for particular uses and summarize best practices to ensure rigorous results were also of particular interest for this special issue. We interpreted the “clinical” in ECD broadly, to include most forms of electronic health data including billing records, EHRs, and so on.

Preparing Electronic Clinical Data for Analysis

Before ECD can be used for analysis, they must first be prepared so that they accurately represent the health risks and outcomes of the patients studied, as well as the care provided by health care system. Two of the papers in this special issue address different aspects of this challenge. Davis and colleagues report on a retrospective observational study in veterans with rheumatoid arthritis to validate diagnostic and procedural codes for identification of acute cardiovascular events. Beyond the specific population and set of outcomes in this example, their analysis demonstrates a validation method that could prove useful in other settings. Reimer and Madigan demonstrate how the development of a fully integrated medical transport record for patients undergoing medical transport—as an example—will offer the ability to address complex questions related to patients’ clinical outcomes in a real-world clinical setting while providing an electronic data infrastructure that can enable high-quality, clinically rich, prospective, and multisite data collection to support CER.

Methods for Analyzing Electronic Clinical Data

Four of the papers in this special issue address methods for analyzing ECD. Stuart and colleagues outline the challenges in estimating causal effects using electronic health data generally, and offer some solutions, with particular attention to propensity score methods. The methods are illustrated with a case study showing how Medicare and Medicaid administrative data can be used to estimate the effect of the Medicare Part D prescription drug program among individuals with serious mental illness. Zurovac and colleagues demonstrate how the concept of multifactorial experiments, drawn from the evaluation literature, offers promise for CER with EHRs. As an example, they focus on quality improvement in physician practices.

Wells and colleagues address the analytical challenges presented by missing data in EHR systems. They examine some of the unique aspects of such missing data, present some statistical advice about how to handle these issues, and provide some suggested areas for future methodological research. A more specific challenge in using ECD is that observations are not scheduled as they would be in a designed randomized clinical trial, but rather occur at irregular intervals coincident with patient visits, which potentially depend on outcomes. Luo and colleagues show how discrete-time hidden Markov models can be used to estimate transition rates in this context, using chronic kidney disease as an example.

Examples of Analyzing ECD for CER

A final paper in this issue provides an example of how methods for preparing and analyzing ECD can come together to enable rigorous CER. Written in the form of a protocol for a prospective, longitudinal cohort study, Sills and colleagues demonstrate how survey methodologies and secondary analysis of existing structured clinical, administrative, and claims data can be used to estimate the effects of Patient-Centered Medical Home characteristics on asthma control in adults and children. The setting for this research is the Scalable Architecture for Federated Translational Inquiries Network (SAFTINet), an AHRQ-funded safety net-oriented practice-based research network, which is designed to measure these variations in delivery system characteristics.

Next Steps

As useful as they are, these papers only begin to address the methodological developments needed to advance the national dialogue on the use of ECD to conduct CER, support QI, and generally to improve outcomes in a learning health care system. These papers offer a beginning snapshot of some critical ideas and innovations shaping the field. eGEMs remains interested in publishing researcher perspectives on methodological challenges resulting from using ECD and lessons learned from facing these challenges, and welcomes future methods-related submissions to our general issue. For more information on submitting papers, please see the eGEMs Instructions for Authors.

I encourage you to read the papers in this collection and think about whether your organization has developed similar approaches—or radical alternatives—and write about them. You might also try some of the approaches discussed here, and let us know how they work out. We are particularly interested in papers that address key topics raised in prior EDM Forum discussions, including strategies for evaluating and addressing missing data and other data quality issues; analytic strategies that can account for granular temporal or spatial information; and new, rigorous methodological approaches to maximizing the use of “big data” to generate new evidence and determine what works best for whom and under what conditions.