Clinical laboratory analytics: Challenges and promise for an emerging discipline.

The clinical laboratory is a major source of health care data. Increasingly these data are being integrated with other data to inform health system-wide actions meant to improve diagnostic test utilization, service efficiency, and "meaningful use." The Academy of Clinical Laboratory Physicians and Scientists hosted a satellite meeting on clinical laboratory analytics in conjunction with their annual meeting on May 29, 2014 in San Francisco. There were 80 registrants for the clinical laboratory analytics meeting. The meeting featured short presentations on current trends in clinical laboratory analytics and several panel discussions on data science in laboratory medicine, laboratory data and its role in the larger healthcare system, integrating laboratory analytics, and data sharing for collaborative analytics. One main goal of meeting was to have an open forum of leaders that work with the "big data" clinical laboratories produce. This article summarizes the proceedings of the meeting and content discussed.

INTRODUCTION

The clinical laboratory is a major source of health care data. Increasingly these data are being integrated with other data to inform health system-wide actions meant to improve diagnostic test utilization, service efficiency, and increase “meaningful use.” Clinical laboratory analytics is the systematic evaluation and communication of clinical laboratory testing data to improve healthcare operations and patient outcomes.

Most of the data created by a clinical laboratory is already coded and transmitted to electronic health records as discrete elements with meaningful flags, making it more amenable to analysis than text-based clinical histories and pathology reports. As health care systems are pressured to improve efficiency and reduce costs while improving patient satisfaction and clinical outcomes it will be increasingly important to leverage clinical laboratory data and the expertise of the pathologists and laboratorians that best understand diagnostic test data.

Clinical laboratory analytics is currently not organized as a discipline. Indeed, a survey of analytics studies of diagnostic test utilization management showed that these studies were published in dozens of general journals, and were conducted by physicians with many specialties.[1] We gathered clinical laboratory analytics experts and interested academics to a special meeting in conjunction with American College of Laboratory Physicians and Scientists (ACLPS) meeting in San Francisco on May 29, 2014. The purpose of the meeting was to discuss the current practices and future directions for clinical laboratory analytics. This paper summarizes the content and conclusions of the discussion on the current and future state of clinical laboratory analytics.

MEETING ORGANIZATION AND STRUCTURE

There were 80 registered participants in a special clinical laboratory analytics session at the ACLPS meeting in San Francisco on May 29, 2014. The meeting featured three short presentations describing the current state, potential applications, and future challenges for clinical laboratory analytics. The meeting also included three panel discussions of experts in the nascent field of clinical laboratory analytics. The meeting concluded with an open discussion of invited panelists and participants about future roles for laboratory physicians and scientists in relation to the larger healthcare system and the “big data” that clinical laboratories produce. This manuscript will summarize the presentations at the ACLPS 2014 satellite meeting on clinical laboratory analytics and synthesize comments from expert panelists and participants about the future of the data that the clinical laboratory produced.

REPORTS OF PRESENTATIONS

Data Science And Laboratory Medicine: Laboratory Data And The Larger Healthcare System

Brian Jackson, vice president and chief medical informatics officer at ARUP Laboratories and Associate Professor of Pathology at the University of Utah proposed a framework for considering what data and metrics are needed for effective management of clinical diagnostics. Such programs will be essential for laboratories to succeed within accountable care and other capitated payment arrangements. The metrics fall into three categories: Cost, process quality, and patient benefit. Within each category, the specific needs range from highly granular data needed for front-line management up to aggregate data needed for executive decision-making. Some of these data and metrics exist already, but most have yet to be developed. For example, most laboratories have reliable cost data at the aggregate level (total lab budget) as well as broken down by accounting category (labor, supplies, equipment). And many laboratories, particularly larger ones, have reasonable estimates of per-test costs. But few if any laboratories have taken costing all the way to the clinical front-line by estimating the cost per case.[2] The second category, process quality measurement, is extremely familiar to laboratorians, but there are opportunities here as well. Analytic quality programs in modern laboratories already cover the full range of tests. Pre and postanalytic quality programs, on the other hand, may only cover a subset of high-visibility areas. A handful of turnaround time measures, for example, cannot realistically reflect the needs of all of the thousands of clinical processes spread across dozens of departments in modern hospitals. And rolling these programs up into unbiased aggregate measures of overall system reliability will also require future work.

The largest and most complex category is measurement of patient benefit. This is an extremely immature domain of clinical medicine. The basic level is patient benefit per case. In some cases, physicians’ diagnostic actions can be measured against guidelines, but only a small proportion of clinical diagnostic scenarios have specifically applicable guidelines. One way to supplement guidelines is to use local expert opinion, either opportunistically or within formalized oversight committees. It can also be extremely useful to measure variation. In a sense, comparing physicians’ actions to those of their peers can be thought of as a sort of crowd-sourced expert opinion. Measuring patient benefit at a test level, as opposed to a clinical case level, requires a different approach. The best example is cost-effectiveness analysis. A key point to consider here is that test benefits are only partially determined by the analytic characteristics of that test. Other considerations can dominate the picture, such as the use of related diagnostic tests and empiric therapy. Finally, global measures of diagnostic care quality are in their early infancy. National programs such as Physician Quality Reporting System and Health plan Employer Data and Information Set contain only a handful of diagnostic measures each, and so they cannot hope to assess in a balanced way the hundreds of thousands of diagnostic-related activities that occur today across the world of clinical medicine. Inventing new clinical quality measurement programs that are economically feasible yet have adequate breadth and balance represents an enormous challenge.

Dr. Jackson was joined by Jason Baron, Assistant in Pathology at the Massachusetts General Hospital and Instructor of Pathology at Harvard Medical School, as well as Enrique Terrazas, Chief of the Laboratory Information Systems and Professor of Laboratory Medicine at the University of California, San Francisco, in a panel discussion about integrating laboratory analytics data into the larger healthcare framework. Dr. Baron suggested that one focus of clinical laboratory analytics should be to provide enhanced decision support at the time of test ordering and test result interpretation. In particular, he highlighted the need to develop a more robust decision support infrastructure capable of implementing both rules-based and machine learning-based algorithms within a clinical environment and workflow. Dr. Baron also provided a few examples of how “offline” clinical laboratory analytics can be useful in analyzing and improving test utilization. For example, analytic approaches can be used to identify inter-clinician variation in test ordering patterns beyond that explainable by clinical factors.

Dr. Terrazas added that the interest level in clinical laboratory analytics points to a definite need in the clinical lab community for better information, not just data, from the LIS. Clinical laboratory analytics is a growing field with vendors starting to fill the void, however, vendor products are expensive and may be out of reach for many laboratories. Dr. Terrazas commented that most of the vendor solutions involve exporting data from the LIS and doing the analytics in a separate environment. Better solutions might be achieved with simple software driven by expert laboratorians and pathologists without the use of external vendor solutions. True business intelligence is understanding the raw data and how it relates to clinical care then displaying these data in a way that others can understand. Using visual analytics software, multidimensional data such as laboratory data (time, value, gender, age, etc.) can be examined and manipulated in real time using off the shelf software.[3] Visual analytics software has been used to examine immunological (flow cytometry) data[4] as well as data mined from a clinical data repository, the analysis of which included comparing erythrocyte sedimentation rate and C-reactive protein across patients.[5] Analysis can be performed on any dimension, thus, for example, a visual representation of turnaround time looking for outliers can be performed.

Integrating Laboratory Analytics: Examples Of Data Use In The Context Of Broader Healthcare Analytics

Bryan Clements, the Director of Finance Decision Support, and Dr. Brad Brimhall, Medical Director of Clinical Laboratories, presented an overview of some analytics projects they have completed at the University of Mississippi Health Care System. The projects largely fell into four categories.

The first category required integration of laboratory and financial data to examine utilization of laboratory tests. One project in this category examined realized savings from tests canceled by providers through duplicate best practice alerts; another examined prospective cost savings from such alerts for 23 unnecessary test combinations (e.g. thyroid-stimulating hormone ordered concurrently with free T4) from the perspective of two hospital systems as well as four regional healthcare payers.

The second category examined the use of laboratory, financial, and other clinical data for determining the true potential cost savings or new net revenue from projects under consideration by the hospital system. One project in this category examined return on investment (ROI) analysis for a matrix-assisted laser desorption ionization time of flight bacterial identification system was presented with limited laboratory data compared to the same analysis including downstream cost savings through decreased patient length of stay. With only laboratory information, the payback period was >20 years compared to <1-year with more comprehensive data integration (clinical integrated with financial and laboratory data).

Projects in the third category required specific clinical integration of wide-scope clinical data with laboratory and financial data. These projects allied the laboratory with other departments of the hospital (e.g. pharmacy, radiology, Intensive Care Unit). One project retrospectively quantified >$375,000 in new annual net revenue and ROI from a project involving radiology and the clinical laboratory for the use of point of care instruments to improve patient flow through radiological procedures requiring contrast. In addition to increased net revenue, patient satisfaction measures also improved dramatically. The importance of quantified projects after their completion was stressed as a way to demonstrate the value to hospital system leaders. A second project quantified the cost of using expensive antibiotics after laboratory culture and sensitivity results showed that the bacterial infection was sensitive to a less expensive antibiotic. Looking at just two antibiotics (linezolid and daptomycin for Gram-positive infections), an additional variable cost was just over $645,000/year.

Many healthcare systems are considering or implementing part of an enterprise data warehouse to include all clinical data and in some cases, financial data. The importance of laboratory involvement in these efforts was the subject of the fourth project category. Both presenters emphasized the collaborative power of including laboratory, finance, and other leaders in enterprise data warehouse development projects.

Dr. Brimhall and Mr. Clements were joined by Geoff Baird, assistant professor of Laboratory Medicine at the University of Washington in Seattle, as well as the Director of Clinical Chemistry at Harborview Medical Center and the Director of Laboratories at Northwest Hospital, as well as Ricky Grisson, Chief of Clinical Chemistry and Toxicology at the VA Puget Sound HCS and Assistant Professor in the Department of Lab Medicine at the University of Washington.

Dr. Grisson commented that the use of digitized, organized health information promises to improve the health of individuals and populations. Laboratory medicine generates abundant time series health information yet deploys few strategies to realize the value of these data. To improve health of the individual patient and the population using laboratory data, outcomes-focused, controlled quasi-experimentation with times series data could be used. He noted that there are many examples of quality improvements strategies, such as Lean, Six Sigma, and VA-TAMMCS using big health system data being used in many areas of the hospital to identify and realize value improvements. Clinical support services, like laboratory medicine, provide abundant healthcare data and effectively use quality control, assurance and improvement strategies to improve lab operations and testing performance. The application of LEAN principles when developing, selecting and introducing new tests has potential to enhance the value laboratory medicine for providers and patients alike. However, the impact of lab medicine on patient and population health or value creation is not well understood or characterized as highlighted by Dr. Jackson.

Dr. Grisson provided an example of the VA, developing new qualitative and quantitative multiplex LC-MS/MS assays for urine drug of abuse testing. Rather than instituting this new test in a manner that adds-on to screening immunoassays, they plan to adapt the available generalizable scientific data to their specific environment using data from their system to guide them in the provision of this panel to replace the immunoassay. In that way, the Batalden model for improvement will be used to enhance the value of their offerings while controlling operational costs.[6]

Data Sharing And Collaborative Analytics

Brian Shirts, Assistant Professor of Laboratory Medicine at the University of Washington presented data illustrating that there is substantial variability in laboratory data on patient, laboratory, and health system levels.[7] Many laboratory utilizations and management proofs of principle papers have been published.[1] Any potential proof of principle project at an individual institution raises questions about generalizability of the findings and the possibility of scaling the interventions to implement similar solutions at other institutions. Potential solutions to these issues are analytics studies that are implemented at multiple institutions or that combine data from multiple institutions.

Dr. Shirts listed several examples of studies that had combined data from multiple sources. One analysis of HPV test utilization analyzed trends over 110 hospitals.[8] Another study looked at enterovirus polymerase chain reaction testing at multiple institutions. These analyses of combined data are more likely to yield results that are generalizable. In addition, if there is sufficient data about methods for obtaining data from multiple institutions it becomes clear how difficult it may be to extract data from different clinical systems and processes are more likely to be amenable to implementation at additional institutions. An additional benefit of analyzing data from multiple institutions is that it allows institution level benchmarking. Shared analysis may also facilitate improved overall analytic quality and the analytic efficiency. Finally, analysis and implementation at multiple institutions can facilitate controlled quasi-experimentation as advocated by Dr. Grisson.

There are several major challenges to clinical laboratory analytics projects that cross-institutional boundaries. One challenge is that institutional review boards by definition function at an institutional level. This means that different levels of institutional and legal review may be necessary for even a relatively simple project. The same project may be classified as a component of healthcare operations by one institution and as a health services research project by another institution. These classifications each have their own regulatory framework, making it difficult for laboratories to collaborate to improve outcomes because of differences in institutional culture. The CAP Q-probes and Q-tracks series illustrate one realized solution to these issues,[910] but this mechanism is not ideal for more complicated, comprehensive, or data intensive analytics such as those proposed by Dr. Jackson, Dr. Brimhall, and others.

Dr. Shirts was joined by Ronald George Hauser, clinical instructor at Yale School of Medicine, and Julie Taylor from the Centers for Disease Control and Prevention for a panel discussion about data sharing and collaborative analytics. They reported on several initiatives to improve collaboration and data sharing in the clinical laboratory. Dr. Hauser emphasized the need for collaboration and unity among the laboratory community interested in the utilization. He reiterated the point by sharing a story of where he serendipitously heard of a colleague working on a project quite similar to his own. He expressed a need to join our efforts to strategically move the discipline of utilization forward. He noted that a subcommittee of the CAP DIHIT group has create a repository of utilization rules and suggested that individuals interested in shared diagnostics utilization resources becoming a member of a community assembling utilization rules currently hosted at “thecan.apphb.com/”.

Dr. Taylor discussed the work of the CDC Laboratory Research and Evaluation Branch's national programs to improve clinical laboratory testing, Clinical Laboratory Integration into Healthcare Collaborative (CLIHC™) and Laboratory Medicine Best Practices (LMBP™).[11] CLIHC™ focuses on improving clinical laboratory test utilization, specifically test selection and results interpretation. Laboratory professionals seeking to improve processes in their own institution may find guidance from the articles, presentations and tools posted on the CLIHC™ webpage.[11] In particular, one recent CLIHC™ publication reported on a national survey about the challenges primary care physicians face related to diagnostic laboratory testing and solutions they believe are helpful and available to them.[12] Physicians reported the helpfulness of many clinical decision support tools, such as electronic prompts and Computerized Physician Order Entry with electronic suggestions to help with test ordering. Trending of results to show past patient results, reflex testing and test characteristics information were also useful to them. Physicians favored report formats that were easy to read and visually displayed the data to highlight the most important results.

The LMBP™ program uses systematic review methods to evaluate evidence of laboratory practice effectiveness, especially in the pre and postanalytic phases.[11] The goal of the program is to improve healthcare quality and patient outcomes through dissemination and uptake of recommendations resulting from evidence reviews of effectiveness that identify evidence-based laboratory medicine “best practices.” The program includes reviews of both published and unpublished evidence. One current project involves reviews of practices to improve test utilization, in particular test ordering practices. The program requests that laboratory professionals submit, through the LMBP™ website, any unpublished quality improvement data for practices that improve test utilization, in particular test ordering practices, including methods to measure laboratory test utilization. The website also provides information and training on how to conduct studies to provide evidence of effectiveness when laboratory professionals want to collect data about the practice.

DISCUSSION: FUTURE DIRECTIONS FOR CLINICAL LABORATORY ANALYTICS

The session ended with an open discussion about with all panelists and meeting participants on the current state and future of clinical laboratory analytics. Several themes emerged in the discussion.

One theme was that it might be challenging to justify the role of laboratory analytics to those not familiar with the field. Dr. Baird commented that it will be vital in the coming decade for academic medical systems to recognize that the study, development, and use of analytic tools and “big data,” specifically as applies to research on laboratory test utilization, are valuable and important activities for laboratory-based faculty members that should be supported on par with other more traditional (i.e. “R01”-focused) scholarly activities. Practically speaking, there must be a financial model that supports this type of work, and it may take some innovation to develop that model fully. However, without such a model in place to support the innovative work of junior faculty in this area, we will not be positioned, as a specialty, to maintain control of the data resources that reside within our departments. Without this, it is possible that this field becomes an administrative field for business mangers that are divorced from clinical and laboratory expertise. Dr. Grisson noted that analytics consumers (i.e. federal and private payers and health service producers) are focused on cost reduction and quality and value improvements in the American healthcare sector. Under these pressures, top-down (Medicare Resource Based Relative Value Scale, Diagnosis Related Groups, Resource Utilization Groups and Veterans Health Administration [VHA] Cost Distribution Report) and bottom-up healthcare costing systems (e.g. activity-based costing and VHA Decision Support System) have been deployed to meet internal financial management, benchmarking, provider profiling, external quality metric reporting and cross facility comparison needs.[13] As we make use of big data to understand laboratory test utilization, both bottom-up and top-down methods could be developed to enable efficient summaries; give short-, medium- and long-term performance information; and to enable system operators implement key improvements. Dr. Baron concurred that key to advancing clinical laboratory decision support is demonstrating its clinical and economic value. Demonstrated value of analytics by laboratory faculty and personnel should help to secure financial support and institutional backing. Many participants agreed that quantifying and communicating the value of laboratory analytics work as performed by laboratory specialists was critically important for future work.

Another theme was that better evidence is needed in analytics studies. Dr. Grisson expressed a solution to produce robust evidence – quasi-experimental studies. Unlike nonexperimental (association) studies, quasi-experimental studies use the same observational data sets but apply an experimental framework to interrogate the data and identify a causal relationship between intervention and outcome. Though limited by the lack of random allocation to intervention or control group, the mere presence of a control group in a quasi-experiment allows better estimation of the main (or treatment) effect. Typically, repeatedly obtained lab operational data (e.g. test searches over time) have been used to illustrate the impact of an intervention on the number of tests ordered.[14] In the absence of a relevant control group, these results can overstate the effect of the intervention. While several quasi-experimental designs are available for use, controlled time-series (A-B), equivalent time-series (A-B-A-B), interrupted time series, regression discontinuity, and multiple baseline (ABBB, AABB, AAAB) are most facile and effective when demonstrating the effect of an intervention using lab data.[15]

A final theme was that better strategies to share information and tools between institutions are needed. Dr. Terrazas commented that there exists an opportunity to follow an open source model to help laboratories achieve what they need in terms of analytics. There are off the shelf generic business intelligence solutions like Tableau that allow data visualization and analysis. If we can define a common export format, and create report templates, for example in a commonly available platform like Crystal Reports, the export process could be achievable across laboratories and across LIS's. Similarly, if templates could be created and shared as a business intelligence off the shelf solution for import of the data and visualization, the protected health information loop would be closed and laboratories would have an achievable methodology for analytics if they have a reasonably technical LIS or IT group. With many labs participating in this type of process, a robust solution may be achievable through collaboration. Collaborative analytics efforts could be used in experimental studies such as those Dr. Grisson described to test the evidence at multiple institutions and also to establish robust guidelines and standards for diagnostic test ordering. If these strategies are used by implementation studies where data is collected at multiple institutions they will address concerns about strength of evidence, and the only limitation will be identifying the most important questions to answer.

An E-mail list will be developed to facilitate further sharing of ideas about clinical laboratory analytics, which is currently maintained by Dr. Hauser. Individuals with an interest in utilization and analytics can contact him to join an online group discussion by sending an E-mail to lab utilization+subscribe@googlegroups.com.