Achieving and sustaining automated health data linkages for learning systems: barriers and solutions.

INTRODUCTION: Delivering more appropriate, safer, and highly effective health care is the goal of a learning health care system. The Agency for Healthcare Research and Quality (AHRQ) funded enhanced registry projects: (1) to create and analyze valid data for comparative effectiveness research (CER); and (2) to enhance the ability to monitor and advance clinical quality improvement (QI). This case report describes barriers and solutions from one state-wide enhanced registry project.
METHODS: The Comparative Effectiveness Research and Translation Network (CERTAIN) deployed the commercially available Amalga Unified Intelligence System™ (Amalga) as a central data repository to enhance an existing QI registry (the Automation Project). An eight-step implementation process included hospital recruitment, technical electronic health record (EHR) review, hospital-specific interface planning, data ingestion, and validation. Data ownership and security protocols were established, along with formal methods to separate data management for QI purposes and research purposes. Sustainability would come from lowered chart review costs and the hospital's desire to invest in the infrastructure after trying it.
FINDINGS: CERTAIN approached 19 hospitals in Washington State operating within 12 unaffiliated health care systems for the Automation Project. Five of the 19 completed all implementation steps. Four hospitals did not participate due to lack of perceived institutional value. Ten hospitals did not participate because their information technology (IT) departments were oversubscribed (e.g., too busy with Meaningful Use upgrades). One organization representing 22 additional hospitals expressed interest, but was unable to overcome data governance barriers in time. Questions about data use for QI versus research were resolved in a widely adopted project framework. Hospitals restricted data delivery to a subset of patients, introducing substantial technical challenges. Overcoming challenges of idiosyncratic EHR implementations required each hospital to devote more IT resources than were predicted. Cost savings did not meet projections because of the increased IT resource requirements and a different source of lowered chart review costs. DISCUSSION: CERTAIN succeeded in recruiting unaffiliated hospitals into the Automation Project to create an enhanced registry to achieve AHRQ goals. This case report describes several distinct barriers to central data aggregation for QI and CER across unaffiliated hospitals: (1) competition for limited on-site IT expertise, (2) concerns about data use for QI versus research, (3) restrictions on data automation to a defined subset of patients, and (4) unpredictable resource needs because of idiosyncrasies among unaffiliated hospitals in how EHR data are coded, stored, and made available for transmission-even between hospitals using the same vendor's EHR. Therefore, even a fully optimized automation infrastructure would still not achieve complete automation. The Automation Project was unable to align sufficiently with internal hospital objectives, so it could not show a compelling case for sustainability.

Introduction

Delivering more appropriate, safer, and highly effective health care is a goal of the learning health care system framework.1 Such a system relies on linking clinical databases from unaffiliated hospitals, which cannot easily share data because they operate with distinct hardware and software systems, use idiosyncratic data models, and are subject to different governance constraints. As part of a strategy to explore these issues, the Agency for Healthcare Research and Quality (AHRQ) funded networks to create enhanced registries that would do the following: (1) create and analyze valid data for comparative effectiveness research (CER), and (2) enhance the ability to monitor and advance quality improvement (QI) of clinical care.

The Comparative Effectiveness Research and Translation Network (CERTAIN)2–4 undertook a three-year project to expand the capability of the Surgical Care and Outcomes Assessment Program (SCOAP) enhanced registry. SCOAP is a clinician-led, performance benchmarking, and QI registry for surgical and interventional procedures.5 Established in 2005, 55 of the 60 hospitals (92 percent) in Washington State participate. At 5 unaffiliated SCOAP-participating hospitals, CERTAIN deployed the commercially available Amalga Unified Intelligence System™ (Amalga) (provided by Caradigm)6 as a central data repository. The project strategy was to achieve the AHRQ goals: (1) by reducing the workflow and staffing burden of medical records review for on-going participation in the QI registry; (2) by demonstrating capacity and scalability to incorporate new clinical disciplines into SCOAP QI, a program that had proven value in improving patient outcomes and reducing health care delivery costs; and (3) by potentiating access to more data on patients across health care encounter types and longitudinal records.

The value proposition to participating hospitals was to reduce the cost of ongoing or expanded participation in QI by reducing the volume of manual chart abstraction work through data automation. Project methods for the automation and validation of electronic data have previously been published.7,8 In this case report, we describe several aspects of implementing the CERTAIN Automation Project, including the following: the central data repository infrastructure design, the hospital recruitment and implementation approach and barriers, the data sharing and ownership approach and barriers, and barriers to successful hospital recruitment and implementation. And—in the context of each of these aspects—we describe our findings regarding sustainability for this project that we believe will apply to similar enhanced QI and research projects in the future.

Background

SCOAP was created by a community of clinicians in Washington State and has been supported by grants from the Life Sciences Discovery Fund to University of Washington (UW) investigators. SCOAP operates under the Foundation for Health Care Quality (FHCQ),9 a nonprofit 501(c)(3) corporation. As a Washington State approved Coordinated Quality Improvement Program,10 participant hospitals are allowed to disclose protected health care information specifically for the program purposes. Hospitals hire and train staff to review medical records and abstract clinical data using SCOAP data collection forms and data dictionaries. The clinical data are manually entered into the SCOAP registry using a web-based form. Participant hospitals receive quarterly QI performance reports that show their data alongside benchmarks and peer performance. The SCOAP continuous data collection and feedback loop is proven to improve the quality and safety of surgical and interventional care while decreasing costs.11,12 There are five SCOAP registries: gastrointestinal and general surgical procedures, oncologic surgical procedures, pediatric surgical procedures, spine surgical procedures, and vascular surgical and interventional procedures. The range of clinical data questions included in each registry ranges from approximately 580 to 880.

SCOAP, its network of participating hospitals, and the rich database of clinical and performance variables comprised a registry that met the requirements for enhancement set by AHRQ. To achieve this, three issues needed to be addressed: (1) scalability—the reliance on manual data abstraction, particularly for already computerized discrete data, limited both the volume of clinical data in the registry and the growth of the registry to include new hospitals and new data types; (2) searchability—the SCOAP registry and governance model were designed under QI principles; it was not designed to undertake ad hoc CER queries that compare outcomes across multiple participating institutions; and (3) cost and sustainability—the training and working time cost for manual data abstractors to add more clinical and process data to the registry placed a nearly prohibitive burden on participating hospitals to grow SCOAP-related QI or to add data types needed for population-level CER. Furthermore, maintaining electronic feeds of data into a centralized data repository after registry enhancement would require a sustainability business model.

Methods

Central Data Repository Infrastructure

A more detailed description of the central data repository infrastructure is published elsewhere.7 CERTAIN approached Microsoft Health Solutions Group (Bellevue, WA) to partner on this project because of prior UW institutional experience with Amalga Unified Intelligence System™ (Amalga), researcher familiarity with the growing use of Amalga in health information exchange (HIE) projects, and the goal of exploring Amalga’s capacity to link disparate electronic health record (EHR) vendor systems. Other vendors were considered but not approached, due to projected cost. The method developed by the NIH-funded Informatics for Integrating Biology and the Bedside (i2b2) project was considered but discarded because it lacked tools to support data extraction and normalization. Single EHR vendor solutions were considered but discarded because the goal was to operate across health systems independent of EHR type. The approaches under development by Health Level 7 International for promoting uniformity in health data delivery, including the Continuity of Care Document (CCD) and the Continuity of Care Record (CCR), were likewise discarded because they lacked standards at the time.

Microsoft would provide installation of Amalga in a hosted environment; implementation of Amalga across participant hospitals, including building data interfaces from hospitals to each hospital’s hosted Amalga repository including data transformation; a SCOAP-specific centralized data repository; and system maintenance during the three-year project. Approximately halfway through the project period, Microsoft Health Solutions Group partnered with General Electric Healthcare to form a new company, Caradigm (Bellevue, WA). This paper refers to Caradigm as the Amalga provider.

Hospital Recruitment and Implementation

CERTAIN’s existing and well-established presence within the Washington State health care community was expected to encourage hospital recruitment into the enhanced registry.13 At the time of CERTAIN’s formation, the principal investigator was serving as the medical director of SCOAP and had been critical to the QI program growth from 5 to 55 participating hospitals, as well as the expansion from one QI registry to five. CERTAIN investigators and staff had existing relationships with many hospitals’ chief medical officers (CMOs), quality directors, and clinicians.

Furthermore, the FHCQ and UW (the parent organization of CERTAIN) had a previously established history of contractual service arrangements for UW to provide QI services such as data collection, data analysis, and data reporting. UW, and specifically CERTAIN investigators, was a trusted collaborator to the FHCQ and to the Washington health care community at large.

At the time of grant submission, CERTAIN investigators had obtained interest letters from 7 hospitals to participate in the project.

CERTAIN created an eight-step process, including hospital recruitment, technical review of EHR systems, hospital-specific interface planning, data ingestion into Amalga, and data validation (Table 1). CERTAIN approached 19 hospitals operating in 12 unaffiliated health care systems from among the 55 SCOAP-participating hospitals. These 19 hospitals were chosen as a convenience sample of hospitals that CERTAIN investigators determined to have sufficient surgical case volume, at least two years of experience in SCOAP, participation in more than one SCOAP registry, and EHR sophistication. The implementation process was co-led by CERTAIN investigators and the following personnel within Caradigm: Vice President of Clinical Solutions, Sales Executive, Customer Relationship Manager, Engagement Manager and Integration Manager.

Table 1

Process for Recruiting and Implementing a Public-Private Health IT Partnership for QI

Step 1	Introductory Meeting with Hospital	Overview of SCOAP, CERTAIN and the Automation Project. Informs CIO, CMO, and Quality Directors about the project purpose and benefits to the hospital.
Step 2	Technical Review Meeting	Secondary meeting for a technical review of interfacing hospital data sources to Amalga and the impact to a hospital for participating in the Automation Project. CIO, CMO, Quality Directors and IT Directors were provided detailed information about automated data feeds, including project definitions. A hospital technical survey was completed before or during this meeting, and any hospital-specific implementation considerations or needs arising were assessed. This also established the hospital team and communication plans for moving forward.
Step 3	Detailed Hospital EHR Assessment	Led by CERTAIN staff, a detailed assessment of the hospital’s EHR was conducted to identify where SCOAP information was stored. This stage did not require that CERTAIN staff access EHRs directly; it was completed by working with current SCOAP QI staff already at the hospital.
Step 4	Interface Analyst Meeting	The detailed, hospital-specific approach to interfacing through Amalga was discussed in this third meeting. At this point, hospital-specific implementation considerations or challenges were addressed in full and an estimate of return benefit to hospitals regarding automation of SCOAP abstraction was completed. After this point, the hospital would decide to move forward with the project.
Step 5	Participant Agreement	A formal consulting-services and business-associate agreement was provided to hospitals for administrative review and signature. This agreement was executed between the hospital and CERTAIN (University of Washington).
Step 6	Project Planning Meeting	After the agreements were signed, a project planning meeting was held to develop a hospital-specific implementation timeline and to plan the assembly of interfaces to feed data to Amalga.
Step 7	Amalga Implementation, led by Caradigm	Using a process provided by Caradigm (Caradigm Implementation Methodology), this step was the technical execution of a hospital-to-Amalga interface, including provision of virtual private network access, provision of lookup codes for the EHR data, creation of data parsing software, and related technical steps.
Step 8	SCOAP Data Validation	Executed by CERTAIN staff. After User Acceptance Testing, three stages of validation were performed in this project: 7 Validation 1: Comparison of raw data displayed in Amalga with that displayed in the EHR to determine performance of ingested data sources and data feeds. Validation 2: Comparison of Amalga SCOAP Extract and EHR sources typically used by a SCOAP abstractor to determine performance of Amalga’s automation. Validation 3: Comparison of Amalga SCOAP Extract and SCOAP manual data abstraction records with verification against the EHR to compare performance of automated and manual abstraction efforts.

Data Sharing and Ownership

Hospitals participating in SCOAP customarily execute an information-sharing agreement (i.e., a data use agreement) and business-associate agreement with the FHCQ. The exchange of data and services explicitly for the Automation Project was permitted through a consulting-services agreement (i.e., a data use agreement) and business-associate agreement directly between CERTAIN’s parent organization, UW, and the participant hospital.

CERTAIN executed a term license agreement with Caradigm for the provision of an Amalga instance for each participant hospital, and therefore Caradigm was acting as a service provider on behalf of UW and was obligated under the same consulting-services and business-associate agreements executed between UW and the participant hospital. Under that agreement, hospitals allowed the use of protected health information for patients eligible for stated QI and research data-use purposes.

The consulting-services agreement between UW and the participant hospital specified that each hospital owned its own data. Unless a hospital had arrangements for continued use of Amalga beyond the three-year project, all Amalga data were to be destroyed in compliance with the Health Insurance Portability and Accountability Act (HIPAA) at the project’s conclusion.

Quality Improvement (QI) Versus Research Data Use

The Automation Project included two component activities: QI and research.7,13 The QI activity for this project involved establishing electronic interfaces from the hospital data sources to the Amalga central data repository in order to partially automate SCOAP data collection. QI designated activities are detailed in Table 1, Steps 7 and 8. To achieve this designation, CERTAIN investigators met with three institutional committees at UW to create a framework that placed a virtual firewall between QI and research activities, and assigned each project investigator and staff to QI or to research, not to both. Those serving on the QI side are authorized, a priori, to view patient-level data for QI purposes. Those serving on the research side see only aggregated, de-identified data. The framework has been codified in a memorandum of understanding and presented to other participant institutions.7,13

Participant Hospital Effort Requirements

Caradigm participated in implementation Steps 1, 2, 4, and 6 (Table 1) and was contracted for the necessary hours to lead Step 7 (i.e., Amalga Implementation) at each hospital participating in the Automation Project.

The goal of having the bulk of the database and interface development and data mapping work performed by Caradigm was to minimize the hours of work required of participating hospitals, thus minimizing an initial barrier to entry into the enhanced registry. The participating hospitals were provided with free installation and license to Amalga during the three-year project, access to the knowledge of CERTAIN biomedical informatics investigators and health IT staff, and processes to partially automate the extraction of SCOAP data to aid their QI practices within hospitals. Participating hospitals were required to provide knowledgeable staff time (e.g., SCOAP data abstractors or EHR subject matter experts) and IT resources to successfully establish and implement the project internally.

We initially estimated that minimal effort would be required during project evaluation and planning phases (Table 1, Steps 1–6) and that 40–80 dedicated hours of work by hospital IT staff would be needed to accomplish each of the project phases shown in Table 2 for Amalga implementation (Table 1, Step 7). There was no work effort anticipated for hospitals to complete SCOAP data validation (Table 1, Step 8).

Table 2

Step 7 of the Implementation Process: Amalga Implementation, With Breakdown of Hospital Effort and Resource Requirements

Project Phase	Resources	Description
Set up Virtual Private Network	Network EngineerInterface AnalystSubject Matter Expert	Connect two virtual private network tunnels (staging connection and production connection) to connect the hospital-side data interface to the Amalga central repository.
Provide Specifications of Feeds	Network EngineerInterface Analyst	Complete a description template detailing electronic data sources at the hospital and expected weekly volume and average message size.
Provide 60 to 90 Days’ Worth of Production Quality Data	Interface Analyst	Securely upload a data sample to allow Caradigm to understand the format of data delivery and create software to properly load inbound data to the Amalga central repository.
Review SCOAP Data Element Description Template	Interface Analyst	Review the description template and provide the location of SCOAP-specific data elements within the hospital data sources.
Verification and Validation of the SCOAP Form Data Elements	Interface AnalystClinical Analyst	Verify that SCOAP data elements have been transmitted and received correctly.
Additional Interface Engine Filters	Interface AnalystFeed Clinical Analyst	Verify that only SCOAP data authorized under the data use agreement is being transmitted.
Connect Live Feeds to Amalga Hosted Environment		Connect live hospital data sources to Amalga and define ongoing support processes.

Implications for Sustainability

The project included three components thought to create a substantial value proposition to be sustainable beyond the funded project period: (1) provide participating hospitals with a cost savings, when compared to their current expenses, to participate in SCOAP; (2) provide to hospitals in-depth experience with a clinical data repository, steering those who lacked a repository to become ongoing customers of the repository vendor; and (3) provide a data repository that would be a cheaper solution (i.e., show a positive return on investment) to internal QI goals if it were maintained.

To estimate cost savings, CERTAIN predicted that 37.5 percent of SCOAP QI registry variables could be targeted for data automation as discrete computable data elements or by using Natural Language Processing (NLP) techniques. NLP is a field of study in human-computer interaction. This project used NLP methods to develop computer algorithms that analyzed free text in the EHR to derive meaning. For example, an algorithm was developed to scan free-text admission notes to determine what kind of smoking history each patient had.

A survey of hospitals conducted in 2010 measured personnel effort necessary to perform data abstraction for SCOAP QI registries. The hospitals ranged in size from 25 to 623 beds and provided information about the average time to review medical records and extract data for the SCOAP registry, and other barriers to scalability. The survey also measured costs from salaries, data management (e.g., identifying SCOAP records to submit to registry, data cleaning), training, staff oversight, within-hospital dissemination of SCOAP data, and general facility or institutional overhead.

From survey results, CERTAIN investigators projected that a medium-sized hospital performing 556 gastrointestinal and general surgical procedures annually would spend approximately $64,000 annually for SCOAP QI registry participation, excluding direct program fees to the FHCQ. Automating data delivery to the central data repository would confer a cost savings of $24,000 annually (37.5 percent of $64,000) for participation in one SCOAP QI registry, and the savings would multiply for participating in multiple SCOAP QI registries.

To convert hospitals to be ongoing Amalga customers, CERTAIN acted as a single client to Caradigm and managed relationships with multiple potential future customers. As described in the methods, Caradigm personnel were present at key points throughout the recruitment and implementation processes.

To show a positive return on investment required completion of data delivery interfaces and a reliable, useful central data repository within Amalga before the end of the project. Hospitals were told that they could access their own data within the project repository and create their own QI reports using Amalga tools independent of the SCOAP Registry.

Findings

Hospitals that completed the Automation Project, compared to those approached that did not complete it, are shown in Table 3. Of 19 hospitals approached, 15 hospitals demonstrated support for project participation at the CMO or CIO level. Of those 15, only 5 completed all process steps to deliver data. The 10 hospitals that expressed CMO or CIO level support did not participate because of competing institutional priorities, as further described below. Also, we approached one organization that provided administrative and medical records services to 22 additional hospitals; the organization expressed engagement and commitment of IT resources, but was unable to overcome significant data governance barriers within the three-year term.

Table 3

Comparison of Hospitals Approached by CERTAIN That Completed the Project Compared to Those That Did Not

	Completed (n = 5)	Did Not Complete (n = 14)
Number of Inpatient Beds, Average (SD)	351 (70)	252 (181)
Annual Surgical Procedures Volume, Average (SD)
Gastrointestinal/General	371 (69)	249 (41)
Vascular	60 (10)	175 (42)
Spine	280 (23)	393 (79)
Number of Years in SCOAP at Project Start, Average (SD)	3.8 (1.2)	2.6 (1.6)
Distance from CERTAIN Headquarters, Average Miles (SD)	6.7 (5.6)	78.9 (75.3)
Percent Considered “Urban”*	100	57
Percent Considered “Teaching Hospital”†	80	43
Percent Considered “Research Hospital”‡	80	72
Percent Self-described “Nonprofit”§	100	93

Definition of urban from http://www.census.gov/population/censusdata/urdef.txt.

Considered “teaching hospital” if affiliated with medical school or contains a residency program.

Considered “research hospital” if it has a research institute or other formal acknowledged research program.

Considered “nonprofit hospital” if organized as a nonprofit corporation.

Many hospitals expected that only data on patients who underwent surgical or interventional procedures included in SCOAP would be exported to the central data repository for this project. Reasons for this restriction included hospitals’ concerns about the need for patient consent in order to permit their data to be used for this project.

The QI and research data-use designations determined there was no human subjects research, thus no additional patient consent was necessary. However, the restriction to export data only for patients included in SCOAP remained in place for the majority of participants.

The restrictions on exporting only SCOAP patient data introduced an unanticipated technical barrier: external data repositories (i.e., Amalga) cannot be configured to screen or filter inbound data based on a given patient. That process must be done either before sending the data, on the hospital side at substantial added cost and time, or after receiving the data into the central data repository and analyzing its content, which was not in the original scope of the grant. Furthermore, at the time data began to accrue in a hospital EHR about a patient, upon admission, it was not always known that the patient would subsequently undergo a qualifying SCOAP procedure. After a patient was identified as a SCOAP patient, a way was needed of going back and obtaining that earlier data. Because the project plan was designed around near-real-time transmission of clinical data, there was no provision for the extra IT work needed to create a method to retrieve selected historic data elements about particular patients.

The memorandum of understanding developed by three institutional committees at UW was presented to all participant hospitals. Three hospitals signed the memorandum. One hospital did not sign, but proceeded forward under the precedent set forth by other participants. One hospital did not require any additional QI or research review.

We found hospitals commonly asked three questions about project participation: (1) How will participation help the hospital achieve existing operational goals, such as meeting deadlines for the Medicare and Medicaid EHR Incentive Programs (Meaningful Use)? Or, can this project push back its timeline until after local IT resources are finished with work on those goals? (2) Since the project isn’t within the hospital IT infrastructure, can the outbound data delivery include only SCOAP-participating patients? (3) How does this project benefit the hospital as a whole, or is it just a surgical QI project? These common questions revealed to us that the fundamental barrier was not a complicated technology gap, but rather the inability to raise the project to a priority level sufficient to obtain enough of the limited hospital-based IT expertise or resources.

First, all hospitals that provided executive-level support were facing Meaningful Use deadlines. These deadlines had both IT impact (e.g., deploying a new EHR) and significant organizational impact (e.g., change management, collection of new outcome metrics). These deadlines were bottom-line financial realities that could not be moved and that required substantial hospital-based IT resources to satisfy. This left very few hospital-based IT experts available to help the CERTAIN team connect to data sources and troubleshoot data-delivery problems.

Seven hospitals approached for the project were implementing completely new EHR infrastructure for their organizations during the project period. These hospitals stated an intent to participate, but recognized that they would be unable to spare any hospital-based IT resources. They requested deferring project initiation until a time past the grant-funded project term, therefore it was not feasible to include them in the project. Another nonparticipating hospital had recently initiated the build of an institution-specific central data repository, but expressed interest in future partnership in distributed network data sharing.

Second, the Amalga instance created for the hospital was not perceived by the hospitals as a new part of their own IT infrastructure that would grow to achieve other operational goals in the future. As the project progressed, we found that limiting the hospital outbound-data delivery to SCOAP-defined patients only required additional and unanticipated IT development work. Building queries, filters, and data tables to limit outbound data could only be done by the hospital IT experts, thus resulting in more time required from hospitals IT resources to meet their requirements in project participation. Hours were not precisely tracked or accounted for by participant hospitals, but we estimate that hospital-based IT experts spent three to four times the original projected effort (i.e., 120 to 320 dedicated IT hours of work) to accomplish the Amalga implementation steps described in Table 2.

Third, upper-level executive support did not always translate to hospital-wide administrative support. CERTAIN investigators identified three primary reasons for this: (1) the QI project was viewed as benefiting a small organizational unit in the hospital (i.e., surgical and procedural service lines) and therefore did not spur hospital-wide changes to priorities; (2) the project timeline was too short to educate enough stakeholders well enough at the hospital in order to elevate the project’s status; and (3) the addition of automation was viewed primarily as a research project on the side of SCOAP, and neither integral to ongoing growth of QI nor containing enough return on investment in its own right. As a result, overarching hospital priorities were not rearranged in a way that prioritized this project as a hospital-wide initiative to enhance data collection for QI and CER. Instead the Automation Project was viewed as a surgical QI project only.

It became clear that, given a limited supply of knowledgeable IT resources, a hospital will always pursue internal, core business priorities first. Participation in SCOAP QI may have been a core business priority for many hospitals because of the proven cost savings, but the argument to add the complexities of IT interface work to a program that was already working by manual abstraction was difficult to make. We observed that this experience mirrored that of challenges in HIE deployments because of similarly mismatched incentives.14

Across SCOAP QI registries, 12.4 percent of variables were successfully automated. (Complete results of the automation process and its validation will be reported separately.) In the example medium-sized hospital performing 556 gastrointestinal and general surgical procedures annually, and spending $64,000 to participate in one SCOAP QI registry, this would confer a cost savings of only $8,000 annually (12.4 percent of $64,000). Furthermore, an unanticipated change occurred during the three-year project—unrelated to CERTAIN or the Automation Project—in the way hospitals were performing manual chart review and data abstraction for SCOAP. Vendors offering contract quality measure-abstraction services emerged on the market, offering participating SCOAP hospitals low-cost services to perform manual chart review and data abstraction. Three of the five hospitals participating in the Automation Project moved to these contracted services, which reduced their internal staff, overhead and resource burden.

Discussion

The Automation Project demonstrated the feasibility in recruiting unaffiliated hospitals to create an enhanced registry to achieve AHRQ goals. Five hospitals delivered data to a central data repository that, together with the rest of the CERTAIN infrastructure, powers a growing network for CER. We believe the eight-step process described above is a generalizable framework for similar projects, if the major barriers we found, discussed in more detail below, could be overcome.

These barriers included a much lower than expected hospital recruitment rate; a reluctance among hospitals to view their project-hosted central data repository as a trusted location to deliver all clinical data, an easily solved question about whether the activity was governed as QI or research, a challenging problem of EHR idiosyncrasies demanding more local IT effort than expected, and, ultimately, a multifaceted challenge to align this work with each hospital’s internal objectives.

The rate of recruitment was much lower than anticipated. We do not believe our recruitment process itself caused any barriers, although some hospitals may not have viewed participation as bringing hospital-wide benefit because of the project’s origins as a surgical QI project. By extending the recruitment timeline or providing more information about the project during the recruiting phase, it is possible that more hospital stakeholders or service lines might have shown interest and demand for participation. It is unlikely, however, that such interest would have raised the Automation Project’s status ahead of Meaningful Use or a new EHR infrastructure roll-out.

A possible solution here is to achieve wider acceptance in a data model that encourages hospitals to view a project-hosted central data repository as a trusted location to deliver all clinical data (discussed further below), and therefore to provide infrastructure for hospital-wide QI projects that could be added later. That, together with a recruitment effort that begins across many hospital clinical services in concert with the QI department, may improve the profile of projects like this and increase their priority level within hospitals.

Trying to accommodate hospitals that wanted to only export data about patients undergoing a SCOAP-included procedure was a tremendous technical challenge. This barrier has been described previously,15 but no definitive solution has yet gained widespread traction. The CERTAIN research team encouraged hospitals to adopt a policy that treats the Amalga instance created for their hospital within this project as an extension of the hospital’s own internal data store that could contain protected health information about any patient. The necessary business-associate agreements and data-security protections were in place to support such a policy.

Allowing the creation of live data-export streams to include all patient data would have shifted the work of finding SCOAP patients, linking their historic data, and cleaning the data to the CERTAIN project team and away from hospital IT staff. Hospitals that exported all patient data would “retain physical control of raw data [each hospital’s hosted Amalga repository] although providing for their aggregation as limited datasets [the SCOAP-specific centralized data repository] to answer specific questions.”15

The strategy to export all patient data to the project’s Amalga repository and treat that as an internal data repository was not accepted by the IT groups at participating hospitals. Not only did the rejection highlight the lack of readiness for hospitals to engage in a broader concept of data aggregation for future and undefined purposes related to research and not QI, but it increased the project costs and substantially raised the burden for sustainability. We believe the concept of extending a hospital’s own physical control of raw data to a hosted repository, from which aggregation to a research data set can be efficiently done, is compelling and is a strong contender for solving this barrier. Stronger national incentives or even requirements for hospitals to engage in this kind of data management are needed.

Quality Improvement Versus Research Data Use

This distinction proved to be a readily solvable barrier. The institution that delayed participation to study the question created a framework for reconciling the coexistence of QI and research goals within the same project. This framework was quickly embraced by other participants, as noted above, and will be described in a future publication as a model that could be adopted nationwide in similar projects.

Participant Hospital Effort

The extra work hours were largely due to unexpected barriers in data access and delivery. In addition to the data sharing barrier discussed above, there were problems from institutional EHR-data handling idiosyncrasies. The limited supply of hospital IT resources enlarged the total impact to the project; the extra work hours that were required, but not accounted for in hospital IT project plans, required that Amalga implementation at participant hospitals be stretched out for months longer than planned. This affected the project by delaying Amalga production deployment until the end of the three-year project, which ultimately prevented hospitals from using Amalga for their own internal real-time QI or other clinical data inquiry. The downstream consequence was the loss of this trial run opportunity to convert a hospital to an ongoing customer for sustainability.

The clinical EHR systems at participating hospitals generally were not configured to facilitate a project like this. This barrier has been described before16 and relates to both inherent limitations of EHRs (i.e., EHRs are not designed for this type of ongoing, live data extraction) as well as idiosyncrasies in patient data introduced by hospital individualization of a widely used EHR brand. These idiosyncrasies prevent accurate comparison of needed hospital effort across institutions, even when the same EHR brand is used at all of them. Further, EHRs do not currently optimize entry or storage of discrete data elements at the point of care; it was determined through this project that a majority of SCOAP QI registry variables would benefit from this type of feature.

Most EHR systems can be configured by the institution or vendor to facilitate structured data capture, as well as to create structured reports of needed data. Many of the SCOAP QI registry variables (e.g., tobacco use history, compliance with medication use at home) are common to several SCOAP QI registries, but these results were rarely available as structured data. By including a concerted program of modifying participating site EHR systems to increase structured data entry, and by building custom reports to deliver those data, the technical barrier of finding, converting, and delivering needed information as discrete data elements could have been reduced. This approach, however, would have added a significant burden of work for the participating sites, and would not have helped us understand whether needed QI data can be gathered by automated processes from various EHR systems as they are in operation today.

Though most data aggregated by EHRs (e.g., laboratory data) are available using well-established transmission methods, such as the widely used Health Level 7 standard, other required data was much more difficult to extract and transmit (e.g., handwritten information scanned into the EHR). As a consequence, data aggregation and integration tools like Amalga cannot always obtain all kinds of data that are contained in the EHR. This is an especially important point, because we found it affected all participants. It means that even a fully optimized automation infrastructure would still not achieve complete data extraction with the inherent limitations of today’s EHR implementations.

Finally, correctly configuring the process to extract, transmit, and load data properly to Amalga could not be completed by CERTAIN or Caradigm alone, but required time from hospital IT staff to explain and address the specific details of data coding and extraction as well as constraints or limitations on transmitted data. These individuals were the only experts who knew the idiosyncrasies of which systems stored clinical data in the most usable state for the Automation Project, and how those data could be delivered from the hospital system to Amalga. They were the only individuals who could troubleshoot the extraction and transmission steps on the hospital side when discrepancies were found among inbound data to Amalga. It became clear that this kind of work was going to be a larger ongoing process for participating hospitals than had been predicted, and this again increased the sustainability burden in an unexpected and previously unreported way.

The research team explored options for obtaining more time by hospital IT experts. Most of the sites approached with this idea, however, noted that funding was not the problem—there simply were not enough people with the depth of local IT knowledge for the work. Hiring new staff would not fix this problem, because of the time needed for them to learn the local system idiosyncrasies before they could be effective.

To sustain the enhanced registry after the three-year project, we expected hospitals would provide financial or workforce resources to maintain data interfaces to the Amalga central data repository, plus pay for annual Amalga license costs. These resource requirements were difficult to estimate at the time of project inception for three reasons: (1) this was a new partnership model proposed to Caradigm, by one client, CERTAIN, which was managing relationships and potential future licenses with hospitals; (2) the scope of work, including the volume of electronic data interfaces between each hospital to Amalga, was difficult to establish prior to the project start. In fact, answering these questions were among the objectives of this project. The scope of work was critical to understanding whether a future license model would allow a hospital full access to Amalga (i.e., unlimited ability to view clinical data, as available through ingested data sources for which interfaces were built during the project) or limited access to a CERTAIN-designed SCOAP view (i.e., access only to view clinical data from SCOAP QI registry forms); and (3) the speed at which AHRQ-enhanced registry projects were funded and expected to move forward, under the American Recovery and Reinvestment Act, required CERTAIN investigators to rapidly initiate project operations, and ongoing resource projections could not be based on long-term historic costs.

Nevertheless, calculated cost savings appeared to be about half that predicted, and with a near-tripling of the demand on hospital IT resources, the project could not mount a compelling argument for a return on investment to hospitals. Lack of inexpensive technologies to facilitate data extraction and transformation from idiosyncratic EHR systems is an ongoing barrier to sustainability. These projects continue to require too much manual work—either manually extracting and transforming the data, or manually configuring and updating the data transfer processes.

Solving this barrier should be a national priority. For example, updated Meaningful Use stage 3 objectives already include a menu item that eligible hospitals and eligible professionals “Electronically transmit data from CEHRT [Certified Electronic Health Record Technology] in standardized form…to one registry.”17 This requirement should be stronger. First, the CEHRT must be capable of normalizing all transmitted data to a nationally standardized data dictionary. Second, eligible hospitals and eligible professionals should store a copy of transmitted data in a central data repository (which could be a regional HIE or institution-based central data repository) capable of producing data sets to answer specific QI and CER questions. Third, a pathway should be established to move from requiring only one registry to standardizing data on all patients.

Conclusion

The most challenging barriers to completing the Automation Project were not primarily technological challenges, but were instead hospital recruitment challenges, lack of alignment with existing health IT projects, restrictive governance and security issues, and unexpected costs raised by the differences in EHR implementation, all of which posed barriers to sustainability. The lessons learned from the CERTAIN Automation Project, and the solutions proposed here, should be both useful and valuable to the informatics community. The challenges we encountered in implementing a centralized data repository across multiple, unaffiliated institutions point to the need to align national IT initiatives, update national policies surrounding data governance and human subjects protections, and advance the field of interoperability standards. Such efforts ought to lower costs and encourage substantial progress toward improving quality and lowering the cost of health care.

Despite the challenges encountered in the Automation Project, over the same project term, CERTAIN experienced multiple successes in the QI and research realms. These successes are attributed to continued pursuit of distributed data-sharing activities that use a variety of data collection methods, including direct communication from patients, automated electronic data extraction, and manual chart review. Three years after network initiation, CERTAIN is operating 12 QI, 5 CER, and 2 dissemination and implementation projects; and it includes over 150 clinicians, 35 clinician offices, and 25 hospitals in the network. Network growth continues to be limited by the human cost of data extraction. It is our hope that our work here and that of others will continue to drive toward the potential of secondary uses of EHR data to drive a learning health care system.