Enhancing the use of EHR systems for pragmatic embedded research: lessons from the NIH Health Care Systems Research Collaboratory.

OBJECTIVE: We identified challenges and solutions to using electronic health record (EHR) systems for the design and conduct of pragmatic research.
MATERIALS AND METHODS: Since 2012, the Health Care Systems Research Collaboratory has served as the resource coordinating center for 21 pragmatic clinical trial demonstration projects. The EHR Core working group invited these demonstration projects to complete a written semistructured survey and used an inductive approach to review responses and identify EHR-related challenges and suggested EHR enhancements.
RESULTS: We received survey responses from 20 projects and identified 21 challenges that fell into 6 broad themes: (1) inadequate collection of patient-reported outcome data, (2) lack of structured data collection, (3) data standardization, (4) resources to support customization of EHRs, (5) difficulties aggregating data across sites, and (6) accessing EHR data. DISCUSSION: Based on these findings, we formulated 6 prerequisites for PCTs that would enable the conduct of pragmatic research: (1) integrate the collection of patient-centered data into EHR systems, (2) facilitate structured research data collection by leveraging standard EHR functions, usable interfaces, and standard workflows, (3) support the creation of high-quality research data by using standards, (4) ensure adequate IT staff to support embedded research, (5) create aggregate, multidata type resources for multisite trials, and (6) create re-usable and automated queries.
CONCLUSION: We are hopeful our collection of specific EHR challenges and research needs will drive health system leaders, policymakers, and EHR designers to support these suggestions to improve our national capacity for generating real-world evidence.

INTRODUCTION

Pragmatic clinical trials (PCTs) are randomized controlled trials designed for generalizability, often involving multiple clinical sites with broad eligibility criteria., Their advantage is the ability to determine if health interventions actually work in practice, and hence such trials can generate “real-world evidence” to inform implementation, clinical practice, and regulatory decision-making., PCTs are “pragmatic” in that they are embedded in the workflows of diverse healthcare systems and aim to leverage existing data streams in the electronic health record (EHR) to limit the costs and burden of research data collection while maximizing the ability to answer important clinical questions and deliver better-quality care.

The Health Care Systems Research Collaboratory (hereafter, “Collaboratory”) is supported by the National Institutes of Health (NIH) Common Fund and was designed to strengthen the national capacity to implement cost-effective, large-scale PCTs that are embedded in routine care and conducted in partnership healthcare systems., The trials often involve cluster randomization (of hospitals, clinics, or primary care providers, etc.), the interventions may be implemented by health system personnel, and data are collected as part of routine clinical care. The Collaboratory is currently supporting 21 large-scale, high-impact PCTs (or Demonstration Projects) that address issues of major health importance, such as hospital-based infections, chronic pain, the opioid crisis, colorectal cancer, medication adherence, and suicide, all in different phases of completion (Table 1). The portfolio includes 6 new projects that are part of the Pragmatic and Implementation Studies for the Management of Pain to Reduce Opioid Prescribing (PRISM), which are part of the HEALSM (Helping to End Addiction Long Term) Initiative. To apply lessons learned from ongoing projects and to provide expertise to the Demonstration Projects, 7 Core working groups were created: EHRs; phenotypes, data standards, and data quality; patient-reported outcomes; healthcare systems interactions; ethics and regulatory; biostatistics and study design; and stakeholder engagement cores. A centralized Coordinating Center and the Core groups assist with the design, conduct, and logistical challenges of the projects and disseminate generalizable knowledge about PCT methods and enabling factors.

Table 1.

Collaboratory projects and EHR functionality

Study name	Project goal	Setting/population	EHR functionality and data types	Status (at the time of survey)
ABATE13 Active Bathing to Eliminate Infection NCT02063867	Determine if using antiseptic bathing for all patients and nasal ointments for patients harboring methicillin-resistant Staphylococcus aureus (MRSA) reduces multidrug-resistant organisms and bloodstream infections	53 hospitals/339 902 patients in adult medical, surgical, oncology, and step-down units	Census, demographics, administrative codes, nursing documentation, and extensive microbiology results.	Completed
ACP PEACE16 Advance Care Planning: Promoting Effective and Aligned Communication in the Elderly NCT03609177	Test whether clinician communication skills training and patient video decision aids increase completion of advance care planning	36 oncology clinics across 3 health systems/Patients >65 years of age with advanced cancer (∼12 000 patients and clinicians)	Free-text notes	Recruiting
BackInAction26 Pragmatic Trial of Acupuncture for Chronic Low Back Pain in Older Adults	Evaluate the effectiveness of acupuncture in older adults with chronic low back pain	4 performance sites/789 adults ≥65 years of age with chronic low back pain	Back-end functionality to extract data on potential recruits, and utilization outcomes data for participants.	Planning
BeatPain Utah Nonpharmacologic Pain Management in Federally Qualified Health Centers (FQHCs) Primary Care Clinics	To compare the effectiveness of nonpharmacologic intervention strategies for patients with back pain.	FQHCs throughout the state of Utah	Demographics, encounter diagnosis/problem list, and medications	Planning
EMBED17 Pragmatic Trial of User-Centered Clinical Decision Support to Implement Emergency Department-Initiated Buprenorphine for Opioid Use Disorder17 NCT03658642	Test the effectiveness of user-centered computerized clinical decision support on rates of emergency department–initiated buprenorphine/naloxone and referral for ongoing medication-assisted treatment in patients with opioid use disorder	4000 patients in 18 sites in 5 large health systems. clinical decision support target is clinicians treating adult patients with opioid use disorder in the emergency department	Dozens of data elements, which are not standardized between healthcare systems	Not yet recruiting (at time of survey; recruiting now)
FM TIPS25 Fibromyalgia TENS in Physical Therapy Study NCT04683042	Test the feasibility and effectiveness of adding transcutaneous electrical nerve stimulation (TENS) nonpharmacologic treatment for pain and fatigue in patients with fibromyalgia (FM)	24 routine physical therapy clinics and 5 health systems in rural and urban settings/∼600 patients with FM	Episode of care data: identification of participation in research study, flag of fibromyalgia diagnosis; identifiers for patient name and date of birth; start and end date for episode of care, ICD 10 diagnostic codes, number of physical therapy visits scheduled, number of physical therapy visits attended. Patient visit level data: CPT Codes per visit; outcome measure patient-specific functional scale on PT Visit 1 and every 30 days while in PT	Planning
GGC4H Pragmatic Trial of Parent-Focused Prevention in Pediatric Primary Care: Implementation and Adolescent Health Outcomes in Three Health Systems (GGC4H: Guiding Good Choices for Health) NCT04040153	Test the feasibility and effectiveness of implementing a universal evidence-based anticipatory guidance curriculum (Guiding Good Choices) for parents of early adolescents	3 large integrated health systems; 75 sites/Pediatric primary care practices serving families with adolescents 11-12 years of age (∼3600 adolescents and families)	Diagnoses, symptoms, procedures, encounters, prescriptions (orders and fills), settings, provider characteristics, and demographics variables (eg, sex, race/ethnicity, insurance type)	Not yet recruiting (at time of survey; recruiting now)
GRACE24 Hybrid Effectiveness-Implementation Trial of Guided Relaxation and Acupuncture for Chronic Sickle Cell Disease Pain	A hybrid type 1 effectiveness-implementation trial to assess the effectiveness of acupuncture and guided relaxation on individuals with sickle cell disease while observing and gathering information on implementation in 3 health systems.	3 performance sites/366 individuals with sickle cell disease and chronic pain	Recruitment of patients, integration of 4 patient-reported outcomes into clinical care	Planning
HiLo27 Pragmatic Trial of Higher vs Lower Serum Phosphate Targets in Patients Undergoing Hemodialysis NCT04095039	Test the effects of liberalizing the serum phosphate target (“Hi”) versus maintaining aggressive phosphate control (“Lo”)	2 large dialysis provider organizations; 100 sites/∼4400 patients with end-stage renal disease receiving maintenance hemodialysis	Automated data on phosphate binder prescription changes	Not yet recruiting (at time of survey; recruiting now)
ICD-Pieces Improving Chronic Disease management with Pieces NCT02587936	Improve care for patients with chronic kidney disease, diabetes, and hypertension by using a novel technology platform (Pieces) that uses the EHR to identify patients and by assigning practice facilitators within primary care practices or community medical homes	4 health systems; 143 sites/∼11 000 patients with multiple comorbid conditions (chronic kidney disease, diabetes, and hypertension)	Medication Rx Vital sign data ICD10CM/ICD10PCD hospital claims data. ICD10CM problem list ICD10CM encounter billed diagnosis	Enrollment completed
LIRE14,23 Lumbar Image Reporting with Epidemiology NCT02015455	Determine if inserting epidemiological benchmarks (essentially representing the normal range) into lumbar spine imaging reports reduces subsequent tests and treatments	4 health systems; 98 clinics/250 401 patients with low back pain	ICD-9 and 10 diagnosis and procedure codes; CPT codes; pharmacy prescription and filled data; radiology images and reports;	Completed
NOHARM Nonpharmacologic Options in Postoperative Hospital-based and Rehabilitation Pain Management NCT04570371	Evaluate the feasibility of EHR-embedded patient- and clinician-facing decision support for nonpharmacologic pain care after surgery	4 large integrated health systems; 22 practice clusters/∼54 000 postoperative patients	Epic PROM collection capabilities via MyChart portal and tablet at point of care; portal questionnaire functionality to embed an interactive conversation guide; Clinical decision support to provide alerts, prompt actions; nursing inpatient educational activities; discharge summaries and processes; registries to drive automated, individualized messaging	Planning
Nudge28,29 Personalized Patient Data and Behavioral Nudges to Improve Adherence to Chronic Cardiovascular Medications NCT03973931	Use pharmacy refill data to test effectiveness of medication reminder nudges delivered to patients’ mobile phones, and test an interactive chat bot mechanism that optimizes content	3 large integrated health systems/14 700 patients with chronic cardiovascular conditions who take medications	ICD diagnosis codes and pharmacy refill data.	Recruiting
OPTIMUM Group-Based Mindfulness for Patients With Chronic Low Back Pain in the Primary Care Setting NCT04129450	Evaluate a group-based mindfulness program (mindfulness-based stress reduction) for patients with chronic low back pain within primary care	Primary care clinics in 3 health systems/∼450 patients with chronic low back pain	Use of EHR-based data to generate recruitment lists; Referral orders/flags for enrollment, routed to study staff Extraction of data on demographics, diagnoses, procedures, medication prescribing, and health service utilization as study covariates and outcomes	Planning
PPACT10 Pain Program for Active Coping and Training NCT02113592	Help patients adopt self-management skills for chronic pain, limit use of opioid medications, and identify factors amenable to treatment in the primary care setting	3 staff model health plans; 106 primary care providers/860 patients with chronic pain on long-term opioid therapy	Patients eligibility was based on long-term opioid treatment receipt for pain, also used for PCP clustering, and for health services/cost analyses and identification of moderators (clinical and demographic characteristics).	Completed
PRIM-ER12,30,31 Primary Palliative Care for Emergency Medicine NCT03424109	Test effectiveness of primary palliative care education, training, and technical support for emergency medicine	33 emergency departments/patients ≥66 years of age in the emergency department with serious, life-limiting illness (4983 providers)	P-CaRES screening process: (1) life-limiting conditions, including end-stage organ disease, advanced cancer, septic shock or multiorgan failure in elderly patients, or a high chance of accelerated death (eg, cardiac arrest); (2) functional decline, uncontrolled symptoms, caregiver distress, or provider gestalt regarding limited prognosis31,33	Not yet recruiting (at time of survey; recruiting now)
PROVEN18,21 Pragmatic Trial of Video Education in Nursing Homes NCT02612688	Determine if showing advanced care planning videos in nursing homes affects the rates of hospitalization	2 nursing home health systems; 359 nursing homes/nursing home health systems serving long-stay (>12 months) patients with advanced comorbid conditions (166 196 patients)	An identical form was introduced into the EMR of each partner	Completed
SPOT19 Suicide Prevention Outreach Trial NCT02326883	Compare outcomes in patients who receive care-management or online skills training for suicide prevention versus usual care	4 large health systems/18 889 individuals at elevated risk for suicide on a depression scale	Encounter diagnoses PHQ9 scores	Analysis in progress
STOP CRC8 Strategies and Opportunities to Stop Colon Cancer in Priority Populations NCT01742065	Improve the rates of colorectal cancer screening by mailing fecal immunochemical testing tests to patients at federally qualified health centers	26 federally qualified health center clinics/62 155 individuals eligible for colorectal screening per the US Preventive Task Force guidelines	A real-time registry in the EHR was used to determine those eligible for screening34. Through a validation study35, accuracy of EHR data was confirmed (88% positive predictive value for identifying eligible participates).	Completed
TiME11 Time to Reduce Mortality in End-Stage Renal Disease NCT02019225	To determine whether increasing hemodialysis session duration reduces mortality and hospitalization rates for patients receiving maintenance hemodialysis care	266 dialysis facilities operated by 2 dialysis provider organizations/7035 adults who have initiated treatment with maintenance hemodialysis within the past 120 days	All of the data elements, including outcomes, were obtained from the EHR, including death, hospitalization rate, predialysis BP, postdialysis hypotension, interdialytic weight gain, fluid removal rate, missed dialysis sessions, and change in quality of life as assessed by the Kidney Disease Quality of Life Short Form-36.11	Completed
TSOS22,32 Trauma Survivors Outcomes and Support NCT02655354	To coordinate care and improve outcomes for trauma survivors with post-traumatic stress disorder (PTSD) and comorbidity and to provide the American College of Surgeons with multisite pragmatic trial evidence that could further inform regulatory policy	25 US level 1 trauma centers/635 trauma survivors with PTSD and comorbidity	Baseline 10-questions PTSD screening, PTSD Checklist, Patient Health Questionnaire, PHQ-9, Alcohol Use Disorder Identification Test (AUDIT), Medical Outcomes Study Short Form (MOS SF) SF-12, Abbreviated Injury Scale (AIS). ICD codes for traumatic brain injury, and comorbid conditions	Completed

EHR: electronic health record.

ABATE

ACP PEACE

BackInAction

EMBED

Pragmatic Trial of User-Centered Clinical Decision Support to Implement Emergency Department-Initiated Buprenorphine for Opioid Use Disorder

FM TIPS

GRACE

HiLo

LIRE,

Nudge,

PPACT

PRIM-ER,,

PROVEN,

SPOT

STOP CRC

TiME

TSOS,

In 2017, the EHR Core reported the challenges from the Collaboratory’s first round of 9 Demonstration Projects. Four broad informatics challenges were identified, including: “(1) using clinical data for research, (2) integrating data from heterogeneous systems, (3) using EHRs to support intervention delivery or health system change, and (4) assessing and improving data capture to define study populations and outcomes.”,

Using qualitative data generated from an electronic survey completed by lead scientists for each of the 21 PCT projects in the Collaboratory portfolio, we update and elaborate on the EHR-related challenges. This collection of examples and prerequisites for successful ePCTs aim to improve the ability to use data collected during routine care to support research and generate new knowledge in the form of implementable clinical interventions to address important public health issues.

MATERIALS AND METHODS

Study design and population

This online cross-sectional and semistructured survey that included 4 open-ended questions was circulated to all 21 Collaboratory Demonstration Projects (including active and completed studies and 2 new projects in early design stage). Collectively, these PCTs in the Collaboratory address a range of critical public health problems, including the opioid crisis, suicide, and colorectal cancer, using a variety of interventions, research designs, settings, and patient populations (Table 1).

Survey content and administration

The semistructured survey was created by members of the EHR Core and consisted of 20 questions, informed by known challenges.,, The survey included 4 open-ended questions: (1) Is there EHR functionality that you would like to have for your study, but didn’t? If yes, please explain. (2) Which EHR data elements are you using for your study? To the best of your knowledge, are they standardized across the EHR systems used in your study? (3) What key question(s) would you ask of EHR vendors to assess their readiness to support research? and (4) What data or functions would you like to see standardized across all EHR systems? Sixteen structured questions pertained to challenges relating specifically to data and data access issues, requiring either “yes” or “no” responses (Supplementary Table S1). If the participants answered “yes”, then they were prompted to elaborate on their experiences. Completion of the survey was voluntary. Data were collected from Collaboratory demonstration project representatives between August 20, 2020, and June 30, 2021.

Data analysis

The responses from participants on the open-ended questions amassed a large amount of rich text, and we conducted a thematic analysis on the resulting data. We coded narrative responses from the 4 open-ended questions directed to each project. First, RLR prepared the data for coding by collating and unitizing the responses. Next, coding was completed by RLR to develop an initial codebook. A second coder (KSM) reviewed the initial coding schema and suggested changes as applicable. Group consensus was achieved for each coding unit, resulting in Cohen’s kappa coefficient of 0.72. The resulting 21 codes were iteratively grouped by RLR and BJD into families, eventually resulting in the final 6 themes. This process is summarized in Figure 1.

Figure 1.

Process for determining themes for EHR challenges in collaboratory projects. EHR: electronic health record.

The consensus themes and underlying codes were compiled and presented to the Collaboratory EHR Core for discussion between November 2020 and December 2020. During these meetings, the EHR Core members provided clarification on the 6 themes and discussed their implications.

Development of prerequisites to facilitate PCTs

The EHR Core used additional calls to brainstorm and discuss strategies and recommendations for enhancing EHR systems to support PCTs. We received input from our expert and diverse members of the EHR Core, Healthcare Systems Interactions Core, and Patient-Reported Outcomes Core. Through the circulation and collaborative writing of this report, members of the EHR, Patient-Reported Outcomes, and HCS Interactions Cores contributed to the list of prerequisites for PCTs and recommended EHR enhancements to improve capacity for pragmatic trials.

RESULTS

A total of 20 out of 21 (95%) Demonstration Projects completed the survey shown in Table 2.

Table 2.

Reported challenges related to using EHR data in pragmatic trials

Survey Question: (Yes/No) Did you experience any challenges related to:	# Projects reporting: n = 20	New projects n = 2 (10%)	Early phase projectsa n = 4 (20%)	Middle phaseb projects n = 7 (35%)	Complete or near complete n = 7 (35%)
IT staff turnover	11 (55%)	1 (5%)	1 (5%)	3 (15%)	6 (30%)
Integrating data from heterogeneous systems	11 (55%)	1 (5%)	2 (10%)	4 (20%)	4 (20%)
Using EHRs to support intervention delivery or health system change	10 (50%)	2 (10%)	1 (5%)	3 (15%)	4 (20%)
Missing data	8 (40%)	0	3 (15%)	0	5 (25%)
Assessing the validity or accuracy of EHR data	7 (35%)	0	2 (10%)	1 (5%)	4 (20%)
Acquiring data	7 (35%)	1 (5%)	2 (10%)	2 (10%)	2 (10%)
Use of free-text or narrative data from EHR	6 (30%)	0	0	2 (10%)	4 (20%)
Data cleaning	6 (30%)	0	1 (5%)	1 (5%)	4 (20%)
Completeness of data across EHRs	6 (30%)	0	2 (10%)	1 (5%)	3 (15%)
Interoperability	5 (25%)	1 (5%)	0	1 (5%)	3 (15%)
Data quality assessment	5 (25%)	0	1 (5%)	0	4 (20%)
Requesting data (ie, knowing which data are available)	5 (25%)	1	2 (10%)	0	2 (10%)
Identification and use of clinical phenotype definitions	4 (20%)	0	1 (5%)	1 (5%)	2 (10%)
Determining or negotiating the frequency of data pulls	4 (20%)	0	0	1 (5%)	3 (15%)
Gaining permission to use data	2 (10%)	0	1 (5%)	0	1 (5%)
Need to upload data or write to EHRs	1 (5%)	0	1 (5%)	0	0

Early-phase projects are just transiting from planning to implementing the trial.

Middle-phase projects are enrolling and collecting data but early; complete or near complete are nearly done with enrollment.

EHR: electronic health record.

The specific EHRs across study sites are presented in Table 3.

Table 3.

Characterization of EHR use across Collaboratory demonstration projects

		PRISM projects					Active projects										Completed projects
	Back in Action	BeatPain Utah	GRACE	FM TIPS	NOHARM	OPTIMUM	ACP-PEACE	EMBED	GGC4H	HiLo	ICD-Pieces	Nudge	PRIM-ER	PROVEN	SPOT	TSOS	ABATE	PPACT	LIRE	STOP-CRC	TiME
Use of EHR data
Determine eligibility	x	x	X		x	x	x	x	x	x	x	x		x	x	x	x	x	x	x	x
Outcomes	x	x	X	x	x	x	x	x	x	x	x	x			x	x	x	x	x	x	x
Patient-reported outcomes					x																x
Part of intervention delivery		x			x			x		x	x		x	x	x	x	x	x	x	x	x
EHR Systems/Data Source
Epic	x	x	X		x	x	x	x	x		x	x	x		x			x	x	x
5 different physical therapy systems				x
Allscripts							x				x
American systems														x
AthenaHealth		x				x
Cerner						x		x					x						x
CPRS (VA)											x	x
Custom
DaVita										x											x
Fresenius																					x
HSCRN-virtual data warehouse																			x
Mayo clinic proprietary																			x
Medicare claims														x
MEDITECH																	x
Point click care														x
Trauma registry																x
eClinicalWorks (eCW)	x
CompuGroup	x

EHR: electronic health record.

The responses to the open-ended questions included 82 unique ideas that were compiled into 21 consensus codes and further refined into 6 broad themes (Figure 1).

Survey themes

Theme 1: Inadequate collection and integration of patient-centered data, including patient-reported outcomes, questionnaires, and documentation of services received

Seven projects—particularly those researching management of pain or psychological trauma—voiced a need for patient-centered data within health systems and EHRs. These studies reported having to collect primary data that were expected to be readily available, including patient-reported outcomes (PROs), previously completed questionnaires, and advance care planning conversations. This led to time- and monetary-consuming adjustments as illustrated by a project using Epic when they said, “We would have appreciated greater ease and flexibility of automatically responding to patient-reported information, including PROs to match participants’ needs. Automatically assigning the intervention to specific patients using Epic EHR logic and based on EHR data was clunky, time consuming, and expensive. We wound up using the MyChart questionnaire functionality, but this was cumbersome, and the user experience was not optimal.”

Another study reported that even if patient-centered data were collected, it was not done in a standardized manner across health systems, preventing the ready use of these data for multisite pragmatic studies.

We hoped to rely on self-report/patient reported outcomes that occur during pediatric well visits… many of the items were different across systems and there was a great deal of missing data for these data at each health system.

Finally, while some PROs were collected, not all the PROs needed for a trial were:

We had 8 patient reported outcomes (PROs) that we wanted patients to enter into the EHR. … Only 4 of the PROs were included in the EHR, the other 4 will be collected by REDCap.

Theme 2: Lack of structured research data collection, standard EHR functions, usable interfaces, and standard workflows

Several respondents commented on their experiences with data collection in the EHR environment. Some reports were positive, as the EHR allowed the structured collection of research data, making it easier to collate, analyze, and share across study sites. One study reported that even when data collection was standardized and integrated into the EHR, there were still varying levels of missing data across sites, and another reported notable errors in the accuracy of structured data. Several projects shared requirements for very specific EHR functionality (eg, prescribing educational videos to patients, notifications to detect changes in medication dosages, customized communications to patients though patient portals, provider alerts regarding state-specific requirements for life-sustaining treatment) that was not readily in place, necessitating the creation of new EHR-based tools or manual extraction of specific data. One respondent suggested that the variability in EHR functionality across sites can introduce potential usability concerns, potentially affect fidelity to the intervention, and hinder scalability of clinical interventions.

We had to use different aspects of EHRs to implement our intervention. Some used the EHR with a static message, another the EHR with a dynamic “pop-up”, one used the Radiology Information System (RIS)/dictation module. Making all of these approaches work smoothly was a challenge.

The EHRs we are working with have little customization capabilities for clinical decision support reminders and alerts. Two of them do not even allow clients to add any new reminders/alerts. Clients can only turn on/off reminder/alerts that are implemented by the vendor and those are often suboptimal. Also, the user interface of reminders/alerts is very basic and does not support actions such as placing an order.

Another project commented on scalability when they stated that, “Better integration of a user-centered interface that is scalable across health systems and EHR vendors [is important for data collection].”

Theme 3: Lack of standardization of structured data for research

Ten studies reported challenges with standardized data across sites. Studies had challenges with standard data elements (n = 5), standard coding (n = 2), inconsistent use of standard coding (n = 2), and lack of standardized or structured (imaging) reports (n = 1). Several studies reported that their study would benefit from the adoption and use of standard code systems—specifically for laboratory test results, procedures, medications, and indications for imaging procedures, even across health systems that use the same EHR product.

The codes are for the most part standardized (same codes used across sites), but their implementation is not. Some settings used ICD-procedure codes and others CPT for the same procedures… One site did not use CPT codes but instead their own proprietary procedure codes. Sites had different limits to the number of diagnosis codes reported for encounters and some sites did not identify primary diagnosis code. Encounter type codes were inconsistent across sites.

Others reported challenges not only in the adoption of standard code systems but also in the consistent use of them. The variation across sites in turn necessitated further processing—and significant resources and time—to aggregate and analyze the data.

We use several EHR data elements. However, they are not fully standardized across systems using same vendor—due to nuances of local build. This is rate limiting in terms of EHR research.

Several projects reported that standard data elements or patient screening and assessment tools would have been useful for their studies. In some cases, these measures exist and in other cases would require the medical and clinical practice communities to develop and advocate for their development and adoption as standard practice. Respondents from a pediatric—adolescent study reported that research and practice would benefit from:

Well Child, Well Adolescent, Adult primary care visit templates using brief, validated, evidence-based screening and assessment tools, … including “gold-standard” behavioral health (mental health and alcohol and drug) measures and clinical decision response algorithms (eg, NIAAA or AUDIT alcohol screening, S2BI or CRAFFT substance use screeners for adolescents, PHQ-2 or 9 + Columbia Suicide Screener, etc.).

The team also suggests we: “Work towards consensus on sets of standardized core clinical measures, perhaps endorsed by bodies such as AAP or AMA or USPSTF [American Academy of Pediatrics, American Medical Association, United States Preventative Services Task Force], perhaps with CMS [Centers for Medicare and Medicaid Services] input, that major health systems could get onboard with, and build them into the platforms.”

Theme 4: Lack of resources to support customization of EHR systems for PCTs

Nine studies identified tools and functions to support research that would be useful for their studies and pragmatic intervention. These included the ability to order an intervention, ability for local customization of EHRs (presumably faster and cheaper than if the EHR company does the work), standardized patient summarizes, configuration of EHR for research (for provider alerts about potentially eligible subjects; tracking patients on protocols; patient portal enhancements, including personalized messaging and randomization; and for research).

Respondents were able to identify several ways that the health system could have supported the EHR configuration for their trial but did not for various reasons, including doubt that the health system had the available resources to do so.

There may be some tension between what helps with clinical research and facility whereas lots of (understandable) caution on clinical delivery system side to allow too much tinkering that may overburden clinical workflow, so I think improved integration of PROs would be terrific but know this can be seen as a double edged sword.

Similarly, another project stated:

Our issues are less about Epic and its capabilities, and more about bandwidth of providers and patients and the supremacy of clinical needs over research needs.

A common feature is that IT departments have demands on them for patient care and billing, and therefore allocating IT resources to research is often not a priority.

Our trial required extensive support from IT programmers and report writers to enable data collection, cleaning, and analysis. EHR vendors allow extensive end user modification and specification that causes end-user engagement in report generation.

Eleven studies reported challenges with IT staff turnover and getting local IT resources to support their project.

Skilled technical staff may leave for better positions—in our health systems or outside companies.

A common feature is that IT departments have demands on them for patient care and billing, and therefore allocating IT resources to research is often not a priority, and a request might take months or years to be started. These challenges seemed to reflect communication issues related to prioritization.

One of the biggest barriers we have experienced is the CMIO and high-level leadership has committed to working on this research study, but when the time comes to actually implement the [EHR alerts] there is a communication gap between the CMIO/Leadership and the IT analyst doing the work. …the person doing the work was not privy to all the previous conversations nor do they realize there is a specific timeline that needs to be followed. This has created some slight delays and a lot of additional and unnecessary conversations.

Another project stated,

Time allocation/resources is very important. If a site has other pressing priorities its difficult sometimes to get an IT analyst dedicated to work on the research items even with a subcontract.

Theme 5: Difficulties aggregating multidata type resources for multisite trials

While many trials were able to collect the data they needed, the aggregation of all the data from multiple sites presented as a common challenge. This presented as both challenges in sites that are not associated with the health system, and as data elements without common models or standards.

We are using pharmacy refill data to identify patients and deliver the intervention to patients. Pharmacy refill data comes from both internal pharmacy data as well as external pharmacies. We are currently able to get daily pharmacy refill data from health system internal pharmacies. For prescriptions filled pharmacies outside of the health system, we are not able to obtain daily pharmacy refill data currently.

Eleven studies faced challenges with the capacity to integrate their data with external sources and to accurately match data across these sources. Among these are challenges with integration with internal/external data sources, access to primary care provider practice data, identity matching across data sources, and lack of integration between care settings (specifically for dosage and medication changes).

[.challenges with]… Integration with specialty care, and procedures. Specifically access to colonoscopy outcomes and pathology lab data from biopsies. Claims data and cancer registry data would also been useful for exploratory outcomes. It would be useful to access data from [Epic's health information exchange module] Care Everywhere for research.

Data harmonization across different EHR products and even the same EHR at different health systems is very challenging and labor intensive.

Theme 6: Difficult and inefficient EHR data access: lack of re-usable and automated queries to support PCTs

Eight studies reported challenges with access to data for analysis. Several survey respondents expressed difficulties with automated data pulls from sites and also coordinating data across sites that update and send data at different intervals. These studies also suggested more widespread use of data warehouses and better access to data, with standard approaches to storing and retrieving data, including common data model (CDM) support.

Using the [site’s] VDW [Virtual Data Warehouse] was problematic due to variation in frequency of updates. Similarly, there were differential and sometimes substantial delays as to when EHR data became available.

…it would be extremely useful to be able to access medication dispensing records through our EHR data.

In addition to standardized/reusable data queries, we had requests for data cleaning support, data quality (including timeliness), and information around data provenance.

It makes a difference if the data is coming from a data warehouse group that has a layer of management of those data rather than a pure export of existing fields. Understanding latency is important. There are also quality differences at the item level, and making that transparent is important.

Other studies reported challenges and lack of tools to pull data from the EHR during the study to support the intervention and conduct of the trials.

We wanted to automate data delivery from EHR systems in order to reduce burden on site research coordinators, but very few study sites used EHRs that were built with an enterprise data warehouse that would permit straightforward data pulls, so complicated workarounds, manual data transcription, and complex electronic reports had to be created.

We had wanted to use the RedCap integration with [EHR product], but it does not pass the Privacy and Safety requirements for 3 of our 4 sites, and for the 4th it won’t pull custom data points from [EHR]. They are limited to pulling predetermined data elements.

Not all of our sites are able to download medication use and healthcare utilization data. One site, a Federally Qualified Health Center (FQHC), has an EHR with limited functionality. In addition, patients from the FQHC may be seen at local hospitals that are not part of the same health system, further complicating the automatic data extractions. Currently, automatic extraction is prohibited across systems via local agreements. Hence, we plan ‘hand extractions’ of EHR data.

PREREQUISITES FOR CONDUCTING PCTS

The EHR Core, PRO Core, and Health Care Systems Interactions (HCS) Core used the thematic EHR-related challenges reported in Collaboratory PCTs to identify prerequisites and recommendations for successful PCTs, which includes enhancing EHR systems to support PCTs. These are provided below.

Integrate the collection of patient-centered data, including PROs, questionnaires, and advance care plans, into EHR systems and clinical workflows to support pragmatic research and personalized clinical care

Healthcare systems need incentives to invest in the routine collection of PROs, and EHR vendors and the Office of the National Coordinator (ONC) for Health Information Technology could better champion, require, enable, or support the collection of PROs. Potential benefits for integrating PROs into the EHR include increasing clinician/patient communication,, improving patient satisfaction with care, and increasing symptom monitoring, which can improve clinical outcomes, such as survival. In addition, participant-prioritized outcomes (eg, anxiety, pain) might differ from traditional clinical outcomes and help with risk prediction. Research demonstrates that integration of PROs into the EHR is possible in a number of different types of care settings,, and facilitating additional measures for research is becoming more of a priority. If PROs have not been established before a PCT is being conducted, clinics could potentially use the trial to help identify domains to include in their EHR, although there is great variability in what patients think is important depending on the disease or condition. However, integrating PROs into the EHR can change workflow and requires buy-in from multiple levels, including input from clinicians and administrators at each site, as well as alignment with the priorities and the mission of healthcare system, and this is not trivial to accomplish. Finally, integration of PROs requires careful consideration and prioritization of what to measure and how to measure it, as sometimes PROs have not shown any effects/improvements in pragmatic settings. Some groups have started to compile recommendations on how to successfully incorporate PROs into the EHR,, and a staged approach is often helpful. Ideally, systems should identify standard mechanisms through which PROs can be added and collected for specific projects, as adding too many PROs can add to burden in the clinical workflow, and it is hard to predict what PROs a future study might need.

Facilitate high-quality structured research data collection by leveraging standard EHR functions, usable interfaces, and standard workflows

Principal Investigators (PIs) and their teams requested the ability for local customization of EHRs, standardized patient screening and assessments, and configuration of EHRs for research (eg, to provide alerts for potentially eligible subjects or providers, track patients on protocols, and patient portal enhancements, including personalized messaging and randomization, and for research). In many cases, the EHR functionality (eg, flagging potential or enrolled research subjects) that researchers requested was available in the EHR, suggesting that organizations were either unaware, under-resourced to implement them, or do not prioritize the use of EHR data in research.

Many of our challenges are not with the EHRs but with the health systems not using functionality. Although low-resourced settings might need a considerable amount of help to implement and maintain EHR functions, leveraging preloaded standardized EHR functions can accelerate implementation of innovative and transformative clinical practices and aid in generation of data from patient populations, which can contribute to improving the delivery of health care, as in a Learning Health System.

Ensure adequate IT and other staff or services to support embedded research

The primary role of healthcare organizations is to provide the best patient care possible at the lowest cost, and conducting research is a secondary goal. Staff turnover has been consistently reported as a challenge for Collaboratory PCTs, particularly IT staff turnover. All of these activities require local IT support, and local IT departments prioritize clinical care and billing, so obtaining IT support can be a challenge, a problem (understandably) exacerbated by the urgent needs created by COVID-19.

Devoting sufficient resources to IT support of PCTs is needed, and some members of our EHR Core suggested that research teams could hire IT staff solely dedicated to research; this could occur at the institutional level and be funded by research dollars. However, some of our investigators who have done this already report that the amount of funding is negligible next to the institution’s bottom line, and this research-charge model might reduce the amount of organizational HIT funding (in response to transitory grant funding which eventually end), possibly acting as a disincentive. One hired personnel with experience with the most common EHRs and used them to work with FQHCs to implement and configure EHR functionality.

Prioritize the creation of structured research data by using and promoting standards

Healthcare organizations could set priorities for IT staff so they increase use EHR data standards to support quality healthcare delivery, continuous quality improvement, and the generation of new knowledge. A main incentive comes from policymakers and regulations that affect payment, such as Uniform Data Systems (UDS) reporting for FQHCs and Meaningful Use requirements by the Centers for Medicare and Medicaid Services (CMS).,

Relevant standards include standard data elements, standardized coding systems and terminology, or structured reports, any of which could be incorporated into common EHR builds. Health systems can request that vendors improve their systems, and perhaps the most efficient way to achieve this is to synergize efforts with agencies that impose requirements tied to healthcare payments, such as CMS/Healthcare Effectiveness Data and Information Set

(HEDIS) quality measures and the United States Core Data for Interoperability (USCDI) data elements set by the ONC/CMS.

Create aggregate, multidata type resources for multisite trials

Because our pragmatic trials were designed to minimize burden of de novo data collection, all make use of current data in the EHRs, but many required other data types and sources, such as commercial pharmacy data, practice data (on provider and practice features), and PROs, which needed to be accessed and linked. The aggregation of all the data from multiple sites presented a common challenge, due to both the fact that different systems might collect data using different data elements and have varying adoption of different CDMs. Both complicate the development of systems and interfaces for researchers to access the data required for multisite trials.

Create re-usable and automated queries to support PCTs

Respondents also reported the need to re-access data throughout the lifespan of their trials, and new queries, or queries of multiple sites, was required. A need for ongoing support for queries (or potentially an automated data pull) was voiced by the respondents. Data are configured differently for the “same” clinical uses across multiple sites, and this makes developing queries even more complex. EHR infrastructure should be designed so that data query and output is efficient and research friendly. If the future is about big data science and using existing data, then we have to a priori create a system that supports that vision.

As we noted in our previous publications,, the lack of standardized data and EHR functions means that programs for data queries, data management (including data quality assessment), study conduct (eg, automated alerts, notes, and dashboards to support the trial itself), and analysis need to be created for every health system participating in a trial. To overcome this need for custom programming at each site, Collaboratory-affiliated researchers recommended increased collaboration between clinical researcher informaticists and healthcare IT operations professionals, as has been previously described by the Collaboratory and others.,, The establishment of multisite registries with data warehouses across health systems, especially for low-resourced settings, would help, and this is happening in several states. For example, in Utah, there is a data warehouse that is automating data feeds from 12 different health systems (FQHCs) using different EHRs. Another similar approach is the creation of a network that uses a CDM, such as the PCORnet,, although transforming data into a CDM can be time consuming and resource intensive.

DISCUSSION

Our findings add examples of specific services and standards, but the overarching question we face remains: how can researchers persuade health care organizations to invest in EHR enhancements and data infrastructure to support PCTs? Our findings from this survey of Collaboratory projects are similar to what we found in 2017: using EHR data for research and obtaining data from multiple sites are still major challenges; the amount of time and resources needed to use the EHR to support the delivery of interventions remains daunting, capturing high-quality data suitable for research is still difficult. While it might seem as if the needle has not moved on any of these issues, it is possible that studies are evolving in their data needs (eg, more complex interventions, more targeted and personalized interventions integrated with EHR systems) and that we are EHR-enabled to answer more sophisticated research questions. To better assess the improvement of these PCT facilitators at a national level, the Collaboratory Coordinating Center explores metrics for interoperability of EHRs and data “readiness”, for pragmatic research. A number of national developments indicated that standardization of EHRs and data are progressing. In addition to the recent ONC report (2020) promoting a research agenda for EHR-enabled research, the use of standards has certainly improved with Meaningful Use certification requirements, and the consolidation in the EHR market has reduced variation in EHR systems. Also, improvements in reporting infrastructure for quality measures have benefits for research.

Our findings also mirror areas of need in the 2020 report from the ONC, which calls for improvements in data quality, harmonization, access to interoperable data, services for efficient storage and data, integration with other health data sources, and data aggregation. The use of EHR systems and data takes effort, costs money, and takes away from other health system priorities. This seems fundamentally unchangeable, so the alternative is to change the “story” and the argument for robust and customizable EHR systems. The story needs to not be about promoting research per se but rather about finding and implementing the best and most effective treatments, and continuously learning within and across health systems. However, building and promoting infrastructure for gathering standardized, re-usable data from the EHR in support of continual learning are not without costs, and it is the joint responsibility of funders, researchers, healthcare systems leaders, EHR vendors, and policymakers to work together to accomplish these goals. The National Academy of Medicine called for the development of a learning health system 11 years ago. It stated that 90% of clinical decisions should supported by timely, up-to-date clinical information, but achieving that goal remains distant and will require “health system leaders to consider rigorous evidence generation a core function of ordinary health care, research funders to prioritize practical questions relevant to population health and to support infrastructure for embedded research.” Currently, patients bear the costs of a lack of evidence, and we must strive to change this. This might be facilitated using expertise and approaches from emerging Implementation Science, which provides a scaffold for evaluation and bringing about change in the healthcare setting. In a recent article by Collaboratory leadership, the authors suggest re-imbursing healthcare systems for research-related costs and supporting re-useable infrastructure in highly engaged systems as a way to mitigate the costs of engaging in research and encouraging healthcare system participation. Similarly, the inclusion of robust cost impact studies might demonstrate a business case for the use of effective treatments (and the deimplementation of ineffective ones). Collectively and over time, these studies might provide compelling justification for continuous pragmatic research in learning health systems.

Research stakeholders (including clinicians, policymakers, research sponsors, patients, and the public/healthy consumers) must persuasively promote the importance of high-quality data. This value is not limited to a specific trial, but extends to other uses such as population health, continuous quality improvement, comparative effectiveness research, pharmacovigilance, and even algorithmic and artificial intelligent safety surveillance in the future. These programs can be conducted more cheaply and effectively with high-quality data. Standards-based EHR systems can empower customers to adopt new practices, which can lead to improvement and implementation in health systems. However, the current status quo is for each instance of an EHR is to not be standardized, even across separate instances or Epic, and the many different data sources (16 across the 21 Collaboratory trials) creates an ongoing challenge for investigators conducting PCTs (ie, for any individual trial, this means that every data pull needs to be configured for each and every participating site). The value of standards-based EHR systems can be promoted to health system leaders as a necessary enabler for integrating evidence-based practices and intervention to improve care and the health of the populations served (ie, a means to link research to quality improvement and continuous learning).

A positive and hopeful development is the HEAL (Helping to End Addiction Long Term) studies in response to the opioid crisis. These 6 studies are supported by the Collaboratory and have helped define a set of common measures, based on research and clinical needs, including a common set of PRO measures. Using this common set of PRO measures will help with comparability across trials, minimize patient burden, as well as provide valid and reliable quality-of-life measures. One next step may be to lobby EHR vendors to include these basic measures and for EHR regulators/standards bodies to endorse them. Should the interventions evaluated by these studies (acupuncture, physical therapy, TENS, guided relaxation group-based mindfulness, etc.) be effective, the use of consistent measures across these studies could help with broader implementation. More recent success with N3C and COVID have shown that with the right incentive (and a bolus of centralized resources for harmonization), investigators can pull together disparate data quickly to understand the disease burden and effective targets for limited resources.

Our study teams also reported the need for configurable EHR systems. Often informaticists tightly control the EHR in an organization, and the key stakeholders, including policymakers, are not fully aware, empowered, or incentivized to champion or facilitate the EHR customizations needed for multisite trials. Enhancing the ability to customize EHRs enables responses to urgent public health needs, delivery of new and evidence-based treatments, and the ability to address emerging local needs and patient-specific care. The number of PCTs funded by NIH and supported by health systems is growing; implementation science has burgeoned as a field; research networks, such as the Patient-Centered Outcomes Research Network (PCORnet) are enabling use of harmonized data; and the FDA is championing the use of real-world evidence. At the same time, the humanitarian and financial issues of poor health metrics, disparities in health, and increases in chronic and multiple diseases are growing. We have not made a dent in the Triple Aim, and the Quadruple aim, which adds the dimension of preventing clinician burnout and frustration with EHR systems.

Driving change for routine PCTs will require meaningful partnerships between healthcare systems, individual clinics, informatics teams, EHR vendors, and researchers, as well as federally driven standards and policies to ensure that EHR systems can support research. Indeed, the 21st Century Cures ACT and other developments are laying the case for this, and we assert that clinical researchers and health services researchers—as representatives of their health delivery organizations—should be leading efforts to develop and promote data standards that can enable sharing and rapid adoption of tools that facilitate the trial. These researchers can also leverage new regulations for quality measures, and weigh in and advocate for the importance of PROs. Embedded PCTs are integral to a learning health system and will enable faster changes in practice and improved patient outcomes. In time, the case for standards will grow in clarity and support as the learning cycles get faster and the benefits become more visible and established.

Limitations

Our sample of studies is limited and may not represent all pragmatic trials, and our sample varies in terms of timing of research, from beginning of enrollment to complete, and was conducted during the COVID-19 pandemic, which may have exacerbated issues related to EHR use and enhancements. The types of studies in our sample vary greatly, as the Collaboratory is funded by the NIH Common Fund with the goal of learning how to conduct high-impact trials that matter to public health, although this variation may not be representative of all PCTs and may affect the conclusions we can draw about EHR system readiness for research. Another limitation is that responses were collected in an asynchronous survey, so classic interview methods could not be done; that is, responses could not be expanded upon.

CONCLUSIONS

The case for high-quality, standardized data goes beyond clinical care, as it can impact the population as a whole by providing better care based on reliable evidence. We argue for tailoring the use of EHR systems to enable the collection of patient perspectives and the extraction of high-quality, robust data to support pragmatic research toward the identification of effective treatments and implementation strategies, and to enable learning within and across healthcare systems.

FUNDING

This work was supported within the National Institutes of Health (NIH) Health Care Systems Research Collaboratory by the NIH Common Fund through cooperative agreement U24AT009676 from the Office of Strategic Coordination within the Office of the NIH Director. This work was also supported by the NIH through the NIH HEAL Initiative under award number U24AT010961. Demonstration Projects within the NIH Health Care Systems Research Collaboratory were supported by the following cooperative agreements with NIH Institutes, Centers, and Offices: ABATE Infection (UH2AT007769, UH3AI113337), ACP PEACE (UG3AG060626, UH3AG060626), EMBED (UG3DA047003, UH3DA047003), GGC4H (UG3AT009838, UH3AT009838), HiLo (UG3DK118748, UH3DK118748), ICD-Pieces (UH2DK104655, UH3DK104655), LIRE (UH2AT007766, UH3AR066795), Nudge (UG3HL144163, UH3HL144163), PPACT (UH2AT007788, UH3NS088731), PRIM-ER (UG3AT009844, UH3AT009844), PROVEN (UH2AG049619, UH3AG049619), SPOT (UH2AT007755, UH3MH007755), STOP CRC (UH2AT007782, UH3CA188640), TiME (UH2AT007797, UH3DK102384), and TSOS (UH2MH106338, UH3MH106338). Demonstration Projects within the NIH HEAL Initiative were supported by the following cooperative agreements with NIH Institutes, Centers, and Offices: BackInAction (UG3AT010739, UH3AT010739), BeatPain Utah (UG3NR019943), FM TIPS (UG3AR076387, UH3AR076387), GRACE (UG3AT011265), NOHARM (UG3AG067593, UH3AG067593), and OPTIMUM (UG3AT010621, UH3AT010621). The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH or its HEAL Initiative.

AUTHOR CONTRIBUTIONS

A study representative and PI reviewed the final summary of each study and its features presented here. All authors listed here participated in the discussion and refinement of the themes and suggestions we present here and provided substantive comments on the manuscript.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.

CONFLICT OF INTEREST STATEMENT

The following authors report no competing interests: ARP, BJD, CG, DD, EBL, GDF, JMB, JRL, KRF, KS, LT, MOE, RLR, CLP, and JMS. ECO reports research grants to her institution from BMS and Novartis. CKZ reports research grants to her institution from F.D.A., NIH, the Foundation for Angelman Syndrome Therapeutics, and the Childhood Arthritis & Rheumatology Research Alliance (CARRA). ADB reports a research grant to his institution from Alike Health and reports consulting through Boyd Health Information Consulting LLC. KM reports a research grant to her institution from Pfizer. KSM reports research grants to his institution from Novartis, Amgen, Bayer, and Boehringer Ingelheim, consulting support from Novartis, and that he is coinventor of Hive Networks, Inc.

DATA AVAILABILITY

The data underlying this article cannot be shared publicly due to the potential to identify individual PIs and participating providers. The data will be shared on reasonable request to the corresponding author.

Click here for additional data file.