Adapting practice-based intervention research to electronic environments: opportunities and complexities at two institutions.

BACKGROUND AND
PURPOSE: Primary care practice-based research has become more complex with increased use of electronic health records (EHRs). Little has been reported about changes in study planning and execution that are required as practices change from paper-based to electronic-based environments. We describe the evolution of a pediatric practice-based intervention study as it was adapted for use in the electronic environment, to enable other practice-based researchers to plan efficient, effective studies.
METHODS: We adapted a paper-based pediatric office-level intervention to enhance parent-provider communication about subspecialty referrals for use in two practice-based research networks (PBRNs) with partially and fully electronic environments. We documented the process of adaptation and its effect on study feasibility and efficiency, resource use, and administrative and regulatory complexities, as the study was implemented in the two networks.
RESULTS: Considerable time and money was required to adapt the paper-based study to the electronic environment, requiring extra meetings with institutional EHR-, regulatory-, and administrative teams, and increased practice training. Institutional unfamiliarity with using EHRs in practice-based research, and the consequent need to develop new policies, were major contributors to delays. Adapting intervention tools to the EHR and minimizing practice disruptions was challenging, but resulted in several efficiencies as compared with a paper-based project. In particular, recruitment and tracking of subjects and data collection were easier and more efficient.
CONCLUSIONS: Practice-based intervention research in an electronic environment adds considerable cost and time at the outset of a study, especially for centers unfamiliar with such research. Efficiencies generated have the potential of easing the work of study enrollment, subject tracking, and data collection.

Introduction

Over the past several years, the implementation of electronic health records (EHRs) has dominated the many changes to outpatient primary care practice, and has resulted in wholesale changes to office workflow as the use of paper for most office functions is phased out.1–3 There is great interest in learning how best to integrate the capabilities of EHRs to make care more effective and efficient. At the same time, practice-based research related to quality improvement and primary care, a priority for some federal funding agencies,4,5 has also evolved quickly. In addition to the practice environment’s “lab” changing from being paper based to being electronic based, study complexity has grown, moving away from predominantly descriptive and observational techniques to those involving interventions to improve care.6–9

While data from practice-based research using EHRs is emerging, knowledge of how to make interventional research feasible for busy practices in this environment is limited to relatively few studies involving clinical alerts and other decision support interventions to facilitate subject recruitment.10–12 While some new knowledge may not be generalizable among different EHR systems, there are some common methodological considerations that could inform the planning and conduct of studies. Some of these may present barriers to research, where others may facilitate the conduct of studies and may even make them more efficient.13,14

In addition to the pragmatics of conducting practice-based research in this new environment, additional administrative and regulatory challenges exist that were not present prior to the advent of EHR use. For example, regulation of research involving EHRs involves potentially conflicting rules regarding patient privacy, data ownership, and data sharing, often necessitating extra agreements such as waivers of the Health Insurance Portability and Accountability Act of 1996 (HIPAA) authorization before data collection begins.12,15,16 Knowledge of likely regulatory and administrative challenges can facilitate study planning and resource allocation.

Although techniques are being developed for practice-based research that has minimal impact on office workflow,6 some observational and most interventional research remain highly dependent on staff or practitioners being involved in data collection and intervention delivery. Unfortunately, demands for increased efficiency in office practice dictated by such factors as payment pressures and increased patient complexity have reduced time for practitioner involvement in research.17 Combined, these trends pose the question: What are the challenges and efficiencies related to implementing an effective, minimally demanding practice-based intervention in a fully electronic environment, as compared with the environment in years past?

The P3RC (Parent-Provider Partnerships in Referral Communication) study (ClinicalTrials.gov #NCT01797497) is a cluster-randomized trial, with randomization by practice using a stepped-wedge design.7 The intervention uses a structured tool (form)18 to enhance communication about subspecialty referrals from primary care providers (PCPs) to subspecialists and subspecialists back to PCPs, with purposeful input from parents and providers, along with brief parent coaching about its use delivered by study or practice staff. The hypothesis is that use of tools to coordinate referrals between physicians and parents will facilitate increased communication between PCPs and specialists, and that parents trained in information exchange and care planning will experience increased self-efficacy in interacting with their child’s physicians. P3RC was originally designed and pilot tested in 2008 and 2009 as a practice-based intervention study in pediatric primary care and outpatient specialty settings that used a paper-based record system. By the time of the project’s launch in 2012, it was necessary to adapt the intervention to the paperless (fully electronic) record environment.

The study protocol (Figure 1) involves the identification of potentially eligible patients and parents by PCPs when the decision to refer is made. The study team contacts the parent via phone to provide orientation regarding the study and to obtain informed consent. For intervention group patients, the communication tool is used by PCPs, subspecialists, and parents. A coaching session for parents about how to use the tool is delivered by PCP practice staff or study staff. Data are then collected from parents and providers through chart review and brief questionnaires. This moderately complex study has the potential to have an impact on practice workflow at several points. The presence of EHR-dictated workflow, as well as the amount of shared EHR data between specialty and primary care, influences each step in the protocol, posing both challenges and opportunities.

Figure 1.

Intervention and Data Collection Protocol for P3RC Referral Communication Study

In this paper, we describe the study’s development for use in the electronic environment, comparing its methods of implementation between paper-based and EHR systems. Specifically, we report lessons learned about converting the paper-based tool to the electronic format; subject recruitment, tracking, and data collection; and administrative and regulatory challenges. We hope to assist other investigators in the planning and development of subsequent practice-based research studies in EHR-based environments.

Environment and Description of Implementation

Setting

The P3RC study is being conducted in two community pediatric-practice-based research networks (PBRNs) in Colorado and Oklahoma and at the children’s referral hospital at which most pediatric subspecialty care in each area takes place. The Colorado Child Outcomes Network (COCONet) and the Oklahoma Child Health Research Network (OCHRN) are Agency for Healthcare Research and Quality (AHRQ)-registered established primary care PBRNs that work extensively with the major pediatric referral center in each state. Participating Colorado primary care practices all share a fully functional EHR (Epic) with their referral center, and some Oklahoma practices share an EHR (Citrix Centricity) with their referral hospital while other Oklahoma practices use different EHRs from various vendors. Despite many shared resources, referral processes including communication have significant room for improvement.

Methods of Implementation: Paper Versus Electronic Systems

As we adapted the P3RC study to the electronic environment, we considered a variety of questions and issues related to each step in the study process. The steps examined included study subject identification, enrollment, and tracking; adaptation of the communication tool to the electronic environment; and data collection techniques. For each step, we considered allocation of resources needed for integration of the study into the electronic workflow at each practice, efficiencies or extra work by practitioners and study staff created as a result of adaptation, and administrative oversight and approvals. We asked the primary question: “What makes this study different from how it would be conducted in a traditional, paper-based system?” In Table 1, aspects of the study using older paper-based methods are compared with those using a fully electronic system, with key learning points and consequences of changes described.

Table 1.

Factors Considered in Adapting the Study to the Electronic Environment

Paper System	Electronic System	Factors
Subject recruitment, identification, tracking Identification by providers using exam room reminders Paper log kept by practices of patients eligible/contacted/refused	Point-of-care reminder pops up in EHR Automated reports at some sites based on referral order data and reminder responses from EHR	It took months and considerable resources to build into the EHR. Practitioner training needed Approval through oversight committee required many months. Data ownership issues, need for extra data sharing agreements
Intervention tool design and delivery Paper communication form (tool) used by practitioners and parents Copy or fax form to communicate with parents and other providers	Electronic template and tool for communication Automatic sending of tool for shared EHR sites; still need copy or fax for disparate EHR	It took months and considerable resources to build into the EHR. Different EHRs use different workflow. Provider workflow was more complicated. Time savings, less lost paperwork
Data collection Questionnaires sent by paper or email to providers Manual chart review at multiple sites by study staff	Secure web-based or email options based on provider preference Remote, web-based access to charts at many sites	Less opportunity for lost or misplaced data Credentialing and data access agreements for study staff Similar format for electronic charts between sites increases data collection efficiency. Decreased time and travel costs

During the first year of the study, we completed development and pilot testing of study procedures in the practice environment, holding three to five meetings with all involved practitioners including at least one meeting on-site at each practice. We also worked closely with research oversight and data management personnel at each site, through at least one in-person meeting followed by subsequent emails, to arrange for programming within the EHR that might facilitate study procedures and to obtain needed authorization for study procedures and data collection. In addition to documenting the extra time and resources needed to adapt the study to the electronic environment, we measured the completeness and accuracy of EHR-generated enrollment reports, the usability of the intervention by patients and practitioners, and the completeness of data collection for enrolled patients.

Findings from Adaptation Process: Pros and Cons of Implementation in the EHR Environment

We encountered many factors that posed challenges to adapting the study to the EHR environment, as well as several factors with the potential to streamline study processes for practitioners and study personnel. The most significant challenges involved decreased study team control over data sources and some study processes, increased time needed for administrative oversight personnel to develop familiarity with practice-based research involving the EHR, and a lack of existing regulatory procedures and protocols to govern some aspects of the research being performed. These challenges resulted in long delays beyond the control of study personnel, as well as potential problems with recruitment. Most potential facilitators involved efficiencies afforded by the EHR in data collection and reporting, potentially saving time and effort by study personnel.

The additional time commitment for meetings with administrative staff that was required to adapt the study to the EHR environment was substantial, and was the biggest challenge for the study team. Initial meetings with practitioners and practice managers were comparable in length and scope to those in the paper environment. However, at both sites (Colorado and Oklahoma), at least three to five meetings with the referral hospitals’ administrative and EHR teams were required once the study was approved by the local Institutional Review Boards (IRBs). At the Colorado site, administrative study approval required extra meetings to familiarize oversight personnel with proposed methods of using the EHR and data from practices, followed by establishment of new data sharing contracts such as Research Associate Agreements (RAAs). The EHR was shared between the hospital and PCP practices, but practice data were owned by the practices that were independent of the hospital. This situation required a lengthy process to reach consensus surrounding data sharing and privacy rules. EHR team meetings included discussion of converting the referral tool template to electronic format, creation of an electronic reminder at the point of care to notify practitioners of eligible patients, and development of reports to track eligible patients and provide practice-level feedback. Each meeting was followed by waits of several weeks to three months for approvals and work by the administrative and EHR teams. The process took approximately one year from initial meeting to full implementation in practices. Two Oklahoma primary care practices also had a different EHR than the referral center. In these practices, there was little ability to modify the EHRs, resulting in extra meetings and use of a hybrid paper and electronic system. Finally, training of practices and providers regarding study procedures at all sites took about twice as long (three to four hours per practice, over two to three sessions) as it took with fully paper-based practices in previous pilot testing.

Subject Recruitment, Identification, and Tracking

As study procedures relied on PCPs to identify potentially eligible children and ask permission to contact the children’s parents, and as PCPs often missed eligible patients during pilot testing, we sought automated methods to identify potentially eligible subjects and to track them once enrolled. As was the case with systems used in other studies,13,14 at the Colorado site we were able to identify potentially eligible participants using the EHR when a referral was ordered by the PCP. The referral order, in turn, generated a real-time provider reminder in the EHR.

Practitioners found this system useful; several mentioned that without it they would not be likely to remember to ask parents about enrolling in the study. While automated prompts were not possible at the Oklahoma site, manual signs on computer terminals and wall signs were used as a provider prompt. Tracking potentially eligible patients and keeping logs of those enrolled was also a challenge, and one that practitioners were reluctant to undertake. Practices requested that the study team develop methods for logging and tracking eligible patients, as well as provide monthly feedback on how many of their patients were eligible. After data sharing agreements were completed, these tracking and feedback functions were made possible. Reports were successful in logging and tracking 100 percent of enrolled subjects, and practitioners felt that their own reports had added value to remind them about the study.

Intervention Tool Design, Development, and Delivery

Both sites were able to integrate the referral tool into their referral centers’ EHRs. This eliminated the cumbersome process of printing and faxing the completed tool by those practices that shared an EHR with the referral center, and it minimized lost documentation. One center was also able to integrate it into parent instructions, which satisfied both the intervention protocol and practice requirements for Meaningful Use of EHRs. For the sites that did not share an EHR with the referral center, the tool was completed on paper with parents, e-faxed to the referral center, then scanned into the center’s EHR.

Adapting the paper referral tool18 to the electronic environment required creativity to fit the tool with existing referral workflows. In a paper system, completing the tool required extra writing, though it was straightforward. In the electronic system, use of the tool had to be flexible enough not to disrupt practitioners’ workflows, including minimizing extra typing and automatically populating parts of the tool with existing information such as medications and allergies. Several practitioners admitted that most of their EHR use occurred after patients had left the office, and that using the EHR to generate referrals while simultaneously providing patient care was difficult, so the protocol had to be revised to allow for flexibility in the timing of tool completion. Unfortunately, due to technological capabilities in the EHR, neither site was able to preserve the attractive format of the paper version. However, one site was able to convert the tool into an easy-to-use, fillable electronic version (Figure 2), while the other was able to convert it into a template that fit within existing workflows of referral ordering and consultation reporting.

Figure 2.

Fillable Electronic Referral Tool Embedded in the EHR

Adapting the intervention and data collection to the EHR also required a subcontract to pay programmers and analysts from the Colorado site’s Information Services department. Adaptation of the referral tool to the EHR, development of eligibility and enrollment reminders for providers, and data reporting related to eligibility and enrollment were included in this subcontract. Finally, as the intervention targeted both English- and Spanish-speaking families, small modifications of the study workflow were needed to accommodate Spanish speakers in an English-based EHR environment.

Data Collection

Data collection involved three main activities: (1) logging and tracking of subjects (described above); (2) chart review; and (3) surveys of parents, PCPs, and specialists. Data collected through EHR chart review were able to be accessed remotely in most cases by study personnel. This eliminated the need for on-site record review at practices, and reduced data collection time requirements substantially. Based on feedback from practitioners, options for collecting survey data were expanded. Many practitioners preferred secure web-based questionnaires or email to the older method of paper questionnaires. And, after about two-thirds of the patients had completed the study, complete data from PCPs, specialists, and parents were available for 94 percent, as compared with about 80 percent in a similarly designed observational study using a paper-based system.19

Discussion

Adaptation of our referral study protocol to the electronic environment was more complex than we initially anticipated. We found that, while the environment enhanced the study in some respects by saving time and resources related to data collection and potentially facilitating recruitment, the electronic environment also required activities that cost significant amounts of time, effort, and money—especially at the start of the study. Several of these activities were likely part of the learning curve for both the study team and the research administrative and regulatory personnel, and could be seen as investments in time and effort that will be reduced in future studies.

Resource and time commitment related to information technology (IT) features and their administrative approval was the aspect of the study that made the electronic environment most different from the paper environment. Though meetings were held far in advance of the study’s launch to anticipate any problems, the money and time needed for programming and meetings with IT personnel both before and after the launch was greater than anticipated, and the time and effort to overcome challenges related to research oversight was much greater than expected. Research oversight personnel mentioned that many procedures were new to them, and that oversight protocols and agreements newly drafted for the study will make the way smoother for similar projects in the future. Investigators contemplating research projects using EHR innovations should assess well in advance of the project’s start date the degree of experience that their institutions’ research oversight groups, applicable IRBs, and related personnel (e.g., compliance) have with such projects. Specifically, they should explore with their institutions the collection of EHR-generated data for recruitment and nonresponse rates, the creation of forms and other innovations for research purposes within the EHR, and the use of EHR data from practices that share an EHR or related resources with the institution. They should share the research protocol well in advance of study initiation, and may well need to allocate substantial extra time and planning prior to launching a study.

Innovations related to recruitment and subject tracking appeared to offer the biggest potential benefit for the study. Point-of-care reminders (in Epic, the “best practice alert”) to practitioners about potential eligibility of patients, and periodic practice-specific feedback about patients whose recruiting might have been missed, were features of the study that were unavailable in a paper-based environment. The effect of point-of-care reminders has been reported previously, and has been associated with increased recruitment rates,13 though recruitment feedback reporting is new. Both reminders and recruitment feedback have the potential to increase recruitment in studies that require practitioners to identify patients, especially in the context of a busy practice. The reporting function in particular was pinpointed by practices in our study as a potential incentive to identify eligible patients, as many described themselves as “naturally competitive” and would compare their performance to that of their peers.

Intervention design and delivery were enhanced by the electronic environment in some ways, but were made somewhat more difficult in others. While entry of clinical information into the referral tool was easy, fitting it into practice workflow (incorporating parent input at the time of the visit) and managing information once it was entered (printing a copy for parents) was sometimes confusing for practitioners, as it required using the EHR slightly differently than they otherwise would. The positive features of not having to manage paper forms and the negative features of having to manipulate electronic referral templates and use them for communication probably canceled one another out in terms of time and work for practitioners. At one site, the referral tool was duplicative of information in the EHR in some subspecialties but not others, highlighting the diversity of information provided to parents and referring providers. Such differences underscore the difficulty of introducing even small changes into practice work-flow in the course of a practice-based study.

Data collection for our study was enhanced by the electronic environment. Chart review benefited from having computerized access. For example, travel time to practices sharing an EHR with the referral center was eliminated, and inefficiencies and data loss generated by missing charts and disparate methods of data organization between practices were minimized. These benefits will likely result in some recovery of the time and costs associated with programming and electronic report generation described earlier. Finally, preliminary data suggest that collection of practitioner questionnaire data by computer may result in an increase in the proportion of subjects for whom data are complete, as compared with traditional paper-based methods.

Limitations

The most significant limitation to our study is that investigators’ experiences at other institutions may be different from our own, given the complex and rapidly changing EHR environments in community practices and affiliated referral centers and the evolution of institutional policies and practices regarding research using EHRs. To our knowledge, there are no national standards for research oversight in this area, as there are for IRBs. When standards emerge, administrative challenges related to practice-based research using EHRs may become more streamlined. Nonetheless, this current snapshot of the opportunities and challenges involved in adapting our intervention study to the electronic health care delivery environment may help other investigators in planning and facilitating practice-based research as it evolves. As we have no comparison group with which to assess the objective results of the innovations implemented in subject logging and data collection, and the clinical trial is ongoing as of the writing of this paper, the impact of the potential facilitators is mainly subjective. However, it is difficult to envision that automating some aspects of these processes would not save time and effort on the part of practitioners and study staff.

Conclusion

Adapting a paper-based, practice-based research intervention for use in electronic practice environments is time-consuming but possible. Facilitating factors generated by elimination of traditional paper-based workflows are counterbalanced by increased administrative oversight and IT resources needed to operationalize the study. Some of these extra burdens may diminish with time and experience on the part of researchers, oversight, and IT personnel. Investigators should consider potential study enhancements and challenges of using the EHR for data collection, such as those identified in the P3RC study, as well as related administrative and regulatory requirements, well in advance as part of study preparations.