A thematic analysis to examine the feasibility of EHR-based clinical decision support for implementing Choosing Wisely ® guidelines.

OBJECTIVE: To identify important barriers and facilitators relating to the feasibility of implementing clinical practice guidelines (CPGs) as clinical decision support (CDS).
MATERIALS AND METHODS: We conducted a qualitative, thematic analysis of interviews from seven interviews with dyads (one clinical expert and one systems analyst) who discussed the feasibility of implementing 10 Choosing Wisely® guidelines at their institutions. We conducted a content analysis to extract salient themes describing facilitators, challenges, and other feasibility considerations regarding implementing CPGs as CDS.
RESULTS: We identified five themes: concern about data quality impacts implementation planning; the availability of data in a computable format is a primary factor for implementation feasibility; customized strategies are needed to mitigate uncertainty and ambiguity when translating CPGs to an electronic health record-based tool; misalignment of expected CDS with pre-existing clinical workflows impact implementation; and individual level factors of end-users must be considered when selecting and implementing CDS tools. DISCUSSION: The themes reveal several considerations for CPG as CDS implementations regarding data quality, knowledge representation, and sociotechnical issues. Guideline authors should be aware that using CDS to implement CPGs is becoming increasingly popular and should consider providing clear guidelines to aid implementation. The complex nature of CPG as CDS implementation necessitates a unified effort to overcome these challenges.
CONCLUSION: Our analysis highlights the importance of cooperation and co-development of standards, strategies, and infrastructure to address the difficulties of implementing CPGs as CDS. The complex interactions between the concepts revealed in the interviews necessitates the need that such work should not be conducted in silos. We also implore that implementers disseminate their experiences.

LAY SUMMARY

Clinical practice guidelines (CPGs) are recommendations for clinical practice, often authored by well-known experts in the field. These recommendations usually address common misconceptions or situations that clinicians face frequently in their day-to-day practice. Because of their utility, health systems are interested in making sure these guidelines are easily accessible and followed when appropriate. An increasingly popular way to deliver this information is through the electronic health record (EHR). Specifically, clinical decision support (CDS) within the EHR is a way to automatically deliver the content of the CPGs to clinicians during their daily workflow. CDS comes in many formats, but often presents in the form of a pop-up alert after certain conditions are met. Unfortunately, translating CPGs into a CDS format can be difficult for several reasons. To advance our knowledge on these issues, we interviewed physician and CDS implementers in pairs so that they could discuss strategies to implement 10 different CPGs. Our participants identified several important considerations when implementing CPGs as CDS, including: addressing issues data quality, changing the way CPGs are authored so that they provide specific guidance when they are implemented, and planning implementations so they do not contradict the way clinicians are used to working.

INTRODUCTION

Clinical practice guidelines (CPGs) have been used for decades to support evidenced-based practice and formalize clinical processes. CPGs are often written and disseminated by professional societies for the purpose of educating clinicians on the most up-to-date standards of practice in their field. While such guidelines may be read and interpreted by clinicians on a case-by-case basis, a more effective and standardized approach for ensuring that CPGs are used in practice and followed as intended is to use clinical decision support (CDS) as the mechanism of delivery. The use of CDS to facilitate CPGs in health systems has many practical benefits, such as overcoming issues with inaccessibility of knowledge at the point-of-care. In addition, the use of CDS improves clinician adherence to a prescribed workflow (in this case, the directions of the guideline), which often shows an improvement in patient outcomes. While the use of EHR-based CDS is an effective approach for implementing CPGs, ongoing implementation challenges inhibit the wide dissemination of CPG-based CDS.

Translating CPGs into sharable, scalable, computable, and actionable CDS requires significant effort if they are not written in a way that effectively leads to CDS implementation. The extra effort required to translate CPGs to CDS may significantly limit their implementation. Moreover, there are no standard approaches to quantify the effort nor characterize the feasibility of implementing CDS. Ad-hoc methods of CDS development along with a paucity of feasibility assessment tools leaves a gap in our understanding of how best to approach CPG-as-CDS implementations. In previous work, we attempted to address this gap by developing a quantitative model to assess implementation feasibility and effort of CPGs as CDS. From this previous work, we discovered that guidelines containing vague concepts and requiring additional data collection from the end-user are significant barriers to implementation feasibility.

As part of our methods to develop the feasibility model, we conducted mixed-methods interviews with clinical experts and system analysts to ascertain quantitative implementation feasibility ratings for 10 widely accepted guidelines from Choosing Wisely (CW). These interviews were conducted in dyads to enable the clinical expert and the system analyst at each site to discuss and form a feasibility rating based on consensus with both technical and end-user-based considerations. The quantitative methods and resulting model are discussed in another manuscript. Here, we report the findings of the qualitative analysis of the interviews conducted with the dyads regarding their experiences and views of CPG implementation.

The objective of our analysis is to identify important barriers and facilitators relating to the feasibility of implementing CPGs as CDS. Although many of these issues are well-known challenges for CDS implementation (given the decades of research in this area), the use of clinical experts and system expert dyads to explore them more deeply in the context of a specific set of CPGs reveals valuable details and examples of considerations to assess the feasibility of implementing CPGs as CDS. These findings can inform organizations that want to adopt guideline driven CDS about the feasibility and challenges.

METHODS AND MATERIALS

Guideline selection

CW guidelines are widely accepted by providers and cover a breadth of topics. The CW initiative has been adopted by 77 professional societies who agree to identify common practices that are not evidence-based, and provide recommendations designed to reduce those practices. These guidelines are the subject of frequent CDS implementation due to their positive reputation and timely subject matter,,, making them an excellent sample to select from. For this study, we selected 10 guidelines. Among the 10 American College of Emergency Physicians (ACEP) CW recommendations, we selected nine that were evidence-based recommendations of procedures to question or avoid for certain types of patients in the emergency department (ED). One recommendation was not used, as it represents a value and policy statement rather than an action, and was replaced with a recommendation developed by the American College of Radiology (imaging approach for suspected appendicitis) relevant to ED practice. Table 1 lists the included recommendations.

Table 1.

CW recommendations selected for discussion

Guideline number	Recommendation
ACEP #1a	Avoid CT scans of the head in emergency department patients with minor head injury who are at low risk based on validated decision rules.
ACEP #2a	Avoid placing indwelling urinary catheters in the emergency department for either urine output monitoring in stable patients who can void, or for patient or staff convenience.
ACEP #4a	Avoid antibiotics and wound cultures in emergency department patients with uncomplicated skin and soft tissue abscesses after successful incision and drainage and with adequate medical follow-up.
ACEP #5a	Avoid instituting intravenous fluids before doing a trial of oral rehydration therapy in uncomplicated emergency department cases of mild to moderate dehydration in children
ACEP #6a	Avoid CT of the head in asymptomatic adult patients in the emergency department with syncope, insignificant trauma, and a normal neurological evaluation.
ACEP #7a	Avoid CT pulmonary angiography in emergency department patients with a low pretest probability of pulmonary embolism and either a negative Pulmonary Embolism Rule-Out Criteria or a negative D-dimer.
ACEP #8a	Avoid lumbar spine imaging in the emergency department for adults with nontraumatic back pain unless the patient has severe or progressive neurologic deficits or is suspected of having a serious underlying condition (such as vertebral infection, cauda equina syndrome, or cancer with bony metastasis).
ACEP #9a	Avoid prescribing antibiotics in the emergency department for uncomplicated sinusitis.
ACEP #10a	Avoid ordering CT of the abdomen and pelvis in young otherwise healthy emergency department patients (<50 years of age) with known histories of kidney stones, or ureterolithiasis, presenting with symptoms consistent with uncomplicated renal colic.
ACR #1b	Don’t do CT for the evaluation of suspected appendicitis in children until after ultrasound has been considered as an option.

ACEP: American College of Emergency Physicians; ACR: American College of Radiology; CT: computed tomography; CW: Choosing Wisely.

Source: http://www.choosingwisely.org/societies/american-college-%20of-emergency-physicians/.

Source: http://www.choosingwisely.org/clinician-lists/american-college-radiology-ct-to-evaluate-appendicitis-in-children/.

Study participants

We interviewed seven dyads consisting of a clinical expert and a system analyst from seven different academic medical centers in the United States. Clinical experts were physicians with experience in either emergency medicine or urgent care settings. System analysts were required to be currently employed in their role and use CDS tools within their EHR system, preferably assigned to ED projects. The pairing of clinical experts and system analysts allowed for a more robust determination of implementation feasibility for the organization, as both technical and clinical deliberations could be considered. The 14 participants from seven sites were recruited as a convenience sample, including those with affiliations or professional contact with the authors.

Study procedures

Prior to the interviews, the 10 guidelines were first transformed into a semi-structured logic format guided by methodology for transitioning CPGs into CDS by Shiffman et al and Tso et al This included the following steps:

Atomize: Extract and refine discrete concepts from narrative recommendations.

Deabstract: Adjust the level of generality for decision variables or actions to enable operationalization.

Disambiguate: Establish a single semantic interpretation for a recommendation statement.

As new concepts arose from the disambiguation step, those concepts were subsequently atomized, deabstracted, and disambiguated as needed. This iterative process was completed until all the concepts were clearly defined and operational. Our previous manuscript where we developed a quantitative model to assess implementation feasibility and effort of CPGs as CDS explains this process in more granular detail.

One researcher (R.L.R.) conducted the interviews between May and August 2018. Each interview lasted 1–1.5 h, guided by a script and slides showing the participants semi-structured logic for 10 CW guidelines.

The primary goal of the interviews was for the dyads to rate each of the 10 CW guidelines for implementation feasibility and data availability. Using open-ended questions, R.L.R. solicited ratings and facilitated discussion, prompting the dyad to discuss the guidelines or elaborate on ambiguous or unclear statements (see Supplementary materials for script and rating scales). Other members of the research team (B.J.D., C.J.S., and G.D.F.) could ask clarifying questions at the end of each interview. Participants had access to their EHR during the interview to make decisions about feasibility ratings prior the end of the interview. All interviews were conducted using WebEx (Cisco, San Jose, CA, United States). Interviews were professionally transcribed and subsequently verified for accuracy by B.J.D. The study was classified as exempt from human subject research by the Duke University Institutional Review Board.

Guiding principles for analysis

This study was conducted to reveal important barriers and facilitators regarding CPG implementation as CDS, according to the perceptions of the clinical experts and systems analysts. Therefore, we elected to use an inductive approach to code the data. To guide this process, we continually referred to our research question: “What are the feasibility considerations of implementing CPGs as EHR-based CDS?”

Data preparation and analysis

Following a standard content analysis methodology outlined by Krippendorff, the study team used the following steps to analyze transcripts:

Each transcript was unitized by B.J.D., ensuring all coders were analyzing the same unit of text and intercoder reliability could be analyzed.

To develop an initial set of codes, inductive (in vivo) coding was performed on one “archetype” transcript. The authors worked in pairs to read, discuss, and develop initial codes for half of the transcript (G.D.F./R.L.R. coded the first half, and B.J.D./C.J.S. coded the second half). Then, the full study team met to discuss findings and the codes discovered in each half of the manuscript. A preliminary codebook was developed from these conversations.

To validate the preliminary codebook, the pairs analyzed the opposite half of the architype transcript using the preliminary codebook to assess the text, noting gaps and desired changes. The full study team met again, modifying the codebook based on group consensus. An initial codebook resulting from this conversation was used moving forward.

Using the initial codebook, the following steps occurred to iteratively enhance the codes and identify representative text.

The next two transcripts were coded in the following manner:

One pair would code an entire transcript; first individually, then the pair would meet to resolve discrepancies. Any desired changes to the codebook were discussed at the weekly team meeting and changes were made based on group consensus.

The other pair would then analyze the same transcript using the updated codebook in a similar manner; individual coding, pair discussion, then group consensus.

The remaining transcripts were each coded by a single pair, bringing discrepancies to the group to be resolved weekly.

We employed iterative, code-recode methods and group consensus to increase our rigor and to ensure thematic saturation and dependability of our results. Coding transcripts was managed using Excel (Microsoft Corporation, Redmond, WA, United States).

Following the completion of coding, the authors met weekly to review each code for consistency, and to begin categorizing codes by families. A dynamic mind map was developed using Lucidchart (Lucid Software Inc., South Jordan, UT, United States) to allow the authors to further organize codes, form code families, and identify relationships between codes. Code families were then grouped to reveal and develop salient themes, which were then classified under the domains of data, knowledge, and sociotechnical to allow for a more organized discussion. These domains emerged through inductive analysis, with no assumptions made a priori.

RESULTS

The 14 participants completed seven dyad interviews and are described in Table 2. Cohen’s kappa was used to calculate intercoder reliability, resulting in an average kappa of 0.41, or moderate agreement with an average increase of 0.16 from the first to second round of coding. Five themes emerged from the analysis, grouped according to three domains: Data, Knowledge, and Sociotechnical. The themes along with supporting quotes and the participant’s role (systems analyst [SA] or clinical expert [CE]) are provided in the following sections.

Table 2.

Description of study participants

	Systems analysts (n = 7)	Clinical experts (n = 7)
Years of experiencea Mean (min, max)	10.3 (5, 20)	16 (5, 35)
Years of EHR experience Mean (min, max)	6.9 (4, 11)	6.8 (2, 12)
Level of training	MS, certificate	MD
Job title	Application Systems Analyst (ASAP), Application Analyst II, RN ASAP Analyst, Sr. Application Analyst, Application Developer, Assoc Med Director for IT Services	Professor of Emergency Medicine, Sr. Physician IT exec, Assistant Professor, Family Medicine Medical Director, ED Medical Director, Attending Physician

For System Analysts, this refers to years of experience supporting CDS implementations. For Clinicians, this refers to years practicing as a physician.

Dynamic attributes of CPG implementation as CDS

The interviews resulted in a wide variety of discussions regarding implementation feasibility, crossing many topics and informatics considerations. To achieve the goal of identifying salient themes, we first examined each code, grouping them into code families, which then were grouped together to reveal five themes within the three domains; data, knowledge, and sociotechnical. This process was iterative and lengthy, as our data showed sophisticated relationships between each code family, theme, and domain. These interactions are paramount to the complexity of implementing CPGs as CDS, furthering a need to consider all the data together, rather than in siloed categories. To better represent these relationships and serve as introduction prior to presenting the key findings, Figure 1 displays the relationships between code families, themes, and domains.

Figure 1.

Interaction of domains with associated themes and code families.

Data domain

In the interviews, participants frequently discussed considerations and concerns regarding their institution’s underlying data, and whether it could support CDS based on the CPGs adequately. Clinical experts would often ask data analysts to look up certain data elements or value sets to determine how much additional work would be needed to facilitate a successful implementation. These conversations often took place prior to discussion of the other two domains, most likely due to the flow of the interviews and use of a script to guide the discussions for the quantitative ratings.

Theme 1: Concern about data quality impacts implementation planning

Participants often discussed data quality concerns and how they relate to the feasibility of CDS implementation. Participants shared concerns about distrust with the accuracy of certain data elements and uncertainty if data would be available at the time it would be needed. Specifically, distrust with data quality manifested in three ways: (a) concern with end-user data entry accuracy, (b) uncertainty about information availability from external EHRs, and (c) potential inability to link to existing patient records when CDS is triggered (i.e. unidentifiable trauma patients).

These concerns about data quality often led participants to consider adding additional data collection within the CDS tool to ensure proper function, even at the cost and provider burden of recollecting some data. Some participants noted that even with additional data collection, concerns would still arise with quality. Other design considerations and modifications to the guideline’s underlying logic were considered to accommodate for unavailable data.

I think I would want to add the data collection to the practitioners’ work at the time of ordering rather than trusting problem lists because of all the issues with problem lists. [SA]

You could always take that out of the decision support logic… but I would hate to be firing that off when it’s really not appropriate because your patient would not have met the criteria to get it and then we would just annoy you. [CE]

Coming from more of a population health type of thing, like I think that would be worth it… I think this BPA* actually may be helpful, but it’s a lot of work [due to poor data quality]. [CE]

*BPA = Best Practice Advisory; Epic EHR nomenclature for their pop-up alert format

Theme 2: the availability of data in a computable format is a primary factor for implementation feasibility

Participants also discussed the impact of data format and data availability on feasibility. Similar to data quality, experts suggested that if the data could not be found in a structured format, the CDS would not work and would not be feasible without additional data collection.

Unless we really say that we’re going to try and capture these data in a structured fashion in order to assess whether somebody has moderate dehydration, then it’s [too difficult to implement]. [SA]

Well, this is similar to the other imaging rules in that we don’t have structured data… for at a time when somebody would be placing this order. So, we basically have to fire an alert for every order for a CT of the abdomen and pelvis asking you for an indication, and then asking you some additional questions to verify that it was appropriate. [SA]

We also noted that participants often discussed a persistent misalignment between concepts referenced in guidelines and available data. To address these issues, the experts explored several solutions including the use of proxy data and asking for additional data from the end-user. Notably, if the CDS addressed a clinical issue that was a high priority for the organization, experts were more likely to allow the CDS to rely on poor-quality data if no other options were readily available.

I actually like the spirit of this BPA though. But that one seems hard, worsening course, like how do you document that as a data element? [CE]

Our EMR would not have the ability in and of itself to identify and capture [the concepts of persistent illness, severe onset, and worsening course] currently just based on the normal workflows [SA]

Knowledge domain

Participants noted that even validated and well-known CPGs found in reputable sources such as CW contain a large amount of ambiguity which cannot translate effectively to a CDS tool. These issues manifested either as complete ambiguity and an inability to execute the guideline without making assumptions, or a need for significant effort to create value sets to operationalize concepts.

Theme 3: Customized strategies are needed to mitigate uncertainty and ambiguity when translating CPGs to an EHR-based CDS tool

For every guideline, our experts identified one or more ambiguous or unclear concepts that were difficult to operationalize in a computable way. Some concepts were noted to be more important than others for ensuring that the automated guidelines would function as intended and with adequate specificity. Despite decades of research in knowledge representation formalisms for computable guidelines, CPGs are still being authored and published in a narrative format, imposing a critical implementation challenge for CDS.

… Is it recent trauma, is it trauma two weeks ago when you had your car accident and never got seen? So, I think this … guideline itself suffers from lack of specificity. We don’t know …which one they’re trying to address and the overall criteria in [the guidelines] are also vague, so … all require clinician input. [CE]

I think that it would be much more difficult and time consuming if we [included] the ‘otherwise healthy’ concept… There’s two different ways to approach this… would we capture ‘otherwise healthy’ as [a new] question, or do you think it makes sense to have that automatically computed? [SA]

There’s no way to define the patient population as ‘minor head injury’ before you get to the place of ordering the test, […] integrating that into the rule would be a significant challenge. [SA]

Sociotechnical domain

For most of the 10 guidelines, participants identified multiple approaches to implementing the guideline based upon workflows, available data, and system features. The clinical and informatics experts had different perspectives on feasibility and suggested approaches to overcome different challenges. In addition, the participants would often ask to clarify what we meant by “feasibility”. Although we often coached them to focus on technical issues which related more closely to our quantitative questions, they repeatedly identified sociotechnical issues such as provider compliance and provider support of the CDS tool. Even if the CDS tools are derived from evidence-based guidelines, they may not reflect a realistic clinical workflow and clinicians may be opposed to using the tool, which may manifest as alert fatigue and erroneous data entry to override an alert.

Theme 4: Misalignment of expected CDS with pre-existing clinical workflows impact implementation

A pervasive point of discussion in the interviews involved mismatches between workflow and expected functionality or outcomes of the CDS tools. One of the common issues in this mismatch concerned clinical and organizational practices that did not align with the recommendations of the guideline. This manifested as conflicts between (a) clinical autonomy, (b) the rigidity of CDS through an EHR, (c) each health system’s individual infrastructure, and (d) hospital executive strategy priorities.

From a workflow standpoint, this is not how this work. You might have ordered your antibiotics prior to that just on how things look like so, this one’s difficult from just a logistics workflow standpoint… the latest literature suggests that the use of antibiotics actually improves outcomes after abscesses, so this would not be very well received anyway. [CE]

In our setting, this would be difficult, but I don’t think it’s technically difficult or infeasible for many organizations… It’s pretty simple rule delivery, but hard in our system. [SA]

But, I’m not even sure you need to build anything with this. It might be kind of wasted effort, but if there’s an overall organizational goal to just create a ton of BPAs then this seems very easy to do. [SA]

Theme 5: Individual level factors of end-users must be considered when selecting and implementing CDS tools

Participants noted that feasibility must consider both sociotechnical and technical feasibility. Discussions of what the interviewers meant by “feasibility” were brought up by interviewees on several occasions. Our original assumptions of this discussion were that feasibility would be defined by “technical” considerations, but the participants revealed several important subthemes of feasibility that extended into end-user behavior. First, participants noted challenges in technology acceptance, with concerns regarding clinicians creating workarounds being the foremost issue. Second, participants expressed potential challenges in change management regarding workflow change, especially with documentation practices. Lastly, because of the need to engineer new knowledge artifacts (such as order sets) to address undefined or “underspecified” concepts in CPGs, participants were worried there would be significant challenges in domain experts reaching consensus on a common definition or what to include. This could issue could also present as end-users refusing to use the CDS tool due to a disagreement with its presentation or resulting recommendations.

Accounting for provider willingness to do this is part of the feasibility. [CE]

Even if I built it in a way that you could capture discretely, I don’t think that we would get everyone to change how they document. [SA]

Are you just talking about technical implementation assuming that you have the [value set], or are you talking about you know, getting the people together to agree on the [value set]? [SA]

DISCUSSION

The results of our analysis have revealed five themes related to the feasibility of implementing CPGs as CDS; the variety of our participants gave way to interesting interpretations of established concepts. In addition, we note in Figure 1 that these domains do not exist independent of each other. Instead, complex interactions between these domains exist and contribute to the complexity of implementing CPGs as CDS.

Data

As is already well established in informatics literature, data are an extremely important consideration for CDS, especially regarding quality. The participants regularly expressed concerns about data quality, but not solely in consideration of the guideline requirements. The outcome of “feasibility” often required an available, reasonable process to continually assess data quality and the ability to assess end-user willingness to enter relevant, accurate data. The fundamental idea expressed was that if the implementers had doubts about the data quality (especially in terms of accuracy, missingness, and availability), then the data would need to be collected prospectively—even at the cost of duplicating previous work or existing data—to be sure that the CDS could function as intended. This is significant when considering the increased amount of time clinicians would need to spend at a computer, increasing the risk of alert fatigue and burnout. However, we noted an exception in our analysis: If a CPG addressed a priority clinical problem of the organization, participants reported they would be more likely to accept poor data quality and a lack of specificity in the CDS tool. This willingness to overlook data quality is potentially problematic, as the underlying assumption of any CPG is that the data used to make the decision is accurate. To mitigate some of these problems, we recommend that clinicians and implementers disseminate their experience with using CPGs as a knowledge base for CDS, giving specific examples of how they addressed problems in data quality and availability. Information available on such specific implementation experiences is sparse in the literature, and this could possibly be due to non-academic implementers and clinicians not prioritizing publication or sharing implementation approaches. In addition, proprietary CDS architectures impose important barriers on organizations interested in sharing CDS logic. Emerging CDS standards, such as CDS Hooks and Substitutable Medical Applications and Reusable Technologies (SMART) on Fast Healthcare Interoperability Resources (FHIR), as well as CDS sharing platforms, such as EHR app stores and cloud-based CDS services, are promising approaches to help reduce those barriers.

Knowledge

Knowledge representation is known to potentially be a barrier in implementation feasibility, especially if the implementer must expend significant effort in tasks such as operationalizing concepts and creating value sets. Participants clearly demonstrated this by noting that CPGs (even highly regarded ones from CW) contained ambiguous concepts that made it difficult to translate them into CDS. Ambiguity was identified as a lack of definition from the guideline; however, our analysis revealed that a significant amount of effort was willing to be spent on operationalizing those definitions depending on organizational goals and the outlook on implementation feasibility. Lack of definition is relatively straightforward, as CPG authors may provide operational definitions for each concept, including computable definitions and value sets. As for concepts that cannot be explicitly measured through the EHR, the question of how this should be implemented remains difficult. In the meantime, each health system must plan for this type of knowledge and create customized solutions to mitigate these issues in CPG translation. In the future, when specification of this type of detail is critical to proper implementation, CPG authors should provide more guidance on how to implement their guidelines in an EHR-driven workflow, possibly noting alternatives or proxies to concepts that are not measured or captured by the EHR. At a minimum, CPG authors should consider the prevalence of CPGs being implemented as CDS and author them in a way that supports the translation of knowledge into computable, actionable CDS in real-world workflows. This may include providing knowledge artifacts that are more computable than narrative guidance, such as value sets, operational definitions, and validated phenotypes depending on the goal of the CPG. Artifacts such as value sets could be published in national repositories such as the Value Set Authority Center (VSAC). In addition, organizations could provide knowledge artifacts using emerging CDS logic representation standards such as the HL7 Clinical Quality Language (CQL) and CPGonFHIR. Open access authoring tools and repositories, such as Agency for Healthcare Research and Quality’s (AHRQ) CDS Connect, are becoming available and could be leveraged to help CPG developers to author and disseminate CPG guidance in computable format.

Sociotechnical

Sociotechnical considerations span across people and technology, informing our understanding of workflow, communication, policies, and regulations. This emerging concept has been well explored in implementation science, requiring unique solutions to overcome local challenges and compliment organizational culture. In our analysis, we noted that the perceived importance of a guideline superseded the challenges expressed regarding underlying data. In other words, our participants expressed that although health systems may prioritize the implementation of CDS based on their business and organizational goals, clinicians adapt their workflow to what they view as important. If a CPG isn’t viewed as useful, this trickles down to changes in their behavior, notably regarding data input. Poor documentation can affect not only a single CDS alert, but several other processes. The rippling effect of implementations that do not consider sociotechnical aspects of CDS use can be significant and foster a culture of mistrust. Of note, our participants also noted that some CPGs may not be needed in their practice, but may be helpful for other institutions, especially if there are a lack of specialists available in the ED.

A one-size-fits-all approach to CPG authoring may not be effective; or at a minimum, each institution may need to adapt guidelines to their available expertise. One solution may be to identify differing levels of knowledge artifacts. The CDS Consortium led by Harvard proposed four levels of CDS knowledge representation: narrative, semi-structured, structured, and executable. CDS authors should strive for the third level when possible (structured), while the fourth level (executable) would be EHR-specific and customizable to the specific health system. To mitigate some further challenges, we also recommend that a learning health system approach to CPG implementation be used. This methodology will help mitigate the sociotechnical aspects we may overlook and facilitate implementers to disseminate their methods and experiences. One such aspect may be that smaller organizations may not have the resources to fully customize their EHR, further hindering integration of CPGs into provider’s workflow, underscoring the importance of data and functional standards.

Limitations

This study has four noteworthy limitations. First, all interviews were based on CW guidelines, namely those relevant to emergency settings. Thus, we cannot be sure if our results generalize to other guidelines. Second, our participants represented a wide range of large health systems and academic medical centers; the results might not apply to other less resourced settings, such as small practices and community health centers. Third, interviews were limited to the perceptions of the interviewees after briefly reviewing the CPGs. They would likely raise other challenges and insights if the guidelines were actually implemented as CDS. Fourth, all guidelines were presented to participants under the assumption that the guideline was to be implemented as CDS. There was no explicit opportunity for participants to argue otherwise.

CONCLUSION

Examining the feasibility of implementing CPGs as CDS is a complex, multifaceted task. Health systems should closely examine barriers and facilitators regarding data, knowledge, and sociotechnical aspects of translating CPGs into actionable CDS. In addition to the implications for CDS implementers, our data reveals a need for change in the way CPGs are written and disseminated. In the era of EHR-driven care, it is not enough to produce guidelines with only descriptive text; guidelines should be accompanied with operationalized definitions, value sets, and clear criteria to facilitate their uptake as CDS tools. A partnership between clinical professional societies and informatics and standards societies would accelerate the transition of new knowledge into clinical practice. As there is already a depth of relevant work in clinical informatics, data management, data standards, knowledge representation and implementation science, these communities must be encouraged to collaborate and address these implementation issues in unison.

Future work should expand to exploring more guidelines (including other guidelines outside of CW) with more diverse health institutions. The exploration of barriers and facilitators to implementation should not be limited to academic medical centers but should be explored in smaller or less specialized (non-academic) hospitals where such tools would be invaluable and help improve patient care and outcomes. Qualitative work is important in exploring the facets of implementation science, as these methods help to empower clinicians and frontline analysts by disseminating their viewpoints and revealing critical issues that may otherwise be overlooked.

FUNDING

This works was supported by the National Library of Medicine (NLM) grant number (R15 LM012335-01A1). B.J.D. is supported by the Robert Wood Johnson Foundation (RWJF) as a Future of Nursing Scholars. The views presented here are solely the responsibility of the authors and do not necessarily represent the official views of the NLM or the National Institutes of Health or the RWJF.

AUTHOR CONTRIBUTIONS

Interviews were conducted by R.L.R., with assistance by B.J.D., C.J.S., and G.D.F. All authors contributed toward coding and thematic analysis, with B.J.D. acting as moderator. B.J.D. conducted intercoder reliability calculations. B.J.D. drafted the manuscript and all authors contributed to the study concept and design, and revision of the manuscript.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.

CONFLICT OF INTEREST STATEMENT

None declared.

DATA AVAILABILITY STATEMENT

The data underlying this article cannot be shared publicly for the privacy of the individuals who participated in the interviews. The data will be shared on reasonable request to the corresponding author.

Click here for additional data file.