Emergency Department Clinician Perspectives on the Data Availability to Implement Clinical Decision Support Tools for Five Clinical Practice Guidelines.

Clinical practice guidelines (CPGs) often serve as the knowledge base for clinical decision support (CDS). While CPGs are rigorously created by medical professional societies, the concepts in each guideline may not be sufficient for translation into CDS applications. In addition, clinicians' perceptions of these concepts may differ greatly, affecting the implementation and impact of CDS within an organization. Five guidelines developed by the American College of Emergency Physicians were systematically explored, generating fifty-one unique clinical concepts. These concepts were presented to two nurses and two physicians, whom were asked to assess and comment on the capture of each clinical concept in the electronic health record (EHR) and the subsequent availability of the data for CDS. Nurses and physicians showed differing perceptions of data availability. These differing perceptions may influence an organizational approach to developing and implementing CDS, potentially informing our understanding of why CDS may not achieve the intended impact.

Introduction

The information seeking behaviors of clinicians perpetuate a need for a growing number of clinical tools[1, 2]. This need is evidenced by the popularity of computerized information repositories (such as UpToDate [http://www.uptodate.com]), the development of InfoButtons, and an emphasis on clinical decision support in both historical and recent contexts (CDS)[3-5]. While CDS is promoted as an essential strategy for improving the delivery and outcomes of care in the coming era, the development and implementation of CDS tools is still a significant challenge for most organizations[6-8]. In the face of these challenges, CDS remains as an effective tool to promote care plan adherence, promoting better patient outcomes[9-11].

For CDS to function, an underlying knowledge base is required to influence medical decisions[12]. Oftentimes clinical practice guidelines (CPGs) serve as the knowledge base, thereby instigating the need for their formal representation within a clinical decision support system (CDSS)[13]. CPGs are themselves viewed as a foundational way to standardize care and improve health, but conversion to CDS has been shown to be challenging due to several factors[14,15]. In particular, the CDSS requires a formal representation of important clinical concepts and the assessments and actions that they drive, as well as a linkage to data elements and patient data as collected in electronic health record (EHR) systems.

This study explores the linkage of patient data to the clinical concepts required for the CDS logic to function as intended, particularly the perceptions of clinician perspectives on data elements within CPGs. This phenomenon is important to understand, as it provides insight into potential barriers affecting development, education, and implementation of CPGs within a CDSS. One portion of data availability seeks to comprehend the cohesion of clinical documentation practices across disciplines, and identify the differences in the perception of available data by the various clinical professions in a formal care setting. Measures for organizational success of CDS often reflect usage and adherence, but the failure of CDS should be more closely considered[10]. The quality of EHR documentation has been shown to vary among practices, which may indicate that accurate and complete documentation could be determined by clinical discipline or individual behaviors[16]. A failure of a CDSS’ implementation may not necessarily result from a poor knowledge base, but potentially a subtle misunderstanding of the availability of the required patient data due to perceptions of the EHR’s ability to capture the clinical process accurately and completely.

The goals of this investigation were to successfully identify clinical concepts within a set of guidelines that could be implemented within a CDSS, examine the similarities and differences of the perception of availability in these concepts among clinicians, and discover phenomenon that could contribute to the understanding of CDS creation, implementation, and maintenance.

Methods

Preparing guidelines to extract clinical concepts for review

The guidelines were selected and processed using a portion of the Shiffman et al.[17] methodology for transitioning clinical guidelines into CDSS, with clarification by Tso et al.[15] as defined by Table 1. The original guidelines were intended for an automated markup of CPGs, but these steps have been systematically explored to include manual modification[15, 17]. Guidelines were selected and dissected to yield the most granular and operational concepts possible, using the supporting literature of each guideline as a reference when applicable. Each of these concepts were presented to the clinicians, and were asked questions regarding their perceptions of how each concept is represented in the EHR. The responses were categorized and compared to draw preliminary conclusions on the perceptions of data availability among these individuals.

Table 1.

Shiffman et al.[17] steps as clarified by Tso et al.[15]

Decision Category	Definition
Select Guidelines	Choice of specific guidelines and choice of specific recommendations within the selected guidelines to be implemented
Atomize	The process of extracting and refining single concepts from the narrative text recommendations
Deabstract	The process of adjusting the level of generality at which a decision variable or action is described to permit operationalization
Disambiguate	The process of establishing a single semantic interpretation for a recommendation statement

For the Select Guidelines step, the Choosing Wisely (http://www.choosingwisely.org) guidelines were queried, and the first set of guidelines developed by the American College of Emergency Physicians (http://acep.org) were selected[15]. The following five Choosing Wisely guidelines by the American College of Emergency Physicians (http://www.choosingwisely.org/societies/american-college-of-emergency-physicians/) were selected due to their breadth of clinical concepts and interest to the research team:

Avoid computed tomography (CT) scans of the head in emergency department patients with minor head injury who are at low risk based on validated decision rules.

Avoid placing indwelling urinary catheters in the emergency department for either urine output monitoring in stable patients who can void, or for patient or staff convenience.

Don’t delay engaging available palliative and hospice care services in the emergency department for patients likely to benefit.

Avoid antibiotics and wound cultures in emergency department patients with uncomplicated skin and soft tissue abscesses after successful incision and drainage with adequate medical follow up.

Avoid instituting intravenous (IV) fluids before doing a trial of oral rehydration therapy in uncomplicated emergency department cases of mild to moderate dehydration in children.

To prepare the five guidelines chosen for this study, the Select Guideline, Atomize, Deabstract, and Disambiguate steps (Table 1) were utilized to create the most granular clinical concepts allowed by the supporting literature of the guidelines to be tested with the clinicians[17]. The Atomization and Deabstraction steps were repeated as needed to clarify any vague or ambiguous concepts (see Figure 1)[17].

Figure 1.

Shiffman et al. Guideline Implementation Gap Model[17]

Procedures for clinician review of clinical concepts from guidelines

Once the concepts were finalized, three questions were developed to explore the breadth of data availability of each concept in the form of a ‘yes’ or ‘no’. The addition of a comments section was added to provide clarification by the clinician of their choice of answers if needed. The following questions were proposed to the participants of each concept:

Is this data captured as discrete data in the EHR?

Is this data possibly captured solely as free-text?

Is this data found in multiple places in the EHR?

Nurses and physicians were recruited from the Duke University Medical Center (DUMC) emergency department on a volunteer basis. The goal of recruitment was to include a total of four participants; two nurses and two physicians. Inclusion would be met on the basis that the clinician has current experience with the DUMC EHR, and is currently practicing in DUMC emergency department. Four participants were recruited: two nurses and two physicians volunteered and met inclusion. Participants were interviewed individually and asked to review each of the final fifty-one clinical concepts, answering the three questions for each concept. They were given ample time to answer each question with clarification provided when prompted. All questions were answered completely in the form of ‘yes’ or ‘no’. Notes were taken on the context of the answers when provided in the comments section.

To better understand the data availability perceptions of each clinician, analysis of the data consisted of an exploration of discipline specific uniformity, nurse-physician agreement, and nurse-physician disagreement between the two groups. To assess discipline specific uniformity, nurse consensus was first determined by calculating the percentage of consensus for each of the questions. For instance, if both nurses reported that “patient age” is captured as a discrete data in the EHR, that would be considered consensus. Similarly, this was done for the physicians. Nurse-physician agreement was assessed by determining the frequency that all participants (both nurses and both physicians) answered identically for a particular concept. For instance, if both nurses and both physicians reported that “patient age” is not possibly captured solely as free-text in the EHR, that would be considered nurse-physician agreement. Lastly, nurse-physician disagreement was determined by calculating the frequency that both nurses answered ‘yes’ for a concept, while both physicians answered ‘no’, or vice versa.

Analysis of findings from clinical review

The degree of consensus between and across nurses and physicians was determined. Nurse and physician consensus were calculated with corresponding Cohen’s Kappa. Consensus was computed by determining the percentage of concepts that were answered unanimously by both participants in their respective groups. Agreement and disagreement between nurses and physicians were also calculated. Similarly, the percentages of concepts that were answered unanimously by all four participants (two nurses and two physicians) were determined for nurse-physician consensus, while nurse-physician disagreement was calculated as the percentage of concepts that were answered as ‘yes’ by both nurses and ‘no’ by both physicians (or vice versa).

Results

Description concepts in guidelines

Twenty-four concepts were identified and categorized from the five guidelines by the first round of Atomization (“Original Count” in Table 2). The phases of CDS were conceptualized for each concept, identifying “Trigger” criteria, “Inclusion and Exclusion” criteria, and “Recommendations” for information or action (“Original Count” in Table 3). Fifty-six concepts were identified by the remaining iterations of Deabstraction and Disambiguation of the Shiffman et al.[17] process, with five being removed due to identical or nearly identical phrasing (“Final Count” in Table 2). Those in the CDS phase of recommendation were removed from the final product of concepts, as these concepts relate to organization-specific implementation and would not necessarily be impacted by clinician perception (“Final Count” in Table 3).

Table 2.

Count of guideline concepts by clinical concept type

Concept Type Name	Original Count	Final Count	Definition	Example
Age	1	1	Criteria regarding a patient’s age	Pediatric patients
Diagnosis/Finding	8	33	Documentation surrounding a clinical finding or formal diagnosis	A history of bleedingdisorders, or delayedcapillary refill
Lab	2	1	Regarding a laboratory order	A wound culture is ordered
Medication	3	3	Regarding a medication	The ordering of an antibiotic
Meta Process/Evaluation	3	6	Reference to a formal or non-formal clinical process in addition to any evaluation that a clinician may conduct	Determining end-of-life needs for the patient
Other Order	2	1	An order not pertaining to any other category	A do not resuscitate status
Procedure	5	6	Regarding diagnostic to therapeutic clinical procedures	Insertion of a Foley catheter

Table 3.

CDS phase count

CDS Phase Name	Original Count	Final Count
Trigger	6	6
Inclusion and Exclusion	12	45
Recommendations	6	0

Assessment of clinician consensus

Discipline specific consensus, nurse-physician agreement rates, disagreement rates, and Cohen’s kappa scores are reported in Table 4. Nurse consensus was the same for questions 1 and 3 at 78%, but varied greatly for question 2 at 57%. Interestingly, physician consensus remained identical for all questions at 78%. Nurse-physician agreement dropped noticeably, at 47%, 35%, and 55% for each question respectively. The nurse-physician disagreement (i.e. both nurses answer ‘yes’ and both physicians answer ‘no’, or vice versa) was 12% and 10% for questions 2 and 3 respectively, with a larger rate of disagreement (18%) for question 1. The Cohen’s kappa statistic varied considerably across the nursing and physician groups, with physicians showing a higher degree of interrater reliability on questions 1 and 2, and nurses showing the highest interrater reliability on question 3. The highest level of agreement (moderate agreement[18]) was reached by physicians (κ =0.56) on question 1 and by nurses (κ =0.50) on question 3.

Table 4.

Consensus and disagreement of nurse and physician assessments of data availability

Question	Nurse Consensus	Nurse Cohen’s Kappa k	Physician Consensus	Physician Cohen’s Kappa k	Nurse-Physician Agreement	Nurse-Physician Disagreement
1. Is this data captured as discrete data in the EHR?	78%	0.24	78%	0.56	47%	18%
2. Is the data possibly captured solely as free-text?	57%	0.22	78%	0.37	35%	12%
3. Is this data found in multiple places in the EHR?	78%	0.50	78%	0.12	55%	10%

Each question is graphically displayed (Figure 2, 3, and 4) to compare the nurse-physician agreement and nurse-physician disagreement count of concepts in each category. Multiple trends can be seen in these representations. Of note, the two groups disagreed solely surrounding discrete data (question 1) categorized as a diagnosis/finding (Figure 2). Aside from age, both groups also disagreed on question 2 regarding procedure data at a significant rate (Figure 4). All instances of medication data’s representation in free text (Figure 3) were in consensus by both groups, with a drop in agreement rate (67%) regarding discrete medication data capture (Figure 1).

Figure 2.

Question 1 (Is this data captured as discrete data in the EHR?) - concept count and percentages by discipline consensus and disagreement

Figure 3.

Question 2 (Is this data possibly captured solely as free-text?) - concept count and percentages by discipline consensus and disagreement

Figure 4.

Question 3 (Is this data found in multiple places in the EHR?) - concept count and percentages by discipline consensus and disagreement

Discussion

For CDS to operate as intended, the relevant clinical data must be available to the CDS application. Currently, there is no standard practice for assessing the availability of data from EHR systems, including which users or clinical experts are most able to make this assessment. From this sample, it is suggestive that nurses and physicians may not have the same perceptions of data availability in the EHR. The first question (Is this data captured as discrete data in the EHR?) asked of the participants examined their perceptions of discrete data. While natural language processing is beginning to create possibilities for free-text data, discrete data are still seen as more favorable for CDS[19]. Both physicians and nurses had consensus at the same rate within their groups regarding this question, but cross-discipline consensus was achieved less than half the time. It is also important to note that this question yielded the highest polarization of disagreement, where 18% of the concepts had opposite nurse-physician consensus. Differences in perceptions of discrete data availability is important to understand for CDS implementation, whether it exists between or within groups.

While the consensus of each question (78%, with the exception of question 2 for the nurses) seems similar for both groups, the Cohen’s Kappa highlights areas where perceptions of data availability may not be as comparable if data were collected from a different (or larger) sample. While a firm conclusion cannot be drawn, the perceptions of discrete data availability within the EHR appears to be variable. The addition of questions regarding free-text and multiple data locations helps to better frame this phenomenon with more context from the user’s perspective.

Data availability drives documentation practices[20]. If perceptions differ, data that could have been captured as discrete data may be documented within a free-text note. If different disciplines show a consistent contradictory understanding of what information is available in the EHR, it may be due to issues in EHR understandability, differences in EHR education and exposure, differences in physician and nurse interfaces, or a variety of other human-computer factors. In complex medical documentation systems, it is difficult to ascertain the degree of influence of these factors on data availability perceptions. These questions are important to address as CDS implementations rely on data quality and good documentation practices[21]. With the increasingly present paradigm of patient-centered care, it is imperative that documentation is cohesive among all disciplines, as a break in the continuity of documentation lessens the potential benefit of the EHR in clinical practice[22]. If a phenomenon is understood and documented differently in the EHR, there exists a barrier to truly achieving automated patient-centric CDS. If an organization understands these differences, steps to mitigate these practices could take place by either the consideration of the data sources, or clinician education.

EHR data is captured in various formats, and it would not be realistic to expect all data to be discrete. Certain aspects of care are more appropriately captured as a narrative, but this presents a challenge when attempting to operationalize certain clinical practice guidelines into CDS. While free text can be appropriate, representing clinical concepts in the EHR with discrete data is currently the most pragmatic approach to maximize data availability and interoperability.

The second question (Is the data possibly captured solely as free-text?) serves two functions. First, it explores the possibility that if a concept is not captured as discrete data, it is at least captured in free-text format. Second, if data is able to be captured as discrete data, the clinicians may identify instances where it is not. From the data collected, there are differences in the consensus of this question between groups. The implications of knowing these responses from the perspective of CDS implementation could help guide data sources and education effort. The second function of this question may also serve to better understand the failure of a CDS implementation. Inappropriate use of free-text space, which is more likely in fast paced environments such as the emergency department, may produce a low utilization rate of CDS, hampering patient outcomes[23].

In addition to the formats of structured and unstructured data, we examined different clinician views of data availability in terms of its readiness for CDS. Documentation practices are not restricted to a binary understanding of discrete data documentation versus free-text data documentation, but instead represent a more complex facet of CDS implementation. The representation of data, even in the preferred format, can complicate the implementation and expected results of CDS if other dynamics of the EHR are not considered.

The third question (Is this data found in multiple places in the EHR?) explores the idea that data could possibly be documented in multiple locations of the EHR. This question had the lowest rate of disagreement between groups at 10%. While the participants were generally in consensus, there appears to be the most discordance in the documentation of procedures (Figure 4). While it is not readily apparent why, additional questions may be asked of the participants to clarify this phenomenon. If the documentation of a procedure is done multiple times, that is less harmful than the procedure being documented once in varying locations dependent on the clinician. Again, this would be beneficial to know from an organization’s perspective as the selection of data sources could be affected by the documentation practices and perceptions of the clinicians.

Limitations of this study exist largely in the sampling methods and sample size. A convenience sample of participants was taken from the DUMC emergency department in Durham, North Carolina. While this was done purposefully to ensure that each participant answered questions regarding a shared experience, the results’ generalizability to other locations and clinical areas is limited. The small convenience sample of two nurses and two physicians also limits the results, thus a larger number of participants would be recommended for replication of this study. The sample of five guidelines was purposeful to limit the number of questions the participants would have to answer, but a larger sample of guidelines would provide more context, especially if trialed in other clinical areas.

Despite these limitations, our methods built on existing methodological processes and demonstrated a feasible approach to answering questions regarding data availability[16]. We wanted to assess differences in clinician perceptions of data availability by way of a brief interview, thus the design of the three questions was purposely crafted to provide maximum insight with minimal burden on the participants. Due to the short duration of the interviews and possibility for the questions to be answered using an online survey, it would not be unreasonable for an organization to utilize this type of data collection to inform CDS implementation. The generalizability of these questions lessens the need for modifications to be used in other institutions and clinical specialties.

Conclusion

The factors of clinician perception of data availability are underappreciated when considering CDS creation, implementation, and maintenance. The implications of this study could modify organizational approaches to CDS and EHR training and follow-up. While this study is limited by its sample size and number of participants, the methods may be replicated in various settings with different CPGs and a larger sample to gain more dependable results. If expanded upon, it would be recommended to include perceptions from a data analyst, as they often serve as the primary source of expertise for CDS development in an organization. Juxtaposition of an analyst’s perceptions against the clinician’s perceptions would allow for more in-depth discussion of organizational strategy surrounding this phenomenon. While this preliminary data supports the notion that different clinical professions have differing understandings of data availability, the reason is not completely understood. As more examinations on this phenomenon are conducted, a better approach to these problems may be developed, leading to stronger CDS implementations and better patient outcomes.