Can emergency department provider notes help to achieve more dynamic clinical decision support?

OBJECTIVE: Assess whether clinical data were present in emergency department (ED) provider notes at time of order entry for cervical spine (c-spine) imaging that could be used to augment or pre-populate clinical decision support (CDS) attributes.
METHODS: This Institutional Review Board-approved retrospective study, performed in a quaternary hospital, included all encounters for adult ED patients seen April 1, 2013-September 30, 2014 for a chief complaint of trauma who received c-spine computed tomography (CT) or x-ray. We assessed proportion of ED encounters with at least 1 c-spine-specific CDS rule attribute in clinical notes available at the time of imaging order and agreement between attributes in clinical notes and data entered into CDS.
RESULTS: A portion of the clinical note was submitted before imaging order in 42% (184/438) of encounters reviewed; 59.2% (109/184) of encounters with note portions submitted before imaging order had at least 1 positive CDS attribute identified supporting imaging study appropriateness; 34.8% (64/184) identified exclusion criteria where CDS appropriateness recommendations would not be applicable. 65.8% (121/184) of encounters had either a positive CDS attribute or an exclusion criterion. Concordance of c-spine CDS attributes when present in both notes and CDS was 68.4% (κ = 0.35 95% CI: 0.15-0.56; McNemar P = 0.23).
CONCLUSIONS: Clinical notes are an underutilized source of clinical attributes needed for CDS, available in a substantial percentage of encounters at the time of imaging order. Automated pre-population of imaging order requisitions with relevant clinical information extracted from electronic health record provider notes may: (1) improve ordering efficiency by reducing redundant data entry, (2) help improve clinical relevance of CDS alerts, and (3) potentially reduce provider burnout from extraneous alerts.

INTRODUCTION

Background

Imaging clinical decision support (CDS; Table 1) is an application of health information technology to inform clinical decision making at the point of care regarding the need for imaging or the optimal diagnostic study based on the best available evidence. Although imaging CDS embedded in the electronic health record (EHR) can effectively deliver timely advice at the bedside, opportunities remain to improve CDS, including timing, sensitivity, and specificity of activation.

Importance

A “commandment” of imaging CDS is to respect ordering provider workflow, particularly by eliminating redundant data entry. To request an imaging study in most commercial electronic health records, physicians must independently enter clinical information in the order entry module via free text and/or structured forms (checkboxes of predetermined salient symptoms or relevant history). These forms do not automatically populate from data in the notes. In a typical workflow, when CDS is integrated within a computerized physician order entry (CPOE) system, a CDS tool is “activated” (presented to the orderer) in response to an action in the CPOE (eg, selecting “trauma” checkbox in the order requisition for a cervical spine [c‐spine] x‐ray activates c‐spine imaging appropriateness CDS). The CDS rule then prompts the orderer to enter structured patient data (“attributes”) separate from clinical notes or the order requisition. In other implementations, orderers are required to enter structured indications at the time of image order entry, irrespective of information documented in the provider note. This process leads to significant workflow interruptions and redundancy of documentation within the same EHR. Incomplete or conflicting information resulting from these redundancies may adversely affect the quality of patient care including propagation of misinformation, lack of ability for providers to identify pertinent information, accuracy of imaging protocol assigned, quality of radiologist's interpretation, or secondary analyses including guideline adherence and research. Low sensitivity of CDS activation may lead to missed opportunities to suggest more appropriate diagnostic studies. Further, exposing providers to low specificity alerts results in alert fatigue.

Image ordering and CDS workflows vary by CDS and electronic health record vendor, and there are a growing number of imaging CDS products in response to the imaging regulations component of the Protecting Access to Medicare Act, in effect as of January 1, 2020. These new federal regulations necessitate use of Appropriate Use Criteria delivered through certified CDS mechanisms for certain high‐cost imaging services in ambulatory settings, including the emergency department (ED), as a requirement for payment. A crisis of alert fatigue and usability deficits may ensue if the implementation of these CDS tools relies on independent provider data entry and does not leverage existing electronic health record data to improve the specificity of their activation. This is in addition to the potential quality of care risks from discordant structured information entered during the provider order entry process, including erroneous CDS guidance, inappropriate imaging protocol selection for the condition being evaluated, and negative impacts on imaging interpretation. ,

CDS can effectively improve ED practice. , , , , The National Emergency X‐Radiography Utilization Study (NEXUS) and the Canadian C‐Spine Rule are well‐validated decision rules based on the highest quality evidence to assist emergency clinicians in reducing unnecessary c‐spine imaging. , , , , In our institution, NEXUS and Canadian C‐Spine rules have been implemented as CDS for c‐spine CT and x‐rays for ED patients with trauma.

The Bottom Line

Clinical decision support for imaging can improve emergency medical practice and support evidence‐based diagnostic decisionmaking. This retrospective study evaluated the proportion of ED encounters in which clinical notes could potentially inform a clinical decision support alert for cervical spine imaging.

Goals of this investigation

We sought to assess the proportion of encounters with at least 1 c‐spine CDS rule attribute documented in clinical notes in the electronic health record at the time of ordering c‐spine CT or x‐ray for ED patients and agreement between attributes in notes and CDS tools in the CPOE.

METHODS

Study design and setting

This study was approved by the Institutional Review Board at Brigham and Women's Hospital (protocol #2015P002169). The requirement for informed consent was waived for this Health Insurance Portability and Accountability Act‐compliant, observational retrospective study of patients who visited the ED of a 793‐bed, quaternary care, Level 1 trauma center, academic hospital with ≈60,000 annual visits.

Selecting participants

We evaluated all encounters for adult patients presenting to the ED between April 1, 2013 and September 30, 2014 with a chief complaint consistent with trauma. After an analysis of chief complaints to identify common terms used for trauma, we defined the cohort by querying the ED chief complaint fields for the following strings and their variations: “fall,” “injury,” “trauma,” “mvc,” “mva,” “bicycle accident,” and “motorcycle accident.” We then excluded encounters that did not have a c‐spine CT or x‐ray performed.

Data collection

In the ED clinical data warehouse, providers separately documented sections of notes such as chief complaint, history of present illness, and assessment and plan. The entire versions of each revised note portion were saved, along with timestamps when each revision was submitted and when the completed note portion was signed. The note sections and their data types are summarized in Table 2. We used the most recent note submission prior to imaging order entry for our analysis.

TABLE 2

Emergency department note sections of interest and their data type

Note section	Data type
Chief complaint	Unstructured free text
History of present illness	Unstructured free text
Mechanism of injury	Structured, including injury due to: bicycle accident motor vehicle accident with rollover motor vehicle accident with ejection from vehicle fall with height and landing surface
Review of systems, positive or negative	Structured
Review of systems, comments	Unstructured free text
Neurological level of consciousness	Structured
Head and neck exam, normal values	Structured
Neurological exam, normal values	Structured
Head and neck exam, comments	Unstructured free text
Neurological exam, comments	Unstructured free text
Initial assessment and plan	Unstructured free text
Attending note	Unstructured free text

We excluded encounters when the trauma team was alerted, as patients immediately receive trauma series imaging studies. We excluded encounters that did not have c‐spine CT or x‐ray. We extracted imaging order timestamps, order requisition details including indications, whether the associated CDS system activated, and clinical attributes input by the ordering provider by querying the institution's radiology CPOE system (Percipio; Medicalis, Kitchener, Ontario, Canada) and radiology information system (IDXrad; GE Healthcare, Burlington, VT) data warehouse. We used the first c‐spine imaging study during an ED visit in our analyses when multiple studies were performed. Demographic and clinical data were collected from the institution's clinical data warehouse, Research Patient Data Registry.

Decision support rule attributes

Our institution's EHR implementation was similar to most commercial EHR implementations in that the imaging order requisition was entered via the CPOE independently of the physician note and did not automatically populate from data in the notes. The order requisition form allowed the ordering provider to enter the indication for imaging as free text and/or to select a structured indication from a customized list of potential attributes (eg, “trauma”) based on the particular imaging study selected. When specific combinations of orders and indications were selected, the CDS tools “activated” by presenting a list of selectable attributes and inclusion/exclusion criteria to the ordering provider as checkboxes. Checking the box indicated the attribute was present (positive) and leaving it unchecked indicated the attribute was absent (or negated), assuming all attributes were addressed. The combination of positive or negated attributes and inclusion/exclusion criteria determined whether the CDS tool provided feedback on the appropriateness of the imaging study based on the evidence‐based CDS rule.

The NEXUS CDS rule contains 5 clinical criteria (“attributes”): (1) posterior midline c‐spine tenderness, (2) evidence of intoxication, (3) alteration in level of alertness, (4) focal neurologic deficit, and (5) presence of a painful, distracting injury. Absence of all 5 of these attributes suggests a very low probability of cervical injury and imaging is not recommended. These attributes are distinct from exclusion criteria where the CDS rules should not be applied. Cervical injury should not be ruled out using NEXUS criteria in patients with penetrating neck trauma. During the study period, NEXUS CDS was activated when a c‐spine CT or 3‐view c‐spine x‐ray study was ordered and the “trauma” checkbox was selected on the order requisition. The Canadian C‐Spine rules have 9 exclusion criteria including Glasgow Coma Scale<15, penetrating neck trauma, and known vertebral disease. Canadian C‐Spine CDS was activated when a standard 6‐view c‐spine x‐ray study was ordered and the “trauma” checkbox was selected on the order requisition. We chose to include both NEXUS and Canadian C‐Spine CDS in our analysis because clinical organizations may choose either of the two CDS rules to apply to c‐spine imaging.

A neurologist and an internist manually reviewed the most recent version of note portions submitted before imaging order entry and documented positive and negated attributes and exclusion criteria of NEXUS and Canadian C‐Spine rules. Any differences between them were discussed until consensus was achieved.

Outcome measures and statistical analyses

The primary outcome was the proportion of encounters with at least 1 c‐spine CDS rule attribute documented in clinical notes in the EHR at the time of imaging order of c‐spine CT or x‐ray for ED patients. Secondary outcome measures included (1) the proportion of encounters with attributes present in the notes stratified by whether the CDS was activated, (2) the proportion of CDS encounters with at least 1 exclusion criterion in the clinical notes available at the time of imaging order entry, (3) the concordance between attributes in clinical notes and those entered in CDS tools in the CPOE system, and (4) the proportion of encounters where the CDS tool was activated when indicated. We assessed for differences in proportions of encounters with CDS attributes when CDS was activated versus not activated with Fisher's exact test. We calculated concordance using kappa statistic of agreement and McNemar test of marginal homogeneity of the paired data. As attributes are present in the CDS tool in all encounters where CDS was activated, concordance was calculated for individual attributes when CDS tool was activated and an attribute was described as positive or negated in the note. Attributes positive or negated in both the CDS tool and notes were graded as concordant. Attributes that were positive in the CDS tool and negated in notes or vice versa were graded as discordant.

Data transformation and comparisons of the timing of orders with the timing of documentation were performed using Microsoft Access 2007 (Microsoft, Redmond, WA) and R version 3.2.2 software (R Project for Statistical Computing, Vienna, Austria). Analyses calculating proportion of encounters with attributes as well as concordance calculations were performed using Microsoft Excel 2007 (Microsoft, Redmond, WA).

RESULTS

Study population

We extracted all versions of note sections of interest (Table 2), with associated authors and timestamps, from 3757 consecutive ED encounters for 3582 unique patients with a trauma chief complaint during the study period. From these encounters, we isolated 3155 consecutive encounters for 2992 unique patients who presented with a trauma chief complaint without trauma team activation (602 encounters for 602 unique patients) during the 18‐month study period (Figure 1). C‐spine imaging (computed tomography [CT] or x‐ray) was performed in 438/3155 (13.9%) of these encounters for 427/2992 (14.3%) unique patients. 55.7% patients were female; the average age was 66.9 years (range: 20–105, standard deviation 20.9).

FIGURE 1

Patient cohort selection flowchart. CDS, clinical decision support; CT, computed tomography; ED, emergency department; NEXUS, National Emergency X‐Radiography Utilization Study; XR, X‐radiography

Note section entry and completion

Rates of note section entry and completion prior to c‐spine imaging order are provided in Table 3. In patients undergoing c‐spine imaging, a submission of any note portion of interest before imaging study ordering was present in 42.0% (184/438) of encounters. There was a submission of a history of present illness, initial assessment and plan, and attending note before imaging study ordering in 34.7% (152/438), 11.0% (48/438), and 8% (35/438) of encounters, respectively.

TABLE 3

Summary of note portions that were submitted and signed prior to the time of image order entry for patients with a chief complaint consistent with trauma

	Entry submitted prior to imaging order N (%)	Signed prior to imaging order N (%)
	438	438
History of present illness	152 (34.7%)	44 (10.05%)
Neurological review of systems (positive or negative)	16 (3.7%)	16 (3.7%)
Neurological review of systems comment	6 (1.4%)	6 (1.4%)
Level of consciousness	49 (11.2%)	49 (11.2%)
Head and neck exam, normal values	36 (8.2%)	32 (7.3%)
Neurological exam, normal values	32 (7.3%)	32 (7.3%)
HEENT exam comments	22 (5.0%)	18 (4.1%)
Neurological exam comments	12 (2.7%)	11 (2.5%)
Initial assessment and plan	48 (11.0%)	16 (3.7%)
Attending note	35 (8.0%)	2 (0.5%)

HEENT, head, eyes, ears, nose, and throat

Proportion of encounters with CDS attributes

A total of 59.2% (109/184) of encounters with note portions submitted before imaging order had at least 1 positive CDS attribute identified in the notes (mean = 0.72/encounter; range = 0–3) and 37.0% (68/184) had at least 1 negated NEXUS attribute (mean = 0.68; range = 0–4) (Table 4). There were no statistically significant differences between proportion of encounters with positive, negated, or any attributes when NEXUS CDS was activated versus without NEXUS CDS activated (positive: 51.8% vs 62.5%, respectively; P = 0.19; negated: 39.3% vs 35.9%, respectively; P = 0.74; any: 67.9% vs 73.4%, respectively; P = 0.48). We found only one of the 184 encounters where all attributes were mentioned in the notes.

TABLE 4

Encounters containing at least 1 NEXUS CDS attribute and/or at least 1 exclusion criterion for NEXUS and Canadian C‐Spine rules among those where note portions were present at the time of image order entry

	Encounters with CDS (%) (N = 56)	Encounters without CDS (%) (N = 128)	P value (Fisher's Exact)	Total encounters (%) (N = 184)	Mean/encounter (N = 184)	Range
Positive attribute	29(51.8%)	80(62.5%)	0.19	109(59.2%)	0.72	0–3
Negated attribute	22 (39.3%)	46 (35.9%)	0.74	68 (37.0%)	0.68	0–4
Any attribute	38 (67.9%)	94 (73.4%)	0.48	132 (71.7%)	1.40	0–5
NEXUS exclusion	3 (5.4%)	12 (9.4%)	0.40	15 (8.2%)
Canadian C‐Spine exclusion	21 (37.5%)	43 (33.6%)	0.62	64 (34.8%)

CDS, clinical decision support.; NEXUS, National Emergency X‐Radiography Utilization Study.

Proportion of encounters with CDS activation

NEXUS CDS was activated in 26.7% (117/438) of the encounters with chief complaint of trauma with c‐spine imaging; Canadian C‐Spine CDS tool was not activated in any encounters, presumably because of low frequency of ordering of 6‐view c‐spine x‐rays ordered for this population at our institution. 47.9% (56/117) of the encounters where NEXUS CDS was activated had note portions submitted before imaging order.

Proportion of CDS encounters with exclusion criteria

A total of 8.2% (15/184) of encounters had at least 1 NEXUS exclusion criterion identified and 34.8% (64/184) of encounters had at least 1 Canadian C‐Spine exclusion criterion identified. There were no statistically significant differences between proportion of encounters with NEXUS exclusion criteria or Canadian C‐Spine exclusion criteria when CDS was activated versus without CDS activated (NEXUS exclusion: 5.4% vs 9.4%, respectively; P = 0.40; Canadian C‐Spine exclusion: 37.5% vs 33.6%, respectively; P = 0.62). A total of 65.8% (121/184) of encounters had either a positive CDS attribute or an exclusion criterion. To assess for potentially suppressible unnecessary CDS activations, only 5.4% (3/56) of encounters with both NEXUS CDS activated and note portions present prior to imaging order entry had an exclusion criterion indicated in the note submission. In these, 37.5% (21/56) indicated in the note submission at least 1 exclusion criterion for the Canadian C‐Spine rule.

Concordance of NEXUS attributes

Table 5 details the concordance of positive or negated NEXUS CDS attributes when present in both the notes and the CDS tool. The overall concordance of NEXUS CDS attributes when present in both sources was 68.4% (κ = 0.35, 95% confidence interval 0.15–0.56) indicating fair agreement. There was no statistically significant difference in the rates of positive or negated attributes obtained from notes versus those obtained via the CDS tool (P = 0.23; McNemar test). Interestingly, the one encounter where all 5 NEXUS attributes (1 positive, 4 negated) were found in the notes had perfect concordance with the CDS tool.

TABLE 5

Concordance of NEXUS CDS attributes with attributes found in note portions submitted prior to image study ordering

		Note submission				Note submission
Midline tenderness		Positive	Negated	Focal Neuro Deficit		Positive	Negated
CDS	Positive	4	5	CDS	Positive	2	0
	Negated	1	10		Negated	4	14
Concordance = 0.70		κ = 0.37		Concordance = 0.80		κ = 0.41

CDS, clinical decision support; Neuro, Neurological; NEXUS, National Emergency X‐Radiography Utilization Study.

LIMITATIONS

Our data must be interpreted in the context of the study design. The study was conducted in a single academic setting making its generalizability unclear. The use of checkboxes for CDS attributes makes an assumption that unchecked boxes indicate negated attributes. But this may also indicate that the provider did not address the attribute. This limits the reliability of our results of concordance and should be addressed in the future with tristate options (yes/no/unsure) or asking the provider to select a yes or no indicator. One NEXUS CDS attribute, “presence of a painful, distracting injury,” is challenging to define and although agreement was reached between reviewers based on documentation in notes, it can be difficult to confirm if an injury is distracting without evaluation of the patient. This may explain why agreement was lowest between notes and the CDS tool for this attribute. Clinical documentation in notes and data entry into CDS are likely clustered and variable based on individual clinician practice; further work should examine these patterns. Our study assessed encounters for patients with trauma chief complaint who received c‐spine imaging, making the generalizability to other complaints unclear, but builds on a previous study on patients with chief complaint of headache. Our study only looked at note portions that were submitted prior to image ordering. As noted in Table 3, this does not include a substantial proportion of assessments and plans where the synthesis of history and findings along with medical decision making are documented. This is where one may expect to find reference to this evidence.

DISCUSSION

This descriptive study utilizing CDS based on highest quality evidence , , demonstrated that high‐impact yet complex CDS attributes and exclusion criteria can be found in unstructured clinical notes present at the time of imaging order entry, building on a previous study that demonstrated clinical notes as an underutilized source of relevant information with the potential to enhance the order entry process. As with practice guideline logic, CDS attributes are often not available in coded and computable EHR data and may not even be mapped to a standard ontology. , They may be found only in notes or would need to be specifically elicited through a structured CDS tool. Additionally, we found the CDS tool was activated in a surprisingly low proportion of encounters when it would have been applicable, making data in the clinical notes even more salient.

Only 1of the 184 encounters mentioned all applicable attributes in the notes. This lack of documentation is concerning as all 5 attributes must be negated to determine that c‐spine imaging is inappropriate. This poses an opportunity to prompt providers to consider attributes that were missing in their notes.

CDS alerts with low specificity lead to high rates of physician override and alert fatigue and reducing alert fatigue is an EHR‐specific patient safety goal. , We identified exclusion criteria for CDS rules in unstructured notes, predictably more often for rules limited by a larger number of exclusion criteria. Additionally, if only 1 c‐spine CDS attribute is positive, c‐spine imaging studies would be appropriate. Harvesting 1 positive attribute or 1 exclusion criteria from the EHR notes may allow suppression of CDS rules not applicable to the current patient, which would free clinicians from unnecessary visual and workflow interruptions by suppressing unnecessary engagement with the CDS tool, thus reducing alert fatigue. We hope that through improved methods and technologies, using data present in notes will improve the precision of delivery of CDS tools for the right patient at the right time. An optimal state may include harvesting from available sources in bidirectional fashion, whether from notes or CDS tools, and requesting the provider to enter absent data, reconcile, and incorporate while using exclusion parameters to suppress unnecessary CDS. Future work is needed in the area of text mining or use of natural language processing (NLP) of these attributes in order to realize a feasible application integrated in the EHR.

An assumption is made of data collected via a CDS tool that they accurately represent the “ground truth” and have been used to supplement the notes in documentation of adherence to CDS rules. , However, lack of concordance between attributes documented both in notes and CDS in our study aligns with previous studies that point to the lack of accuracy and completeness of data in the EHR in either source. , There are opportunities for documentation error in both notes and CDS tools. Future work is needed to investigate which source of information is more accurate where there are conflicts. Although providers may complete only the minimal requirements with structured data entry, this study had insufficient data to assess for similar behavior when completing CDS tools. The differences in rates of positive and negated attributes between note and CDS sources did not reach statistical significance, likely because of the lack of power in the identified sample size. This outcome also reflects findings in a recent study that showed discordance and incompleteness in order requisition indications compared to data from clinical notes, with potential impacts on the selection of imaging protocol to address the indication as well as the interpretation of the imaging studies.

This study illustrates that data entered in the EHR and via CDS fall short in multiple dimensions of data quality including correctness and concordance. Future work is needed to identify methods of improving the quality of data in the EHR and CDS tools, as well as performing validity checks at the point of data entry. This reinforces a potential future state of a semiautomated process of attributes extracted from the notes being verified by the clinician when interacting with order entry or CDS tools. Toward a “data‐driven, ideal care system,” , CDS tools would leverage EHR data, anticipate needs within the physician workflow in real time, trigger alerts in cases that would benefit from physician action, or suppress CDS not applicable to the current patient with a potential to reduce physician burnout. , ,

CONCLUSIONS

Clinician documentation in the EHR is an underutilized resource for CDS attributes and exclusion criteria and is available in a significant percentage of encounters at the point of image ordering. However, low rates of CDS activation and only fair agreement of CDS rule attributes between the data in clinical notes and attributes entered via a CDS tool in the same encounter raise questions of the optimal way to capture high‐quality data in the EHR. Future work via text mining or NLP is needed to automate the extraction of CDS attributes and exclusion criteria to realize a feasible application integrated in the EHR.

CONFLICTS OF INTEREST

The authors declare that they have no conflicts of interest in this research. Please see attached ICMJE conflict of interest disclosure forms for all authors.

AUTHOR CONTRIBUTIONS

Justin F. Rousseau, Ivan K. Ip, Ali S. Raja, and Ramin Khorasani conceived the study and designed the data extraction and the analysis. Justin F. Rousseau conducted the data collection under the guidance of Ivan K. Ip and performed annotation of the data. Justin F. Rousseau performed the statistical analyses with oversight by Ivan K. Ip and Ramin Khorasani. Ali S. Raja and Jeremiah D. Schuur provided clinical subject matter expertise for the clinical context of the data capture in the electronic health records. Justin F. Rousseau drafted the manuscript, and all authors contributed substantially to its revision. Justin F. Rousseau takes responsibility for the paper as a whole.

TABLE 1

Abbreviations and definitions

Abbreviation	Definition
CDS	Clinical decision support
CDS activation	A CDS tool is presented to the ordering provider in response to criteria met in the order entry process (eg, selecting a “trauma” checkbox in the order requisition for a cervical spine [c‐spine] x‐ray activates c‐spine imaging appropriateness CDS).
CDS exclusion criteria	Characteristics that identify patients to whom the CDS rules do not apply as defined by the studies used to create the CDS rules (eg, penetrating neck injury is the only exclusion criterion for the National Emergency X‐Radiography Utilization Study [NEXUS] decision rule).
CDS negated attribute	Clinical criteria (eg, posterior midline c‐spine tenderness) identified as being absent. For NEXUS CDS, if all of 5 attributes are absent, the patient has a very low probability of cervical injury.
CDS positive attribute	Clinical criteria (eg, posterior midline c‐spine tenderness) identified as being present.
CPOE	Computerized physician order entry
ED	Emergency department
EHR	Electronic health record
HEENT	Head, eyes, ears, nose, and throat
NEXUS	National Emergency X‐Radiography Utilization Study
NLP	Natural language processing