A Computational Adverse Event Detection Matrix.

Harms caused during healthcare encounters are pervasive and occur at an alarming rate; therefore, building a set of computational detection methodologies in the adverse event area is urgently needed to address this problem. To understand the entire range of adverse event detection methods currently in practice we have developed a computational adverse event detection matrix. This structure is made of methods used presently at US hospitals to detect patient safety events. It contains adverse event 1) concepts and 2) synthesized detection strategies as well as calculations of overlap of coded data in the subset of algorithms implemented completely computationally. Most importantly, this matrix provides a clear picture of coverage gaps in the detection of adverse events.

Introduction

Harms or Adverse events (AE)s caused during healthcare encounters are pervasive, have a significant impact on patient wellbeing, and occur at a staggering rate. Specifically, one of every ten hospitalized patients develops a healthcare-acquired condition, and twelve million outpatients experience a diagnostic error, which may result in harm, every year. [3]AEs can be categorized as unavoidable (e.g. part of the patient’s condition), potential (near miss or never events) or preventable and can come from active medical errors at the point of care or latent causes present within care settings. [9; 13] The Institute for Healthcare Improvement uses a definition for AEs as “Unintended physical injury resulting from or contributed to by medical care (including the absence of indicated medical treatment), that requires additional monitoring, treatment, or hospitalization, or that results in death”.

This study is designed to determine which of the events designated as harms, or AEs, are currently 1) identifiable algorithmically, 2) which are identifiable using other methods, and 3) which are likely going undetected due to the methodology used. AEs are typically tracked and quantified using incident reporting, medical record review or claims data, although other methods are also available, as shown in Table 1. None of these processes captures all AEs.[10] Comparative studies suggest that the most effective methodology is the Medical record review (MRR).[9] As this process is time-consuming, efforts have been made to increase the efficiency of this work with instruments like the Institute for Healthcare Improvement (IHI) Global Trigger Tool. [3; 7]. To ensure as many AEs are detected as possible while at the same time not overloading healthcare staff automation has been introduced in several of the methodologies listed in Table 1. Detecting AEs using rule-based algorithms makes automation possible. [8] But this automation is only as good as the combination of the phenotype definition and the data it consumes. Not all AE will be detectable using rule-based algorithms. Active errors occur at the point of contact between the patient and the healthcare system and are typically observable whereas latent errors are process errors, where no single event is the source of the problem. AEs arising from active errors are more detectable than those arising from latent causes, and direct process observation or data mining are the best solutions to finding these types of situations. In general, the methods listed in Table 1 can miss as much as 90% of all AE types. [5]

Table 1.

AE Detection Methods Currently in Use

Mcdical Record Review (w/ and w/out Trigger Tools)

Studies based on interviews w/ healthcare providers

Direct Observation

Incident Reporting Systems

External Audit

Studies of legal claims and complaints

Administrative Data

Computational Methods using EHR

Autopsy

We are currently in need of common terminology to quantify the occurrence of and detection methods for AEs. In the US, governmental organizations and Patient safety organizations (PSO), regulated by the US Agency for Healthcare Research and Quality (AHRQ), collect data and do analysis and reporting on patient safety concerns. Many of these organizations have designed approaches to describe, classify and detect patient safety events. Methods used include retrospective surveillance with administrative coded data, observation and voluntary reporting as well as MRR. All of these methods are flawed in that they miss AEs or are excessively expensive so cannot be conducted at scale. [12; 14] Additionally, large scale detection methods are not sensitive enough to show improvement over time. [3]

In an effort to find common ground among the methods currently in practice, we have developed a computational AE detection matrix. This matrix provides a clear picture of what is available and what is missing in the detection of AEs. As part of this process we have also investigated the overlap in computationally implemented algorithms.

Methods

In order to understand where there is overlap in current AE detection methodologies and to discover which AE are not the subject of in practice methodologies, we have classified currently accepted PSO AE detection algorithms into the International Classification for Patient Safety categories from the World Health Organization (WHO). Patient safety information is categorized into standardized concepts that represent descriptions that are valid internationally and can be used for comparison, measurement, monitoring, analysis and interpretation of information for improved patient safety.[15] This framework breaks patient safety events into ten high-level classes which are broken down into very specific but generalizable concepts two of which are directly comparable to the PSO terminology, incident type and patient outcomes. Although, some incidents may be classified in more than one category in the WHO framework we have kept to a one to one relationship with detection algorithm.

Right now, there are a few standard approaches to AE detection and none are both highly efficacious and also computational. Some are completely computational and can detect roughly half of AE [1], and some use MRR, which is very through but limits the number of cases evaluated due the time required. In previous studies commonalities among detection strategies were synthesized and found to be predominantly focused in the areas of surgery and care management.[11] This study seeks to extend the previous work by including areas in healthcare where AE are recognized, but no external detection method is currently in place to identify them. This work includes the extraction of the rule-based definitions. For the computational approaches, the extraction of International Statistical Classification of Diseases, Injuries and Causes of Death (ICD) codes used in the detection of AEs from organizational source documents was necessary to build an understanding of the degree of overlap between computational methodologies. Commonalities have arisen between the ICD codes coming from government bodies The Centers for Medicare and Medicaid Services (CMS) and PSOs (AHRQ). AEs are detected for Patient Safety Indicators (PSI) coming from AHRQ and Human Acquired Conditions (HAC) coming from CMS.

The manual and hybrid approaches to AE detection were extracted from the organization website sources. These cannot be compared directly with the computational methods in terms of data points due to the incorporation of human judgment. The most effective method with respect to efficiency as well as thoroughness is the hybrid approach. Together, review of medical records with computerized monitoring capture conditions that are associated with more preventable and potential AEs than any singular method. [4; 6; 9]

The sources used for these algorithms are (a) CMS, hospital-acquired conditions, (b) IHI, Global Trigger Tool (c) The Agency for Healthcare Research and Quality (AHRQ, PSI), (d) The Joint Commission (TJC, Sentinel Event definitions) and (e) The National Quality Forum (NQF, serious reportable event (SRE) definitions).

Results

Both the CMS HAC codes and the AHRQ PSI rates use ICD codes to detect patient harm. As shown in Figure 1 - 5,702,505 ICD codes are used, and 70,384 are shared. AHRQ uses a large portion of the CMS set plus a great deal more.

Figure 1.

AHRQ and CMS detections methods which use ICD codes exclusively.

This large overlap exists due to the use of the Medicare Severity Diagnosis Related Group (MS-DRG) system for financial considerations by the Centers for Medicare and Medicaid Services. This system, in effect, determines the definition of an AE by associating payment penalties with the code combinations that represent harm conditions. The idea of patient safety and the prevention of AEs was the impetus for the MS-DRG system. That being the case, any computational AE detection methodology in the US will need to also incorporate information from this system.

The AE detection matrix shown in Table 2 classifies rule-based algorithms according to the WHO Patient Safety Conceptual Framework. The WHO incident types are in the leftmost column. The binary patient outcome severity, listed as present or not, is across the top. Both CMS and AHRQ use the MS-DRG system which provides grouping information about severity of an inpatient stay in order to facilitate payment for services. The methodology used in detection is the column title in Table 2 listing the spectrum of methodologies in computational to completely manual.

Table 2.

Computational AE Detection Matrix

	Level of Harm Specified	No	Yes
		EHR Data	EHR Data	Trigger w/MRR	Observation or MRR
WHO Incident Categories	Behavior				NQF/TJC
	Blood/Blood Products		AHRQ/CMS	IHI	NQF/TJC
	Clinical Administration	AHRQ		IHI	NQF/TJC
	Clinical Process/Procedure	AHRQ	AHRQ/CMS	IHI	NQF/TJC
	Healthcare Associated Infection		AHRQ/CMS	IHI	NQF
	Infrastructure/Building/Fixtures				NQF
	Medical Deviee/Equipment				NQF
	Medication/IV Fluids			IHI	NQF
	Oxygen/Gas/Vapor			IHI	NQF
	Patient Accidents		AHRQ/CMS	IHI	NQF/TJC

Increasingly difficult and time consuming to complete → practice from completely

Discussion

This work lays out detection strategies currently in use against an international conceptual framework of AEs, thereby exposing the sparseness of computational solutions in today’s healthcare environment. Each of the fundamental patient safety events in the framework has many subtypes, and out of these 10 categories only 5 are computationally implemented, and one of these doesn’t indicate severity. That leaves half of AE types to voluntary reporting and record review with no external means of being discovered.

Moreover, the computationally implemented detection algorithms rely on coded administrative data. There are concerns about accuracy in this type of data because not all cases of a complication will be captured, leading to both false positives and negatives. Lastly, data quality is a problem in any data source and missing information is the most prominent issue in healthcare data resulting in codes that may not fully reflect the clinical case.[2]

While gaps exist in computational methods, there is good coverage in the application of medical record review. This approach addresses the false positive and false negative concerns but it is intensely laborious and not practical for truly safeguarding patients. The combination of review using a data-driven trigger tool approach has potential as well but is not yet in common practice due to the complexity of the process. [12]

Conclusion

Harms caused during healthcare encounters are ubiquitous and occur at an alarming rate and building a complete set of computational detection methodologies in the AE space is urgently needed to address the problem. This is a many-dimensional problem due to multiple organizations, terminologies, detection methodologies and levels of harm. The matrix of patient safety events developed in this study and the resulting synthesis of associated detection algorithms provides a clear picture of areas where coverage gaps exist in AE detection.