Leveraging electronic health record documentation for Failure Mode and Effects Analysis team identification.

OBJECTIVE: Using Failure Mode and Effects Analysis (FMEA) as an example quality improvement approach, our objective was to evaluate whether secondary use of orders, forms, and notes recorded by the electronic health record (EHR) during daily practice can enhance the accuracy of process maps used to guide improvement. We examined discrepancies between expected and observed activities and individuals involved in a high-risk process and devised diagnostic measures for understanding discrepancies that may be used to inform quality improvement planning.
METHODS: Inpatient cardiology unit staff developed a process map of discharge from the unit. We matched activities and providers identified on the process map to EHR data. Using four diagnostic measures, we analyzed discrepancies between expectation and observation.
RESULTS: EHR data showed that 35% of activities were completed by unexpected providers, including providers from 12 categories not identified as part of the discharge workflow. The EHR also revealed sub-components of process activities not identified on the process map. Additional information from the EHR was used to revise the process map and show differences between expectation and observation.
CONCLUSION: Findings suggest EHR data may reveal gaps in process maps used for quality improvement and identify characteristics about workflow activities that can identify perspectives for inclusion in an FMEA. Organizations with access to EHR data may be able to leverage clinical documentation to enhance process maps used for quality improvement. While focused on FMEA protocols, findings from this study may be applicable to other quality activities that require process maps.

Introduction

Quality improvement approaches often rely on process maps to identify the activities and individuals involved in a clinical workflow. As the process map becomes the basis of improvement, the impact of a particular quality improvement project relies on the map’s accuracy and completeness. Incorrect or incomplete process maps could result in quality improvement projects that do not achieve their desired outcomes. Therefore, improving the quality of process maps could lead to stronger quality improvement projects.

Failure Mode and Effects Analysis (FMEA) is an example of a popular quality improvement method that utilizes process maps of clinical workflows. FMEAs satisfy The Joint Commission’s accreditation standards requiring regular proactive risk assessment, which spurs the approach’s popularity in healthcare. FMEAs proactively identify flaws in high-risk processes, analyze the different ways a process may fail, and prioritize interventions for addressing threats before harm can reach a patient. FMEA starts with the identification of a high-risk process that poses a threat to patient safety, such as hospital discharge. FMEA leaders then assemble a team of topic experts and clinical representatives selected for their familiarity with the high-risk process being analyzed. The FMEA team creates a hand-drawn process map, which includes workflow activities and the clinical provider expected to perform them. The process map then guides the identification of potential threats – or ways a process can fail – and additional individuals who may have insight into the high-risk process that should be consulted. Next, the team calculates Risk Priority Numbers for each threat identified based on the threat’s likelihood of occurrence, likelihood to be detected, and severity of the harm to the patient if not prevented. The FMEA team then uses the Risk Priority Numbers to prioritize interventions based on the most significant threats.

Prior studies question FMEA’s validity due to the approach’s reliance on the knowledge and experience represented during process map creation. Shebl et al. conclude that FMEAs are a useful tool for mapping and understanding a process, but may result in too many inconsistencies and inaccuracies to be a reliable patient safety tool if the FMEA team lacks adequate knowledge. Additionally, FMEAs require a large time commitment and displacement of clinical staff from their other duties, which is seen as another limitation., FMEA accuracy is vital both to prevent harm and to ensure FMEA’s return on investment.,, Such limitations may also be seen in other quality improvement approaches that rely on process maps.

A logical approach to improving FMEA accuracy and enhancing the value derived from the process would be to gather comprehensive perspectives to ensure adequate information is provided about the high-risk process. However, many prior approaches to improving FMEA accuracy focus on strengthening FMEA’s output rather than improving the accuracy of information input to the process., Many suggested improvements fail to address the core issue of starting the FMEA with accurate and complete information to improve the resulting output validity.

Identifying comprehensive perspectives for an FMEA starts by addressing gaps in the process map that guides FMEA planning, threat identification, and subsequent intervention. Ideally, perspectives gathered would comprehensively reflect a diverse range of individuals who are familiar with or involved in a process in order to identify potential ways it can fail. Currently, FMEA team members identify providers expected to perform particular workflow activities. However, literature identifies discrepancies between healthcare worker perception of the clinical team and the actual team as recorded in the electronic health record (EHR)., Additionally, analysis of EHR audit logs identified that clinical providers may overlook the contribution of ancillary medical providers. Findings from literature suggest that clinical staff may not be fully aware of the presence or the activities of other members of the clinical team during a high-risk process, which calls the accuracy of process maps into question.

EHRs represent a new opportunity for strengthening process maps and the resultant quality improvements they inform. For example, EHRs may aid in identifying people involved in a high-risk process whose perspectives may be overlooked by typical FMEA approaches. As indicated previously, emerging literature highlights EHR’s ability to identify unrealized connections among healthcare teams. As providers document their activities in the EHR as part of their daily workflow, the EHR generates metadata including names, titles, times, and activity details. A sophisticated and innovative FMEA approach could leverage secondary use of clinical documentation including orders, forms, and notes to create a more comprehensive FMEA that can more accurately identify threats to a high-risk process. EHR documentation may be able to identify individuals who most frequently perform process activities and who would therefore have knowledge of threats to the process. The EHR could also identify process activities where clinical perception of the provider who performs an activity may differ from reality, and thus result in an overlooked perspective in the FMEA.

Objective

Using FMEA as an example quality improvement approach, this study’s objective was to evaluate whether secondary use of clinical documentation collected by the EHR during daily practice can reveal additional detail about a high-risk process and enhance the accuracy of process maps used to guide improvement. To achieve the objective, we: (1) examined the discrepancies between expected and observed activities and individuals involved in a high-risk process and (2) devised diagnostic measures for understanding discrepancies that may be used to inform perspectives represented in an FMEA.

This study used discharge from an inpatient cardiology unit as an example high-risk process that would be suitable for improvement using the FMEA approach. While this study used FMEA as an example, the strategy described may have relevance to other quality improvement approaches that utilize process maps.

Methods

Data source and variables

Data for this study was derived from the Northwestern Medicine Enterprise Data Warehouse (EDW) for admissions to Northwestern Memorial Hospital’s inpatient cardiology unit between July 1, 2014 and December 31, 2014.

Data selected for extraction was guided by a hand-drawn process map (process map), which was created following typical FMEA process mapping protocols. Following FMEA best practices, we gathered expert clinicians from the unit, including the unit nurse educator and unit director, and created a rough process map for the admission-to-discharge process from the unit. The map included process activities and the providers expected to complete them. We iterated and validated this map with the mock committee until the map was deemed complete and to the desired level of detail. We then observed how users interacted with EHR system during each map activity and took screenshots of where each step would be documented in the EHR. The resulting process map graphically displayed the unit’s usual care activities and the designated providers who would complete them with corresponding screenshots of each activity’s EHR documentation.

We then extracted data from the EDW for variables identified in the EHR screenshots. We matched each process map activity to extracted data by comparing screenshots for each activity to EDW data generated through the EHR. The EDW data extraction process is described in detail elsewhere. Cardiology unit staff then validated the match between process map activity and EHR data. We consulted with EDW programmers for non-matched activities remaining after cardiology staff validation. The resulting match was then re-validated with cardiology unit staff until process map activities and EHR data were fully matched.

While the process map includes the entire cardiology unit process from admission to discharge, this study only focuses on the discharge process from the inpatient cardiology unit. Standard FMEA protocol requires a defined high-risk process with clear boundaries. Clinical partners in the cardiology unit identified discharge as a high-risk process of interest to them that aligned with broader hospital initiatives. Cardiology unit staff identified the discharge activities from the process map to include in this study. Figure 1 displays the process map for the entire workflow from admission to discharge on an inpatient cardiology unit, as well as the process map with discharge activities used in this study isolated.

Figure 1.

Process map displaying the workflow for discharge activities on an inpatient cardiology unit. MD indicates physician, and APC indicates advance practice clinician.

The final dataset used in this study included all activities recorded in the EHR and extracted from the EDW that mapped to discharge-related tasks. For each activity, the dataset included the patient encounter in which the activity occurred and the provider who performed the activity with their provider category.

Analysis

For clarity and consistency, we will use the following terminology throughout the analysis. Encounter refers to the inpatient admission where the workflow was performed. Activity refers to each individual action performed within the discharge workflow. Activity type refers to the classification of those activities as they relate to the workflow. Provider refers to an individual who performed some activity in the discharge workflow. Provider category refers to the professional classification of individual providers.

We calculated Experience as the primary outcome and Diversity, Discordance, and Inaccuracy as secondary outcomes. We computed study outcomes for the entire discharge process and for each activity in the process. Whole process outcomes examine all discharge-related activities as a whole, while per-activity type calculations examine the outcome for each individual activity type in the process. Taken together, the whole-process and per-activity type calculations identify discrepancies between expected and observed activity types and provider categories involved in the process and identify the presence of individual providers who may possess an important but missing perspective on the process. The outcomes can be considered as whole-process and per-activity type diagnostics for comparing process map-based expectation to EHR-based observation.

The primary outcome, Experience, identifies individual providers who performed an activity the most frequently, which indicates they may be the most familiar with a process and its threats. The whole process Experience outcome represents the number of times each individual provider engaged in any activity. Per-activity type outcomes indicate the presence of an individual provider whose frequency performing the activity far exceeded their peers.

Diversity examines the range of different provider categories involved in the process and in each activity type to identify whether provider categories were omitted from the process map. Diversity was calculated by comparing provider categories identified by EHR data to those identified in the process map. Whole process Diversity is presented as the percent increase in provider categories identified by the EHR in comparison to those listed anywhere on process map. Per-activity type Diversity reports the number of different provider categories represented by individual providers who performed the activity type.

Discordance was calculated to determine how closely expectation of provider category completing an activity type aligned with EHR-based observation of who completed that activity type. EHR data was used to identify the frequency of times each activity type was completed by each provider category. Provider categories were then classified as “designated” or “non-designated” for each activity type based on the provider category identified as completing each step in the process map. Whole process Discordance was calculated as the percentage of activities across all activity types performed by an individual provider from a non-designated provider category while per-activity type Discordance reports that calculation for each activity type.

Inaccuracy shows where the EDW-based observation of the process most differs from process map-based expectation. Using the per-activity type Discordance calculation, we determined whether the individual provider who most frequently completed that activity type was of the designated provider category. Whole process Inaccuracy is the percentage of steps where the most experienced provider was from a non-designated provider category. Per-activity type Inaccuracy identifies whether the most experienced provider for that activity type was from a non-designated provider category.

Each diagnostic measure reveals different information about the process map. Discordance and Inaccuracy describe to what extent expectation and observation differ, while Experience and Diversity indicate how they differ. While Discordance identifies the rate at which unexpected provider types perform an activity, Inaccuracy indicates the activity types where expectation most differs from observation. Activities rating high on Discordance and Inaccuracy would be the activities where the extra information added by EHR data may most differ from the information on the process map. For those steps, the Experience and Diversity could then be consulted to determine which perspectives are missing.

Results

Descriptive

The final dataset included 34 939 activities across 2222 encounters. Providers completed 16 (sd = 13; range: 1–200) average activities per encounter. Baseline characteristics of patients served during the analyzed encounters can be found in Supplementary Materials.

Twelve activity types from the process map matched eighteen activity types recorded by the EHR. Table 1 describes the total frequency of process map activity types documented in the EHR and indicates the designated provider category, or the provider expected to complete the activity type, for each.

Table 1.

Total frequency of process map activity types documented in electronic health record with designated provider category indicated

#	Process Map Activity Type	Designated Provider Category	EHR Action Type	Freq.
01a	Cardiac behavioral medicine	Nurse	Note	292
01b	Nutritional therapy assessment	Nutritional Therapist	Form	889
			Note	650
01c	Physical therapy assessment	Physical Therapist	Order	741
			Form	371
			Note	2895
01d	Occupational therapy assessment	Occupational Therapist	Form	168
			Note	1400
01e	Case management assessment	Social Worker	Form	501
			Note	900
02	Discharge determined	Attending/Advance Practice Clinician	Order	4344
03	Follow-up visit scheduled	Physician Referral	Order	6260
04a	Nursing discharge note	Nurse	Order	2129
			Form	2143
			Note	4294
04b	Creating discharge instruction	Attending/ Advance Practice Clinician	Note	2695
04d	Discharge summary	Attending/ Advance Practice Clinician	Note	2191
04e	Medical discharge instructions	Nurse	Form	2076

Overall, process map and EHR activities matched, however EHR data separated certain activity types into their sub-tasks. For example, the process map simplified each activity type as a single action performed by a single individual, while the EHR revealed additional detail in the process by dividing most activities into separate actions that could be performed by different individuals. EHR-identified actions included: an order, where an individual provider requested an activity to be performed; a form, where a provider recorded discrete documentation during an activity; and a note, where a provider recorded narrative documentation. Some activity types were performed multiple times by multiple individual providers during an encounter, while others may not occur for every encounter.

Whole-process Outcomes

The Experience calculation identified that 569 unique individual providers performed activities for the encounters included in the analysis. On average, each individual provider performed 61 activities (sd = 160; range: 1–1177). However, certain individual providers performed activities at a much higher volume than their peers. The distribution of experience was highly skewed, with 90% of providers completing between 0 and 99 activities. Figure 2 presents figures depicting each distribution of experience for providers completing 100 or more activities and the number of unique individual providers that performed discharge-related activities by their provider category can be found in Supplementary Materials.

Figure 2.

Redrawn process map of discharge-related activities on an inpatient cardiology unit with diagnostic characteristics indicated, as determined by data extracted from the institution’s Enterprise Data Warehouse. New activities not identified in the process map are noted in yellow. Highlighted diagnostic measures indicate where observation differs from expectation.

The whole process Diversity calculation identified a 171% increase in provider categories from 7 identified in the process map to 19 identified by EHR data. The 12 provider categories in the EHR who were not identified in the process map included four classifications of physician, two classifications of mid-level practitioner, three classifications of medical Resident or Fellow, student nurses, schedulers, and medical students. While the additional provider categories identified by the EHR were not all clinically relevant during daily practice, they represent perspectives that may be missed during an FMEA based on the process map alone.

Discordance analysis identified that providers from categories other than that designated on the process map completed 35% of activities, while providers from the designated category completed 65% of activities. For 39% of activity types, providers from a non-designated category completed the activity more often than providers from the designated category. However, the process map-designated provider category was completely accurate for 44% of activity types, meaning EHR data never showed an individual of a non-designated provider category completing the activity.

Per-activity Outcomes

We calculated Experience, Diversity, Discordance, and Inaccuracy for each activity type. The four outcomes showed differences in characteristics across the activity types. Table 2 presents each outcome by activity type, sorted by discrepancy between expectation and observation. High discrepancy activity types were those most frequently performed by non-designated providers, which suggests activity types where the EHR and the process map would most disagree. Low discrepancy activities were nearly always completed by a designated provider, which suggests these activity types would be least influenced by EHR data.

Table 2.

Experience, diversity, discordance, and inaccuracy diagnostics by activity type

Activity Type	EHR Action Type	Freq.	Count of Providers	Experience	Diversity	Discordance (%)	Inaccuracy
High Discrepancy
Medical discharge instructions	Form	2076	147	No	9	100	Yes
Cardiac behavioral medicine	Note	292	4	No	2	100	Yes
Discharge determined	Order	4344	328	No	10	86	Yes
Discharge summary	Note	2191	176	No	10	64	Yes
Creating discharge instructions	Note	2695	139	Yes	8	63	No
Physical therapist assessment	Order	741	180	Yes	9	61	No
Follow-up visit scheduled	Order	6260	165	Yes	10	40	No
Low Discrepancy
Nursing discharge note	Order	2129	158	No	5	1	No
	Form	2143	159	No	4	1	No
	Note	4294	157	No	4	1	No
Occupational therapy assessment	Form	168	16	No	1	0	No
	Note	1400	23	Yes	1	0	No
Physical therapist assessment	Form	371	22	Yes	1	0	No
	Note	2895	28	Yes	1	0	No
Nutritional therapy assessment	Form	889	16	Yes	1	0	No
	Note	650	14	Yes	1	0	No
Case management assessment	Form	501	34	Yes	1	0	No
	Note	900	30	Yes	1	0	No

EHR Action Type: Sub-classification of activity type. Frequency: Number of times activity performed. Count of Providers: Number of individual providers who performed activity type. Experience: Presence of individual whose frequency performing activity far exceeded peers. Diversity: Number of provider categories who completed activity type.

Discordance: Percentage of activities performed by non-designated provider category. Inaccuracy: If highest performer for activity type was of a non-designated provider category.

We then used results for the four outcomes to update the process map to reflect the depth of information gleaned from the EHR, shown in Figure 2. The new EHR-informed process map reflects the process map-identified activity types broken into actions reported in the EHR, including orders, forms, and notes. The re-drawn process map provides a quick visual depiction of variation and exceptions in the workflow.

Discussion

We found evidence that secondary use of clinical documentation collected by the EHR during daily practice reveals information about a high-risk process that can be used to enhance the accuracy of process maps used to guide improvement. We were able to use EHR documentation extracted from our institutional EDW to (1) examine the discrepancies between expected and observed activities and individuals involved in a high-risk process and (2) devise diagnostic measures for understanding the discrepancies and informing FMEA planning.

Whole process analysis identified discrepancies between the process map, which would typically guide an FMEA process, and documentation extracted from the EHR. Experience showed that certain individual providers might be more familiar with the discharge process than others. Diversity showed that a greater number of provider categories are involved in the discharge process than expected. Discordance and Inaccuracy showed that human understanding of those involved in the process may not be wholly accurate, as indicated by activity types most frequently completed by non-designated providers. Additionally, the EHR identified discharge sub-processes that were not included in the process map, such as orders, notes, and forms.

Findings suggest the EHR may provide useful information for verifying and enhancing process maps used in quality improvement. For example, findings suggest FMEAs guided by a process map alone may overlook important perspectives and thus miss threats that may result in patient harm. Overall, findings suggest that clinical documentation may identify discrepancies between expectation indicated on the process map and activities as they occur in daily practice that may help improve the accuracy of an FMEA of a high-risk process.

The highlighted diagnostic measures identified in the re-drawn process map found in Figure 2. indicate how observation differs from expectation and provide information about activity types where additional perspectives could benefit an FMEA. Diversity, Discordance, or Inaccuracy highlighted indicate activity types where HDM-based expectation most differs from EDW-based observation. For example, activity types that require the least specialization demonstrate the greatest Discordance and Diversity. Highlighted Experience identifies activity types that particular individuals performed at a higher frequency than their peers, which suggests the presence of someone whose perspective should be represented in an FMEA. For example, the process map accurately identified the provider type completing discipline-specific assessments such as case management or nutritional, physical, or occupational therapy, but the Experience metric indicates the presence of individual providers who performed that activity far more often than others.

While the re-drawn map that resulted from analysis cannot indicate who has the most important perspective regarding a high-risk process, it can identify where different perspectives may be available. FMEA leaders could use information represented in the re-drawn process map in several ways, including: to inform team creation (e.g., “We should invite the most experienced person to the FMEA for the activities where Experience is highlighted.”); to devise specific questions to ask during the FMEA (e.g., “We should ask under what circumstances each provider type would do this activity given its high Diversity.”); to direct where specific questions should be asked (e.g., “Since this step had high Discordance, we should ask nurses about it since they did it more often than who we thought.”); or to select additional providers to vet a completed FMEA (e.g., “Since this step had high Discordance, we should run our findings past a few more providers of different types.”)

Findings from this study showed that discharge from an inpatient cardiology unit is a multidisciplinary process with multiple components. Even when creating the process map, inpatient cardiology unit staff identified a complex process that required involvement from many different providers of diverse disciplines. Our analysis identified that the process may be even more complex than expected, due to the inclusion of multiple unanticipated individual providers in certain activities.

While focused on FMEA, findings from this study may be applicable to other quality activities that require the creation of a process map, such as developing medication safety board protocols or implementing clinical information system changes. Many clinical processes can be examined through the diagnostics suggested – Experience, Diversity, Discordance, and Inaccuracy – to identify discrepancies between expectation and observation as a way to target improvement.

Limitations

Results from this study should be considered in light of limitations. Our analysis was dependent on our approach to creating a process map. However, mapping processes vary widely in the field and may look different across other institutions that use the FMEA approach. Additionally, we were able to identify providers and activities involved in the discharge process despite what turned out to be gaps in our process map. Our proposed approach only addresses one aspect of the FMEA process and in doing so may not address other known limitations of the FMEA approach.

We performed the study using data about workflows relating to patients admitted to a single unit in a single institution, which may limit the generalizability of our specific findings. For example, other institutions may have a different process for patient discharge from an inpatient unit. However, the study provided a proof of concept for an approach for using clinical documentation to inform FMEA. The metrics used to examine the workflow could be generalizable to other institutions where EHR data is available for analysis.

Conclusion

We developed a novel methodology that can be used to inform quality improvement strategies. Organizations with a clinical data warehouse or readily available data extraction may be able to leverage clinical documentation gathered during daily practice to inform proactive risk assessment of high-risk processes. Such information may be used to strengthen FMEA accuracy and validity by more comprehensively identifying individuals involved in a high-risk process who may have information about threats that may result in patient harm.

Click here for additional data file.