Use of EHR-based simulation to diagnose aetiology of information gathering issues in struggling learners: a proof of concept study.

Learners who struggle with clinical decision making are often the most challenging to identify and remediate. While for some learners, struggles can be directly traced to a poor knowledge base, for many others, it is more difficult to understand the reason for their struggles. One of the main component of effective decision making is access to accurate and complete clinical information. The electronic health record (EHR) is the main source of clinical information and, with its widespread adoption, has come increased realisation that a large fraction of users have difficulty in effectively gathering and subsequently processing information out of the EHR. We previously documented that high-fidelity EHR-based simulation improves EHR usability and, when combined with eye and screen tracking, generates important measures of usability. We hypothesised that the same simulation exercise could help distinguish whether learners had difficulty in knowledge, information gathering or information processing. We report the results of the first three struggling learners who participated in this exercise. In each case, the simulation was able to 'diagnose' the aetiology for the learners' struggle and assist in formulating an appropriate solution. We suggest that high-fidelity EHR-based simulation can be a powerful tool in the standard approach to understanding struggling learners.

Introduction

Identification and training of struggling learners have become one of the most challenging aspects for graduate medical education (GME) training programmes.1 Competencies are commonly measured but may not identify specific deficits in troubled learners.2 Changes in GME common programme requirements and the introduction of Next Accreditation System allow for earlier identification of struggling house staff, providing earlier opportunities for intervention. When troubled learners are identified, much of the process of establishing their learning deficit is inferential rather than pursuant to direct, systematic observation.2 It is particularly difficult to define the reasons learners suffer from poor clinical decision making.3 While in some cases signs may point towards poor knowledge acquisition skills (such as poor standardised examination scores), in reality, the issue may be due to complex and subtle deficits that revolve around the learner’s inability to acquire and integrate pertinent information. This may have patient safety implications: one recent study suggested that selective data gathering or interpretation may significantly increase medical errors and harm.4

These issues are compounded by the use of electronic health records (EHRs). Our group and others have documented the negative impact of EHRs on efficiency, and demonstrated that providers at all training levels have significant difficulty in finding and/or recognising important trends in clinical data.5 6 This is exacerbated by the fact that GME programmes recruit trainees with prior exposure to, and different levels of experience with, multiple disparate EHRs that invariably affects their information retrieval skills and subsequently their overall clinical competency.

We previously described the use of high-fidelity simulation activities integrated with screen capture and eye tracking to study resident–EHR interactions,5 7 which led us to hypothesise that we could employ our EHR simulation to identify and assist learners struggling with clinical efficiency and decision making, and specifically assist in delineating whether the learners’ issues lay in suboptimal data retrieval/interpretation and/or clinical knowledge deficits.

Method

The study was approved by the Oregon Health & Science University Institutional Review Board. All data were de-identified and stored securely.

Troubled learners were identified by their Programme Director (PD) and enrolled in this study in a purposive fashion to participate in a high-fidelity EHR use simulation. All simulations were conducted by a member of our study team on a dedicated mobile desktop computer with an attached Tobii Pro X120 eye tracker affixed to the bottom of the monitor (Tobii Systems, Danderyd Municipality, Sweden). This system does not require subjects to wear specialised glasses and is reliable in a variety of lighting conditions with an optimal sampling rate at a user distance of 65 cm. At this distance, the viewing angle is up to 30° with an accuracy of 0.3°.

In addition to the dedicated eye-tracking and screen-tracking software (Tobii Studio), the computer contained the same software package as clinical workstations including access to a dedicated simulation instance of our institution’s EHR (EpicCare, Epic Systems, Madison, Wisconsin, USA) which featured full EHR functionality and all user-specific customisations. The simulation instance was populated with validated medical intensive care unit (ICU) charts that featured rich multiday clinical data, including multiprofessional documentation. Cases were designed to assess both efficient and effective data extraction from the EHR, with effective use being assessed by recognition of a series of built-in patient safety issues and dangerous trends in the patient’s condition, medication errors and failure to adhere to best practices, and a full detail of these safety issues and their performance characteristics have been previously published.5 7 Learners were provided a written sign-out on the patient and then given 10–15 min to review the EHR in the context of their assuming care for the patient in preparation for daily rounds. Screen capture and eye tracking of the subjects’ navigation were recorded using Tobii Studio (Tobii Technologies) to document their EHR usability patterns and guide debriefing.8 After review of the chart, the learner presented the case and was assessed for the number of safety issues identified. Subsequently, we reviewed screen use and eye-tracking patterns with the learners using visualisation maps showing where specific gaze fixations occurred on the EHR screen in a ‘talk aloud’ session to qualify the learners’ thought process regarding data visualisation, interpretation and EHR use. Learners completed a brief survey (see online supplementary table 1) after the sessions to gather feedback, which was analysed to identify themes using a constant comparative method. Exemplar quotes were identified, which are highlighted in the results section below. After the talk aloud session and data analysis, a detailed report was provided to the PD to assist in the development of a personalised learning plan, with primary input from the study authors, that emphasised EHR best use practices specific to the deficiencies identified during the simulation.

Results

Subject #1

This subject was nearing completion of his Intern year in internal medicine (IM). His PD stated:

This is an intern who is struggling with appropriate collection of data from his chart reviews

During the simulation, the subject only identified 14% of relevant safety items. Analysis of EHR navigation patterns revealed that while the subject used screens appropriate for data gathering, there was failure to identify the significance of the data observed on the screen. In essence, the subject would preferentially navigate to screens and hope that relevant data would ‘stand out’ as opposed to having a dedicated plan for searching for specific clinical data items. However, once the information was highlighted, the subject made appropriate decisions, implying adequate knowledge. Subsequent to the simulation debriefing, the learner stated:

I think the biggest gap in my current system is that, as we saw, I tend to review data without much of a critical eye, and as a result I fail to recognize when there are data points of concern. If I could somehow come up with a system that forces stopping points to integrate & interpret data I think I’d keep a much better handle on patients' active issues…

As part of their personalised learning plan, the subject was instructed to create specific templates to facilitate a systematic approach to data collection, focusing on pursing specific data elements within the EHR. Currently, the subject is performing well with no further problems.

Subject #2

Subject was a neurology intern in his preliminary year, with extensive prior exposure to IM services, including the ICU. The PD was concerned about problems with basic medical knowledge and medical decision making.

This subject recognised nearly 78% of the safety items during the simulation. However, while the subject could recognise important trends, this learner’s differential diagnosis was extremely limited, suggesting deficits in medical knowledge.

In this scenario, the personalised learning plan focused on improving general clinical knowledge and recommended participating in additional case-based simulation focused on critical decision making. Unfortunately, the learner continued to struggle with cognitive reasoning despite remediation and resigned from the programme shortly after. The PD noted:

He could find data OK [but] he could not integrate it and that he never developed the core clinical skills in medical school he needed to function as an intern.

This example underscores the premise that some learners may struggle with issues linked to cognitive concerns as opposed to issues with the EHR itself.

Subject #3

This subject was halfway through his preliminary IM internship year. The PD stated his primary issue was that he frequently missed significant and relevant data elements, despite spending an inordinate amount of time in data collection.

During the simulation, the subject recognised 42% of safety issues. However, on review of his navigation, it became apparent that his use pattern did not include viewing EHR screens used by the majority of his house staff peers (as identified by analysing over 200 recorded simulations with peer users); instead, the subject used screens seldom viewed by his peers which also contained significant blind spots in the presentation of critical data elements.

This suggests that prior EHR use patterns (in this case, the learner had used a different EHR in medical school) can result in information retrieval habits that may be detrimental when applied to another EHR. As part of this subject’s personalised learning plan, we provided a script that encouraged the use of high-yield screens for data collection. The subject has since adopted this script and stated:

I found the EMR simulation very helpful, and I’m grateful for the opportunity to partake in it. It was particularly constructive to receive feedback on how my practices are discordant with standard/optimal use

Discussion

In this report, we demonstrate the use of EHR-based simulation to diagnose the aetiology of perceived cognitive issues and deficient medical decision making in identified troubled learners. Interestingly, we discovered distinct phenotypes of learning deficits using this technique.

First, the simulation allowed us to distinguish between learners with difficulty in data processing as opposed to data acquisition. Second, simulation allows us to distinguish troubled learners who engaged in two distinctly different workflows of selective data gathering that resulted in poor clinical decision making.

Of these, our ability to identify learners who struggle to recognise basic trends in data highlights an extreme example of our prior work.5 Such learning deficits may be resolved by prescribing a structured approach to EHR use facilitating data acquisition with the overall aim of the learner successfully adapting a new information retrieval strategy.

In contrast, selective data gathering can also be pursuant to using EHR screens that only partially display pertinent data, resulting in blind spots in data collection. Learners demonstrating this EHR use pattern require in a different remediation strategy focused on best practices for EHR navigation. Interestingly, the ability of one subject to self-identify their learning issue after participating in the simulation suggests that learners may possess latent insight into their deficits that can be metacognitively self-identified as they participate in EHR simulations.

The model of EHR-centric high-fidelity simulation that we have described may assist GME programmes in effectively diagnosing learner deficits rapidly and facilitate the development of personalised, effective solutions that the learners themselves find useful. We are now working on integrating this model into the global GME assessment process to assist struggling learners in our institution. We plan on using the data gathered from additional struggling learners to in turn inform changes in initial EHR training and system redesign, as well as the creation of electronic tools to better identify struggling learners.

Table 1

Qualitative results from post intervention survey

Question	Learner 1	Learner 3
Was the simulation effective?	Effective in that it posed common clinical challenges in a realistic environment	Yes; it was very helpful for identifying systematic weaknesses in my approach to chart review in Epic.
What did you learn from the simulation?	Degree of inefficiency, difficulty processing information in Epic, unfamiliarity with the clinical scenario presented in the simulation	I suboptimally utilised several summary views, particularly the medication, labs, and vitals accordions. I often searched for isolated lab results that did not display data in context. I also had a tendency to miss vent settings and drips (which are not consistently visible in different Epic activities).
Did your use of the EHR change in any way after the simulation? If so, How?	Made me actively seek out further epic training to increase my efficiency	I developed a habit of more frequently referring to the data views in ICU accordion, which are great for systems-focused review.
Have you noticed a difference in patient outcomes after the simulation?	As a result of the above, I’m able to more effectively comb through chart history, assess trends in clinical data. Unclear if this has changed outcomes as there is no metric to evaluate avg LOS, hospital mortality, readmission, etc of patients with whose care I’ve been associated	Yes - fewer misses in changes of titratable infusions - greater appreciation of vitals and lab trends over time - faster and more systematic prerounding and preadmitting in general
Have you noticed a change in your performance and/or evaluation of your performance since the simulation?	Meteoric rise in efficiency stemming from further training and~2 years of extra experience. Lower stress, markedly less frequent ‘misses’, increased overall awareness of patients’ history and current trajectory.	Subjectively, yes. No formal feedback yet available.

Note: learner 2 left the institution before the survey could be administered.