Promoting safety mindfulness: Recommendations for the design and use of simulation-based training in radiation therapy.

There is a need to better prepare radiation therapy (RT) providers to safely operate within the health information technology (IT) sociotechnical system. Simulation-based training has been preemptively used to yield meaningful improvements during providers' interactions with health IT, including RT settings. Therefore, on the basis of the available literature and our experience, we propose principles for the effective design and use of simulated scenarios and describe a conceptual framework for a debriefing approach to foster successful training that is focused on safety mindfulness during RT professionals' interactions with health IT.

Introduction

It is widely believed that health information technology (health IT) can help providers safely deliver the right care at the right time to the right patient and thus should be vigorously embraced.1, 2, 3 However, imperfectly designed and misused health IT can create new hazards in the already complex delivery of health care.4, 5, 6 Suboptimal health IT systems and interfaces can impose additional (and unnecessary) cognitive effort and confusion on health care providers, which increases the potential for patient harm.6, 7, 8, 9, 10 Indeed, there are reports that link patient harm to the suboptimal design and use of health IT,11, 12, 13, 14, 15 including in radiation oncology.16, 17, 18, 19 In fact, information management in health IT has become one of the top patient safety concerns (based on 1.5 million events that were reported to patient safety organizations and a consensus of patient safety experts). Errors in radiation therapy (RT), where most communications and care delivery are essentially completely driven through health IT, are estimated to occur in approximately 5% of the more than 600,000 patients receiving RT per year in the United States. Serious/lethal events occur in approximately 1 in 1000 to 10,000 patients, highlighting the need for improvement.20, 21

Thus, the U.S. government; national associations, including the American Society for Radiation Oncology, American Association of Physics in Medicine, American Society of Radiologic Technologists, American Association of Medical Dosimetrists; and multiple businesses are supporting efforts to optimize the implementation and use of health IT systems (eg, improving interoperability, usability, workflows, and shared learning about safety issues).4, 5, 20, 21 They also emphasize the need to develop and assess innovative educational and training programs to prepare the current and future workforce to safely operate within the health IT sociotechnical system.4, 5, 20, 21

Experts believe that simulation-based training can be preemptively used to yield meaningful improvements during providers' interactions with health IT (eg, procedural compliance, clinical skills, and patient safety)22, 23, 24, 25, 26 including RT settings.27, 28, 29, 30 By allowing trainees to make, recognize, and learn from undesirable behaviors and failures in the simulated training environment, it should be possible to preempt similar unsafe situations and behaviors in clinical environments. Specifically, health care educators have recognized the pivotal role of debriefing in simulation-based training to support the transfer of learning from experiences in the training environment into practice.31, 32, 33, 34, 35, 36, 37, 38, 39, 40 In general, effective debriefing involves a 2-way conversation between the trainee and educators to highlight particularly favorable behaviors/actions and to identify and discuss areas where there is potential for the trainee to improve performance.

For this article, we refer to improvement in performance in terms of safety mindfulness. We define safety mindfulness during RT professionals' interactions with health IT systems (eg, treatment planning and delivery systems, and electronic medical records) as adherence to evidence-based medical procedures including documentation and communication standards while maintaining moment-by-moment appreciation of the potential for latent and active failure pathways (Fig 1).

Figure 1

Consequences of suboptimal safety mindfulness during interactions with health information technology.41, 42

We use the term adherence to emphasize the need for pragmatic improvement efforts, which must be standardized and sustained over time. We use the term moment-by-moment to emphasize the cognitive aspects of safety mindfulness that are involved in monitoring real-time performance, particularly in maintaining awareness of risks and being both willing and able to detect, interpret, and intervene in abnormal and potentially hazardous situations. The premise is that under more standardized systems for doing the work, which encompass the concept of safety mindfulness, failures should interact in a more predictable manner. Thus, a safety-mindful mindset offers the opportunity to operate proactively (eg, something seems wrong here, this seems too perfect to be true) as opposed to reactively (eg, I assumed it was correct, I forgot to communicate), mitigated by using validated cognitive quality assurance (QA) routines and improvement behaviors.41, 42, 43

However, despite the increase in simulation-based training in the medical domain, relatively little research has been carried out to understand how to incorporate, use, and evaluate health IT–specific simulation training. Past research suggests that there are 3 primary barriers to optimal use of simulation-based training.22, 23 Most current training cases that are included in most health IT–related products 1) use overly simplistic data, which differs from real-life patient records that contain an excess of complex and coupled data; 2) are not realistic with respect to the cognitive effort required of providers when they interact with health IT tools; and 3) are focused on the structure and functionality of the health IT tools as the prime anchor for training, rather than on the role and function of those tools in supporting the cognitive tasks the clinician needs to learn and perform.

Therefore, on the basis of the prior literature and our experience, we herein propose principles for the effective design and use of simulated scenarios and describe a conceptual framework for a debriefing approach to foster successful training that is focused on safety mindfulness during RT professionals' interactions with health IT.

Principles for effective design and use of simulation-based training

Our proposed principles for the design and use of simulation-based training build on previous learnings from simulated studies that were performed with radiation oncology professionals27, 43, 44, 45, 46 and general recommendations proposed by scholars.22, 23, 24, 25, 26, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40 The principles also emphasize the need to provide a training experience that requires RT professionals to develop and apply the concepts of safety mindfulness during simulated interactions with health IT systems. Five general principles are proposed:

Simulation cases should be developed by a multidisciplinary team that understands the context of the clinical care that the end-user is expected to provide. Perspectives from the clinical environment should be actively solicited while building cases for simulation. Simulation cases should replicate clinical processes within health IT (eg, treatment planning, treatment delivery, and electronic medical records) with a high level of realism. For example, simulation cases could be designed using an instance of the health IT that has been replicated from the clinical environment. Such an approach would ensure appropriate and realistic functionality and usability of the health IT system. If multiple systems are used clinically, employing all of them might be useful during simulation-based training so that the lessons learned transcend any particular health IT system. As an alternative to simulating actual case studies, one could develop and simulate generic vignettes that represent the essential features of a range of similar clinical problems but do not specifically simulate any actual real-world event.

Development of competence in safety mindfulness should be independent of technology and situation and treated as a required characteristic of the individual that transcends the use of a particular health IT system or specific clinical situations. Individuals who are considered to have acquired a state of safety mindfulness should, in principle, bring that state of mind with them to any future hazardous situations as a component of their professional competence—provided, of course, that the mental state is reinforced and maintained sufficiently and becomes part of the individual's core competence set.

To create situations in which clinicians have the opportunity to develop and practice safety mindfulness, simulated scenarios should range from simple (“bread-and-butter” situations perceived as requiring low cognitive effort, with few unexpected events or unusual or abnormal occurrences; “I have seen/done this before”) to complex (“fire-in-the cockpit” situations perceived as requiring high cognitive effort, including unexpected or abnormal events; “I have never seen/done this before!” “Wow, what just happened!”) cases in which the presence or absence of safety mindfulness would be expected to influence achievement of successful outcomes.27, 43, 44, 45, 46 However, it is important that the simulation cases not be too simplistic or too complex; they should reflect real cognitive effort and performance that is found in clinical environments. For example, Mazur et al created realistic simple and complex simulated scenarios that successfully required lower and higher levels of cognitive effort, respectively (with scores in the ranges of those seen clinically) and better and worse levels of performance of RT professionals. To achieve degradation in safety mindfulness, the simulated cases could include embedded events that realistically occur under standard operating conditions and might go unnoticed under a high level of cognitive effort or when clinicians' thinking is captured by the kinds of cognitive bias, lack of attention to detail, or jumping to conclusions that are associated with what is referred to as System 1 (or “fast”) versus System 2 (or “slow”) thinking47, 48 (eg, cross coverage, rushed jobs, hands-off situations; examples are shown in Table 1).27, 44, 45, 46 This should help educators demonstrate to trainees the underlying need for safety mindfulness in complex systems such as RT in the presence of latent failures (eg, embedded errors).

Table 1

Examples of embedded errors

Type of embedded error	Description of embedded error	Clinical severity if not detected
Documentation	Target lesion location inaccurately documented	Likely none
Usability/functionality	Image registration incorrect	Likely severe
Interoperability	Prior dose not accurately documented	Variable; depending on downstream performance of providers
Workflow	Prescription not approved by physician	None to severe, depending on prescription

Each simulated case should be replicated, with some variations, in multiple instances over time to allow RT professionals to experience changes in cognitive effort and performance and to allow for the multiple debriefing cycles in which pragmatic learning mostly occurs.49, 50, 51 Mosaly et al demonstrated improvements in cognitive effort and performance from repeated simulation cases, representing cross versus regular coverage.

Each simulated scenario should be evaluated based on the elements of the definition of safety mindfulness. The evaluation of adherence to evidence-based medical procedures, including documentation and communication standards, should be at first relatively broad to allow educators to capture factors that contributed to the degradation of procedural compliance. For example, educators should ask trainees about their temporal demands (eg, “Did you feel rushed?”), frustrations (eg, “Were you frustrated?”), overall effort (eg, “Overall, was this scenario difficult for you to complete?”), and perceived performance and procedural compliance (eg, “How do you think you did?”). An application of a formal tool, such as the National Aeronautics and Space Administration Task Load Index could be useful to conduct such initial and broad evaluations.27, 43, 44, 45, 46, 52, 53

Next, a more detailed discussion of the procedures and standards, especially those in question, should be conducted. The overall goal should be to allow trainees to gain the necessary awareness and knowledge of procedural expectations. To evaluate the moment-by-moment appreciation of the potential for latent and active failure pathways, throughout the simulated scenarios, educators must be able to identify and assess issues such as trainees' awareness of risks, their ability to detect and interpret abnormal signals, potentially hazardous situations, and their recognition of situations in which their thinking and decision-making may be prone to the kind of systematic errors of reasoning associated with System 1 (fast) thinking.47, 48 To do this effectively, we recommend recording trainees' interactions with health IT, including data on eye movements, computer mouse movements, and keyboard strokes. Verbalization, either in real time or during a playback of the simulations, may provide powerful insight into trainees' thinking, reasoning, judgements, and decision-making, and perhaps identify situations in which safety mindfulness is suboptimal. Such data are especially useful when discussing cognitive routines and strategies in detail.54, 55, 56, 57, 58, 59

Debriefing tool to promote safety mindfulness

Our debriefing tool is anchored around 2 commonly used educational strategies: learner self-assessment of safety mindfulness and facilitated and focused discussion about safety mindfulness.

Our proposed debriefing tool incorporates examples of scripted language that are specific to the radiation oncology health IT environment for postsimulation debriefing. There is conflicting evidence with regard to the optimal timing of debriefing sessions. In general, scholars recommend that the debriefing sessions be conducted either immediately or relatively soon (eg, within days) after the completion of simulated sessions, when educators and trainees have better recollection of their experiences.38, 39 This requires educators to actively watch and evaluate the trainees' interactions with simulated scenarios in real time. Before the start of each evaluation and debriefing session, educators should consider the level of experience of the trainee, along with their own evaluation and debriefing experience, because these may influence the evaluation and debriefing approach. At minimum, educators must review and be prepared to speak about trainees' strengths and gaps in procedural compliance, and performance related to embedded errors. At the advanced level, educators should be able to speak about the cognitive routines and strategies that trainees used to detect and interpret abnormal signals and potentially hazardous situations. The self-assessment session should start with the following set of generic, open-ended prompts:

“In general, what do you think about the simulated cases?” This question allows learners to express their initial thoughts and feelings.

Next, the educator should invite the trainee to consider sharing 1 or 2 safety events in which the trainee has been personally involved (and perhaps influenced by) in real clinical situations. An option not to share should be offered. Educators should also share with trainees information about safety events that influenced them in their own career. Another option is to highlight some nationally publicized case studies that involved patient harm in RT.16, 17, 18, 19 This can help create a sense of psychologic safety, which is essential during debriefing sessions. To manage time, focusing this discussion on 1 or 2 events can be useful. Finally, the educator could ask trainees to describe in their own words the concept of safety mindfulness during interactions with health IT. The extent to which the provided definition recognizes the core features of safety mindfulness should not be of major concern at this point in time because the next phase is designed to address such a potential lack of knowledge.

Educators should refresh trainees on the scenarios used during simulation-based training. This should be followed by a high-level review of their and other trainees' (in aggregate form) performance. This discussion is a prologue to emphasizing how suboptimal cognitive effort and real-time critical thinking can contribute to degradations in safety mindfulness and thus overall performance. Educators should consider whether critical procedures, communications, and decisions or judgments made by trainees were thoughtful and evidence-based as opposed to, for example, being made unduly quickly (eg, jumping to conclusions with incomplete data, lack of checking, or indications of cognitive biases) and/or without adequate awareness of the context of the current activity (ie, where System 1 [fast] thinking has dominated System 2 [slow] to the detriment of clinical judgment).47, 48, 60 We acknowledge that only skilled educators can perform such comprehensive analyses, especially because thoughtful procedures might be wrong for other reasons. This is where data such as video recordings, verbalization, eye-movement, and computer mouse-click information can be particularly useful to allow educators to better understand trainees' thought processes and strategies.

If possible, educators should acknowledge trainees' objections (when they arise) and build from them to explore trainees' safety mindfulness while trying to normalize and humanize the mutual experience of having made similar errors. For example, if a trainee challenged the responsibility and accountability for a procedural step by saying something like “Someone else should check for that error” or “In the real world, this error would be caught during downstream QA processes,” educators could respond as follows: “Yes, it is true that the downstream QA processes represent another possible layer of safety when they check our work. However, in real life, we know that failures still manage to propagate through the entire QA system for a variety of reasons [existence of latent failure pathways]. We do expect our QA system to catch failures, but we cannot use that as a reason to do our job suboptimally [existence of active failure pathways]. Therefore, it is important for all of us to adopt the mantra of safety mindfulness because we are each an important patient safety layer, and any one of us can prevent the failures from reaching patients.” Educators can also share objective data with trainees on the utility of QA practices because these practices are not as effective as trainees may think they are.61, 62

Deliberations should be focused on the characteristics of safety mindfulness with the premise that although latent and active failure pathways during interactions with health IT are inevitable, they do not need to result in patient harm if they can be detected, understood, and acted upon. We do not use the term inevitable to sound defeatist but rather to realistically emphasize the importance of shared safety mindfulness (across all personnel in the RT workflow) as a cornerstone of a culture that is necessary to achieve zero avoidable patient harm.

Discussion

Despite the continuous growth of health IT system use in RT, there are currently no standardized recommendations and protocols to enhance patient safety during interactions with various treatment planning and delivery systems. Most training is focused on software and hardware functionality and upcoming updates, which is obviously necessary to run daily operations. Simulation-based training focused on safety mindfulness, as outlined herein, offers the potential for progress toward improvements in patient safety.

In this report, we 1) proposed principles for the effective design and use of simulated scenarios; 2) proposed a clarification of the concept of safety mindfulness as it applies to a clinical RT context, including examples of its key characteristics; and 3) described a conceptual framework for a debriefing approach to foster successful training focused on safety mindfulness during RT professionals' interactions with health IT. We acknowledge that rigid adherence to our principles and debriefing approach might not be optimal for every trainee and in some cases could lead to undesirable results, especially if trainees and educators are underperforming and not fully engaged in the training. In reality, educators would adapt the approach and language as they practice and their experience grows. Our proposed principles and debriefing approach offer preliminary structure and guidance. The ultimate goal is for trainees to recognize and understand the importance of safety mindfulness; reflect on the extent to which they possess and use the mental characteristics in their own clinical work; and have the opportunity to experience, develop, and translate safety mindfulness training into clinical practice.

We acknowledge that training on safety mindfulness requires educators to have a particular set of facilitation skills as well as specialist knowledge combined with relentless passion and a drive for patient safety. Without these characteristics, trainees might never fully buy into the underlying concepts. Future research directions could include empiric studies to evaluate the utility of simulation-based training among RT professionals to improve patient safety. Other areas of focus could include topics such as the acquisition and development of debriefing expertise by educators and the role of proposed frameworks and debriefing approaches on overall training quality (Swiss cheese model [Fig 2]; normal accident theory64, 65; information processing theory; and/or other various validated models and techniques from human factors engineering67, 68, 69).

Figure 2

How latent errors (organizational and workspace) might propagate through quality assurance (QA) steps and lead to active error (human error) in a complex sociotechnical system based on the Swiss cheese model.

Conclusions

Simulation-based training can be a powerful tool to help RT professionals recognize, develop, and improve their safety mindfulness skills. The increasing dependence on health IT systems in RT makes this an urgent need. Because much RT work is done in health IT systems, there are ample opportunities to develop high-fidelity, simulation-based training. Obstacles to simulation-based training potentially include relatively high initial development costs (eg, space, technology needs, scenario development, and educator training), buy-in from management and clinicians, compliance with the institutitonal review board/Health Insurance Portability and Accountability Act of 1996 requirements, and a malleable culture.

Initially, simulation-based training could be implemented in real clinical settings with residents and students as targeted trainees and faculty and clinical supervisors as educators. This would require operational and leadership commitment and might be challenging, but it allows for timely knowledge of the potential for improvement in patient safety. However, the psychologic complexity of the concept of safety mindfulness and effective approaches for its development and assessment should not be underestimated. There is a need for good applied research to address these topics. Nevertheless, simulation-based training represents an opportunity for the RT community to further improve patient safety.