Simulation technology for skills training and competency assessment in medical education.

Medical education during the past decade has witnessed a significant increase in the use of simulation technology for teaching and assessment. Contributing factors include: changes in health care delivery and academic environments that limit patient availability as educational opportunities; worldwide attention focused on the problem of medical errors and the need to improve patient safety; and the paradigm shift to outcomes-based education with its requirements for assessment and demonstration of competence. The use of simulators addresses many of these issues: they can be readily available at any time and can reproduce a wide variety of clinical conditions on demand. In lieu of the customary (and arguably unethical) system, whereby novices carry out the practice required to master various techniques--including invasive procedures--on real patients, simulation-based education allows trainees to hone their skills in a risk-free environment. Evaluators can also use simulators for reliable assessments of competence in multiple domains. For those readers less familiar with medical simulators, this article aims to provide a brief overview of these educational innovations and their uses; for decision makers in medical education, we hope to broaden awareness of the significant potential of these new technologies for improving physician training and assessment, with a resultant positive impact on patient safety and health care outcomes.

INTRODUCTION

Irrespective of our clinical specialty or health care profession, we encounter new medical technologies in nearly every facet of modern practice, from diagnostic imaging and laboratory testing techniques to therapeutic devices. The potential for these technologies to improve health care delivery and patient outcomes—as well as the disappointments or failures to deliver these benefits—frequently captures attention in the scientific literature (not to mention the lay press). Perhaps less recognized, however, are the ways that technological advances impact on the fundamental process underlying all clinical practice, that of medical education. Simply witness the widespread use of medical information technology across the continuum of lifelong learning: medical students now view lectures online or via podcasts; residents consult resources stored in personal digital assistants (PDAs) to help make patient management decisions at the point of care; practitioners receive continuing education credits by attending teleconferences broadcast over the internet. Simulations represent another form of technology that medical education has increasingly employed in recent years, and this article aims to provide a general overview of these educational innovations and their uses for training and assessment.

Medical simulations, in general, aim to imitate real patients, anatomic regions, or clinical tasks, and/or to mirror the real-life circumstances in which medical services are rendered. Our discussion here may use the term simulation, which in its broad sense includes any approximation of actual clinical situations (such as mass casualty exercises or standardized patient [SP] encounters), but in keeping with the technology theme (and space limitations) of this special supplement, we will focus more narrowly on simulators, referring to particular simulation devices. These can take many forms and span the range from low to high fidelity, and from devices for individual users to simulations for groups of trainees. A convenient classification scheme groups these various simulators into 3 categories: part task trainers, computer-enhanced mannequins, and virtual reality simulators.1

SIMULATOR TYPES AND FEATURES

Part task trainers consist of 3-D representations of body parts/regions with functional anatomy for teaching and evaluating particular skills, such as plastic arms for venipuncture or suturing. In most cases, the interface with the user is passive (i.e., the device is examined, or procedures are performed on it, with little more than rudimentary responses from the simulator). Although more sophisticated part task trainers may contain computerized components, we nonetheless distinguish them from computer-enhanced mannequins (CEMs) because the latter reproduce not only the anatomy, but also normal and pathophysiologic functions. With CEMs the interface with the user is more often active or even interactive: in the latter case, the simulator response will vary according to user actions (for example, heart rate and blood pressure will change appropriately depending on the dose of a particular drug administered intravenously). Training and assessment using these simulators can focus on individual skills (e.g., ability of a resident to intubate) or the effectiveness of teams (e.g., a “code blue” resuscitation scenario).

Virtual reality (VR) simulations are even newer innovations in which a computer display simulates the physical world, and user interactions are with the computer within that simulated (virtual) world. Existing technologies now allow for very high-fidelity simulations, ranging from desktop computer-generated environments (much like those in 3-D computer games) to highly immersive VR (e.g., CAVE simulations where the user wears goggles and sensor-containing gloves and sits within a specially designed display). Sound and visual feedback are often highly realistic in these simulations, with recent progress in “haptic” (touch and pressure feedback) technology improving the tactile experience as well. Employed most commonly for examination, surgical, and endoscopic procedures training and assessment, we can use VR simulations (like CEMs) to evaluate both individual and collaborative skills.

In all of these examples, the learner is required to react to the simulation as he or she would under real-life circumstances; of course, we realize that the fidelity of a simulation is never completely identical to “the real thing”. Some reasons are obvious: engineering limitations, psychometric requirements, cost and time constraints.2 Nonetheless, technological advancement leading to higher fidelity and increasingly realistic simulators has been a significant contributor to the recent rise in the use of this technology throughout medical education.

FACTORS INFLUENCING USE OF SIMULATION-BASED EDUCATION

This shift to simulation-based training and assessment constitutes a significant departure from the traditional “see one, do one” approach and the customary reliance on real patients for education. In addition to developments in simulator technology per se, other factors have influenced this evolution. Changes in health care delivery (e.g., outpatient management of many conditions for which inpatient treatment was previously indicated, higher acuity of illnesses and shorter hospital stays for patients who are admitted) have reduced patient availability as learning opportunities at academic medical centers; at the same time, resident work hour reforms and changes in staff compensation make it increasingly difficult for both trainees and clinical faculty to balance their service obligations with time for education and evaluation.3,4 Many simulators, by contrast, are ideally suited for independent learning and, thus, can save faculty time. Moreover, unlike real patients who are frequently “off the ward” when instructors and learners arrive to perform their assessments, simulators can be readily available at any time and can reproduce a wide variety of clinical conditions and situations on demand. This transforms curricular planning from an ad hoc process (dependent on finding real patients with specific conditions of interest) to a proactive scheme with great flexibility for educators. In addition, simulators do not become tired or embarrassed or behave unpredictably (as might real, especially ill, patients), and therefore they provide a standardized experience for all.5

Mastery of clinical tasks involving innovative diagnostic and therapeutic technologies, such as those featured in this issue (for example, deployment of medical devices via minimally invasive or endovascular techniques), often follows a steep learning curve; this has obvious implications for patient safety, particularly when novices are performing invasive procedures on real patients. Recent reports6,7 have highlighted the problem of medical errors and the need not only to prevent mistakes by individuals, but also to correct faults in the systems of care.8

Other professions with high-risk performance environments already have successfully incorporated simulation technology into their training and assessment programs. Examples include flight simulators for pilots and astronauts, war games and training exercises for military personnel, and technical operations scenarios for nuclear power plant personnel.9–11 Simulations such as these improve skills acquisition by placing trainees in lifelike situations and by providing immediate feedback about decisions and actions.3 Such simulation-based programs enhance not only the development and evaluation of individual skills, but also effective collaboration in teams and the building of a safety-oriented culture.

Adopting these models in medical education, specialties such as anesthesiology, critical care, and emergency medicine have led the way in using simulation modalities, especially for teaching and testing the skills needed to manage rare and/or critical incidents. Examples of the effectiveness of such simulation-based training include the mastery of advanced cardiac life support skills by Internal Medicine residents,12 and a systematic literature review details other features and uses of high-fidelity medical simulations that lead to improved educational outcomes in multiple domains.13

Closely related to these safety issues are important ethical questions about the appropriateness of “using” real (even standardized) patients as training or assessment resources. Such debate often centers on instructional or evaluation settings that involve sensitive tasks (e.g., pelvic examination) or risk of harm to patients (e.g., endotracheal intubation). Use of patient substitutes, such as cadavers or animals, raises ethical concerns of its own and faces additional challenges (such as availability, cost, and maintaining an adequately realistic clinical environment). Use of simulators, conversely, circumvents most of these ethical obstacles: trainees can make mistakes and learn to recognize and correct them in the forgiving environment of the simulation, without fear of punishment or harm to real patients. At the same time, the educational experience becomes truly learner-centered, instead of focused on the patient, as is appropriate in actual clinical settings.

SIMULATION FOR OUTCOMES-BASED EDUCATION

Finally, to understand fully all the influences driving the increased use of simulation in medical training today, we must consider them within a broader new context: “While student learning is clearly the goal of education, there is a pressing need to provide evidence that learning or mastery actually occurs.”14 This statement reflects a recent worldwide shift in focus toward outcomes-based education throughout the health care professions. This paradigm change derives in part from attempts by academic institutions and professional organizations to self-regulate and set quality benchmarks, but chiefly it represents a response to public demand for assurance that doctors are competent.15 Accordingly, medical schools, postgraduate training programs, hospital and health system credentialing committees, and licensing and specialty boards are all placing greater emphasis on using simulation modalities for the evaluation of competence across multiple domains.16–23 Thus, beyond its scope for teaching and learning, simulation technology offers potential advantages in the realm of clinical assessment.

The new outcomes-based educational paradigm serves as a suitable framework for considering the best applications of simulation technology for testing purposes. The Accreditation Council for Graduate Medical Education (ACGME) in the US describes 6 domains of clinical competence: 1) patient care, 2) medical knowledge, 3) practice-based learning and improvement, 4) interpersonal and communication skills, 5) professionalism, and 6) systems-based practice.24 Evaluators may use simulations to assess various knowledge, skills, and attitudes within these domains.

During a ward rotation for Internal Medicine residents, for example, faculty can test aspects of trainees’ patient care: using a cardiology patient simulator, demonstrate the ability to perform a focused cardiac examination and identify a fourth heart sound or a murmur. We can evaluate medical knowledge: using a full-body simulator during a simulated cardiac arrest, verbalize the correct steps in the algorithm for treatment of pulseless electrical activity. We can assess interpersonal and communication skills and professionalism: during a simulation integrating an SP with a plastic mannequin arm, demonstrate how to draw blood cultures while explaining to the patient the indications for the procedure.

This last example highlights the reality that actual clinical encounters often require practitioners to bring to bear their abilities in multiple domains simultaneously. Formal assessments have traditionally focused on isolated clinical skills: e.g., perform a procedure on a simulator at 1 station in an Objective Structured Clinical Examination (OSCE), obtain a history or deliver bad news with an SP at another station. More recently, very innovative work features evaluations more reflective of real clinical practice by combining simulation modalities—for instance, a trainee must interact (gather some history, obtain consent, explain the procedure) with a female SP, who is draped below the waist, while performing a bimanual exam on a pelvic simulator placed beneath the drape—for simultaneous assessment of both technical and communication skills.25

SIMULATION FOR COMPETENCY ASSESSMENT

Additionally, within any of the domains of competence, we can assess learners at 4 different levels, according to the pyramid model conceptualized by Miller.26 These levels are: a) knows (knowledge)—recall of basic facts, principles, and theories; b) knows how (applied knowledge)—ability to solve problems, make decisions, and describe procedures; c) shows how (performance)—demonstration of skills in a controlled setting; and d) does (action)—behavior in real practice.

Various assessment methods are more or less well suited to evaluation at these different levels of competence; for example, written instruments, such as exams consisting of multiple-choice questions, are efficient tools for assessing what a student “knows”. Conversely, it makes little sense (despite longstanding custom) to test the ability to perform a procedure by writing about it. Rather, for evaluation of those outcomes that require trainees to demonstrate or “show how” they are competent to perform various skills, the ACGME Toolbox of Assessment Methods27 suggests that simulations are the most appropriate instruments.

In the patient care domain, for example, the toolbox ranks simulations among “the most desirable” methods for assessing ability to perform medical procedures and “the next best method” for demonstrating how to develop and carry out patient management plans. Within the medical knowledge competency, examiners can devise simulations to judge trainees’ investigatory/analytic thinking or knowledge/application of basic sciences. Simulations are “a potentially applicable method” to evaluate how practitioners analyze their own practice for needed improvements (practice-based learning and improvement) and, in the realm of professionalism, simulations are among the methods listed for assessing ethically sound practice.27

One of the strengths of simulators for testing purposes is their generally high degree of reliability: because of their programming, simulators consistently present evaluation problems in the same manner for every examinee and minimize the variability inherent in actual clinical encounters. This reproducibility becomes especially important when high-stakes decisions (e.g., certification and licensure) hinge on these assessments. Use of simulators for such examinations is already occurring in several disciplines: for instance, the Royal College of Physicians and Surgeons of Canada is utilizing computer-based and mannequin simulations in addition to SPs for their national Internal Medicine certification (oral) exams,23 and the American Board of Internal Medicine employs similar simulations in the Clinical Skills Module that is part of their Maintenance of Certification Program.28

Numerous published studies offer evidence of validity (usually “face”, “construct”, or “content validity”) for various medical simulators, but whereas determination of these psychometric properties is important, research often has not addressed the perhaps more important question of “predictive validity” (i.e., will performance on a given assessment predict future performance in actual practice?). Only recently have reports of newer simulation devices for testing (e.g., virtual reality systems for minimally invasive surgery29,30) spoken to these considerations that are fundamental to the competency-based education model.

CONCLUSION

Spanning the continuum of educational levels and bridging multiple health care professions, medical simulations are increasingly finding a place among our tools for teaching and assessment. Technological advances have created a diverse range of simulators that can facilitate learning and evaluation in numerous areas of medical education. Simulation technology holds great promise to improve physician training and, thereby, to impact patient safety and health care outcomes in a positive and significant way.