Metric-based simulation training to proficiency in medical education:- what it is and how to do it.

High profile error cases and reduced work hours have forced medicine to consider new approaches to training. Simulation-based learning for the acquisition and maintenance of skills has a growing role to play. Considerable advances have been made during the last 20 years on how simulation should be used optimally. Simulation is also more than a technology learning experience for supplanting the traditional approach of repeated practice. Research has shown that simulation works best when it is integrated into a curriculum. Learning is optimal when trainees receive metric-based feedback on their performance. Metrics should unambiguously characterize important aspects of procedure or skill performance. They are developed from a task analysis of the procedure or skills to be learned. The outcome of the task analysis should also shape how the simulation looks and behaves. Metric-based performance characterization can be used to establish a benchmark (i.e., a level of proficiency) which trainees must demonstrate before training progression. This approach ensures a more homogeneous skill-set in graduating trainees and can be applied to any level of training. Prospective, randomized and blinded clinical studies have shown that trainees who acquired their skills to a level of proficiency on a simulator in the skills laboratory perform significantly better in vivo in comparison to their traditionally trained colleagues. The Food and Drug Administration in the USA and the Department of Health in the UK have candidly indicated that they see an emergent and fundamental role for simulation-based training. Although a simulation-based approach to medical education and training may be conceptually and intellectually appealing it represents a paradigm shift in how doctors are educated and trained.

Background

During the Annual meeting of the American Surgical Association (ASA) in April 2002 at the Homestead (VA, USA) researchers from Queen's University Belfast and Yale University (USA) reported the results from the first prospective, randomized, double-blinded trial of virtual reality (VR) training for the operating room (OR). Surgical residents randomized to the VR training arm of this study subsequently performed the actual dissection of the gallbladder from the liver bed portion of a laparoscopic cholesystectomy (LC) 26% faster than traditionally trained surgical residents and made six times fewer objectively assessed intra-operative errors1. This study (or VR to OR as it has become known2) was important because for the first time it unambiguously demonstrated the potential of (VR) simulation as a powerful training methodology for the acquisition of procedural skills outside the OR which directly impacted on in vivo OR performance. Previous studies had compared VR training to no, or traditional approaches to training (i.e. on real patients), but these studies were conducted wholly in the skills laboratory and not on real patients.3-7

Although simulation had been used in other industries such as aviation for decades8 and in medical disciplines such as anaesthetics for many years9 its potential as a training device in surgery and procedure-based medicine was not taken seriously until a series of high profile and impactful events forced medicine and surgery to consider a new way of training. High profile error cases in the UK10 and the USA11 as well reduced work (and training) hours12, n forced medicine to consider new training paradigms. Bad experiences following the introduction of minimally invasive surgery (MIS) ensured that surgeons were already sensitised to the need to improve training.. The introduction of MIS, particularly LC, was accompanied by an increased frequency of complications, many life-threatening, particularly during the early experiences.14 That these problems could occur when experienced surgeons, well versed in open techniques and with knowledge of anatomy and pitfalls embraced new techniques, heightened concerns about the training of novices who lacked such a background in open surgery. But the agenda was now set, surgery needed to develop new methods for training the novice in surgical techniques in general and for training experienced surgeons in the newer techniques.15

Dr. Richard Satava, a US army general surgeon first proposed the idea of VR simulation for the acquisition of surgical skills in the early 1990s whilst on secondment to the Defence Advanced Research Projects Agency (DARPA).16 Until the VR to OR study results were reported his proposal was at best considered ‘eccentric’ and VR simulation a technology of marginal significance. In March 2002 at a closed door meeting in Boston College, hosted by Dr. Gerry Healy (President of the American College of Surgeons 2007 - 2008) and the American College of Surgeons (ACS) and in light of the imminent VR to OR study report at the ASA, the decision was taken for the ACS to ‘champion surgical simulation'.17 The impact of this brave and enlightened decision was enormous. True to its word the ACS championed simulation based training which has led to the establishment of Accredited Education Institutes (within the USA and globally) of which simulation is an important pedagogical component.18

What is Simulation?

Simulations have been given many definitions over the years but it is fair to say that ‘simulation’ is usually thought of as VR. In 1993 Satava16 originally proposed that “Virtual reality [simulation] is a fully three-dimensional computer-generated “world” in which a person can move about and interact as if he actually were in this imaginary place.”... “A world can be anything from a kitchen to an automobile to an abdomen ∼ anything which can be drawn can be experienced three dimensionally”. In contrast, high fidelity simulations in disciplines such as anaesthetics utilized very realistic materials and equipment to represent the task(s) that the candidate must perform19. Anaesthetic simulations centred on computerized, interactive, life-sized manikin that could be programmed to provide realistic patient responses and outcomes. Low fidelity simulations would be ones in which the candidate is presented with a verbal description of a hypothetical work situation and then asked to describe how he/she would deal with the situation rather than having the candidate perform the actions he/she would take.20 Thus, simulation fidelity has been construed by trainers and educators as the degree of similarity (and apparent technical sophistication) of the simulation to the real world situation that was being trained. Inanimate simulation tasks such as animal tissue would be low fidelity and increasing sophistication and or face validity of the simulation represented increasing simulation fidelity. This issue will be returned to when the concept of metrics is discussed. It suffices to state at this point that both of these views of simulation fidelity are now incomplete in light of a more sophisticated and comprehensive theoretical understanding of what a simulation is and how it is configured and implemented for the efficient and effective learning of skilled performance.

What is Skill?

One of the most important functions of a simulation is to facilitate the effective and efficient training of skill outside the clinical situation thus minimising the risk to the patient from at least part of the novice's learning curve. But what is skill? Failure by medicine to explicitly answer this question has been one of the major impediments to the development of good simulations and simulation-based training. When United States Supreme Court Justice Potter Stewart was asked describe his threshold test for obscenity in the case of Jacobellis v. Ohio (1964) he infamously ruled that ‘he knew it when he saw it'.21 Similarly, most doctors have an opinion as to what skill is but few can robustly define it. A parsimonious definition of skill might be ‘it is what skilled individuals do'. However, this definition does not advance a specific testable model of skilled performance that could be used to characterise performance. In contrast, psychologists have tackled the same problem by subjecting the ‘skill’ to be characterised to a detailed task analysis and then operationally defining (not describing) important aspects of performance which constitute skill.15,22 They then quantitatively validate whether their characterisation fits with what is known about the skill they have analysed. Do more skilled individuals perform better on their assessments than less skilled or experienced individuals (construct validity)? Do individuals who perform well on their evaluations also perform well on a variety of similar and related tasks (concurrent validity)? Do their assessments predict future skilled performance (predictive validity)? These task-analysis derived characterizations of skilled performance do not have to capture every aspect of performance but should at least allow for ordinal differentiation between different levels of performance as described by Dreyfus and Dreyfus23 and summarised in Figure 115.

Fig 1

Dreyfus & Dreyfus Figure23

What are Metrics?

Based on the task analysis process outlined above, the units of performance that have been identified (and validated) as integral to skilled task performance are the metric units of task execution. This means that these performance units should be used to define and shape the configuration of any simulation developed to train skilled task performance. Metric units must be unambiguously defined so that they can be scored as occurring or not occurring. These metric units should capture the essence of procedure performance and might include the steps that the procedure should be performed in, the instruments used and what should be done with them. Crucially, the metrics should also describe for each procedure step what should not be done thus characterising performance that deviates from optimal performance (or errors).24 Metric errors are some of the most important performance units for simulation based training.15 Training should concentrate on what should be done and the order in which it should be done but it should also target performance errors for at least reduction, preferably elimination. This means that operational definitions of performance units or metrics need to be unambiguous. For example, table 1 shows the operational definitions of metrics for the dissection of the gallbladder from the liver bed portion of a LC including a start and end point procedure markers.1 They unambiguously ‘define’ rather than describe when each metric error has occurred. This approach considerably facilitates the reliable scoring of metric-based performance units across a variety of functions from skills training25-28 at different experience levels.29,30 It has also been shown that this approach works well as part of the process for selection into higher training31 and considerably enhances assessment reliability levels in comparison to Likert-scale assessments.32

Table 1

Metric Errors

Metric errors/criteria of injury assessment	Operational Definition
Procedure START	first contact of diathermy with tissue
Procedure END	last attachment is divided
FAILURE TO PROGRESS	No progress made in excising the gallbladder for an entire minute of the dissection. Dealing with the consequences of a predefined error represents lack of progress if no progress is made in excising the gallbladder during this period.
GALLBLADDER INJURY	There is gallbladder wall perforation with or without leakage of bile. Injury may be incurred with either hand.
LIVER INJURY	Necessitates capsule penetration and may have bleeding.
BURN NONTARGET TISSUE	Any application of electrocautery to non-target tissue with the exception of the final part of the fundic dissection where some current transmission may occur.
TEARING TISSUE	Uncontrolled tearing of tissue with the dissecting or retracting instrument.
INCORRECT PLANE OF DISSECTION	The dissection is conducted outside the recognized plane between the gallbladder and the liver (i.e. in the sub-mucosal plane on the gallbladder, or sub-capsular plane on the liver).
INSTRUMENT OUT OF VIEW	The dissecting instrument is placed outside the field of view of the telescope such that its tip is un-viewable and can potentially be in contact with tissue. No error will be attributed to an incident of a dissecting instrument out of view as the result of a sudden telescope movement.
ATTENDING TAKEOVER	The supervising attending surgeon takes the dissecting instrument (right hand), or retracting instrument (left hand) from the resident and performs a component of the procedure.

In discussions with different groups of physicians and surgeons from around the world there appears to be a consensus appears that reaching agreement on performance metrics is all but impossible. This may however, only apply to agreement on ‘everything'. The majority of doctors very experienced and proficient in the performance a specific procedure can very quickly identify and agree what should be done, how it should be done and what most certainly should not be done for most parts of a procedure. The problem is that doctors rarely think about procedures in that level of detail. Practitioners who are ‘proficient’ in the performance of a procedure (with average to good outcomes) will already exhibit many if not all of the important performance characteristics to perform the procedure well. They have however, automated to many of these and how it is they are performed (much like the complex skills required to ride a bicycle). A primary function of the task analysis process is to identify and define what these performance characteristics are. This should be done initially fora ‘reference procedure', i.e., a straightforward procedure that can be performed without complications or deviations under ideal circumstances. (An optimal approach to learning should ensure that trainees are capable of performing an uncomplicated procedure before they have to deal with procedure variations). The task analysis should seek consensus (not necessarily agreement) between procedure experts on the characteristics of the reference procedure and instruct them to characterise the reference procedure and not unusual or interesting variants of it. Procedure performance should be guided by a) professional guidelines, b) manufacturer guidelines on device usage and c) results from empirical studies. In the absence of a consensus between the experts on the items (a) to (c) above then individual procedure practices that they may have developed from years of practice wisdom should be employed. Errors are defined as procedure actions which deviate from optimal practice and are not necessarily bad but are potentially unsafe. Critical errors in contrast are procedure actions which are most certainly unsafe but may not always lead to a bad outcome.15 The underlying philosophy of this approach is that bad outcomes do not happen by accident but usually from the coalescence of deviations from optimal procedure protocol.33 The task analysis stage of the development of a simulation is crucial as metrics are the fundamental building blocks of a good training program. Metrics thus define how the simulation should be characterised and performed by the trainee and must afford the opportunity for meaningful performance assessment. Assuming that the metric identification and definition process, simulation operationalizaion and implementation goes well these performance characteristics should be easily validated, as distinguishing between experts and novices (i.e., construct validity) and predictive of acquired skills post training (predictive validity). Other validation processes are necessary but these two are probably the most important for training purposes. The construct validity study will inform the training process which metrics best distinguish between experienced/expert and novic performances and will guide the skills benchmarking process or ‘proficiency level’ which trainees should acquire before progressing to in vivo practice.15,34

Simulation Defined

Equipped with an understanding of the importance of metrics for the characterization and configuration of a simulation, simulation can thus be defined as i) an artificially created or configured ‘learning’ situation that allows for the practice or rehearsal of all or salient aspects of a procedure. The artificial learning situation must ii) provide the span of appropriate sensory responses to learner physical actions that are behaviourally consistent with what would be experienced in real life (including the opportunity to enact both appropriate and inappropriate learner actions (i.e., errors)). The simulation should also afford the opportunity to iii) perform the procedure iv) in the same order and v) with the same devices that the procedure would normally be performed. Crucially, it should also afford vi) reliable and valid metric-based assessment of performance. Assessments must at a minimum vii) allow summative but preferably formative feedback on procedure performance proximate to task execution, particularly for metric errors.

What is Proficiency-Based Training?

Traditionally medicine has trained doctors to be ‘competent’ and this fact has been reinforced by high profile medical errors cases, e.g., The Bristol Case.10 However, what is probably less well known outside of (academic) medicine is that this level of competency is in fact ‘minimum’ competence. Indeed over the years medicine has developed robust statistical processes for the definition of minimum competency levels (Figure 2a).35 As shown in Figure 3a competency definitions can vary considerably between and even within institutions using these statistical methods. The concern that many surgeons, interventional cardiologists and radiologists and other procedurists express is that this level of skill may be too low. Also of concern to them is the variability in competence thresholds. They also express concern about the failure of medicine to define a measured skill level that is objective, transparent and fair. Furthermore, they believe that training practitioners to a level of proficiency23 may be a more conservative but superior approach. They are however faced with the same problem of benchmark definition.

Fig 2a

the Wijen method of competency assessment

The “I know it when I see it” approach does not work well for understanding and operationally denning what skill is but, it will serve more than adequately for helping to define a proficiency benchmark, i.e., proficiency is the level of performance exhibited by proficient individuals. Thus, validated metric-based simulations can serve as benchmarking devices. The philosophy being that individuals who are very experienced at performing a specific task or procedure e.g., a consultant/master surgeon, are at least competent, probably proficient and some will be expert.15, 34 This means that a level of proficiency benchmarked on the mean performance level of these individuals on the validated metric-based simulation is a fairly conservative criterion level. Furthermore, it has considerable face validity. Rather than benchmarking on some abstract performance level reached by consensus in a committee the training pass level is defined on the performance levels of individuals who are actually very experienced at performing the procedure clinically. This does not imply that individuals who demonstrate this benchmark in training have the same ‘performance capacity’ as the individuals on whom the level was established. It simply means that they were able to demonstrate that performance level (e.g.., technical or knowledge or both) on consecutive trials. They will not (yet) have acquired the procedural wisdom of the experienced operator. This approach to training ensures a much more heterogeneous skill set from trainees as ALL trainees must reach AT LEAST the proficiency level (on two consecutive trials) before progressing to in vivo practice (Figure 2b). The other advantage of this approach (summarized in Figure 3) is that it eliminates the issue of number of training trials or time in training before progressing. It also puts the onus on the trainee to demonstrate proficiency before progressing. The onus on the trainer is to provide the facilities and access to training to demonstrate proficiency. The results from this approach to training skills have shown that proficiency-based progression trainees significantly outperform traditionally trained doctors,1, 25,36 even for advanced procedures.28

Fig 2b

Competency Vs Proficiency

Fig 3

The Proficiency-based progression method

What is the Difference Between Repeated and Deliberate Practice?

Medicine is currently grappling with the mechanics of a proficiency-based progression approach to training (e.g., development of metrics, metric validation, proficiency definition etc).15 It circumvents some very sticky problems such as how to define competency? If an individual has demonstrated a proficiency benchmark they must by default have demonstrated competency as proficiency is a more advanced skill level as proposed by Dreyfus and Dreyfus.23 It has also forced medicine to define precisely what is meant by ‘skill’ (for a reference approach at least). Another benefit from the process of defining skill and deviations from optimal performance, i.e., metrics, has been the evolution of an understanding how metrics could be used to define, shape and configure a specific simulation. Thus a simulation becomes a vehicle for the delivery of metric-based training rather than some abstract entity. Furthermore, metric-based simulation training can be configured in such a way as to make training much more efficient and effective. Traditionally skills in medicine have been acquired through repeated practice in the clinical setting. How quickly they learned depended on how frequently they encountered the learning situation, whether they have the opportunity to implement or practice their skills and whether their supervisor had the opportunity (or inclination) to use the setting as a training occasion. Thus, the learning accrued from such opportunities could be very variable. Simulation affords the trainee to acquire their skills in the simulation laboratory in a much more systematic way. It also means that each time the trainee engages with a metric-based simulation their learning can be optimized. Each time the trainee performs the procedure they will receive feedback on their performance. Summative feedback at the end of the procedure will facilitate learning but metric-based formative feedback on performance is a much more powerful aid. This will inform trainees if they are performing the procedure incorrectly or in the wrong order and it should let them know if they are using the devices inappropriately. Trainees should receive this feedback proximate to the performance error. This approach to learning is called deliberate practice.37

The extended, structured and motivated practice by trainees first described Ericsson, Krampe and Tesch-Römer37 were important explanations of learning skilled performance but one crucial aspect of their accounts to training was missing, i.e., performance feedback. In addition to engaging in deliberate practice the trainee musicians studied by Ericsson and colleagues also had access to immediate feedback from their tutor. They would have been informed by their tutor (or recognised themselves) when they played a note wrong. Engaging in frequent practice sessions would mean that the trainee had ample opportunity to practice and rehearse their playing with equivalent opportunities to correct and improve performance. Implicit to this process and presumed performance improvement (i.e., learning) is the concept of performance feedback. The same is true for the acquisition of skill in medicine. This approach to learning is more efficient and effective than the traditional approach of repeated practice and is made possible with metric-based simulations. The pre-requisites for this approach to simulation training are the identification and definition of optimal and sub-optimal performance by procedural experts and the capability of the simulation to model the procedure and operator performance interaction in real-time. It is then simply a matter of benchmarking proficiency using the actual performance of proficient doctors. Thus the experienced doctor is the starting point for the development of a simulation and the quality assurance of it by benchmarking on their performance thus circumventing some potentially complicated, convoluted and thorny issues such as what is competence or how is it defined?

The Future (Now)

Metric-based simulation ensures that training sessions are more than just simulated clinical ‘experiences'. It ensures that there is no ambiguity about the progress of training for the trainee (and the trainer). Simulation-based training in medicine now has a quantitatively validated clinical function. This foothold is considerably better than even a decade ago. So much so that the Food and Drug Administration in the USA now requires simulation based training as part of device approval38,39 and the Department of Health in the UK have issued guidance which appears to suggest that trainees should not be performing a procedure on a real patient the first time they perform the procedure.40 Implementation of simulation based training will continue to evolve but for certain, simulation based training will form a fundamental part of acquisition and maintenance of skills in medicine and healthcare. In time it will also (probably) be part of the General Medical Councils reaccreditation process. It is therefore essential that medicine has a thorough understanding of what simulation is and the imperative of metrics. Quantitative characterization that has been validated for a proficiency-based progression training function is not a million miles away from quantitative assessment of performance by experienced doctors, i.e., re-validation.

Summary

Proficiency-based progression training works and it works because of well proven principles and practices of learning.15 To ensure the optimal effectiveness of a proficiency-based progression training program does not require a radical change in the current curriculum content. However, what it does require is a radical change in how that curriculum is delivered and implemented. Simulation is very powerful training tool for the delivery of deliberate practice coupled to formative and summative metric-based feedback on performance. In the absence of computer generated simulation, formative metrics on training performance need to be delivered by a trainer who is very experienced at performance assessment. A training program that has a clear end point (i.e., level of proficiency) must provide the facilities and opportunities for the learner to meet the level of proficiency. A deliberate practice training regime affords the opportunity for independent pacing of skill acquisition; a coherent curriculum with appropriately sequenced learning material and a variety of learning experiences. Although this approach to medical education and training may be conceptually and intellectually appealing it represents a paradigm shift in how doctors are educated and trained.