The evolution of surgical training in the UK.

Surgical training in the UK has undergone major reforms over the last few decades. The focus has shifted from time based training to competency based training programs. This paper discusses the transformation of assessment in surgical training in the UK from the apprenticeship model to a more objective workplace-based assessment model. The paper describes the different milestones during this transformation process and discusses the assessment of surgical and nonsurgical skills in a measurable way; moreover, it highlights the strengths and weaknesses of different assessment tools.

Background

Surgical training in the UK has passed several milestones. For a long time, surgical training and the acquisition of surgical skills relied on a model based on apprenticeship. Young surgeons rotated through a number of units throughout the UK ranging from small units in district general hospitals to larger tertiary centers. During these attachments, surgical trainees were supervised by senior trainees and the consultants running the unit. The surgical career path was both lengthy and ill-defined. Trainees progressed from the old Senior House Officer (SHO) grade to becoming a Registrar and then a Senior Registrar (SR), gaining both theoretical and practical experience along this protracted course of training. The appointment to any of these grades was largely decided according to availability of vacancies for service provision. Although trainees were required to pass the Fellowship of the Royal College of Surgeons (FRCS), progress along the surgical career pathway was dependent on the opinion of the supervising consultants and senior surgeons both through formal and informal reports. Assessment of trainee skills was achieved through observation and informal feedback by the consultant both in theaters and during provision of daily clinical duties in the unit.1 The FRCS examination was an assessment tool to ensure that trainees had the knowledge required to become surgeons, but it did not assess their technical skills. Many other nonsurgical skills considered essential for the practice of surgery in the modern era such as communication skills and team working were not formally assessed. The course of surgical training was protracted, with trainees spending more than 15 years before obtaining a consultant post. Training was devoid of any educational framework or formal assessment of the end product.

Calman training

In 1993, the report of the working group on specialist medical training was produced. The group was led by the Chief Medical Officer for England, Kenneth Calman. The report recommended the introduction of a new certificate of completion of specialist training, the award of which requires achievement of a definitive educational end point. There was a clearly defined curriculum for each specialty, including surgery, set by the respective royal colleges. The postgraduate dean ensured delivery of the curriculum. The report had specific emphasis on improving assessment and feedback. The Record of In Training Assessment (RITA) came to life. This annual review of performance was carried out by members of the Specialty Advisory Committee (SAC). The RITA was not designed to be an assessment per se but rather a review of achievements of the preceding year of training. The purpose of the RITA was to ensure that trainees have achieved the competencies required to progress to the next year of training. Trainees provided their log books as evidence of their operative experience.2

Galsko and McKay agree that log books are essential for assessing the progress of surgical trainees. Log books provide the number of procedures performed by the trainee and the level of supervision in theater but with little emphasis on details and technical skills.3 The log book was later complemented by the operative competency (Op Comp) form; this is completed by the supervising consultant at the end of the training period. It assessed the ability of the trainee to perform an index procedure either independently or under supervision and identifies areas where improvement is required. Unfortunately, the Op Comp form was often filled retrospectively and therefore did not allow trainees to use the feedback given to them in a constructive way to improve on the job.4 It was a subjective assessment and it lacked accuracy and reliability.5

Modernising Medical Careers (MMC)

In 2005, Sir Liam Donaldson, the Chief Medical Officer for England, introduced the Modernizing Medical Careers (MMC) project.6 This was a major shake up to the way postgraduate medical training was conducted in the UK. The introduction of a 2-year foundation program followed by a specialty training programs also marked the start of a new era in evaluation and assessment in postgraduate medical training. These changes were introduced to improve patient care through improving medical education and creating a transparent and efficient career pathway for doctors. As a result, higher surgical training and specialty training in general was shortened and streamlined.6 The implementation of Modernizing Medical Careers was soon followed by the adoption of the European Working Time Directive (EWTD) in the UK resulting in a shorter working week of a maximum of 48 hours.7 The reduction of operative experience caused by these two factors had a significant impact on the technical skills of trainees in some specialties like surgery and Obstetrics & Gynecology. Purcell Jackson and Tarpley8 suggested in their article published in the British Medical Journal that 20,000 hours are needed to train a surgeon. However, in a EWTD compliant 5-year training programme, this is potentially reduced to a maximum of 11,520 hours. This reduction in the operative experience of trainees meant that operative competence could no longer be measured on the basis of experience alone.9

Intercollegiate Surgical Curriculum (ISCP)

To facilitate the transformation of surgical training in the new era of MMC, the four Royal Colleges of Surgeons in UK and Ireland developed a new surgical curriculum called the Intercollegiate Surgical Curriculum Project (ISCP; www.iscp.ac.uk). This was supervised by the Postgraduate Medical Education and Training Board (PMETB) which is now part of the General Medical Council (GMC). The purpose of the new curriculum was to streamline surgical training across all surgical sub-specialties and to develop competence-based training and assessment. This curriculum covered 4 levels of activity:

setting standards for what trainees should know, be able to do, and be committed to;

developing national regulatory systems, informed by these standards;

developing educational resources to support local training programs; and

supporting learning and teaching in both their formal and their informal aspects.

The new curriculum framework has been adopted by the 9 surgical subspecialties in the UK and presents the foundation years and the beginning, middle, and end of specialist training as 4 “blocks” or modules that together comprise “seamless training.” Each module is defined by explicit standards, with clear “way-points” for the development of competent practice. This framework assesses the progress of trainees during different stages of training across four main domains, namely:

clinical judgment;

technical and operative skills;

specialty-based knowledge; and

generic professional skills.

Surgical trainees are required to demonstrate satisfactory progression of competence along all four domains at annual reviews of competence progression (ARCP) which replaced the old RITA.

Given the time constraints within the NHS, training programmes had to be more focused with more supervision and mentoring of trainees to allow them to achieve maximum use of the training opportunities available to them. Equally, there was more support from training authorities for consultants to train juniors. Health professionals across all grades were encouraged to enhance their knowledge and skills in teaching and to engage in the provision of training and teaching in their units.9 The utilization of clinics, wards and theater sessions for opportunistic learning was encouraged and trainees were allowed to share the experience within units. The various forms of assessment were scrutinized improving various kinds of feedback; and developing approaches to multi-professional and team-based learning.10

Workplace-based assessment (WPBA)

The new reforms in postgraduate medical training and the move from the apprenticeship model to structured training programs in all 9 surgical specialties resulted in changes in the way trainees are assessed. These changes were also driven by increasing public concerns about doctors’ performance and patient safety, as well as increased accountability to the public and funding agencies.11–13 The growing need for more formative assessment for doctors in training led to the introduction of workplace-based assessment (WPBA).

The GMC defines WPBA as “the assessment of working practices based on what trainees actually do in the workplace and predominantly carried out in the workplace itself. The key virtue of WPBA is that it considers actual performance in practice, given that competence alone is generally regarded as an insufficient marker of attainment”. A range of assessment methods should therefore contribute to the overall judgment of the trainee; these may include feedback from both patients and colleagues through multisource feedback and patient satisfaction surveys. WPBA integrates learning with assessment, with an increased emphasis on effective feedback as an aid to the learning process. There is strong evidence that well-constructed credible feedback can lead to improvement in performance.14 Feedback is a core element of any formative assessment, as discussed by Sadler15 and Branch.16 It can promote student learning in three ways:17

it informs trainees of their progress or lack thereof;

it advises trainees regarding observed learning needs and resources available to facilitate their learning; and

it motivates trainees to engage in appropriate learning.

The ISCP is a performance-based curriculum based on Miller’s pyramid (Figure 1), integrating knowledge with the day-to-day performance of trainees including the application of the acquired knowledge and skills to solve patient problems and the use of nontechnical skills in the workplace. Thus, WPBAs assess the higher two levels of Miller’s pyramid; performance “Shows how” and the actual demonstration of tasks “Does,” as described by George Miller.18

Figure 1

Miller’s pyramid of clinical competence.

Notes: Miller GE. The assessment of clinical skills/competence/performance. Acad Med. 1990;65(9 Suppl):S63–S67.18

Abbreviations: PBAs, procedure-based assessments; DOPS, direct observation of procedural skills; mini-CEX, mini clinical evaluation exercises; EMQs, MCQs, multiple choice questions.

The ISCP defined the purpose and aims of WPBA, which include:

provide feedback to trainers and trainees as part of the learning cycle;

provide formative guidance on practice;

encompass the assessment of skills, knowledge, behaviour and attitudes during day-to-day surgical practice;

provide a reference point on which current levels of competence can be compared with those at the end of a particular stage of training; and

contribute toward a body of evidence held in the web-based learning portfolio and made available for the Annual Review of Competence Progression (ARCP) panel and planned educational reviews.

There are many types of WPBA applicable to postgraduate medical training. The ISCP requires trainees to complete a range of assessments including: mini clinical evaluation exercises (mini-CEX), case-based discussions (CBD), direct observation of procedural skills (DOPS), procedure-based assessments (PBAs) and 360° appraisals using a multi-source feedback (MSF) tool. The WPBA implementation document produced by the GMC states that WPBA explores all four steps in Miller’s pyramid but provides little evidence to support the reliability of WPBA as an assessment tool.19 Norcini20 suggests that the number of encounters observed the number of assessors and the aspects of performance assessed all impact on the reliability of an assessment tool. Trainees have to be observed doing the same or similar tasks by multiple assessors or by the same assessor in different settings.20 The Joint Committee on Higher Surgical Training (JCHST) requires trainees to complete a minimum number of assessments during each clinical attachment. This is also true of the foundation programme. Beard viewed this as a misuse of WPBA, as most trainees undertook the minimum number of assessments. Infrequent assessments led to WPBAs being regarded as “mini-exams”. Trainees practised informally and only asked for an assessment when they felt confident of achieving a good score. This also puts pressure on the assessors to give a good score, which they usually did. Thus, the main purpose of WBA as an assessment for learning was lost, and it was often regarded as being little more than a tick-box exercise.21

In 2009, shortly after the introduction of WPBA, an online survey of surgical trainees highlighted the dissatisfaction of trainees with WPBA and the fact that they consider them as an administrative burden rather than an educational tool. Of the 539 respondents, 49% described WPBAs as “poor” or “very poor.” Overall, 6% stated that the ISCP had reflected negatively on their training, with 41% reporting a negative effect overall on their training. Only 6% reported a positive effect on training.22 It is important that trainees and trainers change the way they view WPBA to maximize their utili-zation as a learning method within the job. The GMC has recently recommended the introduction of new assessment tool similar to WPBA. They suggest that “supervised learning events” (SLE) should be performed as formative assessments. These are designed to aid the learning and professional development of trainees through reflection and structured feedback.23 Unlike WPBA, supervised learning events will not be used as evidence of summative assessment.

With the exception of multisource feedback (MSF), which does not require training, assessor training is essential if WPBAs are to be utilized appropriately. The aim of training is to improve the quality of constructive feedback and, more importantly, improve the reliability and validity of assessment through a thorough understanding of the individual methods and the standard required for the completion of each level or module of training. The engagement of trainers in formal training courses such as Training the Trainers course organized by the Royal College of Surgeons and deaneries has been poor. In one study, Canavan et al24 showed that only 11.5% participated in the training offered to them.

Procedure-based assessment

The introduction of WPBA in surgical education faced multiple challenges in its early stages. In deciding which qualities of tomorrow’s surgeons need to be tested and monitored, trainers had to choose from a list of technical and nontechnical skills. It was difficult to design a reproducible and objective tool to test for a wide range of skills. While surgical skills can be tested in the operating theater using different validated tools, it was more challenging to test nontechnical skills such as leadership, team work and communication skills.

The performance of surgical trainees in theater is assessed using two methods of WPBA: DOPS for the assessment of basic surgical procedures and PBA for the assessment of more complex procedures. PBA was originally developed by the Orthopedic Curriculum and Assessment Project (OCAP).25 ISCP has adopted PBA as the principal WPBA method for technical skills. PBA was developed based on a structure similar to the two-part Structured Technical Skills Assessment Form (STSF) described by Winckel.26 Part 1 consists of the essential components of the procedure, ie, generic skills (Table 1).

Table 1

Part 1 of the PBA form, assessment of generic surgical skills

Generic skills	N/D/S
Follows an agreed, logical sequence or protocol for the procedure
Consistently handles tissue well with minimal damage
Controls bleeding promptly by an appropriate method
Demonstrates a sound technique of knots and sutures/ staples
Uses instruments appropriately and safely
Proceeds at appropriate pace with economy of movement
Anticipates and responds appropriately to variation, eg, anatomy
Uses assistant(s) to the best advantage at all times

Abbreviations: N, not observed or not appropriate; D, development required; S, satisfactory standard for CCT (no prompting or intervention required); CCT, certificate of completion of training.

Part 2 consists of more specific technical skills that concentrate on certain aspects of a specific operation, eg, laparoscopic cholecystectomy (Table 2).

Table 2

Part 2 of the PBA form, assessment of specific skills in laparoscopic cholecystectomy

Specific skills	N/D/S
Creates a pneumoperitoneum safely (eliminated from simulation)
Safely inserts an appropriate number of ports
Dissects cholecystectomy triangle safely
Safely ligates and divides cystic duct
Safely ligates and divides cystic artery
Carefully mobilizes gallbladder off the liver
Safely extracts gallbladder from a port site

Abbreviations: N, not observed or not appropriate; D, development required; S, satisfactory standard for CCT (no prompting or intervention required); CCT, certificate of completion of training.

PBA forms have been developed and validated for all index procedures across the nine surgical sub-specialities. Trainees are expected to fill in a PBA form each time they perform a procedure and together with their log book, the completed PBA forms help their assigned educational supervisor to make a judgment about their operative skills and the ability to perform a specific operation.

The concept of assessing the technical skills of general surgical trainees by directly observing them completing a task was examined and tested by Beard et al.27 They compared the inter-rater reliability between direct observation and blinded videotape assessment of 33 trainees. They concluded that the assessment of surgical competence by direct observation and video recording is feasible and reliable and that such assessments could be used for both formative and summative assessment. In their article, Pitts et al25 discuss the validity of performance-based assessment within the orthopedics and trauma sub-speciality. They used the OCAP tool to assess the performance of trainees and concluded that OCAP is relevant and easily usable tool and that performance-based assessment is realistic.25 Mar-riott et al28 examined the reliability and validity of PBA by assessing 81 trainees across six different surgical specialties performing common procedures. Seven hundred and forty-nine PBAs were submitted covering 348 operations. They concluded that PBA demonstrated good overall validity and exceptionally high reliability.

In 2001, the National Institute for Health Research funded a large study to assess the skills of surgical trainees in theater.29 They compared user satisfaction and acceptability, reliability and validity of three WPBA methods for assessing the surgical skills of trainees in the operating theater across a range of different surgical specialties and index procedures. The three methods examined were PBA, Objective Structured Assessment of Technical Skills (OSATS) and Nontechnical Skills for Surgeons (NOTSS). Despite having problems with recruitment early during the study, the authors were able to analyze data for over 400 patients. Their results showed that the PBA process is highly reliable for assessing the same index procedure but good PBA reliability for a mix of index procedures can be achieved only by using large numbers of cases and assessors, owing to strong procedure-specific variance in scores. They concluded that PBA has high utility for assessing technical skills in theater whereas NOTSS should be considered as an assessment tool of nonsurgical skills within the assessment framework of surgical specialties. They argue that these methods help assessors in making reliable judgments. PBA/OSATS had high acceptability and user satisfaction for providing feedback. The only limitation of this study was that it was conducted across one city.

Another validated tool in surgical assessment is the Zwisch scale, which was developed by Joseph Zwischen-berger in the USA. It is a simple model that provides both trainees and trainers with specific stages of supervision during an index procedure. This allows for a safe training in a graduated manner. The four stages of the Zwisch model of supervision are:

show and tell;

active help;

passive help; and

supervision only.

George et al30 showed that the Zwisch score can be deployed on a smartphone-based system and used to make reliable and valid measurements of faculty guidance and resident autonomy.30

It can be concluded from the discussion above that a wide range of technical and nontechnical attributes are required to be a competent surgeon. The question remains which of these attributes are to be assessed and what methods of assessment are to be used. In the future, laparoscopic and visual reality simulators may play an important role in the assessment of surgical skills. These methods provide a more objective way of testing surgical skills by eliminating the patient factor. They are currently widely used in training, especially in laparoscopic and robotic surgery, but in the future, more sophisticated software can be developed and implemented in surgical training.

Summary

Surgical training in the UK has undergone major reforms over the last few decades. The long protracted unstructured career path has been slowly replaced by a shorter more structured surgical syllabus. The introduction of the Calman Specialist Registrar grade in 1993 was the first step in these reforms. This was followed by the introduction of the new Intercollegiate Surgical Curriculum Program (ISCP) in 2007. Implementation of this new curriculum led to major changes in the way surgical competency is assessed. WPBAs were introduced as part of the MMC project to ensure that the new competency-based curriculum produces doctors fit for purpose and to promote public trust in the health system. Within this framework, surgical competency is judged by two main tools: DOPS and PBA. PBA was found to be a valid and reliable tool to assess surgical as well as nontechnical skills in theater.