The impact of feedback of intraoperative technical performance in surgery: a systematic review.

OBJECTIVES: Increasing patient demands, costs and emphasis on safety, coupled with reductions in the length of time surgical trainees spend in the operating theatre, necessitate means to improve the efficiency of surgical training. In this respect, feedback based on intraoperative surgical performance may be beneficial. Our aim was to systematically review the impact of intraoperative feedback based on surgical performance.
SETTING: MEDLINE, Embase, PsycINFO, AMED and the Cochrane Database of Systematic Reviews were searched. Two reviewers independently reviewed citations using predetermined inclusion and exclusion criteria. 32 data-points per study were extracted. PARTICIPANTS: The search strategy yielded 1531 citations. Three studies were eligible, which comprised a total of 280 procedures by 62 surgeons.
RESULTS: Overall, feedback based on intraoperative surgical performance was found to be a powerful method for improving performance. In cholecystectomy, feedback led to a reduction in procedure time (p=0.022) and an improvement in economy of movement (p<0.001). In simulated laparoscopic colectomy, feedback led to improvements in instrument path length (p=0.001) and instrument smoothness (p=0.045). Feedback also reduced error scores in cholecystectomy (p=0.003), simulated laparoscopic colectomy (p<0.001) and simulated renal artery angioplasty (p=0.004). In addition, feedback improved balloon placement accuracy (p=0.041), and resulted in a smoother learning curve and earlier plateau in performance in simulated renal artery angioplasty.
CONCLUSIONS: Intraoperative feedback appears to be associated with an improvement in performance, however, there is a paucity of research in this area. Further work is needed in order to establish the long-term benefits of feedback and the optimum means and circumstances of feedback delivery. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://group.bmj.com/group/rights-licensing/permissions.

Systematic review, minimising likelihood of relevant papers being missed.

Detailed extraction of data from studies.

Very few relevant studies in the literature despite the importance of the topic.

Introduction

In light of increasing patient demands, costs and emphasis on safety, surgeons and their outcomes have become the subject of increased expectations and scrutiny.1 Coupled with this, time spent in the operating theatre by surgical trainees is declining worldwide due to regulations that have reduced the legal number of working hours;2 3 this is particularly alarming in light of the now well-established relationship between surgical volume and surgical outcomes,4–6 and recent work has also directly linked intraoperative technical skill to complication and mortality rates.7 Such challenges necessitate increased efficiency of surgical training programmes such that an equivalent or superior level of surgical proficiency can be achieved in spite of the shorter length of time spent in the operating theatre.

One means by which surgical skills acquisition could be enhanced is via the dissemination of feedback on intraoperative performance. Studies in medical students performing basic surgical skills such as suturing, knot tying and basic laparoscopic tasks, have demonstrated that feedback can improve skill acquisition.8–10 Additionally, proficiency in simulated laparoscopic salpingectomy is accelerated in medical students when they receive instructor feedback,11 and feedback improves colonoscopy performance in gastroenterologists.12 Thus, provision of feedback on intraoperative surgical performance to surgical trainees may also be associated with improved performance and/or a more rapid acquisition of skills, and hence formalised feedback should potentially serve as a key component of future surgical training programmes. Although feedback of intraoperative skill and technique can be a common occurrence in the operating theatre, the impact of this on performance and requirements for optimal training have thus far not been reviewed. We therefore conducted a systematic review to evaluate the impact of feedback of technical skill in the operating theatre and in the context of simulation. For the purpose of this study, feedback was defined as the provision of information pertaining to the operator's surgical performance with the aim of improving subsequent performance.

Methods

Data sources and search strategy

The systematic review was conducted in accordance with PRISMA guidelines. A comprehensive search was undertaken to determine the impact of feedback on surgical performance via the Ovid SP interface. The following databases were searched from inception to February 2013: MEDLINE, Embase, PsycINFO, AMED and the Cochrane Database of Systematic Reviews.

We used two different domains of MeSH-terms and key words combined by ‘AND’, and within each domain the terms were combined by ‘OR’. The first domain contained terms related to surgical skill and performance, while the second contained terms related to the impact of feedback. A detailed search strategy can be found in online supplementary appendix 1. The search was limited to English publications with no other restrictions.

Study selection

Two reviewers independently reviewed citations and selected eligible studies based on predetermined inclusion and exclusion criteria. Publications were selected for review if they satisfied the following inclusion criteria: the article was published in a peer-reviewed journal; the article described a study involving surgical patients or simulation; the article investigated the impact of feedback of intraoperative surgical performance; the article used a statistical unit that was patient-focused or procedure-focused. The following exclusion criteria were applied to search results: the article was a conference abstract, editorial, letter, opinion, audit or review; the population studied was non-surgical (eg, pathology, medicine); the article described methods of feedback, not the impact of feedback; the article utilised a medical student population. Two authors (MM, AT) independently examined all retrieved articles for inclusion. Any disagreements over inclusion or exclusion were resolved by discussion between authors. References in relevant papers were also reviewed in order to identify any additional studies that may have been missed by the search strategy.

Data extraction

Thirty two data-points per study were extracted using a predesigned data collection form including: first author, year of publication, study aim, study type, study design (eg, prospective, retrospective, experimental, observational, cross-sectional, longitudinal), study population, population setting (eg, hospital), surgical specialty, surgical procedure analysed, number of surgeons, types of feedback dissemination, content of feedback, frequency of feedback, measured outcomes and interventions following feedback. The full data extraction from the studies can be found in online supplementary appendix 2.

Results

Study identification and selection

Our search yielded 1531 citations, of which 1185 articles were excluded. After detailed evaluation of the 346 remaining articles, three studies remained eligible, which comprised of a total of 280 procedures by 62 surgeons.13–15 A flow diagram of the search results is illustrated in figure 1.

Figure 1

Summary of search strategy for identification of relevant studies.

Study characteristics

All three studies were performed on surgical trainees, one involving live cholecystectomy cases, one involving simulated laparoscopic colectomy and one involving simulated renal artery angioplasty. Two studies were two-armed RCTs (with one arm receiving feedback and the control arm receiving no feedback),13 15 while one study was a three-armed RCT (with one arm receiving expert feedback, another arm receiving non-expert feedback and one arm receiving no feedback).14 The studies included in this review are shown in table 1, and their basic characteristics are summarised in table 2.

Table 1

Overview of studies included and impact of feedback on performance

Reference	Country	Specialty	Surgical procedure	Number of participating surgeons	Total number of cases	Study design
Boyle et al13	Ireland	General surgery	Simulated laparoscopic colectomy	28	5 per surgeon	RCT
Boyle et al14	Ireland	General surgery	Simulated renal artery angioplasty	18	6 per surgeon	RCT (three arms)
Grantcharov et al15	Denmark	General surgery	Cholecystectomy	16	2 per surgeon	RCT

RCT, randomised controlled trial.

Table 2

Basic characteristics of studies included

Study characteristics	Number of studies	References
All studies	3	13–15
Feedback dissemination
Oral	3	13–15
Written	0
Self-assessment	1	13
Feedback contents
Outcomes	3	13–15
Benchmarking relative to peers	1	13
Comparable literature-reported figures	0
Feedback frequency
After each procedure	3	13–15
Video footage utilisation
Assessment participant performance	2	14 15
Dissemination of feedback	1	15

Feedback dissemination

In all studies, feedback was delivered orally after each procedure.13–15 No written feedback was provided in any of the studies and one study required participants to self-assess their performance after each case in addition to receiving oral feedback.13 One study utilised video footage in facilitating feedback.15

Feedback contents

The two examined studies involving simulation provided participants with feedback relevant to the exercise, including standard instrument metrics, procedural time and errors, accompanied with a description of correct methods where necessary.13 14 For the study of live surgery, feedback was facilitated by review of a videotape recording of the operation and a 60 min structured feedback session, during which technical deficiencies and possible errors were covered and instructions for improvement offered.15 One study provided benchmarking relative to peers13 and no studies provided surgeons with comparable data from the literature.

In one study, feedback was provided solely by a single surgeon with significant operative and teaching experience in laparoscopy.15 One study assessed the impact of expert (consultant surgeons) and non-expert (inexperienced surgical trainees) instructor feedback.14 There was no difference to be found between expert and non-expert feedback in all outcomes assessed other than error scores, which were lower when using the Vascular Interventional Surgical Trainer (VIST) error metrics and scoring during the operation (p=0.009), but not so when using a custom, more extensive scoring sheet completed by a single expert on reviewing video footage of the procedures.14

Impact of feedback

All three studies identified improvements in one or more of the outcomes assessed. Table 3 shows outcomes assessed across the three studies with associated p values. In addition, the study of simulated renal artery angioplasty assessed procedure-specific outcomes including contrast volume (mL), fluoroscopic time (seconds), balloon placement accuracy (mm), residual stenosis and lesion coverage (%).14 Of these, balloon placement accuracy was shown to be significantly improved in those receiving feedback (p=0.041).14 Although not reaching statistical significance, contrast volume utilised was 24.9 mL in control group, and 9.55 mL in those receiving feedback.14 While not demonstrated via statistical methods, a smoother learning curve and earlier plateau in performance was noted in the group with feedback.14

Table 3

Key outcomes in included studies

Study	Procedure	Procedure time	Instrument path length	Instrument smoothness	Economy of movement	Error scores
Boyle et al13	Simulated laparoscopic colectomy	–	0.001	0.045	–	<0.001
Boyle et al14	Simulated renal artery angioplasty	Ns	–	–	–	0.004
Grantcharov et al15	Cholecystectomy	0.022	–	–	<0.001	0.003

‘–’=outcome not assessed.

p Values shown are p values for improvement in that outcome in feedback group when compared to control group with no feedback.

Ns, not significant.

Discussion

Our review included three studies assessing the impact of feedback of intraoperative surgical performance. Feedback was consistently found to be a powerful method for improving surgical performance in terms of operative metrics such as error scores and instrument movement metrics, as well as metrics specific to the procedure being undertaken. Feedback could thus represent a simple but powerful means by which efficiency and safety could be improved, thereby allowing for the attainment of surgical skills to a greater level of proficiency and/or in a shorter length of time in the context of training. This is of particular relevance as, with the exception of video and virtual reality simulator training, training methods known to enhance performance in the operating theatre are few and far between.16 17

Only three studies were included in this review, reflecting the dearth of research in this area despite the significant benefits that feedback could bring; there appear to be many studies in the literature that describe how to assess or rate technical skill,18–20 but very few that actually assess how this data should be used. Limitations of this study include the fact that the search was conducted in February 2013, that conference abstracts were excluded and that study quality was not formally assessed.

All three studies were randomised controlled trials, however, two of these three involved simulated, as opposed to live, procedures.13 14 Future studies should look to further assess the impact of feedback related to live surgery such that the broader implications of feedback can be appreciated.

The impact of feedback on long-term skill acquisition was not studied; all studies only assessed surgical performance with between one and five procedures after the first feedback was provided. Thus, studies taking place over a longer time scale are necessary. It is also important to establish the clinical significance of feedback; none of the studies included assessed whether the improvement in technical skill was associated with an improvement in clinical outcomes, although one might suspect it would, particularly in light of recent findings that technical skills rated by experts based on video footage correlate with surgical outcomes.7

In one study, feedback included a review of a videotape recording and a 60 min structured feedback session with a senior surgeon.15 Although extensive feedback sessions have been suggested (but not shown) to be effective,21 provision of feedback in this manner may be resource intensive and hence cost- and time-effectiveness must also be considered. The finding from a study involving simulation that non-expert delivered feedback is still effective14 may broaden options for educationalists and time-pressed senior surgeons, although one must be careful not to implement counter-productive feedback initiatives.

Although there are a few studies on this subject, all studies included in our analysis were randomised controlled trials. Given the consistent benefit of feedback demonstrated, this supports further research on this topic and implementation of structured intraoperative feedback initiatives.

The small number of studies included in this review highlights the need for more substantive research in this area in order to establish the optimum means and circumstances of feedback dissemination such that standardised methods for future widespread implementation can be attained, and future studies should consider the effect of the following study variables

Source (oral/written), facilitator (expert/non-expert), frequency (every procedure/once daily/weekly/monthly) and duration of feedback (months/years).

Surgeon involvement in feedback (either active or passive), standardised means of assessing surgical performance (which may be both generic and procedure-specific), content of feedback, timing of feedback relative to the procedure (intraoperatively/postoperatively), and the opportunities available for discussion, correction and learning.

Benchmarking (relative to peers as well as to literature data) and feedback based on intraoperative recordings reviewed at a later time point.

Other interventions utilised, such as guidelines, education and review of instructional videos. The contributions of these interventions, and the additive effect they may have with feedback on performance and outcomes are poorly understood.

It should also be borne in mind that in some circumstances or when delivered inappropriately, feedback may not be effective; for instance, although a number of studies in medical students have found feedback to improve acquisition of basic surgical skills,8–10 some have failed to find this,22 and the effect of feedback may plateau.23 24 Frameworks have been suggested in order to ensure appropriate dissemination of feedback,25 which is particularly important given the fact that trainees often feel they are provided with inadequate feedback despite senior surgeons feeling their feedback provision is adequate.26 27

In an era of increasing demands and scrutiny of surgeons in which surgical trainees are simultaneously spending less time in the operating theatre, methods to improve the efficiency of surgical performance are needed. The findings from this review suggest that feedback of intraoperative performance is an effective means by which this might be achieved; however, despite the potential impact, there is a paucity of research in this area, and further work is needed in order to establish the optimum circumstances and means by which feedback can be delivered in a time-effective and cost-effective manner.