Retrospective adjustment of self-assessed medical competencies - noteworthy in the evaluation of postgraduate practical training courses.

AIM: The efficacy of postgraduate practical training courses is frequently evaluated by self-assessment instruments. The present study analyses the effect of a basic course in laparoscopic surgery on self-assessed medical competencies.
METHODS: The 3-day course included teaching of knowledge and training of practical skills. In relation to course evaluation, a questionnaire for self-assessment was applied at the beginning of the course ('pre-course'), at the end of the course ('post-course') and at the end of the course to reassess pre-course competencies ('retrospective pre-course').
RESULTS: 89 out of 110 participants (81%) attending 10 courses completed all the questionnaires; 83% were postgraduate trainees in surgery and 82% were inexperienced as an independent surgeon. At the beginning of the course most trainees rated themselves as 'moderately competent' or 'fully competent' with respect to the various task levels as well as to specific areas of medical competencies. At the end of the course however pronounced retrospective revisions of self-assessment to lower ratings became apparent. Statistically significant differences were seen for the task 'performing surgical procedures under supervision' and for most of the practical skills trained during the course (p <0.01). In contrast, no significant differences were observed for knowledge taught during the course as well as for 'ability to work in a team' and 'ability to concentrate', which were not foci of the course.
CONCLUSIONS: Surgeons with little experience change their self-assessment of pre-course competencies to a lower level after participation in a practical postgraduate training course. Evaluations comparing 'pre-course' and 'post-course' ratings only - without 'retrospective pre-course' ratings - may underestimate the training effects. This phenomenon needs to be taken into account when evaluations are dependent exclusively on self-assessment instruments.

Introduction

Efficacy and the achievement of learning objectives are essential considerations while conducting training programmes for medical competencies [1], [2]. In his model of evaluation, Kirkpatrick suggests evaluating the efficacy of training programmes using four levels. On the first level, the investigator asks for the perception of the participants ('Reaction'). Level two will evaluate the amount of learning that has occurred ('Learning'). Level three will evaluate the transfer that has occurred to the trainees’ everyday environment ('Behaviour') and level four measures the impact of the training to the participants’ professional life ('Outcomes'/ 'Results') [3], [4]. Kirkpatrick's model is being discussed because important factors influencing the learning process are not considered: personality traits, culture of learning and type of teaching aids [5]. Furthermore, no causal relationship between the different levels has been determined to date [5]. In addition, there is a limited amount of data available to support the concept that level four evaluations are more important than those of level 1 to 3 [5]. Nevertheless, using the four-level-model, Kirkpatrick was able to simplify and categorise the complex processes of evaluating training programmes as well as to establish measures of 'Behaviour' and 'Outcome' as important criteria [5]. However, application of these latter criteria is often not feasible due to the complexity of the object of investigation or cost constraints. Thus, training programmes are mostly evaluated on level 1 and 2 using self-assessment instruments.

The ability to assess one's own competence is recognised as a key factor for medical competencies [6]. It is essential for medical professionals to be able to identify their personal strengths and weaknesses, to have confidence in their expertise and to be able to restrict their medical practice activities as appropriate [7]. With regard to the learning process, self-assessment skills are required for the determination of aims and to verify existing skills. However, it has been illustrated in a recent review article that physicians are predominantly unable to assess their abilities accurately. The authors were particularly worried about those professionals with the fewest skills but who overestimated their abilities the most [8]. These observations have been confirmed by other investigations, which revealed that the least competent individuals were unable to assess the competence level of either their own skills or those of other individuals accurately [9], [10], [11]. Self-assessment at the beginning of a training programme may contain an over-estimation of skills due to inexperience [12]. Appropriate training programmes may though improve not only the competence of the trainees, but furthermore self-assessment skills [12], [13]. Therefore, the data from previous investigations suggest that evaluation of training programmes using self-assessment measures may contain systematic errors due the effect mentioned above.

Medical education is traditionally organised as an apprenticeship, but recently more and more training courses are held [14]. Training courses are needed particularly in surgical education because advanced technical skills are required for highly specialised procedures [15], [16]. Several studies have demonstrated the efficacy of practical training courses, especially in laparoscopic surgery [17], [18], [19].

Aim

Using the example of a basic course in laparoscopic surgery, we aimed to investigate whether and how self-assessment scores of medical competencies changed as the participants progressed through the postgraduate practical training courses. Furthermore, we intended to discuss the possible effects on course evaluation measures. We considered the following questions:

How do participants rate

their competencies at the beginning of the course (‘pre-course’),

their competencies at the end of the course (‘post-course’) and

their pre-course competencies at the end of the course (‘retrospective pre-course’)?

Is there an alteration in self-assessment over the course of the training (‘post-course’ versus ‘pre-course’ and ‘retrospective pre-course’ versus ‘pre-course’, respectively) and which medical competencies are affected?

Are there differences in the changes reported between areas of competencies, which are trained and those that are not?

Methods

The present investigation was conducted for 10 basic courses in laparoscopic surgery at the ‘European Surgical Institute’ of the Ethicon Endo-Surgery Company in Norderstedt, Germany.

Practical postgraduate training course

Three-day courses were performed to impart knowledge and train practical skills in a complex operative setting. In the first phase, basic knowledge was taught and essential skills trained. Lectures of 3¼ hours were held on basic anatomical, physiological and clinical principles as well as surgical procedures. In a video demonstration of 1½ hours possible risks and sources of error were discussed. The handling of laparoscopic instruments was shown in a practical demonstration (1 hour). Exercises in practical skills comprised of suturing and knotting techniques in a microsurgical setting (2 hours) and in a computer simulation (2 hours). In the second phase, the skills learnt were transferred into the laparoscopic surgery setting; suturing exercises were performed on a simulated abdomen with an opened or closed abdominal wall (‘Pelvitrainer’). In the last phase, an animal organ was added (pig’s liver) and a laparoscopic cholecystectomy was performed by the trainees (6 hours). The training was provided within a teamwork setting and feedback was given continuously. Overall, the aim of the courses was to enhance medical competencies with regard to knowledge and practical skills in a laparoscopic setting.

Questionnaire and data acquisition

A questionnaire was developed for self-assessment of different aspects of laparoscopic medical competencies (see Table 1 (Tab. 1)). A complete item is illustrated in Table 2 (Tab. 2). The confidence of the trainees was assessed, for example, by the question ‘Do you feel confident to perform laparoscopic procedures independently?’ and they were asked to rate themselves according to four task levels using a 5-point scale ranging from 0 (‘not applicable’) to ‘4 (‘fully applicable’). In addition, confidence was assessed regarding different areas of competency: ‘How do you rate your competencies with regard to…?’. A 5-point scale ranging from 0 (‘low’) to 4 (‘very high’) was used by the participants to indicate their responses. The questions were then reformulated to assess knowledge, practical skills, judgment and professional behaviour (see Table 1 (Tab. 1)). The questionnaire has not yet been evaluated with regard to psychometric properties.

Table 1

Different task levels and specific areas of medical competencies of the self-assessment questionnaire

Table 2

Exemplary illustration of a complete item of the self-assessment questionnaire

Participants of 10 basic courses in laparoscopic surgery in 2000 and 2001 were asked to complete the questionnaire. They rated their current laparoscopic medical competencies at the beginning of the course (‘pre-course’), at the end of the 3 day course (‘post-course’) and re-assessed their pre-course competencies at the end of the course (‘retrospective pre-course’).

Data analysis

The data were pseudonymised and data analysis was performed using SPSS® 15.

For presentation purposes with regard to question 1, the ratings 0 and 1 were merged into the category ‘minimally competent’, rating 2 to ‘moderately competent’ and ratings 3 and 4 to ‘fully competent’ and displayed in table form. The Wilcoxon test was used for analysis regarding questions 3 and 4 (comparison of self-assessment ‘post-course’ vs. ‘pre-course’ and ‘retrospective pre-course’ vs. ‘pre-course’ respectively) because significant differences from normal distribution were found. Bonferroni-Holm correction for multiple measurements was used as multiple items in the questionnaire were compared. In this case, the maximum statistical value of the single comparisons was evaluated on a significance level of α`=α / m (i.e. 0.05 / number of single comparisons). If this statistical value was significant, the next value was evaluated on a level of α / (m-1) and so on, until no significance was found [20].

Results

The basic characteristics of the study cohort as well as the experiences of the surgeons gained before commencing the course are displayed in Table 3 (Tab. 3).

Table 3

Basic characteristics and ‚pre-course‘ experiences of the participants

Self-assessment of medical competencies

more than half of the participants rated their competencies as low (‘minimally competent’) if asked about independently performed surgical procedures, whereas most of the trainees rated themselves as ‘moderately competent’ or ‘fully competent’ for surgical procedures performed under supervision, as a surgical first assistant or as a camera assistant (see Table 4 (Tab. 4)). With regard to specific areas of medical competencies more than 90% of the trainees assessed themselves ‘pre-course’ as ‘moderately competent’ or ‘fully competent’ regarding ‘knowledge about laparoscopic procedures’, ‘manual skills’, ‘mastering basic surgical techniques’, ‘careful manipulation of the tissue’ and ‘orientation and navigation in the surgical field’. Similarly, most trainees rated their skills ‘pre-course’ in the following aspects of social behaviour ‘problem management and handling of errors’, ‘ability to work in a team’ and ‘ability to concentrate and handle stressful situations’ as ‘moderately competent’ or ‘fully competent’. More than half of the trainees regarded their ‘knowledge of laparoscopic instruments and technical equipment’ as ‘moderate’ (see Table 5 (Tab. 5)).

Table 4

Adjustment of the self-assessment of laparoscopic competencies with regard to different task levels – frequency distribution

Table 5

Adjustment of the self-assessment of laparoscopic competencies with regard to specific areas of medical competencies – frequency distribution

a significant increase of self-assessments was found with regard to different task levels (except for ‘work as a camera assistant’; see Table 6 (Tab. 6)). Furthermore, a significant increase was seen ‘post-course’ for the following areas of medical competencies: ‘knowledge on laparoscopic procedures’, ‘knowledge of laparoscopic instruments and technical equipment’, ‘orientation and navigation in the surgical field’. Self-assessment was comparable to the ‘pre-course’ ratings in the remaining areas of medical competencies (see Table 7 (Tab. 7)).

Table 6

Adjustment of the self-assessment of laparoscopic competencies with regard to different task levels – statistical analysis

Table 7

Adjustment of the self-assessment of laparoscopic competencies with regard to specific areas of medical competencies – statistical analysis

The ratings indicated a significant adjustment of self-assessment with regard to the task level ‘performing laparoscopic procedures under supervision’: ‘pre-course’ 74% of the trainees rated themselves as ‘fully competent’, which decreased to 57% on the ‘retrospective pre-course’ questionnaire (see Table 4 (Tab. 4); Table 5 (Tab. 5)). Such significant adjustment was in fact seen in most of the areas of medical competencies trained during the course (‘manual skills’, ‘mastering basic surgical techniques’, ‘careful manipulation of the tissue’ but not in ‘orientation and navigation in the surgical field’) as well as ‘problem management and handling of errors’ and ‘clinical reasoning’. The proportion of participants, who rated their competencies ‘pre-course’ as ‘minimally competent’ increased in the ‘retrospective pre-course’ assessment. In contrast, no significant difference was seen with regard to the knowledge taught (‘knowledge of laparoscopic procedures‘, ‘knowledge of laparoscopic instruments and technical equipment’) as well as with regard to 'ability to work in a team', which was not the focus of the course (see Table 6 (Tab. 6), Table 7 (Tab. 7)).

Discussion

The present investigation provides evidence that surgeons with limited experience overestimate their competencies. First, participants rated themselves as ‘moderately competent’ or ‘fully competent’ before attending the course with regard to different task levels (with the exception of ‘performing laparoscopic procedures independently) as well as specific areas of medical competencies. This is astonishing because the participants had applied for training of laparoscopic competencies and only a minority already had finished postgraduate surgical training. Second, our data revealed substantial retrospective revisions to lower ratings over the course of the training. More than 85 percent of the trainees rated their ‘pre-course’ competencies as ‘moderately competent’ or ‘fully competent’ in 9 out of 10 areas of medical competencies. Relevant adjustments were seen in the ‘retrospective pre-course’ assessment: Only about a quarter of the trainees (or a third, respectively) rated themselves as “moderately” or “fully” competent in 5 (or 3, respectively) competency areas. These adjustments were statistically significant in those tasks most important to the focus of the training course ('performing surgical procedures under supervision') and in 5 out of 10 specific areas of medical competencies. No statistically significant difference was seen regarding the knowledge taught (‘knowledge of laparoscopic procedures’, ‘knowledge of laparoscopic instruments and technical equipment’). Likewise, no significant difference was seen in the competencies ‘ability to work in a team’ and ‘ability to concentrate and handle stressful situations’. The latter aspects of social behaviour were not the focus of the training course and no feedback was applied to the participants.

Our results confirm previous data on the tendency of minimally competent surgeons to overestimate their abilities [10], [13]. Furthermore, our data support previous publications which suggest that trainings lead to increased ‘metacognitive abilities’ and more realistic self-assessment abilities [12], [13] via improved competencies [17], [18], [19] and the feedback obtained. It cannot, however, be overlooked that retrospective assessments may also have socio-psycological influences (eg. absence of threats to self esteem) and distorting phenomena (eg Hindsight bias) [21].

In the present study we investigated the adjustment of self-assessment ratings exclusively in a typical postgraduate practical training course. An additional objective assessment of competence improvement would have added to the validity of the results. However, instruments for objective assessment of surgical competencies are very limited and require extensive resources [22]. Therefore, an objective assessment of surgical competencies was not made.

Our data suggest that the initial over-estimation of trainees’ abilities will influence the results of self-assessment instruments to evaluate the effectiveness of training. In particular a comparison restricted to ‘pre-course’ vs. ‘post-course’ assessments may lead to an under-estimation of the training effect that has occurred. Instead, a ‘retrospective pre-course’ assessment should also be considered if self-assessment instruments are in use. ‘Retrospective pre-course’ assessments are not a substitute for an objective measure or ‘gold standard’. However, it may reveal valuable information regarding the level of over-estimation of personal capabilities at the outset. If such over-estimation could be recognized and incorporated into the curriculum of training programs it may play an important role in the learning process of trainees.

Acknowledgement

The authors thank Ethicon Endo-Surgery, Norderstedt (Germany) and Mr. Thomas Bürger for the opportunity to conduct this investigation over the 10 basic courses in laparoscopic surgery run at their institution. Furthermore, we thank Dr. Anja Rogausch for the revision of the statistical analyses.

Competing interests

The authors declare that they have no competing interests.