Toward an objective assessment of technical skills: a national survey of surgical program directors in Saudi Arabia.

BACKGROUND: After almost a decade of implementing competency-based programs in postgraduate training programs, the assessment of technical skills remains more subjective than objective. National data on the assessment of technical skills during surgical training are lacking. We conducted this study to document the assessment tools for technical skills currently used in different surgical specialties, their relationship with remediation, the recommended tools from the program directors' perspective, and program directors' attitudes toward the available objective tools to assess technical skills.
METHODS: This study was a cross-sectional survey of surgical program directors (PDs). The survey was initially developed using a focus group and was then sent to 116 PDs. The survey contains demographic information about the program, the objective assessment tools used, and the reason for not using assessment tools. The last section discusses the recommended tools to be used from the PDs' perspective and the PDs' attitude and motivation to apply these tools in each program. The associations between the responses to the assessment questions and remediation were statistically evaluated.
RESULTS: Seventy-one (61%) participants responded. Of the respondents, 59% mentioned using only nonstandardized, subjective, direct observation for technical skills assessment. Sixty percent use only summative evaluation, whereas 15% perform only formative evaluations of their residents, and the remaining 22% conduct both summative and formative evaluations of their residents' technical skills. Operative portfolios are kept by 53% of programs. The percentage of programs with mechanisms for remediation is 29% (19 of 65).
CONCLUSION: The survey showed that surgical training programs use different tools to assess surgical skills competency. Having a clear remediation mechanism was highly associated with reporting remediation, which reflects the capability to detect struggling residents. Surgical training leadership should invest more in standardizing the assessment of surgical skills.

Background

There has been a recent, significant shift in residency training toward competency-based programs, such as the Canadian Medical Education Directives for Specialists (CanMEDS) framework1 and the Accreditation Council for Graduate Medical Education (ACGME) Outcome Project.2 These projects have identified certain competencies that residents should acquire during their training (Table 1). Surgical competency is covered under the medical expert and patient care sections in the CanMEDS and ACGME frameworks, respectively. However, after the initiation of these projects, program directors (PDs) have been struggling to identify an optimal assessment tool to measure this competency that meets certain criteria.3,4 The first criterion, reliability, refers to the reproducibility, consistency, and generality of the results over time. The second criterion, feasibility, reasons that, in order to be accepted, a new tool should be cost-effective, easy to execute, and not time-consuming. The third criterion, validity, asks if the tool assesses what it is supposed to assess. Validity is divided into five sub-categories: predictive validity, which asks if the tool can predict future performance; face validity, which asks if the tool reflects real life; construct validity, which asks if the tool measures what it is designed to measure; content validity, which asks if the domain that we are assessing is assessed by the tool; and criterion-related validity, which asks if the result assessed by a tool correlates with that measured by the current gold standard tool.

Table 1

CanMEDS and ACGME core competencies

CanMEDS	ACGME
• Medical expert	• Medical knowledge
• Communicator	• Communication and interpersonal skills
• Collaborator	• Patient care
• Manager	• System-based practice
• Health advocate	• Practice-based learning and improvement
• Scholar	• Professionalism
• Professionalism

Abbreviations: CanMEDS, Canadian Medical Education Directives for Specialists; ACGME, Accreditation Council for graduate Medical Education.

After almost a decade of implementing competency-based programs in postgraduate training programs, the assessment of technical skills remains more subjective than objective, relying mainly on the gut feelings of the assessor and on nonstandardized direct observation of the resident while the resident performs a procedure. These subjective assessment methods have poor reliability, as manifested by measuring interrater and test–retest reliability.3,5 Dr Carlos A Pellegrini, the former president of the American Surgical Association, questioned in his presidential letter: “Is performance measurement possible and accurate?”6 He sought a valid, reliable, measurable, and feasible assessment tool. In recent decades, surgical educators around the world have developed and validated objective assessment tools to measure surgical residents’ technical skills. Martin et al invented a new method that resembles the Objective Structured Clinical Examination and named it the Objective Structured Assessment of Technical Skills (OSATS).7 The future of training is moving toward the use of more technology; Fried et al8 developed the McGill Inanimate System for Training and Evaluation of Laparoscopic Skills (MISTELS). In addition, Datta et al9 developed the Imperial College Surgical Assessment Device (ICSAD), which assesses surgical residents’ dexterity by electromagnetic means.

The Saudi Commission For Health Specialties (SCFHS)10 was established under a royal decree in 1983. Its responsibilities are to organize, evaluate, and set policies for training program accreditation; in addition, the SCFHS is the accreditation and qualifying agency of the trainees. Thirty-seven residency and fellowships training programs in multiple health specialties have been recognized by the SCFHS. One of the main aims of the SCFHS is to supervise and assure the quality of graduates by offering a fair assessment to all trainees.

To enact a new change in a complex organization system, Heifetz and Laurie11 recommend “getting on the balcony” to have a clear view of the current situation. Data on the assessment of technical skills in surgical specialties in national settings are lacking. We conducted this research to explore the current tools used by different surgical programs for technical skills assessment, their relationship with remediation (ie, the action taken by a program director toward a resident who does not fulfill the criteria for promotion to the next level of training), the recommended tools from the PDs’ perspective, and PDs’ attitudes toward the available objective tools for technical skills assessment.

Methods

Study design

A cross-sectional study was carried out between January 27, 2010 and March 24, 2010 and involved all of the surgical PDs accredited by the SCFHS.

Population, sample size, and sampling

This census study involved all of the PDs of all surgical programs accredited by the SCFHS (general surgery, orthopedics surgery, urology, otolaryngology, plastic surgery, maxillofacial surgery, obstetrics, and gynecology). The inclusion criterion was a surgical PD accredited by the SCFHS. The exclusion criteria were PDs not identified by the SCFHS or with invalid or incorrect email addresses on the SCFHS website. Ophthalmology and fellowship PDs were also excluded.

The survey

For the survey, we used a modified version of a previously used survey after receiving author permission.12 The modified survey was designed by the primary investigator and the coauthors. A focus group was developed to assess the content and face validity of the survey. The focus group was composed of five surgeons from different surgical specialties who hold degrees in epidemiology and/or medical/surgical education. These surgeons were non-PDs to avoid contamination of the final sample.

The survey contains demographic information about the program (specialty, number of residents evaluated each year, number of residents accepted each year, number of full-time staffers, and the presence of a simulation lab). Respondents were provided a list of the available objective assessment tools with a brief description of each and asked to identify which of the tools they used and explain the reason for not using any of the available tools, if applicable. Remediation was also covered (the presence of a clear remediation mechanism based on poor surgical performance, information on when it takes place, and the number of residents that required remediation). The last section involves the importance and the recommended tools from the respondent’s perspective, his/her attitudes and motivations to apply those tools in his/her training program, and any further comments.

The survey was sent to the participants’ emails through the SurveyMonkey® website. A reminder email was sent to participants who failed to respond within 2 weeks.

Statistical analysis

All data were collected electronically and downloaded as an Excel (Microsoft Corporation, Redmond, WA) spreadsheet. From there, they were incorporated into the SPSS program (v 17; IBM Corporation, Armonk, NY). Descriptive statistics were tabulated, as were mean and median calculations for the scoring of various questions. The response to each question was considered as a continuous variable, and Chi-squared and Fisher’s exact tests were used where appropriate.

We then conducted a correlation analysis using the Pearson correlation coefficient with the remediation rate for each assessment tool. The associations were also calculated using odds ratios for each assessment tool. Both P-values less than 0.05 and 95% confidence intervals that did not overlap with 1 were considered statistically significant.

Results

Response rate

A total of 116 PDs were surveyed in the study, and 71 directors responded to our survey, for a response rate of 61%. Of these 71 directors, 65 respondents (56%) completed the survey. Six incomplete responses were excluded. The response rate after the first mailing was 18 (16% of the total) respondents; after the second mailing, another 53 responded (46%).

Demographic data

Urology PDs comprised 25% (n = 16) of the respondents (Figure 1). The mean number of residents evaluated each year by a single PD was 11.4 (median = 8), with the highest number in general surgery. The mean number of residents accepted each year per institution was 2.74 (median = 2), with the highest number in general surgery. Fourteen percent of PDs reported having a simulation lab in their institution. The complete descriptive characteristics of the studied programs are shown in Table 2.

Figure 1

Distribution of respondents by specialty.

Abbreviations: GS, general surgery; OBGY, obstetrics and gynecology; ORL, otorhinolaryngology; OMF, oral maxillofacial surgery.

Table 2

Demographic data of the programs surveyed

Specialty (number of respondents/number of total program directors surveyed)	Mean number of residents evaluate/year (range)	Mean number of residents accepted each year (range)	Mean number of consultants (range)	Consultant to resident ratio	Number of programs with a simulation lab
Urology (n = 16/20)	5.6	1.18	5.56	1:1	4
	(1–20)	(1–2)	(2–11)
Neurosurgery (n = 8/11)	5.5	2	4.5	1:1.2	2
	(1–8)	(1–4)	(3–12)
GS (n = 11/18)	26	3.9	11.63	1: 2.24	3
	(5–60)	(2–6)	(5–24)
OBGY (n = 5/16)	15.2	3.2	13.4	1:1.33	2
	(4–22)	(2–5)	(7–21)
OMF (n = 3/8)	7.33	3	4	1:1.83	0
	(4–12)	(2–5)	(3–5)
ORL (n = 7/14)	8.1	3.71	5.86	1:1.38	1
	(3–12)	(1–10)	(3–14)
Plastic surgery (n = 4/5)	10.75	3.5	4.25	1: 2.53	1
	(2–20)	(1–6)	(3–6)
Orthopedics (n = 11/24)	14.36	3.18	7.3	1:1.97	1
	(4–30)	(1–10)	(3–12)

Abbreviations: GS, general surgery; OBGY, obstetrics and gynecology; ORL, otorhinolaryngology; OMF, oral maxillofacial surgery.

Surgical skills evaluation and feedback

Of the respondents, 39 (60%) only performed summative evaluations of their residents (ie, end of rotation/year evaluation), whereas 10 (15%) only conducted formative evaluations of their residents (ie, continued evaluation throughout the rotation/year). The remaining 14 (22%) used both summative and formative evaluations of residents’ technical skills. The evaluation and feedback was provided orally by 37 (57%) respondents and was written for 48 (74%) respondents. The most common evaluation used was an end-of-rotation, written summative evaluation (70%). The distribution of evaluation techniques by specialty is shown in Figure 2.

Figure 2

Distribution of evaluation technique by specialty.

Abbreviations: GS, general surgery; OBGY, obstetrics and gynecology; ORL, otorhinolaryngology; OMF, oral maxillofacial surgery.

Technical skills assessment tools

Of the respondents, 42% thought that they were using more than subjective technical skills assessment, whereas the remaining 59% mentioned using only nonstandardized, subjective, direct observation for technical skills assessment. Eighteen (26%) respondents were using at least one of the assessment tools listed in the survey (ie, OSATS, videotaped analysis, and resident surgical portfolios). Resident surgical portfolios refer to the log books containing all surgical cases in which the resident participated, either as the primary surgeon or the assistant, during his/her residency training. Eight (12%) respondents reported using hand-motion analysis, and five (8%) respondents reported using simulation to assess residents’ technical skills. Table 3 shows the distribution of assessment tools used by specialty.

Table 3

Distribution of assessment tools used by specialty

Specialty	OSATS	Hand motion analysis	Videotaped analysis	Residents surgical portfolios
GS	1	2	4	5
Urology	1	2	3	10
ORL	1	0	1	3
Neurosurgery	0	1	0	3
OMF	0	1	1	3
Plastic	0	1	0	4
OBGY	1	1	0	2
Orthopedics	3	0	1	5

Abbreviations: OSATS, Objective Structured Assessment of Technical Skills; GS, general surgery; OBGY, obstetrics and gynecology; ORL, otorhinolaryngology; OMF, oral maxillofacial surgery.

OSATS

Seven (11%) respondents were using the OSATS, and orthopedic surgeons used it the most. Those respondents who never used it (59, 89%) attributed their non-use to the fact that it was not validated in their specialty (23, 39%), that they were not aware of the supporting literature (21, 36%), and that it was either not cost-effective (3, 5%) or not practical (6, 10%). In addition, eight respondents (14%) attributed its non-use to availability issues, whereas four (7%) attributed its non-use to lack of institutional support and resources. Of the respondents, 80% judged the OSATS as important, whereas only 1.5% thought that it was not important. Furthermore, 5% and 14% stated that they did not know or were neutral regarding the importance, respectively. Many participants (85%) recommended that the OSATS be included as part of a technical skills assessment program, whereas the other participants were neutral (11%) or did not know (5%).

Videotaped analysis

Ten (15%) respondents used videotaped analysis, and general surgery programs used this tool the most. Of those respondents who never used it (55, 85%), reasons given were that it was not validated in their specialty (21, 38%); they were not aware of the supporting literature (19, 35%); it was not cost-effective (7, 13%), practical (14, 26%), or available (4, 7%), and other reasons (8, 15%). Of the respondents, 60% thought that videotaped analysis was important; 32% thought it had no effect, whereas 6% thought that it was not important. Many of the participants (63%) recommended that videotaped analysis be part of a technical skills assessment program, whereas 29% were neutral, and 3% did not recommend its use.

Resident surgical portfolios

A total of 35 (54%) respondents used resident surgical portfolios. Urology PDs used them the most. Of those respondents who never used these portfolios (30, 46%), the reasons provided were that they were not validated in their specialty (11, 37%), that they were not aware of the supporting literature (12, 40%), that they were either not cost-effective (3, 10%) or practical (5, 17%), and other reasons (5, 17%). Of the respondents, 79% thought that resident surgical portfolios were important and should be part of a technical skills assessment program, whereas only 19% thought that portfolios were neutral. No respondents thought that portfolios were not important or did not recommend their use.

Remediation

Nineteen programs reported having a clear mechanism for remediation based on poor surgical performance. These remediation mechanisms take place at the end of the rotation, at the end of the academic year, and at the end of training at frequencies of 68%, 26%, and 5%, respectively. Thirty-one (48%) PDs reported having had at least one remediation case in the past 5 years based on poor surgical performance. The mean number of residents in all programs remediated in the past five years was 1.77 (1–4), and a total of 55 residents were required to remediate. Using videotaped analysis as an objective assessment tool was significantly associated with reporting remediation (P = 0.026), but there was no significant association with the use of the OSATS or surgical residents’ portfolios and remediation (Table 4). However, having a clear remediation mechanism based on poor surgical performance was highly significantly associated with reporting remediation (P = 0.001). However, no association was identified between reporting remediation and the different modes of evaluation and feedback.

Table 4

The association between remediation and both different assessment tools and the presence of a remediation mechanism

		Remediation		Odds ratio (95% CI)	P-value
		Yes	No
OSATS	Yes	4	3	1.53	0.596
	No	27	31	(0.314–7.457)
Videotaped analysis	Yes	8	2	9.85	0.026
	No	23	32	(1.080–28.675)
Surgical residents’ portfolios	Yes	17	18	1.08	0.878
	No	14	16	(0.406–2.865)
Remediation	Yes	15	4	7.03	0.001
Mechanism	No	16	30	(1.997–24.759)

Abbreviations: CI, confidence interval; OSATS, Objective Structured Assessment of Technical Skills.

Satisfaction, attitude, motivation

Only 39% of respondents are satisfied (Figure 3) with the methods they have been using to assess technical skills, and this satisfaction was significantly associated with the use of surgical residents’ portfolios (P = 0.035). Of the respondents, 91% agreed that our survey increased their knowledge about the objective assessment of technical skills; 89% and 88% of the respondents agreed that our survey motivated them to learn more about and to apply objective assessment tools in their programs, respectively. A total of 94% of the participants expressed interest in attending a workshop on objective assessment tools. The comments provided by the participants were very favorable, and, in general, the participants found the survey to be extremely helpful. As an example of how participants valued the survey, one participant stated, “All members of examination committees in different programs should have continuous updates of their methods of evaluation.” Most comments also involved expressing interest in attending any further workshops or training pertaining to the matter of the objective assessment of surgical skills, e.g. “I think we need a workshop on acceptable methods for all trainers in all programs in Saudi Arabia.”

Figure 3

Program directors’ satisfaction regarding technical skills assessment.

Discussion

Technical surgical errors comprise the majority of surgical errors, with more than half occurring due to a lack of competence.13,14 Reports have suggested that objective assessments be used to minimize medical errors.15 The SCFHS pays a great deal of attention to standardizing training and assures the quality control of graduates by adapting specific criteria for the assessment of the quality of training centers and by conducting annual national exams for all postgraduate medical training programs in the different specialties.10 Despite this great interest in highly standardized training programs, national exams only assess clinical skills and knowledge by using multiple choice questions and an Objective Structured Clinical Examination, whereas the assessment of surgical residents’ technical skills is completely ignored and left to each individual institution. Due to this nonstandardized technical skills assessment, graduate residents’ surgical competency varies widely. We hope our study will serve as a cornerstone for developing a standardized objective assessment program for surgical residents’ technical skills, both nationally and internationally.

The detection of struggling residents during training is one of the main aims of PDs. Subjective evaluation, which has poor reliability,3,5 may not detect this issue. Therefore, the use of objective assessment tools such as the OSATS, MISTELS, and electromagnetic hand motion analysis could be a “safeguard” against this dilemma, as all trainees would be tested fairly and objectively using validated tools.5,16 Applying these tools could be another struggle; most of our respondents are motivated and willing to apply new and innovative methods for assessment but lack knowledge and information about these tools. Thus, adapting faculty enhancement programs directed toward surgical education could solve this problem.

Having clear policies and mechanisms for remediation is helpful in detecting struggling trainees. The association between implementing such policies and reporting residents requiring remediation is confirmed in this study. However, the use of objective assessment tools did not display any association with reporting remediation, possibly because they are relatively new tools and their implementation is recent. Videotaped analysis is not a consistent tool, but using it with a validated checklist increases its reliability, validity, and objectivity.17 However, if used without checklists, subjective direct observation continues to have disadvantages.

Bandiera et al18 preferred using simulations, direct observation and in-training evaluation reports, logbooks, and objective structured assessment tools such as the OSATS and MISTELS for technical skills assessment. The ACGME2 recommend that checklists and 360° evaluation be the first methods used for performance assessments of medical procedures. However, most of our participants recommended that the OSATS and residents’ surgical portfolios be used first. Moreover, although more than half of the PDs surveyed use portfolios, the quality of these portfolios is unknown. In addition, the literature has demonstrated poor reliability and validity for this instrument.3 Therefore, the assessment should not depend primarily on portfolios, and other assessment tools should be included;19 portfolios provide an idea of the quantity, but not the quality, of the procedures performed.3 The reliability and construct validity of the OSATS are established,20 but only a minority of programs use it, either due to a lack of knowledge and/or inadequate resources.

Recent reports on incorporating technology and health informatics into postgraduate training and assessment are fascinating. Reports from Tufts University21 on adapting the OpRate application to follow the progress of surgical residents’ technical skills demonstrate that adapting such formative written application programs is useful in identifying residents who need special consolidated training in one area or another. Another reliable application adapted by Southern Illinois University22 is the operative performance rating system, which is an internet-based, procedure-based assessment including ten checklist items that are used to evaluate residents’ technical and decision-making skills for six different general surgical procedures.

Dexterity-based analysis devices have been covered extensively in the literature. The ICSAD has been validated as a hand motion analyzer in both open and laparoscopic surgeries.23,24 The ICSAD tracks hand motion efficiency by measuring the number and speed of hand motions, the time taken to complete the task, and the distance traveled by the surgeon’s hand. The Advanced Dundee Endoscopic Psychomotor Tester25 is a computer-controlled endoscopic performance assessment system that measures the time taken to complete the task, the contact error time, the route of the props, and the number of completed tasks within a time limit. This system has reasonable reliability26 and contrast validity27 as an endoscopic assessor. The third system, the Hiroshima University Endoscopic Surgical Assessment Device measures the time taken to complete the task and the deviation of the instrument tip in the horizontal and vertical axes. Its initial reliability and validity reports are encouraging.28

Simulators have gained great popularity in training and, recently, in assessment. The minimally invasive surgical trainer-virtual reality (MIST-VR) simulator is one of the first simulators developed for training basic laparoscopic skills.29 MIST-VR has been validated for assessing basic laparoscopic skills.30 Maithel et al31 compared three commercially available virtual reality simulators, MIST-VR, Endotower, and the Computer Enhanced Laparoscopic Training System (CELTS). All three simulators met the criteria for construct and face validity, and CELTS had the highest scores. Fried et al8 also developed MISTELS, which includes five laparoscopic task stations (peg transfer, precision pattern cutting, securing a ligating loop, intracorporeal knotting, and extra-corporeal knotting) performed in standardized laparoscopic training boxes. The reliability and validity of this system have been confirmed.8,32–36 In addition, MISTELS has been shown to be superior to subjective in-training evaluation reports in identifying residents with poor laparoscopic skills.37 Recently, a cognitive component was incorporated into MISTELS through the fundamentals to laparoscopic surgery (FLS) module, and FLS has been reported to have reasonable reliability.38 However, the validity of the FLS is questionable39 because the cognitive element of the assessment cannot differentiate between experts and novices.

Few published reports have focused on the real situation of technical skills assessment. Brown et al12 surveyed otolaryngology PDs in the United States and found that most of the participants used subjective evaluation. Similarly, they identified a significant correlation between having a clear remediation mechanism and the remediation rate, which confirms that implementing clear mechanisms and policies will enhance the identification of residents requiring remediation.

This study has several limitations. First, as in any survey study, the response rate is a major limitation and affects the calculated response rate for each specialty. Therefore, we sought to overcome this issue by frequently sending the survey to the participants, communicating via phone and on-site visits, and extending the survey time from 4 weeks to 6 weeks. Second, we assumed that the majority of participants would have a poor understanding of all of the assessment tools, so we added a brief description of each assessment tool in the survey, which may have caused a biased response. Third, the variety of specialties surveyed and the limited number of available programs make it difficult to generalize the results. Fourth, although we developed this survey scientifically using focus group methods and validated it prior to distribution to our target population, the survey has not been formally validated. Using the survey in a different population, such as with North American surgical PDs or any other regional PDs, would add credence to our results and will further validate both the survey and the results of our study.

Conclusion

The survey revealed that surgical training programs use different tools to assess surgical skills competency. Having a clear remediation mechanism was highly associated with reporting remediation, which reflects the capability to detect struggling residents. Surgical training leadership should invest more in standardizing the assessment of surgical skills.