BACKGROUND: The establishment of assessment reliability at the level of the individual trainee is an important attribute of assessment methodologies, particularly for doctors who have been failed. This issue is of particular importance for the process of competence assessment in the USA, UK, Australia and New Zealand. METHODS: We use data from 19 applicants for higher surgical training in 2008 at the Royal College of Surgeons in Ireland to compare: (i) the objective structured assessment of technical skills (OSATS) method; and (ii) a procedure-specific checklist to assess surgical technical skills in the excision of a sebaceous cyst task by two experienced senior surgeons. RESULTS: The overall interrater reliability (IRR) of the OSATS assessment as determined by a correlation coefficient was 0.507 (P < 0.03) and 0.67 with coefficient alpha, considerably below the accepted 0.8 level of IRR. The checklist's overall IRR was 0.89. Individually, only five (26%) of the OSATS assessments reached the 0.8 level of IRR in contrast to 18 (95%) of the checklist assessments. DISCUSSION: We propose binary procedure-based assessment checklists as more reliable assessment instruments with more robust reproducibility.
BACKGROUND: The establishment of assessment reliability at the level of the individual trainee is an important attribute of assessment methodologies, particularly for doctors who have been failed. This issue is of particular importance for the process of competence assessment in the USA, UK, Australia and New Zealand. METHODS: We use data from 19 applicants for higher surgical training in 2008 at the Royal College of Surgeons in Ireland to compare: (i) the objective structured assessment of technical skills (OSATS) method; and (ii) a procedure-specific checklist to assess surgical technical skills in the excision of a sebaceous cyst task by two experienced senior surgeons. RESULTS: The overall interrater reliability (IRR) of the OSATS assessment as determined by a correlation coefficient was 0.507 (P < 0.03) and 0.67 with coefficient alpha, considerably below the accepted 0.8 level of IRR. The checklist's overall IRR was 0.89. Individually, only five (26%) of the OSATS assessments reached the 0.8 level of IRR in contrast to 18 (95%) of the checklist assessments. DISCUSSION: We propose binary procedure-based assessment checklists as more reliable assessment instruments with more robust reproducibility.
Authors: Karthikeyan Kallidaikurichi Srinivasan; Anthony Gallagher; Niall O'Brien; Vinod Sudir; Nick Barrett; Raymund O'Connor; Francesca Holt; Peter Lee; Brian O'Donnell; George Shorten Journal: BMJ Open Date: 2018-10-15 Impact factor: 2.692
Authors: Till Johannes Bugaj; Christoph Nikendei; Jan Benedikt Groener; Jan Stiepak; Julia Huber; Andreas Möltner; Wolfgang Herzog; Ansgar Koechel Journal: BMC Med Educ Date: 2018-11-12 Impact factor: 2.463