OBJECTIVE: The tracheostomy is an essential component of otolaryngology surgery training. There is a pressing need for a reliable, valid, intra-operative assessment tool for assessing the tracheostomy skills of residents in training programs. This study aimed to develop an assessment tool for surgical tracheostomy that was valid, reliable, and feasible. METHODOLOGY: Based on a previously developed tool, the Objective Structured Assessment of Technical Skills, we developed an objective structured tool specifically for assessing tracheostomy skills (OSTS). Our panel included Otorhinolaryngology Head & Neck Surgery faculty members, academic otolaryngologists, and medical education experts. To test the OSTS, 8 board certified otolaryngologists assessed 44 residents from 5-year residency programs in the operating theater, from July 2010 to May 2012, in three tertiary level academic institutions. Residents received constructive feedback after the assessments. RESULTS: The OSTS displayed good reliability, in terms of internal consistency (α = 0.761). Its validity was established based on significant improvement in senior resident scores compared to junior resident scores, assessed with an ANOVA (F = 5.31, p = 0.001). The tool also displayed an overall trend of score improvement with an individual's advancement through each year of postgraduate training. The evaluators considered the OSTS easy to understand, complete, practical, and time-efficient, which established its face validity. CONCLUSION: The OSTS is a valid, reliable tool for evaluating surgical competency of residents performing tracheostomy. It provided systematic and comprehensive feedback. The examiners and examinees both displayed positive attitudes towards the instrument.
OBJECTIVE: The tracheostomy is an essential component of otolaryngology surgery training. There is a pressing need for a reliable, valid, intra-operative assessment tool for assessing the tracheostomy skills of residents in training programs. This study aimed to develop an assessment tool for surgical tracheostomy that was valid, reliable, and feasible. METHODOLOGY: Based on a previously developed tool, the Objective Structured Assessment of Technical Skills, we developed an objective structured tool specifically for assessing tracheostomy skills (OSTS). Our panel included Otorhinolaryngology Head & Neck Surgery faculty members, academic otolaryngologists, and medical education experts. To test the OSTS, 8 board certified otolaryngologists assessed 44 residents from 5-year residency programs in the operating theater, from July 2010 to May 2012, in three tertiary level academic institutions. Residents received constructive feedback after the assessments. RESULTS: The OSTS displayed good reliability, in terms of internal consistency (α = 0.761). Its validity was established based on significant improvement in senior resident scores compared to junior resident scores, assessed with an ANOVA (F = 5.31, p = 0.001). The tool also displayed an overall trend of score improvement with an individual's advancement through each year of postgraduate training. The evaluators considered the OSTS easy to understand, complete, practical, and time-efficient, which established its face validity. CONCLUSION: The OSTS is a valid, reliable tool for evaluating surgical competency of residents performing tracheostomy. It provided systematic and comprehensive feedback. The examiners and examinees both displayed positive attitudes towards the instrument.