Simon R Turner1, Basil S Nasir2, Hollis Lai3, Kazuhiro Yasufuku4, Colin Schieman5, Brian E Louie6, Eric L R Bédard7. 1. Department of Thoracic Surgery, University of Alberta, Edmonton, Alberta, Canada. Electronic address: sturner@ualberta.ca. 2. Department of Thoracic Surgery, Université de Montréal, Montréal, Québec, Canada. 3. Faculty of Medicine and Dentistry, University of Alberta, Edmonton, Alberta, Canada. 4. Department of Thoracic Surgery, University of Toronto, Toronto, Ontario, Canada. 5. Department of Thoracic Surgery, University of Calgary, Calgary, Alberta, Canada. 6. Department of Thoracic Surgery, Swedish Cancer Institute and Medical Center, Seattle, Washington. 7. Department of Thoracic Surgery, University of Alberta, Edmonton, Alberta, Canada.
Abstract
BACKGROUND: To develop and evaluate a surgical trainee competency assessment instrument for invasive mediastinal staging, including cervical mediastinoscopy and endobronchial ultrasound (EBUS), a comprehensive instrument was developed, the Thoracic Competency Assessment Tool-Invasive Staging (TCAT-IS), using expert review and simulated and clinical pilot-testing. METHODS: Validity and reliability evidence were collected, and item analysis was performed. Initially, a 27-item instrument was developed, which underwent expert review with members of the Canadian Association of Thoracic Surgeons (n = 86) in 2014 to 2015 (response rate, 57%). TCAT-IS was refined to 29 items in 4 competency areas: preoperative, general operative, mediastinoscopy, and EBUS. Further refinements were made based on simulated use. The final version was then used to assess competency of 5 thoracic trainees performing invasive mediastinal staging in live patients. RESULTS: Participants were assessed during 20 mediastinoscopy and 8 EBUS procedures, with 47 total assessments completed. Reliability (Cronbach's alpha = 0.94), interrater reliability (κ = 0.80), and correlation with an established global competency scale (κ = 0.75) were high. The most difficult items were "set up and adjust EBUS equipment" and "identify vascular anatomy (EBUS)." Feedback questionnaires from trainees (response rate, 80%) and surgeons (response rate, 100%) were consistently positive regarding user friendliness, utility as an assessment tool, and educational benefit. Participants believed the tool "facilitated communicating feedback to the trainee with specific areas to work on." CONCLUSIONS: TCAT-IS is an effective tool for assessing competence in invasive staging and may enhance instruction. This initial test establishes early validity and reliability evidence, supporting the use of TCAT-IS in providing structured, specific, formative assessments of competency.
BACKGROUND: To develop and evaluate a surgical trainee competency assessment instrument for invasive mediastinal staging, including cervical mediastinoscopy and endobronchial ultrasound (EBUS), a comprehensive instrument was developed, the Thoracic Competency Assessment Tool-Invasive Staging (TCAT-IS), using expert review and simulated and clinical pilot-testing. METHODS: Validity and reliability evidence were collected, and item analysis was performed. Initially, a 27-item instrument was developed, which underwent expert review with members of the Canadian Association of Thoracic Surgeons (n = 86) in 2014 to 2015 (response rate, 57%). TCAT-IS was refined to 29 items in 4 competency areas: preoperative, general operative, mediastinoscopy, and EBUS. Further refinements were made based on simulated use. The final version was then used to assess competency of 5 thoracic trainees performing invasive mediastinal staging in live patients. RESULTS:Participants were assessed during 20 mediastinoscopy and 8 EBUS procedures, with 47 total assessments completed. Reliability (Cronbach's alpha = 0.94), interrater reliability (κ = 0.80), and correlation with an established global competency scale (κ = 0.75) were high. The most difficult items were "set up and adjust EBUS equipment" and "identify vascular anatomy (EBUS)." Feedback questionnaires from trainees (response rate, 80%) and surgeons (response rate, 100%) were consistently positive regarding user friendliness, utility as an assessment tool, and educational benefit. Participants believed the tool "facilitated communicating feedback to the trainee with specific areas to work on." CONCLUSIONS: TCAT-IS is an effective tool for assessing competence in invasive staging and may enhance instruction. This initial test establishes early validity and reliability evidence, supporting the use of TCAT-IS in providing structured, specific, formative assessments of competency.
Authors: Simon R Turner; Hollis Lai; Basil S Nasir; Kazuhiro Yasufuku; Colin Schieman; James Huang; Eric L R Bédard Journal: Ann Thorac Surg Date: 2019-11-09 Impact factor: 4.330