Domenico Veneziano1,2,3, Antonio Canova4, Michiel Arnolds5, John D Beatty6, Chandra S Biyani7, Federico Dehò8,9, Cristian Fiori10, Giles O Hellawell11, J F Langenhuijsen12, Giovannalberto Pini13, Oscar Rodriguez Faba14, Giampaolo Siena15, Andreas Skolarikos16, Theodoros Tokas17, Ben S E P Van Cleynenbreugel18, Christian Wagner19, Giovanni Tripepi20, Bhaskar Somani21, Bhaskar Lima2,3. 1. Life and Health Sciences Research Institute (ICVS), School of Medicine, University of Minho, Braga, Portugal. 2. ICVS/3B's, PT Government Associate Laboratory, Braga/Guimarães, Portugal. 3. Department of Urology and Kidney Transplant, Grande Ospedale Metropolitano, Reggio Calabria, Italy. 4. Human Resources Management, MBDA Italia SpA, Rome, Italy. 5. Department of Urology, Gelre Ziekenhuis, Apeldoorn, The Netherlands. 6. Department of Urology, Leicester General Hospital, University of Leicester, Leicester, UK. 7. Department of Urology, St James's University Hospital Leeds Teaching Hospitals NHS Trust, Leeds, UK. 8. Department of Urology, San Raffaele Scientific Institute, University Vita-Salute, Milan, Italy. 9. Division of Oncology/Unit of Urology, URI, IRCCS Ospedale San Raffaele, Milan, Italy. 10. Department of Oncology, A.O.U. San Luigi Gonzaga, Università degli Studi di Torino, Orbassano, Italy. 11. North West London Hospitals NHS Trust, Imperial College Healthcare NHS Trust, London, UK. 12. Department of Urology, Radboud University Medical Centre, Nijmegen, The Netherlands. 13. Department of Urology, Ospedale San Raffaele - Turro, Milano, Italy. 14. Department of Urology, Fundaciò Puigvert, Barcelona, Spain. 15. Department of Urology, Azienda Ospedaliero-Universitaria di Careggi, Firenze, Italy. 16. Department of Urology, University of Athens, Athens, Greece. 17. Department of Urology and Andrology, General Hospital Hall i.T., Hall in Tirol, Austria. 18. Department of Urology, UZ Leuven, Leuven, Belgium. 19. Department of Urology and Oncology, St. Antonius-Hospital, Gronau, Germany. 20. CNR IFC, U.O. of Reggio Calabria, Reggio Calabria, Italy. 21. Department of Urology, University Hospital Southampton NHS Foundation Trust, Southampton, UK.
Abstract
OBJECTIVE: To evaluate the variability of subjective tutor performance improvement (Pi) assessment and to compare it with a novel measurement algorithm: the Pi score. MATERIALS AND METHODS: The Pi-score algorithm considers time measurement and number of errors from two different repetitions (first and fifth) of the same training task and compares them to the relative task goals, to produce an objective score. We collected data during eight courses on the four European Association of Urology training in Basic Laparoscopic Urological Skills (E-BLUS) tasks. The same tutor instructed on all courses. Collected data were independently analysed by 14 hands-on training experts for Pi assessment. Their subjective Pi assessments were compared for inter-rater reliability. The average per-participant subjective scores from all 14 proctors were then compared with the objective Pi-score algorithm results. Cohen's κ statistic was used for comparison analysis. RESULTS: A total of 50 participants were enrolled. Concordance found between the 14 proctors' scores was the following: Task 1, κ = 0.42 (moderate); Task 2, κ = 0.27 (fair); Task 3, κ = 0.32 (fair); and Task 4, κ = 0.55 (moderate). Concordance between Pi-score results and proctor average scores per participant was the following: Task 1, κ = 0.85 (almost perfect); Task 2, κ = 0.46 (moderate); Task 3, κ = 0.92 (almost perfect); Task 4 = 0.65 (substantial). CONCLUSION: The present study shows that evaluation of Pi is highly variable, even when formulated by a cohort of experts. Our algorithm successfully provided an objective score that was equal to the average Pi assessment of a cohort of experts, in relation to a small amount of training attempts.
OBJECTIVE: To evaluate the variability of subjective tutor performance improvement (Pi) assessment and to compare it with a novel measurement algorithm: the Pi score. MATERIALS AND METHODS: The Pi-score algorithm considers time measurement and number of errors from two different repetitions (first and fifth) of the same training task and compares them to the relative task goals, to produce an objective score. We collected data during eight courses on the four European Association of Urology training in Basic Laparoscopic Urological Skills (E-BLUS) tasks. The same tutor instructed on all courses. Collected data were independently analysed by 14 hands-on training experts for Pi assessment. Their subjective Pi assessments were compared for inter-rater reliability. The average per-participant subjective scores from all 14 proctors were then compared with the objective Pi-score algorithm results. Cohen's κ statistic was used for comparison analysis. RESULTS: A total of 50 participants were enrolled. Concordance found between the 14 proctors' scores was the following: Task 1, κ = 0.42 (moderate); Task 2, κ = 0.27 (fair); Task 3, κ = 0.32 (fair); and Task 4, κ = 0.55 (moderate). Concordance between Pi-score results and proctor average scores per participant was the following: Task 1, κ = 0.85 (almost perfect); Task 2, κ = 0.46 (moderate); Task 3, κ = 0.92 (almost perfect); Task 4 = 0.65 (substantial). CONCLUSION: The present study shows that evaluation of Pi is highly variable, even when formulated by a cohort of experts. Our algorithm successfully provided an objective score that was equal to the average Pi assessment of a cohort of experts, in relation to a small amount of training attempts.
Authors: Guglielmo Mantica; Juan Gomez Rivas; Diego M Carrion; Moises E Rodriguez-Socarrás; Francesco Esperto; Giovanni E Cacciamani; Domenico Veneziano Journal: Cent European J Urol Date: 2020-04-06
Authors: Mohammad S A Amin; Abdullatif Aydin; Nurhan Abbud; Ben Van Cleynenbreugel; Domenico Veneziano; Bhaskar Somani; Ali Serdar Gözen; Juan Palou Redorta; M Shamim Khan; Prokar Dasgupta; Jonathan Makanjuoala; Kamran Ahmed Journal: Surg Endosc Date: 2020-08-26 Impact factor: 4.584