BACKGROUND: The ability of an assessment to predict performance would be of major benefit to residency programs, allowing for early identification of residents at risk. OBJECTIVE: We sought to establish whether passing the Objective Structured Assessment of Technical Skills (OSATS) examination in postgraduate year 1 (PGY-1) predicts future performance. METHODS: Between 2002 and 2012, 133 PGY-1 surgery residents at the University of Toronto (Toronto, Ontario, Canada) completed an 8-station, simulated OSATS examination as a component of training. With recently set passing scores, residents were assigned a pass/fail status using 3 standards setting methods (contrasting groups, borderline group, and borderline regression). Future in-training performance was compared between residents who had passed and those who failed the OSATS, using in-training evaluation reports from resident files. A Mann-Whitney U test compared performance among groups at PGY-2 and PGY-4 levels. RESULTS: Residents who passed the OSATS examination outperformed those who failed, when compared during PGY-2 across all 3 standard setting methodologies (P < .05). During PGY-4, only the contrasting groups method showed a significant difference (P < .05). CONCLUSIONS: We found that PGY-1 surgical resident pass/fail status on a technical skills examination was associated with future performance on in-training evaluation reports in later years. This provides validity evidence for the current PGY-1 pass/fail score, and suggests that this technical skills examination may be used to predict performance and to identify residents who require remediation.
BACKGROUND: The ability of an assessment to predict performance would be of major benefit to residency programs, allowing for early identification of residents at risk. OBJECTIVE: We sought to establish whether passing the Objective Structured Assessment of Technical Skills (OSATS) examination in postgraduate year 1 (PGY-1) predicts future performance. METHODS: Between 2002 and 2012, 133 PGY-1 surgery residents at the University of Toronto (Toronto, Ontario, Canada) completed an 8-station, simulated OSATS examination as a component of training. With recently set passing scores, residents were assigned a pass/fail status using 3 standards setting methods (contrasting groups, borderline group, and borderline regression). Future in-training performance was compared between residents who had passed and those who failed the OSATS, using in-training evaluation reports from resident files. A Mann-Whitney U test compared performance among groups at PGY-2 and PGY-4 levels. RESULTS: Residents who passed the OSATS examination outperformed those who failed, when compared during PGY-2 across all 3 standard setting methodologies (P < .05). During PGY-4, only the contrasting groups method showed a significant difference (P < .05). CONCLUSIONS: We found that PGY-1 surgical resident pass/fail status on a technical skills examination was associated with future performance on in-training evaluation reports in later years. This provides validity evidence for the current PGY-1 pass/fail score, and suggests that this technical skills examination may be used to predict performance and to identify residents who require remediation.
Authors: L Southgate; R B Hays; J Norcini; H Mulholland; B Ayers; J Woolliscroft; M Cusimano; P McAvoy; M Ainsworth; S Haist; M Campbell Journal: Med Educ Date: 2001-05 Impact factor: 6.251
Authors: Barbara A Goff; Peter E Nielsen; Gretchen M Lentz; Greg E Chow; Robert W Chalmers; Dee Fenner; Lynn S Mandel Journal: Am J Obstet Gynecol Date: 2002-04 Impact factor: 8.661
Authors: Barbara Goff; Lynn Mandel; Gretchen Lentz; Amy Vanblaricom; Anne-Marie Amies Oelschlager; David Lee; Andrew Galakatos; Matthew Davies; Peter Nielsen Journal: Am J Obstet Gynecol Date: 2005-04 Impact factor: 8.661
Authors: Omer Burak Argun; Kristin Chrouser; Sanket Chauhan; Manoj Monga; Bodo Knudsen; Geoffrey N Box; David I Lee; Matthew T Gettman; Lauren H Poniatowski; Qi Wang; Troy E Reihsen; Robert M Sweet Journal: J Urol Date: 2015-05-27 Impact factor: 7.450