Lawrence Fulton1, Cristian Lieneck2, Zo Ramamonjiarivelo2, Clemens Scott Kruse2, Matthew S Brooks2. 1. Department of Health Administration, 601 University Drive, Texas State University, San Marcos, TX, 78666, USA. lf25@txstate.edu. 2. Department of Health Administration, 601 University Drive, Texas State University, San Marcos, TX, 78666, USA.
Abstract
BACKGROUND: Assessing competencies or program learning outcomes in educational programs is often a leadership challenge. This case study reports medical education program's efforts to document undergraduate competency attainment using a pre-post, third-party, objective testing service that allows for inter-university comparison, a testing service that is being adopted by some certification and accrediting bodies. METHODS: Students completed a pre-test after program acceptance and a post-test at the end of the last didactic semester (1.5 years later) just prior to their required internships. Scores and subscores were evaluated using t-tests (Holm-adjusted p-values). MANOVA models of sub-competency difference scores were also evaluated. RESULTS: Results indicate competency improvement for each of the 12 areas based on the n = 55 student sample, (p < .001 for all scores). These improvements were independent of ethnicity, age, gender, and grades. The average student improved by 12.85 points (95% CI of 10.52 to 15.18) with the largest improvements in strategic planning and leadership competency areas (21.30 and 18.33 percentage points, respectively). CONCLUSIONS: The third-party pre-post has some face validity given that student performance improved after completing a related curriculum as would be expected. Congruent with earlier studies, we find that repeated testing helps document competency attainment and that a single method for assessment is insufficient. We further document limitations of this 3d-party exam.
BACKGROUND: Assessing competencies or program learning outcomes in educational programs is often a leadership challenge. This case study reports medical education program's efforts to document undergraduate competency attainment using a pre-post, third-party, objective testing service that allows for inter-university comparison, a testing service that is being adopted by some certification and accrediting bodies. METHODS: Students completed a pre-test after program acceptance and a post-test at the end of the last didactic semester (1.5 years later) just prior to their required internships. Scores and subscores were evaluated using t-tests (Holm-adjusted p-values). MANOVA models of sub-competency difference scores were also evaluated. RESULTS: Results indicate competency improvement for each of the 12 areas based on the n = 55 student sample, (p < .001 for all scores). These improvements were independent of ethnicity, age, gender, and grades. The average student improved by 12.85 points (95% CI of 10.52 to 15.18) with the largest improvements in strategic planning and leadership competency areas (21.30 and 18.33 percentage points, respectively). CONCLUSIONS: The third-party pre-post has some face validity given that student performance improved after completing a related curriculum as would be expected. Congruent with earlier studies, we find that repeated testing helps document competency attainment and that a single method for assessment is insufficient. We further document limitations of this 3d-party exam.
Authors: Judith G Calhoun; Pamela L Davidson; Marie E Sinioris; Eric T Vincent; John R Griffith Journal: Qual Manag Health Care Date: 2002 Impact factor: 0.926
Authors: C P M van der Vleuten; L W T Schuwirth; E W Driessen; J Dijkstra; D Tigelaar; L K J Baartman; J van Tartwijk Journal: Med Teach Date: 2012 Impact factor: 3.650