Daniel J Tancredi1, Klea D Bertakis, Anthony Jerant. 1. Department of Pediatrics, Center for Healthcare Policy and Research, University of California Davis School of Medicine, Sacramento, California 95817, USA.
Abstract
PURPOSE: The annual U.S. News & World Report (USN&WR) Primary Care Medical School (PCMS) ranking attracts considerable attention, but its measurement properties have not been published. The authors examined the short-term stability of the PCMS ranking and the PCMS score from which it derives, along with the short-term spread of schools' rankings. METHOD: The authors employed published data and methods to reconstruct the 2009-2012 PCMS scores and rankings. They used mixed-effects models to assess the within-school, between-year reliability (short-term stability) of the PCMS score and ranking, yielding intraclass correlation coefficients (ICCs). They defined short-term spread as the median within-school range in ranking across the four-year study period. RESULTS: Reconstructed PCMS scores correlated highly with published scores all four years (Pearson correlations≥98.9%). Most schools' mean annual PCMS scores were tightly clustered near the center of the score distribution. ICCs for the PCMS score and ranking were, respectively, 94% and 90%. The median difference between the best and worst ranking over the study period was 4 for the 18 schools with an average annual ranking of 1 to 20, and 17 for the other 89 schools (P<.001, Kruskal-Wallis test). CONCLUSIONS: The short-term stability of the USN&WR PCMS score and ranking were reasonably good. However, the short-term spread in PCMS rankings was large, particularly among schools with mean annual rankings below the top 20. The variability is greater than could be plausibly attributed to actual changes in training quality. These findings raise questions regarding the ranking's validity and usefulness.
PURPOSE: The annual U.S. News & World Report (USN&WR) Primary Care Medical School (PCMS) ranking attracts considerable attention, but its measurement properties have not been published. The authors examined the short-term stability of the PCMS ranking and the PCMS score from which it derives, along with the short-term spread of schools' rankings. METHOD: The authors employed published data and methods to reconstruct the 2009-2012 PCMS scores and rankings. They used mixed-effects models to assess the within-school, between-year reliability (short-term stability) of the PCMS score and ranking, yielding intraclass correlation coefficients (ICCs). They defined short-term spread as the median within-school range in ranking across the four-year study period. RESULTS: Reconstructed PCMS scores correlated highly with published scores all four years (Pearson correlations≥98.9%). Most schools' mean annual PCMS scores were tightly clustered near the center of the score distribution. ICCs for the PCMS score and ranking were, respectively, 94% and 90%. The median difference between the best and worst ranking over the study period was 4 for the 18 schools with an average annual ranking of 1 to 20, and 17 for the other 89 schools (P<.001, Kruskal-Wallis test). CONCLUSIONS: The short-term stability of the USN&WR PCMS score and ranking were reasonably good. However, the short-term spread in PCMS rankings was large, particularly among schools with mean annual rankings below the top 20. The variability is greater than could be plausibly attributed to actual changes in training quality. These findings raise questions regarding the ranking's validity and usefulness.
Authors: Julie M Donohue; Sharon-Lise T Normand; Marcela Horvitz-Lennon; Aiju Men; Ernst R Berndt; Haiden A Huskamp Journal: J Ment Health Policy Econ Date: 2016-06