William H Replogle1, William D Johnson. 1. Department of Family Medicine, University of Mississippi Medical Center, Jackson, 32916, USA. wrepogle@familymed.umsmed.edu
Abstract
BACKGROUND AND OBJECTIVES: The American Board of Family Practice In-training Examination (ABFP ITE) is a cognitive examination similar in content to the ABFP Certification Examination (CE). The ABFP ITE is widely used in family medicine residency programs. It was originally developed and intended to be used for assessment of groups of residents. Despite lack of empirical support, however, some residency programs are using ABFP ITE scores as individual resident performance indicators. This study's objective was to estimate the positive predictive value of the ABFP ITE for identifying residents at risk for poor performance on the ABFP CE or a subsequent ABFP ITE. METHODS: We used a normal distribution model for correlated test scores and Monte Carlo simulation to investigate the effect of test reliability (measurement errors) on the positive predictive value of the ABFP ITE. RESULTS: The positive predictive value of the composite score was .72. The positive predictive value of the eight specialty subscales ranged from .26 to .57. CONCLUSIONS: Only the composite score of the ABFP ITE has acceptable positive predictive value to be used as part of a comprehension resident evaluation system. The ABFP ITE specialty subscales do not have sufficient positive predictive value or reliability to warrant use as performance indicators.
BACKGROUND AND OBJECTIVES: The American Board of Family Practice In-training Examination (ABFP ITE) is a cognitive examination similar in content to the ABFP Certification Examination (CE). The ABFP ITE is widely used in family medicine residency programs. It was originally developed and intended to be used for assessment of groups of residents. Despite lack of empirical support, however, some residency programs are using ABFP ITE scores as individual resident performance indicators. This study's objective was to estimate the positive predictive value of the ABFP ITE for identifying residents at risk for poor performance on the ABFP CE or a subsequent ABFP ITE. METHODS: We used a normal distribution model for correlated test scores and Monte Carlo simulation to investigate the effect of test reliability (measurement errors) on the positive predictive value of the ABFP ITE. RESULTS: The positive predictive value of the composite score was .72. The positive predictive value of the eight specialty subscales ranged from .26 to .57. CONCLUSIONS: Only the composite score of the ABFP ITE has acceptable positive predictive value to be used as part of a comprehension resident evaluation system. The ABFP ITE specialty subscales do not have sufficient positive predictive value or reliability to warrant use as performance indicators.