Danette W McKinley1, John R Boulet. 1. Research and Evaluation Educational Commission for Foreign Medical Graduates, 3624 Market Street, 4th Floor, Philadelphia, PA 19104, USA. dmckinley@ecfmg.org
Abstract
BACKGROUND: Relatively little research has been published examining the use of factor analytic techniques to improve the psychometric qualities of performance assessments. The purpose of the current investigation is to illustrate the potential use of structural equation modeling (SEM) with scores from a standardized patient examination. METHOD: Checklist items for a single case were examined to compare two models within a SEM framework: a single, underlying "data gathering" construct and a two-factor model consisting of medical history interviewing and physical examination maneuvers. RESULTS: The chi2 statistics obtained were statistically significant for both the single factor (chi2=1645.8, p<.001) and the two-factor model, (chi2=1373.1; p<.001), indicating marginal model-data fit. A difference statistic was calculated (chi2=320.9; p<.001), suggesting that the two-factor model has better fit. CONCLUSIONS: The parameters estimated by each model could be used to evaluate checklist item performance. The results of this investigation illustrate the utility of this method in the analysis of checklist item data.
BACKGROUND: Relatively little research has been published examining the use of factor analytic techniques to improve the psychometric qualities of performance assessments. The purpose of the current investigation is to illustrate the potential use of structural equation modeling (SEM) with scores from a standardized patient examination. METHOD: Checklist items for a single case were examined to compare two models within a SEM framework: a single, underlying "data gathering" construct and a two-factor model consisting of medical history interviewing and physical examination maneuvers. RESULTS: The chi2 statistics obtained were statistically significant for both the single factor (chi2=1645.8, p<.001) and the two-factor model, (chi2=1373.1; p<.001), indicating marginal model-data fit. A difference statistic was calculated (chi2=320.9; p<.001), suggesting that the two-factor model has better fit. CONCLUSIONS: The parameters estimated by each model could be used to evaluate checklist item performance. The results of this investigation illustrate the utility of this method in the analysis of checklist item data.