Generalizability of a study sample assessment procedure for entrance selection for medical school.

To optimize the validity of instruments for the selection of students for admission to medical school a close resemblance between selection criteria and activities in medical school and patient care is proposed. A study sample assessment procedure (SSAP), focusing on independent studying, collaboration with peers and providing information to standardized patients, was designed and has been applied as a selection tool at UMC Utrecht Medical School since 2001. The interviews with standardized patients are observed and rated on the quality of information provided and the quality of communication. This study investigates the psychometric properties of this observational procedure. Generalizability theory was applied to estimate the reliability of the SSAP and to compare it with other procedures carried out in the same populations, such as the rating of application forms and a structured interview procedure. Data from three years were analysed. The G-coefficients for the SSAP (0.84 to 0.90) were higher than those for the interview and the application form (0.74 to 0.83 and 0.53 to 0.61 respectively). In conclusion, the SSAP appears to be a feasible and reliable procedure. The number of raters could, if necessary, be reduced from three to two.