The impact of the inclusion of simulated patient ratings on the reliability of OSCE assessments under the borderline regression method.

BACKGROUND: In Objective Structured Clinical Examinations (OSCEs), the use of simulated patients (SPs) at many stations is a key aspect of the assessment. Often the SPs are asked to provide formal feedback (ratings) of their experience with the students under examination. AIMS: This study analyses whether and how exactly SP data can be best used to enhance the robustness of the formal standard setting process.
METHODS: A retrospective statistical investigation into the relationship between SP ratings and those provided by the clinical assessors (criterion-based checklist scores and overall grades for each station) is presented. In addition, the paper also includes a study into the impact of the inclusion of the SP ratings in the formal standard setting process for OSCEs, particularly when pass marks are calculated using the borderline regression method.
RESULTS: The general results of the analysis, including discussion of two distinct methods for the combining of the SP ratings and assessor judgements, are presented, and demonstrate that the inclusion of this additional data can have important effects on individual student results.
CONCLUSION: It is possible for the overall quality of the OSCE assessment process to be improved, with increased reliability, by combining assessor checklist scores and SP ratings.