A I Rothman1, M Cusimano. 1. Department of Medicine, University of Toronto, Toronto, Ontario, Canada. arthur.rothman@utoronto.ca
Abstract
PURPOSE: To assess the quality of ratings of interviewing skills and oral English proficiency provided on a clinical skills OSCE by physician examiners, standardized patients (SPs), and communication skills experts. METHOD: In 1998, 73 candidates to the Ontario International Medical Graduate (OIMG) Program completed a 29-station OSCE-type clinical skills selection examination. Physician examiners, SPs, and communication skills experts assessed components of oral English proficiency and interview performance. Based on these results, the frequency and generalizability of English-language flags, physician examiners' indications that spoken English skills were bad enough to significantly impede communication with patients; the reliability of the OIMG's Interview and Oral Performance Scales and generalizability of overall interview and oral performance ratings; and comparisons of repeated assessments by experts were calculated. Principal-components analysis was applied to the panels' ratings to determine a more economical expression of the language proficiency and interview communication skills results. RESULTS: The mean number of English-language flags per candidate was 2.1, the median was 1.0, and Cronbach's alpha of the ratings was 0.63. Means, SDs, and alphas of the physician examiners' and SPs' ratings of the interview performance scale were 9.15/10, 0.43, 0.36, and 9.30/10, 0. 56, 0.50, respectively. Corresponding values for overall interview performance ratings were 3.08/4, 0.30, 0.33, and 3.34/4, 0.32, 0.47. Means, SDs, and alphas of the physician examiners' and SPs' ratings of the oral performance scale were 8.54/10, 0.74, 0.78, and 8.74/10, 1.00, 0.76. Corresponding values for overall ratings of oral performance were 3.85/5, 0.51, 0.68, and 4.08/5, 0.60, 0.68. For the two experts' ratings of two contiguous five-minute interview stations, internal consistencies were 0.88 and 0.78. For the two experts' ratings of standardized ten-minute interviews, internal consistencies were 0.81 and 0.92. Correlations between the mean values of the experts' ratings of the ten- and five-minute stations were 0.45 and 0.51. Three factors emerged from the PCA, language proficiency, physician examiners' ratings of interview proficiency, and SPs' ratings of interview proficiency. CONCLUSIONS: Consistency between the physician examiners' and SPs' ratings of English proficiency was observed; less agreement was observed in their ratings of interviewing skills, and little agreement was observed between the experts' ratings. Communication skills results may be validly expressed by three measures: one overall global rating of language proficiency provided by physician examiners or SPs, and overall global ratings of interview proficiency provided separately by physician examiners and SPs.
PURPOSE: To assess the quality of ratings of interviewing skills and oral English proficiency provided on a clinical skills OSCE by physician examiners, standardized patients (SPs), and communication skills experts. METHOD: In 1998, 73 candidates to the Ontario International Medical Graduate (OIMG) Program completed a 29-station OSCE-type clinical skills selection examination. Physician examiners, SPs, and communication skills experts assessed components of oral English proficiency and interview performance. Based on these results, the frequency and generalizability of English-language flags, physician examiners' indications that spoken English skills were bad enough to significantly impede communication with patients; the reliability of the OIMG's Interview and Oral Performance Scales and generalizability of overall interview and oral performance ratings; and comparisons of repeated assessments by experts were calculated. Principal-components analysis was applied to the panels' ratings to determine a more economical expression of the language proficiency and interview communication skills results. RESULTS: The mean number of English-language flags per candidate was 2.1, the median was 1.0, and Cronbach's alpha of the ratings was 0.63. Means, SDs, and alphas of the physician examiners' and SPs' ratings of the interview performance scale were 9.15/10, 0.43, 0.36, and 9.30/10, 0. 56, 0.50, respectively. Corresponding values for overall interview performance ratings were 3.08/4, 0.30, 0.33, and 3.34/4, 0.32, 0.47. Means, SDs, and alphas of the physician examiners' and SPs' ratings of the oral performance scale were 8.54/10, 0.74, 0.78, and 8.74/10, 1.00, 0.76. Corresponding values for overall ratings of oral performance were 3.85/5, 0.51, 0.68, and 4.08/5, 0.60, 0.68. For the two experts' ratings of two contiguous five-minute interview stations, internal consistencies were 0.88 and 0.78. For the two experts' ratings of standardized ten-minute interviews, internal consistencies were 0.81 and 0.92. Correlations between the mean values of the experts' ratings of the ten- and five-minute stations were 0.45 and 0.51. Three factors emerged from the PCA, language proficiency, physician examiners' ratings of interview proficiency, and SPs' ratings of interview proficiency. CONCLUSIONS: Consistency between the physician examiners' and SPs' ratings of English proficiency was observed; less agreement was observed in their ratings of interviewing skills, and little agreement was observed between the experts' ratings. Communication skills results may be validly expressed by three measures: one overall global rating of language proficiency provided by physician examiners or SPs, and overall global ratings of interview proficiency provided separately by physician examiners and SPs.