OBJECTIVE: To explore the validity and reliability of the affective competency score (ACS), compared to a global rating measure to predict overall competency to perform a death disclosure in a standardized patient exercise and to investigate useful thresholds of the ACS. METHODS: Thirty-seven fourth-year students underwent standardized patient training in death disclosure during a fourth-year emergency medicine clerkship. Students were evaluated using a checklist, an ACS, and a global rating assessment. ACS interrater reliability, interitem reliability, item-total reliability, and split-half reliability were calculated. Area under the curve (AUC) measurements were used to establish criterion validity. RESULTS: For the ACS, item-total correlations ranged from 0.76 to 0.85, 0.76 to 0.93, and 0.42 to 0.87; the split-half reliability was 0.82 (p = 0.0001), 0.86 (p = 0.0001) and 0.55 (p = 0.0007) for the standardized patient (SP), the faculty and the medical students, respectively. Interitem correlations were adequate. A moderate interrater correlation of the ACS was observed between the faculty observer and the SP (r = 0.47; p = 0.04); however, the medical students' self evaluation did not correlate significantly with either the SP (r = -0.04; p = 0.79), or the faculty observer (r = 0.00; p = 0.99). The AUC for was 0.98 (95% confidence interval [CI] 0.94 to 1.00), 0.87 (95% CI 0.73 to 0.99), and 0.74 (95% CI 0.53 to 0.95) for the faculty, SP, and medical student, respectively. CONCLUSIONS: The ACS may be a valid, reliable, and useful measure to assess communication skills by faculty or SPs in this setting. At an ACS score of 16, 19, and 21 points for faculty, SPs, and medical students, respectively, there is 100% specificity for the detection of competency assessed on a global rating. However, the ACS appears to have limited reliability and validity when used by medical students.
OBJECTIVE: To explore the validity and reliability of the affective competency score (ACS), compared to a global rating measure to predict overall competency to perform a death disclosure in a standardized patient exercise and to investigate useful thresholds of the ACS. METHODS: Thirty-seven fourth-year students underwent standardized patient training in death disclosure during a fourth-year emergency medicine clerkship. Students were evaluated using a checklist, an ACS, and a global rating assessment. ACS interrater reliability, interitem reliability, item-total reliability, and split-half reliability were calculated. Area under the curve (AUC) measurements were used to establish criterion validity. RESULTS: For the ACS, item-total correlations ranged from 0.76 to 0.85, 0.76 to 0.93, and 0.42 to 0.87; the split-half reliability was 0.82 (p = 0.0001), 0.86 (p = 0.0001) and 0.55 (p = 0.0007) for the standardized patient (SP), the faculty and the medical students, respectively. Interitem correlations were adequate. A moderate interrater correlation of the ACS was observed between the faculty observer and the SP (r = 0.47; p = 0.04); however, the medical students' self evaluation did not correlate significantly with either the SP (r = -0.04; p = 0.79), or the faculty observer (r = 0.00; p = 0.99). The AUC for was 0.98 (95% confidence interval [CI] 0.94 to 1.00), 0.87 (95% CI 0.73 to 0.99), and 0.74 (95% CI 0.53 to 0.95) for the faculty, SP, and medical student, respectively. CONCLUSIONS: The ACS may be a valid, reliable, and useful measure to assess communication skills by faculty or SPs in this setting. At an ACS score of 16, 19, and 21 points for faculty, SPs, and medical students, respectively, there is 100% specificity for the detection of competency assessed on a global rating. However, the ACS appears to have limited reliability and validity when used by medical students.