Diane B Wayne1, Elaine Cohen, Gregory Makoul, William C McGaghie. 1. Department of Medicine, Northwestern University Feinberg School of Medicine, Galter 3-150, 201 East Huron Street, Chicago, IL 60611, USA. dwayne@northwestern.edu
Abstract
BACKGROUND: Patient surveys such as the Communication Assessment Tool (CAT) are recommended to assess physicians. Optimal methods to select judges and set a minimum passing standard (MPS) for a patient survey are unknown. METHOD: Fifty-eight judges in five groups provided item-based (Angoff) and group-based (Hofstee) judgments for the CAT on two occasions. Judges were communication experts, program directors, trainees, and patients (two groups). Ratings were used to calculate MPSs for the CAT. Interrater reliability and test-retest reliability (stability) were estimated. MPSs were applied to data from a CAT pilot study. RESULTS: Judges produced different MPSs depending on their background. Patients and communication experts were most stringent in their MPS decisions; program directors and trainees were most lenient. Failure rates from the pilot study ranged from 0% to 47%. CONCLUSIONS: Judge expertise and background dramatically influenced MPSs for a patient survey of physician communication skills. Judge selection is a key decision when setting an MPS for a clinical skills evaluation.
BACKGROUND:Patient surveys such as the Communication Assessment Tool (CAT) are recommended to assess physicians. Optimal methods to select judges and set a minimum passing standard (MPS) for a patient survey are unknown. METHOD: Fifty-eight judges in five groups provided item-based (Angoff) and group-based (Hofstee) judgments for the CAT on two occasions. Judges were communication experts, program directors, trainees, and patients (two groups). Ratings were used to calculate MPSs for the CAT. Interrater reliability and test-retest reliability (stability) were estimated. MPSs were applied to data from a CAT pilot study. RESULTS: Judges produced different MPSs depending on their background. Patients and communication experts were most stringent in their MPS decisions; program directors and trainees were most lenient. Failure rates from the pilot study ranged from 0% to 47%. CONCLUSIONS: Judge expertise and background dramatically influenced MPSs for a patient survey of physician communication skills. Judge selection is a key decision when setting an MPS for a clinical skills evaluation.
Authors: Jeffrey H Barsuk; Rebecca S Harap; Elaine R Cohen; Kenzie A Cameron; Kathleen L Grady; Jane E Wilcox; Kerry B Shanklin; Diane B Wayne Journal: Clin Simul Nurs Date: 2018-11-27 Impact factor: 2.391
Authors: Kristin L Rising; Rhea E Powell; Kenzie A Cameron; David H Salzman; Dimitrios Papanagnou; Amanda M B Doty; Lori Latimer; Katherine Piserchia; William C McGaghie; Danielle M McCarthy Journal: Acad Med Date: 2020-07 Impact factor: 6.893
Authors: David H Salzman; Kristin L Rising; Kenzie A Cameron; Rhea E Powell; Dimitri Papanagnou; Amanda Doty; Katherine Piserchia; Lori Latimer; William C McGaghie; Danielle M McCarthy Journal: J Grad Med Educ Date: 2020-02
Authors: Roberta Bowen; Kate M Lally; Francine R Pingitore; Richard Tucker; Elisabeth C McGowan; Beatrice E Lechner Journal: PLoS One Date: 2020-03-09 Impact factor: 3.240