BACKGROUND: The UK Department of Health is considering a single, generic multi-source feedback (MSF) questionnaire to inform revalidation. METHOD: Evaluation of an implementation pilot, reporting: response rates, assessor mix, question redundancy and participants' perceptions. Reliability was estimated using Generalisability theory. RESULTS: A total of 12,540 responses were received on 977 doctors. The mean time taken to complete an MSF exercise was 68.2 days. The mean number of responses received per doctor was 12.0 (range 1-17) with no significant difference between specialties. Individual question response rates and participants' comments about questions indicate that some questions are less appropriate for some specialities. There was a significant difference in the mean score between specialities. Despite guidance, there were significant differences in the mix of assessors across specialties. More favourable scores were given by progressively more junior doctors. Nurses gave the most reliable scores. CONCLUSIONS: It is feasible to electronically administer a generic questionnaire to a large population of doctors. Generic content is appropriate for most but not all specialties. The differences in mean scores and the reliability of the MSF between specialties may be in part due to the specialty differences in assessor mix. Therefore the number and assessor mix should be standardised at specialty level and scores should not be compared across specialties.
BACKGROUND: The UK Department of Health is considering a single, generic multi-source feedback (MSF) questionnaire to inform revalidation. METHOD: Evaluation of an implementation pilot, reporting: response rates, assessor mix, question redundancy and participants' perceptions. Reliability was estimated using Generalisability theory. RESULTS: A total of 12,540 responses were received on 977 doctors. The mean time taken to complete an MSF exercise was 68.2 days. The mean number of responses received per doctor was 12.0 (range 1-17) with no significant difference between specialties. Individual question response rates and participants' comments about questions indicate that some questions are less appropriate for some specialities. There was a significant difference in the mean score between specialities. Despite guidance, there were significant differences in the mix of assessors across specialties. More favourable scores were given by progressively more junior doctors. Nurses gave the most reliable scores. CONCLUSIONS: It is feasible to electronically administer a generic questionnaire to a large population of doctors. Generic content is appropriate for most but not all specialties. The differences in mean scores and the reliability of the MSF between specialties may be in part due to the specialty differences in assessor mix. Therefore the number and assessor mix should be standardised at specialty level and scores should not be compared across specialties.
Authors: John L Campbell; Martin Roberts; Christine Wright; Jacqueline Hill; Michael Greco; Matthew Taylor; Suzanne Richards Journal: BMJ Date: 2011-10-27
Authors: Eva K Hennel; Ulrike Subotic; Christoph Berendonk; Daniel Stricker; Sigrid Harendza; Sören Huwendiek Journal: BMC Med Educ Date: 2020-10-12 Impact factor: 2.463