OBJECTIVES: The continuity-structured clinical observation tool was developed and used to conduct a multiple-observer evaluation to assess residents in the core competency areas of patient care, interpersonal and communication skills, and professionalism. The objectives were to assess pediatric resident performance in a continuity clinic by using direct observation and to compare evaluations among preceptors, residents, and parents. METHODS: Pediatric residents in a large continuity clinic participated between August and December 2005. The continuity-structured clinical tool included items in the following domains: opening the interview, history taking, relationship skills, personal manner, negotiation or management, and physical examination. Each resident was directly observed during 1 entire patient encounter. Parents, preceptors, and residents completed evaluations by selecting 1 of 4 possible responses for performance of each item. We dichotomized responses as "yes" versus "no/partial" and analyzed aggregate scores for individual items and domains among the 3 evaluators by McNemar test, percentage agreement, and interoberserver agreement (kappa). RESULTS: Fifty-four of 57 eligible residents had all 3 evaluations completed. Parents rated residents the highest and showed least variability (only 2 items for which parents indicated the task was completed in <90% of the encounters). Residents rated themselves the lowest (35 items with <90%). In comparing the residents and preceptors, the domains that had the lowest percentage of agreement were history taking (range: 61%-91%) and negotiation or management (range: 51%-88%). All of the evaluators scored residents the lowest in the domain of negotiation or management, with the following lowest-score items: probe for decision-makers, assess willingness and barriers, and use of visual aids. CONCLUSIONS: Compared with parents, residents and preceptors demonstrated greater variability in resident performance evaluations. All of the evaluators scored residents lowest in the domain of negotiation or management during continuity-clinic visits. Residency programs should strongly consider emphasizing skill development in this area.
OBJECTIVES: The continuity-structured clinical observation tool was developed and used to conduct a multiple-observer evaluation to assess residents in the core competency areas of patient care, interpersonal and communication skills, and professionalism. The objectives were to assess pediatric resident performance in a continuity clinic by using direct observation and to compare evaluations among preceptors, residents, and parents. METHODS: Pediatric residents in a large continuity clinic participated between August and December 2005. The continuity-structured clinical tool included items in the following domains: opening the interview, history taking, relationship skills, personal manner, negotiation or management, and physical examination. Each resident was directly observed during 1 entire patient encounter. Parents, preceptors, and residents completed evaluations by selecting 1 of 4 possible responses for performance of each item. We dichotomized responses as "yes" versus "no/partial" and analyzed aggregate scores for individual items and domains among the 3 evaluators by McNemar test, percentage agreement, and interoberserver agreement (kappa). RESULTS: Fifty-four of 57 eligible residents had all 3 evaluations completed. Parents rated residents the highest and showed least variability (only 2 items for which parents indicated the task was completed in <90% of the encounters). Residents rated themselves the lowest (35 items with <90%). In comparing the residents and preceptors, the domains that had the lowest percentage of agreement were history taking (range: 61%-91%) and negotiation or management (range: 51%-88%). All of the evaluators scored residents the lowest in the domain of negotiation or management, with the following lowest-score items: probe for decision-makers, assess willingness and barriers, and use of visual aids. CONCLUSIONS: Compared with parents, residents and preceptors demonstrated greater variability in resident performance evaluations. All of the evaluators scored residents lowest in the domain of negotiation or management during continuity-clinic visits. Residency programs should strongly consider emphasizing skill development in this area.