Comparing Resident Self-Report to Chart Audits for Quality Improvement Projects: Accurate Reflection or Cherry-Picking?

BACKGROUND: Resident engagement in quality improvement is a requirement for graduate medical education, but the optimal means of instruction and evaluation of resident progress remain unknown.
OBJECTIVE: To determine the accuracy of self-reported chart audits in measuring resident adherence to primary care clinical practice guidelines.
METHODS: During the 2010-2011 academic year, second- and third-year internal medicine residents at a single, university hospital-based program performed chart audits on 10 patients from their primary care clinic to determine adherence to 16 US Preventive Services Task Force primary care guidelines. We compared residents' responses to independent audits of randomly selected patient charts by a single external reviewer.
RESULTS: Self-reported data were collected by 18 second-year and 15 third-year residents for 330 patients. Independently, 70 patient charts were randomly selected for review by an external auditor. Overall guideline compliance was significantly higher on self-reported audits compared to external audits (82% versus 68%, P < .001). Of 16 guidelines, external audits found significantly lower rates of adherence for 5 (tetanus vaccination, osteoporosis screening, colon cancer screening, cholesterol screening, and obesity screening). Chlamydia screening was more common in audited charts than in self-reported data. Although third-year residents self-reported higher guideline adherence than second-year residents (86% versus 78%, P < .001), external audits for third-year residents found lower overall adherence (64% versus 72%, P  =  .040).
CONCLUSIONS: Residents' self-reported chart audits may significantly overestimate guideline adherence. Increased supervision and independent review appear necessary to accurately evaluate resident performance.