Jorge Hermida1, Edward I Broughton, Lynne Miller Franco. 1. University Research Co, LLC/HCI, Avenida ColonE8-57 y Diego Almagro, Edificio El Cisne, Sto Piso Oficina C-D, Quito, Ecuador. jhermida@urc-chs.com
Abstract
OBJECTIVE: Health care quality improvement (QI) efforts commonly use self-assessment to measure compliance with quality standards. This study investigates the validity of self-assessment of quality indicators. DESIGN: Cross sectional. SETTING: A maternal and newborn care improvement collaborative intervention conducted in health facilities in Ecuador in 2005. PARTICIPANTS: Four external evaluators were trained in abstracting medical records to calculate six indicators reflecting compliance with treatment standards. INTERVENTIONS: About 30 medical records per month were examined at 12 participating health facilities for a total of 1875 records. The same records had already been reviewed by QI teams at these facilities (self-assessment). MAIN OUTCOME MEASURES: Overall compliance, agreement (using the Kappa statistic), sensitivity and specificity were analyzed. We also examined patterns of disagreement and the effect of facility characteristics on levels of agreement. RESULTS: External evaluators reported compliance of 69-90%, while self-assessors reported 71-92%, with raw agreement of 71-95% and Kappa statistics ranging from fair to almost perfect agreement. Considering external evaluators as the gold standard, sensitivity of self-assessment ranged from 90 to 99% and specificity from 48 to 86%. Simpler indicators had fewer disagreements. When disagreements occurred between self-assessment and external valuators, the former tended to report more positive findings in five of six indicators, but this tendency was not of a magnitude to change program actions. Team leadership, understanding of the tools and facility size had no overall impact on the level of agreement. CONCLUSIONS: When compared with external evaluation (gold standard), self-assessment was found to be sufficiently valid for tracking QI team performance. Sensitivity was generally higher than specificity. Simplifying indicators may improve validity.
OBJECTIVE: Health care quality improvement (QI) efforts commonly use self-assessment to measure compliance with quality standards. This study investigates the validity of self-assessment of quality indicators. DESIGN: Cross sectional. SETTING: A maternal and newborn care improvement collaborative intervention conducted in health facilities in Ecuador in 2005. PARTICIPANTS: Four external evaluators were trained in abstracting medical records to calculate six indicators reflecting compliance with treatment standards. INTERVENTIONS: About 30 medical records per month were examined at 12 participating health facilities for a total of 1875 records. The same records had already been reviewed by QI teams at these facilities (self-assessment). MAIN OUTCOME MEASURES: Overall compliance, agreement (using the Kappa statistic), sensitivity and specificity were analyzed. We also examined patterns of disagreement and the effect of facility characteristics on levels of agreement. RESULTS: External evaluators reported compliance of 69-90%, while self-assessors reported 71-92%, with raw agreement of 71-95% and Kappa statistics ranging from fair to almost perfect agreement. Considering external evaluators as the gold standard, sensitivity of self-assessment ranged from 90 to 99% and specificity from 48 to 86%. Simpler indicators had fewer disagreements. When disagreements occurred between self-assessment and external valuators, the former tended to report more positive findings in five of six indicators, but this tendency was not of a magnitude to change program actions. Team leadership, understanding of the tools and facility size had no overall impact on the level of agreement. CONCLUSIONS: When compared with external evaluation (gold standard), self-assessment was found to be sufficiently valid for tracking QI team performance. Sensitivity was generally higher than specificity. Simplifying indicators may improve validity.