Bradford D Winters1, Aamir Bharmal, Renee F Wilson, Allen Zhang, Lilly Engineer, Deidre Defoe, Eric B Bass, Sydney Dy, Peter J Pronovost. 1. *Johns Hopkins School of Medicine Armstrong Institute for Patient Safety and Quality, Baltimore, MD Departments of †Health Policy and Management, Johns Hopkins School of Medicine ‡Anesthesiology and Critical Care Medicine, Johns Hopkins School of Medicine §Johns Hopkins University Bloomberg School of Public Health ∥Department of Internal Medicine, Johns Hopkins School of Medicine, Baltimore, MD.
Abstract
BACKGROUND: The Agency for Health Care Research and Quality Patient Safety Indicators (PSIs) and Centers for Medicare and Medicaid Services Hospital-acquired Conditions (HACs) are increasingly being used for pay-for-performance and public reporting despite concerns over their validity. Given the potential for these measures to misinform patients, misclassify hospitals, and misapply financial and reputational harm to hospitals, these need to be rigorously evaluated. We performed a systematic review and meta-analysis to assess PSI and HAC measure validity. METHODS: We searched MEDLINE and the gray literature from January 1, 1990 through January 14, 2015 for studies that addressed the validity of the HAC measures and PSIs. Secondary outcomes included the effects of present on admission (POA) modifiers, and the most common reasons for discrepancies. We developed pooled results for measures evaluated by ≥3 studies. We propose a threshold of 80% for positive predictive value or sensitivity for pay-for-performance and public reporting suitability. RESULTS: Only 5 measures, Iatrogenic Pneumothorax (PSI 6/HAC 17), Central Line-associated Bloodstream Infections (PSI 7), Postoperative hemorrhage/hematoma (PSI 9), Postoperative deep vein thrombosis/pulmonary embolus (PSI 12), and Accidental Puncture/Laceration (PSI 15), had sufficient data for pooled meta-analysis. Only PSI 15 (Accidental Puncture and Laceration) met our proposed threshold for validity (positive predictive value only) but this result was weakened by considerable heterogeneity. Coding errors were the most common reasons for discrepancies between medical record review and administrative databases. POA modifiers may improve the validity of some measures. CONCLUSION: This systematic review finds that there is limited validity for the PSI and HAC measures when measured against the reference standard of a medical chart review. Their use, as they currently exist, for public reporting and pay-for-performance, should be publicly reevaluated in light of these findings.
BACKGROUND: The Agency for Health Care Research and Quality Patient Safety Indicators (PSIs) and Centers for Medicare and Medicaid Services Hospital-acquired Conditions (HACs) are increasingly being used for pay-for-performance and public reporting despite concerns over their validity. Given the potential for these measures to misinform patients, misclassify hospitals, and misapply financial and reputational harm to hospitals, these need to be rigorously evaluated. We performed a systematic review and meta-analysis to assess PSI and HAC measure validity. METHODS: We searched MEDLINE and the gray literature from January 1, 1990 through January 14, 2015 for studies that addressed the validity of the HAC measures and PSIs. Secondary outcomes included the effects of present on admission (POA) modifiers, and the most common reasons for discrepancies. We developed pooled results for measures evaluated by ≥3 studies. We propose a threshold of 80% for positive predictive value or sensitivity for pay-for-performance and public reporting suitability. RESULTS: Only 5 measures, Iatrogenic Pneumothorax (PSI 6/HAC 17), Central Line-associated Bloodstream Infections (PSI 7), Postoperative hemorrhage/hematoma (PSI 9), Postoperative deep vein thrombosis/pulmonary embolus (PSI 12), and Accidental Puncture/Laceration (PSI 15), had sufficient data for pooled meta-analysis. Only PSI 15 (Accidental Puncture and Laceration) met our proposed threshold for validity (positive predictive value only) but this result was weakened by considerable heterogeneity. Coding errors were the most common reasons for discrepancies between medical record review and administrative databases. POA modifiers may improve the validity of some measures. CONCLUSION: This systematic review finds that there is limited validity for the PSI and HAC measures when measured against the reference standard of a medical chart review. Their use, as they currently exist, for public reporting and pay-for-performance, should be publicly reevaluated in light of these findings.
Authors: Yanru Qiao; Christina A Spivey; Junling Wang; Ya-Chen Tina Shih; Jim Y Wan; Julie Kuhle; Samuel Dagogo-Jack; William C Cushman; Marie Chisholm-Burns Journal: J Pharm Health Serv Res Date: 2018-09-07
Authors: Eline F de Vries; Jeroen N Struijs; Richard Heijink; Roy J P Hendrikx; Caroline A Baan Journal: BMC Health Serv Res Date: 2016-08-18 Impact factor: 2.655
Authors: Marie-Annick Le Pogam; Catherine Quantin; Oliver Reich; Philippe Tuppin; Anne Fagot-Campagna; Fred Paccaud; Isabelle Peytremann-Bridevaux; Bernard Burnand Journal: JMIR Res Protoc Date: 2017-05-11
Authors: Saul N Weingart; Jason Nelson; Benjamin Koethe; Omar Yaghi; Stephan Dunning; Albert Feldman; David Kent; Allison Lipitz-Snyderman Journal: Cancer Med Date: 2020-04-13 Impact factor: 4.452