OBJECTIVE: To demonstrate how failure to account for measurement error in an outcome (dependent) variable can lead to significant estimation errors and to illustrate ways to recognize and avoid these errors. DATA SOURCES: Medical literature and simulation models. STUDY DESIGN/DATA COLLECTION: Systematic review of the published and unpublished epidemiological literature on the rate of preventable hospital deaths and statistical simulation of potential estimation errors based on data from these studies. PRINCIPAL FINDINGS: Most estimates of the rate of preventable deaths in U.S. hospitals rely upon classifying cases using one to three physician reviewers (implicit review). Because this method has low to moderate reliability, estimates based on statistical methods that do not account for error in the measurement of a "preventable death" can result in significant overestimation. For example, relying on a majority rule rating with three reviewers per case (reliability approximately 0.45 for the average of three reviewers) can result in a 50-100 percent overestimation compared with an estimate based upon a reliably measured outcome (e.g., by using 50 reviewers per case). However, there are statistical methods that account for measurement error that can produce much more accurate estimates of outcome rates without requiring a large number of measurements per case. CONCLUSION: The statistical principles discussed in this case study are critically important whenever one seeks to estimate the proportion of cases belonging to specific categories (such as estimating how many patients have inadequate blood pressure control or identifying high-cost or low-quality physicians). When the true outcome rate is low (<20 percent), using an outcome measure that has low-to-moderate reliability will generally result in substantially overestimating the proportion of the population having the outcome unless statistical methods that adjust for measurement error are used.
OBJECTIVE: To demonstrate how failure to account for measurement error in an outcome (dependent) variable can lead to significant estimation errors and to illustrate ways to recognize and avoid these errors. DATA SOURCES: Medical literature and simulation models. STUDY DESIGN/DATA COLLECTION: Systematic review of the published and unpublished epidemiological literature on the rate of preventable hospital deaths and statistical simulation of potential estimation errors based on data from these studies. PRINCIPAL FINDINGS: Most estimates of the rate of preventable deaths in U.S. hospitals rely upon classifying cases using one to three physician reviewers (implicit review). Because this method has low to moderate reliability, estimates based on statistical methods that do not account for error in the measurement of a "preventable death" can result in significant overestimation. For example, relying on a majority rule rating with three reviewers per case (reliability approximately 0.45 for the average of three reviewers) can result in a 50-100 percent overestimation compared with an estimate based upon a reliably measured outcome (e.g., by using 50 reviewers per case). However, there are statistical methods that account for measurement error that can produce much more accurate estimates of outcome rates without requiring a large number of measurements per case. CONCLUSION: The statistical principles discussed in this case study are critically important whenever one seeks to estimate the proportion of cases belonging to specific categories (such as estimating how many patients have inadequate blood pressure control or identifying high-cost or low-quality physicians). When the true outcome rate is low (<20 percent), using an outcome measure that has low-to-moderate reliability will generally result in substantially overestimating the proportion of the population having the outcome unless statistical methods that adjust for measurement error are used.
Authors: E J Thomas; D M Studdert; W B Runciman; R K Webb; E J Sexton; R M Wilson; R W Gibberd; B T Harrison; T A Brennan Journal: Int J Qual Health Care Date: 2000-10 Impact factor: 2.038
Authors: Greg Arling; Mathew Reeves; Joseph Ross; Linda S Williams; Salomeh Keyhani; Neale Chumbler; Michael S Phipps; Christianne Roumie; Laura J Myers; Amanda H Salanitro; Diana L Ordin; Jennifer Myers; Dawn M Bravata Journal: Circ Cardiovasc Qual Outcomes Date: 2011-12-06
Authors: Ariann F Nassel; Elisabeth D Root; Jason S Haukoos; Kevin McVaney; Christopher Colwell; James Robinson; Brian Eigel; David J Magid; Comilla Sasson Journal: Resuscitation Date: 2014-09-27 Impact factor: 5.262
Authors: James P Marcin; Patrick S Romano; Madan Dharmar; James M Chamberlain; Nanette Dudley; Charles G Macias; Lise E Nigrovic; Elizabeth C Powell; Alexander J Rogers; Meridith Sonnett; Leah Tzimenatos; Elizabeth R Alpern; Rebecca Andrews-Dickert; Dominic A Borgialli; Erika Sidney; Charlie Casper; Jonathan Michael Dean; Nathan Kuppermann Journal: Health Serv Res Date: 2017-11-16 Impact factor: 3.402
Authors: Elizabeth M Viglianti; Sean M Bagshaw; Rinaldo Bellomo; Joanne McPeake; Xiao Qing Wang; Sarah Seelye; Theodore J Iwashyna Journal: Intensive Care Med Date: 2020-06-04 Impact factor: 17.440
Authors: Deborah R Kaye; Rodney L Dunn; Jonathan Li; Lindsey A Herrel; James M Dupree; David C Miller; Chad Ellimoottil Journal: J Surg Res Date: 2018-12-04 Impact factor: 2.192