Rohan Khera1, Suveen Angraal2, Tyler Couch1, John W Welsh2, Brahmajee K Nallamothu3, Saket Girotra4, Paul S Chan5,6, Harlan M Krumholz2,7,8. 1. Division of Cardiology, University of Texas Southwestern Medical Center, Dallas. 2. Center for Outcomes Research and Evaluation, Yale-New Haven Hospital, New Haven, Connecticut. 3. Division of Cardiology, Department of Internal Medicine, University of Michigan, Ann Arbor. 4. Division of Cardiovascular Medicine, Department of Internal Medicine, University of Iowa Carver College of Medicine, Iowa City. 5. Saint Luke's Mid America Heart and Vascular Institute, Kansas City, Missouri. 6. University of Missouri-Kansas City, Kansas City, Missouri. 7. Section of Cardiovascular Medicine, Department of Internal Medicine, Yale School of Medicine, New Haven, Connecticut. 8. Department of Health Policy and Management, Yale School of Public Health, New Haven, Connecticut.
Abstract
Importance: Publicly available data sets hold much potential, but their unique design may require specific analytic approaches. Objective: To determine adherence to appropriate research practices for a frequently used large public database, the National Inpatient Sample (NIS) of the Agency for Healthcare Research and Quality (AHRQ). Design, Setting, and Participants: In this observational study of the 1082 studies published using the NIS from January 2015 through December 2016, a representative sample of 120 studies was systematically evaluated for adherence to practices required by AHRQ for the design and conduct of research using the NIS. Exposures: None. Main Outcomes and Measures: All studies were evaluated on 7 required research practices based on AHRQ's recommendations and compiled under 3 domains: (1) data interpretation (interpreting data as hospitalization records rather than unique patients); (2) research design (avoiding use in performing state-, hospital-, and physician-level assessments where inappropriate; not using nonspecific administrative secondary diagnosis codes to study in-hospital events); and (3) data analysis (accounting for complex survey design of the NIS and changes in data structure over time). Results: Of 120 published studies, 85% (n = 102) did not adhere to 1 or more required practices and 62% (n = 74) did not adhere to 2 or more required practices. An estimated 925 (95% CI, 852-998) NIS publications did not adhere to 1 or more required practices and 696 (95% CI, 596-796) NIS publications did not adhere to 2 or more required practices. A total of 79 sampled studies (68.3% [95% CI, 59.3%-77.3%]) among the 1082 NIS studies screened for eligibility did not account for the effects of sampling error, clustering, and stratification; 62 (54.4% [95% CI, 44.7%-64.0%]) extrapolated nonspecific secondary diagnoses to infer in-hospital events; 45 (40.4% [95% CI, 30.9%-50.0%]) miscategorized hospitalizations as individual patients; 10 (7.1% [95% CI, 2.1%-12.1%]) performed state-level analyses; and 3 (2.9% [95% CI, 0.0%-6.2%]) reported physician-level volume estimates. Of 27 studies (weighted; 218 studies [95% CI, 134-303]) spanning periods of major changes in the data structure of the NIS, 21 (79.7% [95% CI, 62.5%-97.0%]) did not account for the changes. Among the 24 studies published in journals with an impact factor of 10 or greater, 16 (67%) did not adhere to 1 or more practices, and 9 (38%) did not adhere to 2 or more practices. Conclusions and Relevance: In this study of 120 recent publications that used data from the NIS, the majority did not adhere to required practices. Further research is needed to identify strategies to improve the quality of research using the NIS and assess whether there are similar problems with use of other publicly available data sets.
Importance: Publicly available data sets hold much potential, but their unique design may require specific analytic approaches. Objective: To determine adherence to appropriate research practices for a frequently used large public database, the National Inpatient Sample (NIS) of the Agency for Healthcare Research and Quality (AHRQ). Design, Setting, and Participants: In this observational study of the 1082 studies published using the NIS from January 2015 through December 2016, a representative sample of 120 studies was systematically evaluated for adherence to practices required by AHRQ for the design and conduct of research using the NIS. Exposures: None. Main Outcomes and Measures: All studies were evaluated on 7 required research practices based on AHRQ's recommendations and compiled under 3 domains: (1) data interpretation (interpreting data as hospitalization records rather than unique patients); (2) research design (avoiding use in performing state-, hospital-, and physician-level assessments where inappropriate; not using nonspecific administrative secondary diagnosis codes to study in-hospital events); and (3) data analysis (accounting for complex survey design of the NIS and changes in data structure over time). Results: Of 120 published studies, 85% (n = 102) did not adhere to 1 or more required practices and 62% (n = 74) did not adhere to 2 or more required practices. An estimated 925 (95% CI, 852-998) NIS publications did not adhere to 1 or more required practices and 696 (95% CI, 596-796) NIS publications did not adhere to 2 or more required practices. A total of 79 sampled studies (68.3% [95% CI, 59.3%-77.3%]) among the 1082 NIS studies screened for eligibility did not account for the effects of sampling error, clustering, and stratification; 62 (54.4% [95% CI, 44.7%-64.0%]) extrapolated nonspecific secondary diagnoses to infer in-hospital events; 45 (40.4% [95% CI, 30.9%-50.0%]) miscategorized hospitalizations as individual patients; 10 (7.1% [95% CI, 2.1%-12.1%]) performed state-level analyses; and 3 (2.9% [95% CI, 0.0%-6.2%]) reported physician-level volume estimates. Of 27 studies (weighted; 218 studies [95% CI, 134-303]) spanning periods of major changes in the data structure of the NIS, 21 (79.7% [95% CI, 62.5%-97.0%]) did not account for the changes. Among the 24 studies published in journals with an impact factor of 10 or greater, 16 (67%) did not adhere to 1 or more practices, and 9 (38%) did not adhere to 2 or more practices. Conclusions and Relevance: In this study of 120 recent publications that used data from the NIS, the majority did not adhere to required practices. Further research is needed to identify strategies to improve the quality of research using the NIS and assess whether there are similar problems with use of other publicly available data sets.
Authors: Brenda Motheral; John Brooks; Mary Ann Clark; William H Crown; Peter Davey; Dave Hutchins; Bradley C Martin; Paul Stang Journal: Value Health Date: 2003 Mar-Apr Impact factor: 5.725
Authors: Xinsong Du; Jae Min; Chintan P Shah; Rohit Bishnoi; William R Hogan; Dominick J Lemas Journal: Int J Med Inform Date: 2020-04-15 Impact factor: 4.046
Authors: Paul T Kröner Florit; Juan E Corral Hurtado; Karn Wijarnpreecha; Enrique F Elli; Frank J Lukens Journal: Obes Surg Date: 2019-11 Impact factor: 4.129
Authors: David Metcalfe; Cheryl K Zogg; Elliott R Haut; Timothy M Pawlik; Adil H Haider; Daniel C Perry Journal: Int J Epidemiol Date: 2019-12-01 Impact factor: 7.196
Authors: Saraschandra Vallabhajosyula; Saarwaani Vallabhajosyula; Shannon M Dunlay; Sharonne N Hayes; Patricia J M Best; Jorge A Brenes-Salazar; Amir Lerman; Bernard J Gersh; Allan S Jaffe; Malcolm R Bell; David R Holmes; Gregory W Barsness Journal: Mayo Clin Proc Date: 2020-09 Impact factor: 7.616