BACKGROUND: Observational healthcare data offer the potential to enable identification of risks of medical products, and the medical literature is replete with analyses that aim to accomplish this objective. A number of established analytic methods dominate the literature but their operating characteristics in real-world settings remain unknown. OBJECTIVES: To compare the performance of seven methods (new user cohort, case control, self-controlled case series, self-controlled cohort, disproportionality analysis, temporal pattern discovery, and longitudinal gamma poisson shrinker) as tools for risk identification in observational healthcare data. RESEARCH DESIGN: The experiment applied each method to 399 drug-outcome scenarios (165 positive controls and 234 negative controls across 4 health outcomes of interest) in 5 real observational databases (4 administrative claims and 1 electronic health record). MEASURES: Method performance was evaluated through Area Under the receiver operator characteristics Curve (AUC), bias, mean square error, and confidence interval coverage probability. RESULTS: Multiple methods offer strong predictive accuracy, with AUC > 0.70 achievable for all outcomes and databases with more than one analytical approach. Self-controlled methods (self-controlled case series, temporal pattern discovery, self-controlled cohort) had higher predictive accuracy than cohort and case-control methods across all databases and outcomes. Methods differed in the expected value and variance of the error distribution. All methods had lower coverage probability than the expected nominal properties. CONCLUSIONS: Observational healthcare data can inform risk identification of medical product effects on acute liver injury, acute myocardial infarction, acute renal failure and gastrointestinal bleeding. However, effect estimates from all methods require calibration to address inconsistency in method operating characteristics. Further empirical evaluation is required to gauge the generalizability of these findings to other databases and outcomes.
BACKGROUND: Observational healthcare data offer the potential to enable identification of risks of medical products, and the medical literature is replete with analyses that aim to accomplish this objective. A number of established analytic methods dominate the literature but their operating characteristics in real-world settings remain unknown. OBJECTIVES: To compare the performance of seven methods (new user cohort, case control, self-controlled case series, self-controlled cohort, disproportionality analysis, temporal pattern discovery, and longitudinal gamma poisson shrinker) as tools for risk identification in observational healthcare data. RESEARCH DESIGN: The experiment applied each method to 399 drug-outcome scenarios (165 positive controls and 234 negative controls across 4 health outcomes of interest) in 5 real observational databases (4 administrative claims and 1 electronic health record). MEASURES: Method performance was evaluated through Area Under the receiver operator characteristics Curve (AUC), bias, mean square error, and confidence interval coverage probability. RESULTS: Multiple methods offer strong predictive accuracy, with AUC > 0.70 achievable for all outcomes and databases with more than one analytical approach. Self-controlled methods (self-controlled case series, temporal pattern discovery, self-controlled cohort) had higher predictive accuracy than cohort and case-control methods across all databases and outcomes. Methods differed in the expected value and variance of the error distribution. All methods had lower coverage probability than the expected nominal properties. CONCLUSIONS: Observational healthcare data can inform risk identification of medical product effects on acute liver injury, acute myocardial infarction, acute renal failure and gastrointestinal bleeding. However, effect estimates from all methods require calibration to address inconsistency in method operating characteristics. Further empirical evaluation is required to gauge the generalizability of these findings to other databases and outcomes.
Authors: Joshua J Gagne; Bruce Fireman; Patrick B Ryan; Malcolm Maclure; Tobias Gerhard; Sengwee Toh; Jeremy A Rassen; Jennifer C Nelson; Sebastian Schneeweiss Journal: Pharmacoepidemiol Drug Saf Date: 2012-01 Impact factor: 2.890
Authors: Jeffrey S Brown; Martin Kulldorff; K Arnold Chan; Robert L Davis; David Graham; Parker T Pettus; Susan E Andrade; Marsha A Raebel; Lisa Herrinton; Douglas Roblin; Denise Boudreau; David Smith; Jerry H Gurwitz; Margaret J Gunter; Richard Platt Journal: Pharmacoepidemiol Drug Saf Date: 2007-12 Impact factor: 2.890
Authors: Marc A Suchard; Ivan Zorych; Shawn E Simpson; Martijn J Schuemie; Patrick B Ryan; David Madigan Journal: Drug Saf Date: 2013-10 Impact factor: 5.606
Authors: Robert S Bresalier; Robert S Sandler; Hui Quan; James A Bolognese; Bettina Oxenius; Kevin Horgan; Christopher Lines; Robert Riddell; Dion Morton; Angel Lanas; Marvin A Konstam; John A Baron Journal: N Engl J Med Date: 2005-02-15 Impact factor: 91.245
Authors: Patrick B Ryan; Martijn J Schuemie; Emily Welebob; Jon Duke; Sarah Valentine; Abraham G Hartzema Journal: Drug Saf Date: 2013-10 Impact factor: 5.606
Authors: Preciosa M Coloma; Paul Avillach; Francesco Salvo; Martijn J Schuemie; Carmen Ferrajolo; Antoine Pariente; Annie Fourrier-Réglat; Mariam Molokhia; Vaishali Patadia; Johan van der Lei; Miriam Sturkenboom; Gianluca Trifirò Journal: Drug Saf Date: 2013-01 Impact factor: 5.606
Authors: John S Brownstein; Shawn N Murphy; Allison B Goldfine; Richard W Grant; Margarita Sordo; Vivian Gainer; Judith A Colecchi; Anil Dubey; David M Nathan; John P Glaser; Isaac S Kohane Journal: Diabetes Care Date: 2009-12-15 Impact factor: 19.112
Authors: Martijn J Schuemie; M Soledad Cepeda; Marc A Suchard; Jianxiao Yang; Yuxi Tian; Alejandro Schuler; Patrick B Ryan; David Madigan; George Hripcsak Journal: Harv Data Sci Rev Date: 2020-01-31
Authors: Martijn J Schuemie; Rosa Gini; Preciosa M Coloma; Huub Straatman; Ron M C Herings; Lars Pedersen; Francesco Innocenti; Giampiero Mazzaglia; Gino Picelli; Johan van der Lei; Miriam C J M Sturkenboom Journal: Drug Saf Date: 2013-10 Impact factor: 5.606
Authors: Paul E Stang; Patrick B Ryan; J Marc Overhage; Martijn J Schuemie; Abraham G Hartzema; Emily Welebob Journal: Drug Saf Date: 2013-10 Impact factor: 5.606