BACKGROUND: Methodological research to evaluate the performance of methods requires a benchmark to serve as a referent comparison. In drug safety, the performance of analyses of spontaneous adverse event reporting databases and observational healthcare data, such as administrative claims and electronic health records, has been limited by the lack of such standards. OBJECTIVES: To establish a reference set of test cases that contain both positive and negative controls, which can serve the basis for methodological research in evaluating methods performance in identifying drug safety issues. RESEARCH DESIGN: Systematic literature review and natural language processing of structured product labeling was performed to identify evidence to support the classification of drugs as either positive controls or negative controls for four outcomes: acute liver injury, acute kidney injury, acute myocardial infarction, and upper gastrointestinal bleeding. RESULTS: Three-hundred and ninety-nine test cases comprised of 165 positive controls and 234 negative controls were identified across the four outcomes. The majority of positive controls for acute kidney injury and upper gastrointestinal bleeding were supported by randomized clinical trial evidence, while the majority of positive controls for acute liver injury and acute myocardial infarction were only supported based on published case reports. Literature estimates for the positive controls shows substantial variability that limits the ability to establish a reference set with known effect sizes. CONCLUSIONS: A reference set of test cases can be established to facilitate methodological research in drug safety. Creating a sufficient sample of drug-outcome pairs with binary classification of having no effect (negative controls) or having an increased effect (positive controls) is possible and can enable estimation of predictive accuracy through discrimination. Since the magnitude of the positive effects cannot be reliably obtained and the quality of evidence may vary across outcomes, assumptions are required to use the test cases in real data for purposes of measuring bias, mean squared error, or coverage probability.
BACKGROUND: Methodological research to evaluate the performance of methods requires a benchmark to serve as a referent comparison. In drug safety, the performance of analyses of spontaneous adverse event reporting databases and observational healthcare data, such as administrative claims and electronic health records, has been limited by the lack of such standards. OBJECTIVES: To establish a reference set of test cases that contain both positive and negative controls, which can serve the basis for methodological research in evaluating methods performance in identifying drug safety issues. RESEARCH DESIGN: Systematic literature review and natural language processing of structured product labeling was performed to identify evidence to support the classification of drugs as either positive controls or negative controls for four outcomes: acute liver injury, acute kidney injury, acute myocardial infarction, and upper gastrointestinal bleeding. RESULTS: Three-hundred and ninety-nine test cases comprised of 165 positive controls and 234 negative controls were identified across the four outcomes. The majority of positive controls for acute kidney injury and upper gastrointestinal bleeding were supported by randomized clinical trial evidence, while the majority of positive controls for acute liver injury and acute myocardial infarction were only supported based on published case reports. Literature estimates for the positive controls shows substantial variability that limits the ability to establish a reference set with known effect sizes. CONCLUSIONS: A reference set of test cases can be established to facilitate methodological research in drug safety. Creating a sufficient sample of drug-outcome pairs with binary classification of having no effect (negative controls) or having an increased effect (positive controls) is possible and can enable estimation of predictive accuracy through discrimination. Since the magnitude of the positive effects cannot be reliably obtained and the quality of evidence may vary across outcomes, assumptions are required to use the test cases in real data for purposes of measuring bias, mean squared error, or coverage probability.
Authors: Preciosa M Coloma; Gianluca Trifirò; Martijn J Schuemie; Rosa Gini; Ron Herings; Julia Hippisley-Cox; Giampiero Mazzaglia; Gino Picelli; Giovanni Corrao; Lars Pedersen; Johan van der Lei; Miriam Sturkenboom Journal: Pharmacoepidemiol Drug Saf Date: 2012-02-08 Impact factor: 2.890
Authors: Marc A Suchard; Ivan Zorych; Shawn E Simpson; Martijn J Schuemie; Patrick B Ryan; David Madigan Journal: Drug Saf Date: 2013-10 Impact factor: 5.606
Authors: Preciosa M Coloma; Paul Avillach; Francesco Salvo; Martijn J Schuemie; Carmen Ferrajolo; Antoine Pariente; Annie Fourrier-Réglat; Mariam Molokhia; Vaishali Patadia; Johan van der Lei; Miriam Sturkenboom; Gianluca Trifirò Journal: Drug Saf Date: 2013-01 Impact factor: 5.606
Authors: Mickael Arnaud; Francesco Salvo; Ismaïl Ahmed; Philip Robinson; Nicholas Moore; Bernard Bégaud; Pascale Tubert-Bitter; Antoine Pariente Journal: Drug Saf Date: 2016-03 Impact factor: 5.606
Authors: Rave Harpaz; Alison Callahan; Suzanne Tamang; Yen Low; David Odgers; Sam Finlayson; Kenneth Jung; Paea LePendu; Nigam H Shah Journal: Drug Saf Date: 2014-10 Impact factor: 5.606
Authors: Martijn J Schuemie; M Soledad Cepeda; Marc A Suchard; Jianxiao Yang; Yuxi Tian; Alejandro Schuler; Patrick B Ryan; David Madigan; George Hripcsak Journal: Harv Data Sci Rev Date: 2020-01-31
Authors: Paul E Stang; Patrick B Ryan; J Marc Overhage; Martijn J Schuemie; Abraham G Hartzema; Emily Welebob Journal: Drug Saf Date: 2013-10 Impact factor: 5.606