Criteria revision and performance comparison of three methods of signal detection applied to the spontaneous reporting database of a pharmaceutical manufacturer.

BACKGROUND AND
OBJECTIVE: Several statistical methods exist for detecting signals of potential adverse drug reactions in spontaneous reporting databases. However, these signal-detection methods were developed using regulatory databases, which contain a far larger number of adverse event reports than the databases maintained by individual pharmaceutical manufacturers. Furthermore, the composition and quality of the spontaneous reporting databases differ between regulatory agencies and pharmaceutical companies. Thus, the signal-detection criteria proposed for regulatory use are considered to be inappropriate for pharmaceutical industry use without modification. The objective of this study was to revise the criteria for signal detection to make them suitable for use by pharmaceutical manufacturers.
METHODS: A model comprising 40 drugs and 1000 adverse events was constructed based on a spontaneous reporting database provided by a pharmaceutical company and used in a simulation to investigate appropriate criteria for signal detection. In total, 1000 pseudo datasets were generated with this model, and three statistical methods (proportional reporting ratio [PRR], Bayesian Confidence Propagation Neural Network [BCPNN] and multi-item gamma Poisson shrinker [MGPS]) for signal detection were applied to each dataset. The sensitivity and specificity of each method were evaluated using these pseudo datasets. The optimum critical value for signal detection (i.e. the value that achieved the highest sensitivity with 95% specificity) was identified for each method. The optimum values were also examined with the adverse events classified into two categories according to frequency. The three original detection methods and their revised versions were applied to a real pharmaceutical company database to detect 173 known adverse reactions of four drugs.
RESULTS: The 1000 pseudo datasets consisted of an average of 81 862 reports and 11,407 drug-event pairs, including 1192 adverse drug reactions. The sensitivities of PRR, BCPNN and MGPS methods were 49%, 45% and 26%, respectively, whereas their specificities were 95%, 99.6% and 99.99%, respectively; these sensitivities were unacceptably low for pharmaceutical manufacturers, whereas the specificities were acceptable. The highest sensitivity for each method, obtained by changing critical values and maintaining specificity at 95%, was 44%, 62% and 62%, respectively. When adverse events were classified into two categories, sensitivities as high as 75% for regular events and 39% for rare events were achieved with the revised BCPNN method. The critical values of the information component minus two standard deviations (IC - 2SD) index of the revised BCPNN method were greater than -0.7 for regular events and greater than -0.6 for rare events. The revised BCPNN method yielded 51% sensitivity and 89% specificity for the real dataset.
CONCLUSION: A lower critical value may be needed when signal-detection methodology is applied to the spontaneous reporting databases of pharmaceutical manufacturers. For example, it is recommended that pharmaceutical manufacturers use the BCPNN method with IC - 2SD criteria of greater than -0.7 for regular events and greater than -0.6 for rare events.