OBJECTIVE: We sought to validate 3 methods for automated safety monitoring by evaluating clinical trials with elevated adverse events. METHODS: An automated outcomes surveillance system was used to retrospectively analyze data from 2 randomized, TIMI multicenter trials. Trial A was stopped early due to elevated 30-day mortality rates in the intervention arm. Trial B was not stopped early, but there was transient concern regarding 30-day intracranial hemorrhage rates. We compared statistical process control (SPC), logistic regression risk adjusted SPC (LR-SPC), and Bayesian updating statistic (BUS) methods with a standard prospective 2-arm event rate analysis. Each method compares observed event rates to alerting boundaries established with previously collected data. In this evaluation, the control arms approximated prior data, and the intervention arms approximated the observed data. RESULTS: Trial A experienced elevated 30-day mortality rates beginning 7 months after the start of the trial and continuing until termination at month 14. Trial B did not experience elevated major bleeding rates. Combining the alerting performance of each method across both trials resulted in sensitivities and specificities of 100% and 85% for SPC, 0% and 100% for BUS, and 100% and 93% for both LR-SPC models, respectively. CONCLUSION: Both SPC and LR-SPC methods correctly identified the majority of months during which the cumulative event rates were elevated in trial A but were susceptible to false positive alerts in trial B. The BUS method did not result in any alerts in either trial and requires revision.
RCT Entities:
OBJECTIVE: We sought to validate 3 methods for automated safety monitoring by evaluating clinical trials with elevated adverse events. METHODS: An automated outcomes surveillance system was used to retrospectively analyze data from 2 randomized, TIMI multicenter trials. Trial A was stopped early due to elevated 30-day mortality rates in the intervention arm. Trial B was not stopped early, but there was transient concern regarding 30-day intracranial hemorrhage rates. We compared statistical process control (SPC), logistic regression risk adjusted SPC (LR-SPC), and Bayesian updating statistic (BUS) methods with a standard prospective 2-arm event rate analysis. Each method compares observed event rates to alerting boundaries established with previously collected data. In this evaluation, the control arms approximated prior data, and the intervention arms approximated the observed data. RESULTS: Trial A experienced elevated 30-day mortality rates beginning 7 months after the start of the trial and continuing until termination at month 14. Trial B did not experience elevated major bleeding rates. Combining the alerting performance of each method across both trials resulted in sensitivities and specificities of 100% and 85% for SPC, 0% and 100% for BUS, and 100% and 93% for both LR-SPC models, respectively. CONCLUSION: Both SPC and LR-SPC methods correctly identified the majority of months during which the cumulative event rates were elevated in trial A but were susceptible to false positive alerts in trial B. The BUS method did not result in any alerts in either trial and requires revision.
Authors: Lucila Ohno-Machado; Vineet Bafna; Aziz A Boxwala; Brian E Chapman; Wendy W Chapman; Kamalika Chaudhuri; Michele E Day; Claudiu Farcas; Nathaniel D Heintzman; Xiaoqian Jiang; Hyeoneui Kim; Jihoon Kim; Michael E Matheny; Frederic S Resnic; Staal A Vinterbo Journal: J Am Med Inform Assoc Date: 2011-11-10 Impact factor: 4.497
Authors: Frederic S Resnic; Arjun Majithia; Danica Marinac-Dabic; Susan Robbins; Henry Ssemaganda; Kathleen Hewitt; Angelo Ponirakis; Nilsa Loyo-Berrios; Issam Moussa; Joseph Drozda; Sharon-Lise Normand; Michael E Matheny Journal: N Engl J Med Date: 2017-01-25 Impact factor: 91.245
Authors: Frederic S Resnic; Thomas P Gross; Danica Marinac-Dabic; Nilsa Loyo-Berrios; Sharon Donnelly; Sharon-Lise T Normand; Michael E Matheny Journal: JAMA Date: 2010-11-10 Impact factor: 56.272
Authors: Sharon E Davis; Jeremiah R Brown; Chad Dorn; Dax Westerman; Richard J Solomon; Michael E Matheny Journal: Circ Cardiovasc Qual Outcomes Date: 2022-08-12
Authors: Frederic S Resnic; Arjun Majithia; Sanket S Dhruva; Henry Ssemaganda; Susan Robbins; Danica Marinac-Dabic; Kathleen Hewitt; Lucila Ohno-Machado; Matthew R Reynolds; Michael E Matheny Journal: Circ Cardiovasc Qual Outcomes Date: 2020-04-14
Authors: Amit Kumar; Michael E Matheny; Kalon K L Ho; Robert W Yeh; Thomas C Piemonte; Howard Waldman; Pinak B Shah; Richard Cope; Sharon-Lise T Normand; Sharon Donnelly; Susan Robbins; Frederic S Resnic Journal: Circ Cardiovasc Qual Outcomes Date: 2014-12-09
Authors: Michael E Matheny; Sharon-Lise T Normand; Thomas P Gross; Danica Marinac-Dabic; Nilsa Loyo-Berrios; Venkatesan D Vidi; Sharon Donnelly; Frederic S Resnic Journal: BMC Med Inform Decis Mak Date: 2011-12-14 Impact factor: 2.796
Authors: Lucila Ohno-Machado; Zia Agha; Douglas S Bell; Lisa Dahm; Michele E Day; Jason N Doctor; Davera Gabriel; Maninder K Kahlon; Katherine K Kim; Michael Hogarth; Michael E Matheny; Daniella Meeker; Jonathan R Nebeker Journal: J Am Med Inform Assoc Date: 2014-04-29 Impact factor: 4.497
Authors: Arjun Majithia; Michael E Matheny; Sourbha S Dani; Jessica K Paulus; Danica Marinac-Dabic; Susan Robbins; Henry Ssemaganda; Kathleen Hewitt; Angelo Ponirakis; Nilsa Loyo-Berrios; Issam Moussa; Joseph Drozda; Sharon-Lise Normand; Frederic S Resnic Journal: BMJ Surg Interv Health Technol Date: 2020-11-11