Shaun M Comfort1, Bruce Donzanti2, Darren Dorrell3, Sunita Dhar3, Chris Eden3, Francis Donaldson4. 1. Roche-Genentech, Inc., A Member of the Roche Group, 1 DNA Way, South San Francisco, CA, 94080, USA. comforts@gene.com. 2. Donzanti PV Services, LLC., Orlando, FL, USA. 3. Roche-Genentech, Inc., A Member of the Roche Group, 1 DNA Way, South San Francisco, CA, 94080, USA. 4. Roche-Genentech, Inc., Welwyn Garden City, UK.
Abstract
INTRODUCTION: In 2018, we published the MONARCSi algorithmic decision support tool showing high inter-rater agreement, moderate sensitivity, and high specificity compared with drug-event pairs (DEPs) previously reviewed using current, industry-established approaches. Following publication, MONARCSi was implemented as a prototype system to facilitate medical review of individual case safety reports (ICSRs). This paper presents subsequent evaluation of MONARCSi-supported causality assessments against an independent, best achievable reference standard. OBJECTIVE: This paper describes the development of an independent reference standard (i.e., reference comparator) using a sample of DEPs evaluated by Roche subject matter experts (SMEs) and subsequent performance analysis for both the reference standard and MONARCSi. METHODS: Roche collected a random sample of 131 DEPs evaluated by an external vendor using the MONARCSi prototype during 2020, and collectively referred to as the VMON (Vendor using the MONARCSi system for medical review) dataset. An internal group of causality SMEs (aka CAUSMET) were recruited and trained to assess the same DEPs independently using the MONARCSi structure with Global Introspection to determine their individual assessments of causality. The CAUSMET final causality was determined using a majority voting rule. RESULTS: Binary comparison of the aggregate results showed substantial agreement (Gwet kappa = 0.81) between the VMON and reference standard CAUSMET assessments. Bayesian latent class modeling showed that both the reference standard and VMON assessments exhibited similar high posterior mean sensitivity and specificity (CAUSMET: 89 and 93%, respectively; VMON: 87 and 94%, respectively). Finally, comparison of the sensitivity and specificity suggested no obvious difference across groups. CONCLUSION: Analysis of causality results from the assessments by independent internal SMEs using MONARCSi shows there is no obvious difference in performance between the aggregate CAUSMET and VMON assessments based on the comparison of specificity and sensitivity. These results further support use of MONARCSi as a decision support tool for evaluating drug-event causality in a consistent and documentable manner.
INTRODUCTION: In 2018, we published the MONARCSi algorithmic decision support tool showing high inter-rater agreement, moderate sensitivity, and high specificity compared with drug-event pairs (DEPs) previously reviewed using current, industry-established approaches. Following publication, MONARCSi was implemented as a prototype system to facilitate medical review of individual case safety reports (ICSRs). This paper presents subsequent evaluation of MONARCSi-supported causality assessments against an independent, best achievable reference standard. OBJECTIVE: This paper describes the development of an independent reference standard (i.e., reference comparator) using a sample of DEPs evaluated by Roche subject matter experts (SMEs) and subsequent performance analysis for both the reference standard and MONARCSi. METHODS: Roche collected a random sample of 131 DEPs evaluated by an external vendor using the MONARCSi prototype during 2020, and collectively referred to as the VMON (Vendor using the MONARCSi system for medical review) dataset. An internal group of causality SMEs (aka CAUSMET) were recruited and trained to assess the same DEPs independently using the MONARCSi structure with Global Introspection to determine their individual assessments of causality. The CAUSMET final causality was determined using a majority voting rule. RESULTS: Binary comparison of the aggregate results showed substantial agreement (Gwet kappa = 0.81) between the VMON and reference standard CAUSMET assessments. Bayesian latent class modeling showed that both the reference standard and VMON assessments exhibited similar high posterior mean sensitivity and specificity (CAUSMET: 89 and 93%, respectively; VMON: 87 and 94%, respectively). Finally, comparison of the sensitivity and specificity suggested no obvious difference across groups. CONCLUSION: Analysis of causality results from the assessments by independent internal SMEs using MONARCSi shows there is no obvious difference in performance between the aggregate CAUSMET and VMON assessments based on the comparison of specificity and sensitivity. These results further support use of MONARCSi as a decision support tool for evaluating drug-event causality in a consistent and documentable manner.
Authors: Ana Filipa Macedo; Francisco Batel Marques; Carlos Fontes Ribeiro; Frederico Teixeira Journal: Pharmacoepidemiol Drug Saf Date: 2005-12 Impact factor: 2.890
Authors: C A Naranjo; U Busto; E M Sellers; P Sandor; I Ruiz; E A Roberts; E Janecek; C Domecq; D J Greenblatt Journal: Clin Pharmacol Ther Date: 1981-08 Impact factor: 6.875
Authors: Lateef M Khan; Sameer E Al-Harthi; Abdel-Moneim M Osman; Mai A Alim A Sattar; Ahmed S Ali Journal: Saudi Pharm J Date: 2015-01-10 Impact factor: 4.330