Chung-You Tsai1, Shi-Heng Wang2, Min-Huei Hsu3, Yu-Chuan Jack Li4. 1. Institute of Biomedical Informatics, National Yang-Ming University, Taipei, Taiwan; Division of Urology, Department of Surgery, Far Eastern Memorial Hospital, New Taipei, Taiwan. 2. Institute of Epidemiology and Preventive Medicine, College of Public Health, National Taiwan University, Taipei, Taiwan; Graduate Institute of Biostatistics, China Medical University, Taichung, Taiwan. 3. Office of International Cooperation, Ministry of Health and Welfare, Taiwan; College of Medicine Science and Technology (CoMST), Taipei Medical University, Taiwan; International Center for Health Information Technology (ICHIT), Taipei Medical University, Taiwan. 4. College of Medicine Science and Technology (CoMST), Taipei Medical University, Taiwan; International Center for Health Information Technology (ICHIT), Taipei Medical University, Taiwan; Department of Dermatology, Wan Fang Hospital, Taipei, Taiwan. Electronic address: jack@tmu.edu.tw.
Abstract
OBJECTIVES: False positive alerts in patient-safety-related clinical decision support systems (CDSS) are defined as alerts which incorrectly prompt when no-risk patients are encountered. It is an unfavorable condition which may potentially mislead physicians. The aim is to investigate physician responses toward false positive (FP) and true positive (TP) alerts in CDSS for the prevention of contrast-induced nephropathy (CIN). METHODS: A two-arm cluster randomized controlled trial was conducted in university hospitals. Eligible physicians were randomized to receive alert intervention or no intervention (groups 1 and 2, respectively). The alert system was embedded with a deliberately non-specific risk detection tool in order to generate TP and FP alerts. The naïve alert system would alert the physician to cancel the order regardless of the patient being at-risk or not at-risk. CIN risk was stratified as at-risk and no-risk according to a patient's pre-existing renal function. Contrast imaging order-cancellation rate was measured as primary outcome. RESULTS: 3802 contrast-enhanced examination orders from 66 physicians were analyzed. Demographic data and risk distributions of patients were similar and well-balanced between two groups. In the intervention group, a total of 1892 alerts were generated (332 TP alerts and 1560 FP alerts). Order-cancellation rates were 5.1% versus 1.4% in groups 1 and 2 for at-risk patients (relative risk [RR] = 3.69) from TP alerts, and 1.0% versus 1.4% for no-risk patients (RR = 0.71) from FP alerts. Using generalized linear model with generalized estimating equation, the FP alerts had no order-cancellation effect when compared to the control arm (adjusted RR = 0.69; 95%CI, 0.36-1.32). The TP alerts had a larger order-cancellation effect than that of the control arm (adjusted RR = 2.95; 95%CI, 0.94-9.27), which revealed a marginal trend toward significance. However, the effect was not statistically significant (adjusted RR = 1.24; 95%CI, 0.71-2.18) if TP and FP alerts were mixed. CONCLUSIONS: Physicians are not likely to adopt recommendations provided by false positive alerts in patient-safety-related CDSS. If reporting only the adoption rate of CDSS as a whole without differentiating between TP and FP alerts, the effects of TP and FP alerts will be mixed, and thus, will lead to an underestimation of system effectiveness.
RCT Entities:
OBJECTIVES: False positive alerts in patient-safety-related clinical decision support systems (CDSS) are defined as alerts which incorrectly prompt when no-risk patients are encountered. It is an unfavorable condition which may potentially mislead physicians. The aim is to investigate physician responses toward false positive (FP) and true positive (TP) alerts in CDSS for the prevention of contrast-induced nephropathy (CIN). METHODS: A two-arm cluster randomized controlled trial was conducted in university hospitals. Eligible physicians were randomized to receive alert intervention or no intervention (groups 1 and 2, respectively). The alert system was embedded with a deliberately non-specific risk detection tool in order to generate TP and FP alerts. The naïve alert system would alert the physician to cancel the order regardless of the patient being at-risk or not at-risk. CIN risk was stratified as at-risk and no-risk according to a patient's pre-existing renal function. Contrast imaging order-cancellation rate was measured as primary outcome. RESULTS: 3802 contrast-enhanced examination orders from 66 physicians were analyzed. Demographic data and risk distributions of patients were similar and well-balanced between two groups. In the intervention group, a total of 1892 alerts were generated (332 TP alerts and 1560 FP alerts). Order-cancellation rates were 5.1% versus 1.4% in groups 1 and 2 for at-risk patients (relative risk [RR] = 3.69) from TP alerts, and 1.0% versus 1.4% for no-risk patients (RR = 0.71) from FP alerts. Using generalized linear model with generalized estimating equation, the FP alerts had no order-cancellation effect when compared to the control arm (adjusted RR = 0.69; 95%CI, 0.36-1.32). The TP alerts had a larger order-cancellation effect than that of the control arm (adjusted RR = 2.95; 95%CI, 0.94-9.27), which revealed a marginal trend toward significance. However, the effect was not statistically significant (adjusted RR = 1.24; 95%CI, 0.71-2.18) if TP and FP alerts were mixed. CONCLUSIONS: Physicians are not likely to adopt recommendations provided by false positive alerts in patient-safety-related CDSS. If reporting only the adoption rate of CDSS as a whole without differentiating between TP and FP alerts, the effects of TP and FP alerts will be mixed, and thus, will lead to an underestimation of system effectiveness.
Authors: Roman Koposov; Sturla Fossum; Thomas Frodl; Øystein Nytrø; Bennett Leventhal; Andre Sourander; Silvana Quaglini; Massimo Molteni; María de la Iglesia Vayá; Hans-Ulrich Prokosch; Nicola Barbarini; Michael Peter Milham; Francisco Xavier Castellanos; Norbert Skokauskas Journal: Eur Child Adolesc Psychiatry Date: 2017-04-28 Impact factor: 4.785
Authors: Kaveh G Shojania; Alison Jennings; Alain Mayhew; Craig R Ramsay; Martin P Eccles; Jeremy Grimshaw Journal: Cochrane Database Syst Rev Date: 2009-07-08