OBJECTIVES: Determine if clinical decision support (CDS) malfunctions occur in a commercial electronic health record (EHR) system, characterize their pathways and describe methods of detection. METHODS: We retrospectively examined the firing rate for 226 alert type CDS rules for detection of anomalies using both expert visualization and statistical process control (SPC) methods over a five year period. Candidate anomalies were investigated and validated. RESULTS: Twenty-one candidate CDS anomalies were identified from 8,300 alert-months. Of these candidate anomalies, four were confirmed as CDS malfunctions, eight as false-positives, and nine could not be classified. The four CDS malfunctions were a result of errors in knowledge management: 1) inadvertent addition and removal of a medication code to the electronic formulary list; 2) a seasonal alert which was not activated; 3) a change in the base data structures; and 4) direct editing of an alert related to its medications. 154 CDS rules (68%) were amenable to SPC methods and the test characteristics were calculated as a sensitivity of 95%, positive predictive value of 29% and F-measure 0.44. DISCUSSION: CDS malfunctions were found to occur in our EHR. All of the pathways for these malfunctions can be described as knowledge management errors. Expert visualization is a robust method of detection, but is resource intensive. SPC-based methods, when applicable, perform reasonably well retrospectively. CONCLUSION: CDS anomalies were found to occur in a commercial EHR and visual detection along with SPC analysis represents promising methods of malfunction detection.
OBJECTIVES: Determine if clinical decision support (CDS) malfunctions occur in a commercial electronic health record (EHR) system, characterize their pathways and describe methods of detection. METHODS: We retrospectively examined the firing rate for 226 alert type CDS rules for detection of anomalies using both expert visualization and statistical process control (SPC) methods over a five year period. Candidate anomalies were investigated and validated. RESULTS: Twenty-one candidate CDS anomalies were identified from 8,300 alert-months. Of these candidate anomalies, four were confirmed as CDS malfunctions, eight as false-positives, and nine could not be classified. The four CDS malfunctions were a result of errors in knowledge management: 1) inadvertent addition and removal of a medication code to the electronic formulary list; 2) a seasonal alert which was not activated; 3) a change in the base data structures; and 4) direct editing of an alert related to its medications. 154 CDS rules (68%) were amenable to SPC methods and the test characteristics were calculated as a sensitivity of 95%, positive predictive value of 29% and F-measure 0.44. DISCUSSION: CDS malfunctions were found to occur in our EHR. All of the pathways for these malfunctions can be described as knowledge management errors. Expert visualization is a robust method of detection, but is resource intensive. SPC-based methods, when applicable, perform reasonably well retrospectively. CONCLUSION:CDS anomalies were found to occur in a commercial EHR and visual detection along with SPC analysis represents promising methods of malfunction detection.
Entities:
Keywords:
Clinical decision support; alerts; electronic health record; electronic medical record; errors; malfunction
Authors: Sidney C Smith; Emelia J Benjamin; Robert O Bonow; Lynne T Braun; Mark A Creager; Barry A Franklin; Raymond J Gibbons; Scott M Grundy; Loren F Hiratzka; Daniel W Jones; Donald M Lloyd-Jones; Margo Minissian; Lori Mosca; Eric D Peterson; Ralph L Sacco; John Spertus; James H Stein; Kathryn A Taubert Journal: Circulation Date: 2011-11-03 Impact factor: 29.690
Authors: Nidhi R Shah; Andrew C Seger; Diane L Seger; Julie M Fiskio; Gilad J Kuperman; Barry Blumenfeld; Elaine G Recklet; David W Bates; Tejal K Gandhi Journal: J Am Med Inform Assoc Date: 2005-10-12 Impact factor: 4.497
Authors: Adam Wright; Dean F Sittig; Joan S Ash; Joshua Feblowitz; Seth Meltzer; Carmit McMullen; Ken Guappone; Jim Carpenter; Joshua Richardson; Linas Simonaitis; R Scott Evans; W Paul Nichol; Blackford Middleton Journal: J Am Med Inform Assoc Date: 2011-03-17 Impact factor: 4.497
Authors: Lorenzo Moja; Koren H Kwag; Theodore Lytras; Lorenzo Bertizzolo; Linn Brandt; Valentina Pecoraro; Giulio Rigon; Alberto Vaona; Francesca Ruggiero; Massimo Mangia; Alfonso Iorio; Ilkka Kunnamo; Stefanos Bonovas Journal: Am J Public Health Date: 2014-10-16 Impact factor: 9.308
Authors: Joan S Ash; Dean F Sittig; Richard Dykstra; Adam Wright; Carmit McMullen; Joshua Richardson; Blackford Middleton Journal: Stud Health Technol Inform Date: 2010
Authors: Tiffani J Bright; Anthony Wong; Ravi Dhurjati; Erin Bristow; Lori Bastian; Remy R Coeytaux; Gregory Samsa; Vic Hasselblad; John W Williams; Michael D Musty; Liz Wing; Amy S Kendrick; Gillian D Sanders; David Lobach Journal: Ann Intern Med Date: 2012-07-03 Impact factor: 25.391
Authors: David Lobach; Gillian D Sanders; Tiffani J Bright; Anthony Wong; Ravi Dhurjati; Erin Bristow; Lori Bastian; Remy Coeytaux; Gregory Samsa; Vic Hasselblad; John W Williams; Liz Wing; Michael Musty; Amy S Kendrick Journal: Evid Rep Technol Assess (Full Rep) Date: 2012-04
Authors: Jeritt G Thayer; Jeffrey M Miller; Alexander G Fiks; Linda Tague; Robert W Grundmeier Journal: Appl Clin Inform Date: 2019-04-03 Impact factor: 2.342
Authors: Pierre Elias; Eric Peterson; Bob Wachter; Cary Ward; Eric Poon; Ann Marie Navar Journal: Appl Clin Inform Date: 2019-11-27 Impact factor: 2.342
Authors: Evan W Orenstein; Naveen Muthu; Asli O Weitkamp; Daria F Ferro; Mike D Zeidlhack; Jason Slagle; Eric Shelov; Marc C Tobias Journal: Appl Clin Inform Date: 2019-10-30 Impact factor: 2.342
Authors: Juan D Chaparro; Jonathan M Beus; Adam C Dziorny; Philip A Hagedorn; Sean Hernandez; Swaminathan Kandaswamy; Eric S Kirkendall; Allison B McCoy; Naveen Muthu; Evan W Orenstein Journal: Appl Clin Inform Date: 2022-05-25 Impact factor: 2.762
Authors: Barbara E Jones; Dave S Collingridge; Caroline G Vines; Herman Post; John Holmen; Todd L Allen; Peter Haug; Charlene R Weir; Nathan C Dean Journal: Appl Clin Inform Date: 2019-01-02 Impact factor: 2.342
Authors: Mujeeb A Basit; Krystal L Baldwin; Vaishnavi Kannan; Emily L Flahaven; Cassandra J Parks; Jason M Ott; Duwayne L Willett Journal: JMIR Med Inform Date: 2018-04-13