Sen Yang1, Aleksandra Sarcevic2, Richard A Farneth3, Shuhong Chen4, Omar Z Ahmed5, Ivan Marsic6, Randall S Burd7. 1. Electrical and Computer Engineering Department, Rutgers University, 94 Brett Road, Piscataway, NJ 08854, USA. Electronic address: sy358@rutgers.edu. 2. College of Computing and Informatics, Drexel University, 3141 Chestnut Street, Philadelphia, PA 19104, USA. Electronic address: aleksarc@drexel.edu. 3. Children's National Medical Center, 111 Michigan Ave NW, Washington, DC 20010, USA. 4. Electrical and Computer Engineering Department, Rutgers University, 94 Brett Road, Piscataway, NJ 08854, USA. Electronic address: sc1624@rutgers.edu. 5. Children's National Medical Center, 111 Michigan Ave NW, Washington, DC 20010, USA. Electronic address: oahmed3@childrensnational.org. 6. Electrical and Computer Engineering Department, Rutgers University, 94 Brett Road, Piscataway, NJ 08854, USA. Electronic address: marsic@rutgers.edu. 7. Children's National Medical Center, 111 Michigan Ave NW, Washington, DC 20010, USA. Electronic address: rburd@childrensnational.org.
Abstract
MOTIVATION: Prior research has shown that minor errors and deviations from recommended guidelines in complex medical processes can accumulate to increase the likelihood that a major error will go uncorrected and lead to an adverse outcome. Real-time automatic and accurate detection of process deviations may help medical teams better prevent or mitigate the effect of errors and improve patient outcomes. Our goal was to develop an approach for automatic detection of errors and process deviations in trauma resuscitation. METHODS: Using video review, we coded activity traces of 95 pediatric trauma resuscitations collected in a Level 1 trauma center over two years (2014-2016). Twenty-four randomly selected activity traces were compared with a knowledge-driven model of trauma resuscitation workflow using a phase-based conformance checking algorithm for detecting true and false deviations (alarms). An analysis of false alarms identified three types of causes: (1) model gaps or discrepancies between the model ("work as imagined") and actual practice ("work as done"), (2) errors in activity traces coding, and (3) algorithm limitations. We repaired the system to remove model gaps, reduce coding errors, and address algorithm limitations. The repaired system was first evaluated with another 20 traces and then applied to the entire dataset of 95 traces. RESULTS: During the training, we detected 573 process deviations in 24 activity traces that include 1099 activities. Among these deviations, only 27% represented true deviations and the remaining 73% were false alarms. This initial deviation detection accuracy was only 66.6%, with a F1-score of 0.42. Detection accuracy of the repaired system increased to 95.2% (0.85 F1-score) during system validation and to 98.5% (0.96 F1-score) during testing. After deploying the repaired deviation detection system to all 95 activity traces, we detected 1060 process deviations in 5659 activities (11.2 deviations per resuscitation). Among the 5659 activities in these traces, 4893 fit the repaired knowledge-driven workflow model, 294 were errors of omission, 538 were errors of commission, and 228 were scheduling errors. CONCLUSION: Our approach to automatic deviation detection provides a method for identifying repeated, omitted and out-of-sequence activities that can be included in the design of decision support systems for complex medical processes. Our findings show the importance of assessing detected deviations for repairing a knowledge-driven model that best represents "work as done."
MOTIVATION: Prior research has shown that minor errors and deviations from recommended guidelines in complex medical processes can accumulate to increase the likelihood that a major error will go uncorrected and lead to an adverse outcome. Real-time automatic and accurate detection of process deviations may help medical teams better prevent or mitigate the effect of errors and improve patient outcomes. Our goal was to develop an approach for automatic detection of errors and process deviations in trauma resuscitation. METHODS: Using video review, we coded activity traces of 95 pediatric trauma resuscitations collected in a Level 1 trauma center over two years (2014-2016). Twenty-four randomly selected activity traces were compared with a knowledge-driven model of trauma resuscitation workflow using a phase-based conformance checking algorithm for detecting true and false deviations (alarms). An analysis of false alarms identified three types of causes: (1) model gaps or discrepancies between the model ("work as imagined") and actual practice ("work as done"), (2) errors in activity traces coding, and (3) algorithm limitations. We repaired the system to remove model gaps, reduce coding errors, and address algorithm limitations. The repaired system was first evaluated with another 20 traces and then applied to the entire dataset of 95 traces. RESULTS: During the training, we detected 573 process deviations in 24 activity traces that include 1099 activities. Among these deviations, only 27% represented true deviations and the remaining 73% were false alarms. This initial deviation detection accuracy was only 66.6%, with a F1-score of 0.42. Detection accuracy of the repaired system increased to 95.2% (0.85 F1-score) during system validation and to 98.5% (0.96 F1-score) during testing. After deploying the repaired deviation detection system to all 95 activity traces, we detected 1060 process deviations in 5659 activities (11.2 deviations per resuscitation). Among the 5659 activities in these traces, 4893 fit the repaired knowledge-driven workflow model, 294 were errors of omission, 538 were errors of commission, and 228 were scheduling errors. CONCLUSION: Our approach to automatic deviation detection provides a method for identifying repeated, omitted and out-of-sequence activities that can be included in the design of decision support systems for complex medical processes. Our findings show the importance of assessing detected deviations for repairing a knowledge-driven model that best represents "work as done."
Authors: Filip Caron; Jan Vanthienen; Kris Vanhaecht; Erik Van Limbergen; Jochen De Weerdt; Bart Baesens Journal: Comput Biol Med Date: 2013-11-07 Impact factor: 4.589
Authors: Rachel B Webman; Jennifer L Fritzeen; JaeWon Yang; Grace F Ye; Paul C Mullan; Faisal G Qureshi; Sarah H Parker; Aleksandra Sarcevic; Ivan Marsic; Randall S Burd Journal: J Trauma Acute Care Surg Date: 2016-10 Impact factor: 3.313
Authors: J R Clarke; B Spejewski; A S Gertner; B L Webber; C Z Hayward; T A Santora; D K Wagner; C C Baker; H R Champion; T C Fabian; F R Lewis; E E Moore; J A Weigelt; A B Eastman; C Blank-Reid Journal: Acad Emerg Med Date: 2000-11 Impact factor: 3.451
Authors: Deirdre C Kelleher; R P Jagadeesh Chandra Bose; Lauren J Waterhouse; Elizabeth A Carter; Randall S Burd Journal: J Am Coll Surg Date: 2013-11-26 Impact factor: 6.113
Authors: Omar Z Ahmed; Sen Yang; Richard A Farneth; Aleksandra Sarcevic; Ivan Marsic; Randall S Burd Journal: J Surg Res Date: 2019-05-14 Impact factor: 2.192
Authors: Emmanuel Helm; Anna M Lin; David Baumgartner; Alvin C Lin; Josef Küng Journal: Int J Environ Res Public Health Date: 2020-02-19 Impact factor: 3.390