Katharina Schultebraucks1,2,3, Vijay Yadav4, Arieh Y Shalev2, George A Bonanno5, Isaac R Galatzer-Levy2,4. 1. Department of Emergency Medicine, Vagelos School of Physicians and Surgeons, Columbia University Irving Medical Center, New York, New York, USA. 2. Department of Psychiatry, New York University Grossman School of Medicine, New York, New York, USA. 3. Data Science Institute, Columbia University, New York, New York, USA. 4. AiCure, New York, New York, USA. 5. Department of Counseling and Clinical Psychology, Teachers College, Columbia University, New York, New York, USA.
Abstract
BACKGROUND: Visual and auditory signs of patient functioning have long been used for clinical diagnosis, treatment selection, and prognosis. Direct measurement and quantification of these signals can aim to improve the consistency, sensitivity, and scalability of clinical assessment. Currently, we investigate if machine learning-based computer vision (CV), semantic, and acoustic analysis can capture clinical features from free speech responses to a brief interview 1 month post-trauma that accurately classify major depressive disorder (MDD) and posttraumatic stress disorder (PTSD). METHODS: N = 81 patients admitted to an emergency department (ED) of a Level-1 Trauma Unit following a life-threatening traumatic event participated in an open-ended qualitative interview with a para-professional about their experience 1 month following admission. A deep neural network was utilized to extract facial features of emotion and their intensity, movement parameters, speech prosody, and natural language content. These features were utilized as inputs to classify PTSD and MDD cross-sectionally. RESULTS: Both video- and audio-based markers contributed to good discriminatory classification accuracy. The algorithm discriminates PTSD status at 1 month after ED admission with an AUC of 0.90 (weighted average precision = 0.83, recall = 0.84, and f1-score = 0.83) as well as depression status at 1 month after ED admission with an AUC of 0.86 (weighted average precision = 0.83, recall = 0.82, and f1-score = 0.82). CONCLUSIONS: Direct clinical observation during post-trauma free speech using deep learning identifies digital markers that can be utilized to classify MDD and PTSD status.
BACKGROUND: Visual and auditory signs of patient functioning have long been used for clinical diagnosis, treatment selection, and prognosis. Direct measurement and quantification of these signals can aim to improve the consistency, sensitivity, and scalability of clinical assessment. Currently, we investigate if machine learning-based computer vision (CV), semantic, and acoustic analysis can capture clinical features from free speech responses to a brief interview 1 month post-trauma that accurately classify major depressive disorder (MDD) and posttraumatic stress disorder (PTSD). METHODS: N = 81 patients admitted to an emergency department (ED) of a Level-1 Trauma Unit following a life-threatening traumatic event participated in an open-ended qualitative interview with a para-professional about their experience 1 month following admission. A deep neural network was utilized to extract facial features of emotion and their intensity, movement parameters, speech prosody, and natural language content. These features were utilized as inputs to classify PTSD and MDD cross-sectionally. RESULTS: Both video- and audio-based markers contributed to good discriminatory classification accuracy. The algorithm discriminates PTSD status at 1 month after ED admission with an AUC of 0.90 (weighted average precision = 0.83, recall = 0.84, and f1-score = 0.83) as well as depression status at 1 month after ED admission with an AUC of 0.86 (weighted average precision = 0.83, recall = 0.82, and f1-score = 0.82). CONCLUSIONS: Direct clinical observation during post-trauma free speech using deep learning identifies digital markers that can be utilized to classify MDD and PTSD status.
Entities:
Keywords:
Computer vision; deep learning; depression; digital biomarker; emergency department; landmark feature; posttraumatic stress; resilience; voice analysis
Authors: Ayse S Cakmak; Erick A Perez Alday; Giulia Da Poian; Ali Bahrami Rad; Thomas J Metzler; Thomas C Neylan; Stacey L House; Francesca L Beaudoin; Xinming An; Jennifer S Stevens; Donglin Zeng; Sarah D Linnstaedt; Tanja Jovanovic; Laura T Germine; Kenneth A Bollen; Scott L Rauch; Christopher A Lewandowski; Phyllis L Hendry; Sophia Sheikh; Alan B Storrow; Paul I Musey; John P Haran; Christopher W Jones; Brittany E Punches; Robert A Swor; Nina T Gentile; Meghan E McGrath; Mark J Seamon; Kamran Mohiuddin; Anna M Chang; Claire Pearson; Robert M Domeier; Steven E Bruce; Brian J O'Neil; Niels K Rathlev; Leon D Sanchez; Robert H Pietrzak; Jutta Joormann; Deanna M Barch; Diego A Pizzagalli; Steven E Harte; James M Elliott; Ronald C Kessler; Karestan C Koenen; Kerry J Ressler; Samuel A Mclean; Qiao Li; Gari D Clifford Journal: IEEE J Biomed Health Inform Date: 2021-08-06 Impact factor: 7.021