Interpretability of time-series deep learning models: a study in cardiovascular patients admitted to Intensive Care Unit.

Interpretability is fundamental in healthcare problems and the lack of it in deep learning models is currently the major barrier in the usage of such powerful algorithms in the field. The study describes the implementation of an attention layer for Long Short-Term Memory (LSTM) neural network that provides a useful picture on the influence of the several input variables included in the model. A cohort of 10,616 patients with cardiovascular diseases is selected from the MIMIC III dataset, an openly available database of electronic health records (EHRs) including all patients admitted to an ICU at Boston's Medical Centre. For each patient, we consider a 10-length sequence of 1-hour windows in which 48 clinical parameters are extracted to predict the occurrence of death in the next 7 days. Inspired from the recent developments in the field of attention mechanisms for sequential data, we implement a recurrent neural network with LSTM cells incorporating an attention mechanism to identify features driving model's decisions over time. The performance of the LSTM model, measured in terms of AUC, is 0.790 (SD=0.015). Regard our primary objective, i.e. model interpretability, we investigate the role of attention weights. We find good correspondence with driving predictors of a transparent model (r=0.611, 95% CI [0.395, 0.763]). Moreover, most influential features identified at the cohort-level emerge as known risk factors in the clinical context. Despite the limitations of study dataset, this work brings further evidence of the potential of attention mechanisms in making deep learning model more interpretable and suggests the application of this strategy for the sequential analysis of EHRs.