Neural correlates to automatic behavior estimations from RGB-D video in epilepsy unit.

To augment neural monitoring, a minimally intrusive multi-modal capture system was designed and implemented in the epilepsy clinic. This system provides RGB-D audio-video synchronized with patient electrocorticography (ECoG), which records neural activity across cortex. We propose an automated approach to studying the human brain in a naturalistic setting. We demonstrate coarse functional mapping of ECoG electrodes correlated to contralateral arm movements. Motor electrode mapping was generated by analyzing continuous movement data recorded over several hours from epilepsy patients in hospital rooms. From these recordings we estimate the kinematics of patient hand movement behaviors using computer vision algorithms. We compare movement behaviors to neural data collected from ECoG, specifically high-γ (70-110 Hz) spectral features. We present a functional map of electrode responses to natural arm movements, generated using a statistical test. We demonstrate that our approach has the potential to aid in the development of automated functional brain mapping using continuous video and neural recordings of patients in clinical settings.