Learning Depth with Convolutional Spatial Propagation Network.

Depth prediction is one of the fundamental problems in computer vision. In this paper, we propose a simple yet effective convolutional spatial propagation network (CSPN) to learn the affinity matrix for various depth estimation tasks. We can append this module to any output from a state-of-the-art (SOTA) network to improve their performances. In practice, we further extend CSPN in two aspects: 1) take a sparse depth map as additional input, which is useful for the task of sparse to dense (a.k.a depth completion); 2) we propose 3D CSPN to handle features with one additional dimension, which is effective in the task of stereo matching using 3D cost volume. For the tasks of depth completion, we experimented the proposed CPSN conjunct algorithms over NYU v2 and KITTI datasets, where we show that our proposed algorithms not only produce high quality (e.g., 30% more reduction in depth error), but also run faster (e.g., 2 to 5 × faster) than previous SOTA spatial propagation network. We also evaluated our stereo matching algorithm on the Scene Flow and KITTI Stereo datasets, and rank 1st on both the KITTI Stereo 2012 and 2015 benchmarks, which demonstrates the effectiveness of the proposed module again.