Distributed Variational Representation Learning.

The problem of distributed representation learning is one in which multiple observations X1,…,XK are processed separately to learn as much information as possible about some source Y. We investigate this problem from information-theoretic grounds, through a generalization of Tishby's centralized Information Bottleneck (IB) method to the distributed setting. Specifically, K ≥ 2 encoders, compress their observations X1,…,XK separately such that, collectively, the produced representations preserve as much information as possible about Y. We study both discrete memoryless (DM) and vector Gaussian data models. For the discrete model, we establish a single-letter characterization of the optimal tradeoff for a class of memoryless sources. For the vector Gaussian model, we provide an explicit characterization of the optimal complexity-relevance tradeoff. Furthermore, we develop a variational bound on the complexity-relevance tradeoff which generalizes the evidence lower bound (ELBO) to the distributed setting. We provide two algorithms to compute this bound: i) a Blahut-Arimoto type iterative algorithm which computes optimal complexity-relevance mappings by iterating over a set of self-consistent equations, and ii) a variational inference type algorithm in which the encoding mappings are parametrized by neural networks, the bound approximated by Markov sampling and optimized with stochastic gradient descent. Numerical results on synthetic and real datasets are provided to support the efficiency of the approaches and algorithms developed in this paper.