Gibbs ensembles for incompatible dependency networks.

In most statistical applications, the Gibbs sampler is the method of choice for inference regarding conditionally specified distributions that are compatible. Compatibility ensures that a unique Gibbs distribution exists. For machine learning of complex models such as dependency networks, the conditional models are sometimes incompatible. In this paper, we review an ensemble approach using the Gibbs sampler as the base procedure. A Gibbs ensemble consists of many joint distributions resulting from different scan orders of the same conditional model, and the solution is a weighted sum of the ensemble. The algorithm is scalable and can handle large data sets of high dimensionality. The proposed approach provides joint distributions that conform with the conditional specifications better than the solutions obtained by linear programming and by a fixed-scan Gibbs sampler alone. Owing to incompatibility, the invariant distribution of a Gibbs sampler is scan-order dependent. A Gibbs ensemble is the collection of joint distributions estimated from the Gibbs samples of different scan orders.