Research Area:  Machine Learning

Abstract:

We present a new approximate inference algorithm for Deep Boltzmann Machines (DBMs), a generative model with many layers of hidden variables. The algorithm learns a separate “recognition” model that is used to quickly initialize, in a single bottom-up pass, the values of the latent variables in all hidden layers. We show that using such a recognition model, followed by a combined top-down and bottom-up pass, it is possible to efficiently learn a good generative model of high-dimensional highly-structured sensory input. We show that the additional computations required by incorporating a top-down feedback plays a critical role in the performance of a DBM, both as a generative and discriminative model. Moreover, inference is only at most three times slower compared to the approximate inference in a Deep Belief Network (DBN), making large-scale learning of DBMs practical. Finally, we demonstrate that the DBMs trained using the proposed approximate inference algorithm perform well compared to DBNs and SVMs on the MNIST handwritten digit, OCR English letters, and NORB visual object recognition tasks.

Keywords:  
Artificial Intelligence
Statistics
Deep Boltzmann Machines
SVM
MNIST

Author(s) Name:  Ruslan Salakhutdinov, Hugo Larochelle

Publisher name:  PMLR

DOI:  

Volume Information:   Volume 9

Paper Link:   https://proceedings.mlr.press/v9/salakhutdinov10a.html