Amazing technological breakthrough possible @S-Logix

Office Address

  • 2nd Floor, #7a, High School Road, Secretariat Colony Ambattur, Chennai-600053 (Landmark: SRM School) Tamil Nadu, India
  • +91- 81240 01111

Social List

Mode-assisted unsupervised learning of restricted Boltzmann machines - 2020

Mode-Assisted Unsupervised Learning Of Restricted Boltzmann Machines

Research Area:  Machine Learning


Restricted Boltzmann machines (RBMs) are a powerful class of generative models, but their training requires computing a gradient that, unlike supervised backpropagation on typical loss functions, is notoriously difficult even to approximate. Here, we show that properly combining standard gradient updates with an off-gradient direction, constructed from samples of the RBM ground state (mode), improves training dramatically over traditional gradient methods. This approach, which we call mode-assisted training, promotes faster training and stability, in addition to lower converged relative entropy (KL divergence). We demonstrate its efficacy on synthetic datasets where we can compute KL divergences exactly, as well as on a larger machine learning standard (MNIST). The proposed mode-assisted training can be applied in conjunction with any given gradient method, and is easily extended to more general energy-based neural network structures such as deep, convolutional and unrestricted Boltzmann machines.


Author(s) Name:  Haik Manukian, Yan Ru Pei, Sean R. B. Bearden & Massimiliano Di Ventra

Journal name:  Communications Physics

Conferrence name:  

Publisher name:  Springer Nature


Volume Information:  Vol 3, Article number: 105