On the Training Algorithms for Restricted Boltzmann Machines

Abstract

Deep learning techniques have been studied extensively in the last years due to their good results related to essential tasks on a large range of applications, such as speech and face recognition, as well as object classification. Restrict Boltzmann Machines (RBMs) are among the most employed techniques, which are energy-based stochastic neural networks composed of two layers of neurons whose objective is to estimate the connection weights between them. Recently, the scientific community spent much effort on sampling methods since the effectiveness of RBMs is directly related to the success of such a process. Thereby, this work contributes to studies concerning different training algorithms for RBMs, as well as its variants Deep Belief Networks and Deep Boltzmann Machines. Further, the work covers the application of meta-heuristic methods concerning a proper fine-tune of these techniques. Moreover, the validation of the model is presented in the context of image reconstruction and unsupervised feature learning. In general, we present different approaches to training these techniques, as well as the evaluation of meta-heuristic methods for fine-tuning parameters, and its main contributions are: (i) temperature parameter introduction in DBM formulation, (ii) DBM using adaptive temperature, (iii) DBM meta-parameter optimization through meta-heuristic techniques, and (iv) infinity Restricted Boltzmann Machine (iRBM) meta-parameters optimization through meta-heuristic techniques.