Options
Natural Reweighted Wake–Sleep
Publikationstyp
Journal Article
Date Issued
2022-11
Sprache
English
Author(s)
Institut
Journal
Volume
155
Start Page
574
End Page
591
Citation
Neural Networks 155: 574-591 (2022-11)
Publisher DOI
Scopus ID
PubMed ID
36208615
Helmholtz Machines (HMs) are a class of generative models composed of two Sigmoid Belief Networks (SBNs), acting respectively as an encoder and a decoder. These models are commonly trained using a two-step optimization algorithm called Wake–Sleep (WS) and more recently by improved versions, such as Reweighted Wake–Sleep (RWS) and Bidirectional Helmholtz Machines (BiHM). The locality of the connections in an SBN induces sparsity in the Fisher Information Matrices associated to the probabilistic models, in the form of a finely-grained block-diagonal structure. In this paper we exploit this property to efficiently train SBNs and HMs using the natural gradient. We present a novel algorithm, called Natural Reweighted Wake–Sleep (NRWS), that corresponds to the geometric adaptation of its standard version. In a similar manner, we also introduce Natural Bidirectional Helmholtz Machine (NBiHM). Differently from previous work, we will show how for HMs the natural gradient can be efficiently computed without the need of introducing any approximation in the structure of the Fisher information matrix. The experiments performed on standard datasets from the literature show a consistent improvement of NRWS and NBiHM not only with respect to their non-geometric baselines but also with respect to state-of-the-art training algorithms for HMs. The improvement is quantified both in terms of speed of convergence as well as value of the log-likelihood reached after training.
Subjects
Helmholtz machine
Information geometry
Natural gradient
Wake–Sleep