Parametrisation Independence of the Natural Gradient in Overparametrised Systems
Publikationstyp
Conference Paper
Date Issued
2021-07
Sprache
English
Author(s)
Institut
First published in
Number in series
12829 LNCS
Start Page
726
End Page
735
Citation
International Conference on Geometric Science of Information (GSI 2021)
Contribution to Conference
Publisher DOI
Scopus ID
ISBN of container
978-303080208-0
In this paper we study the natural gradient method for overparametrised systems. This method is based on the natural gradient field which is invariant with respect to coordinate transformations. One calculates the natural gradient of a function on the manifold by multiplying the ordinary gradient of the function by the inverse of the Fisher Information Matrix (FIM). In overparametrised models, the FIM is degenerate and therefore one needs to use a generalised inverse. We show explicitly that using a generalised inverse, and in particular the Moore-Penrose inverse, does not affect the parametrisation independence of the natural gradient. Furthermore, we show that for singular points on the manifold the parametrisation independence is not even guaranteed for non-overparametrised models.
Subjects
Deep learning
Information geometry
Natural gradient
Riemannian metric