Parametrisation Independence of the Natural Gradient in Overparametrised Systems

Publikationstyp
Conference Paper
Date Issued
2021-07
Sprache
English
Author(s)
Oostrum, Jesse van  
Ay, Nihat  
Institut
Data Science Foundations E-21  
TORE-URI
http://hdl.handle.net/11420/10161
First published in
Lecture notes in computer science  
Number in series
12829 LNCS
Start Page
726
End Page
735
Citation
International Conference on Geometric Science of Information (GSI 2021)
Contribution to Conference
5th International Conference on Geometric Science of Information, GSI 2021  
Publisher DOI
10.1007/978-3-030-80209-7_78
Scopus ID
2-s2.0-85112585821
In this paper we study the natural gradient method for overparametrised systems. This method is based on the natural gradient field which is invariant with respect to coordinate transformations. One calculates the natural gradient of a function on the manifold by multiplying the ordinary gradient of the function by the inverse of the Fisher Information Matrix (FIM). In overparametrised models, the FIM is degenerate and therefore one needs to use a generalised inverse. We show explicitly that using a generalised inverse, and in particular the Moore-Penrose inverse, does not affect the parametrisation independence of the natural gradient. Furthermore, we show that for singular points on the manifold the parametrisation independence is not even guaranteed for non-overparametrised models.
Subjects
Deep learning
Information geometry
Natural gradient
Riemannian metric