Options
Invariance properties of the natural gradient in overparametrised systems
Citation Link: https://doi.org/10.15480/882.4812
Publikationstyp
Journal Article
Date Issued
2023-06
Sprache
English
Author(s)
Institut
TORE-DOI
Journal
Volume
6
Issue
1
Start Page
51
End Page
67
Citation
Information Geometry 6 (1): 51-67 (2023-06)
Publisher DOI
Scopus ID
Publisher
Springer
The natural gradient field is a vector field that lives on a model equipped with a distinguished Riemannian metric, e.g. the Fisher–Rao metric, and represents the direction of steepest ascent of an objective function on the model with respect to this metric. In practice, one tries to obtain the corresponding direction on the parameter space by multiplying the ordinary gradient by the inverse of the Gram matrix associated with the metric. We refer to this vector on the parameter space as the natural parameter gradient. In this paper we study when the pushforward of the natural parameter gradient is equal to the natural gradient. Furthermore we investigate the invariance properties of the natural parameter gradient. Both questions are addressed in an overparametrised setting.
Subjects
Deep learning
Information geometry
Natural gradient
Riemannian metric
DDC Class
600: Technik
620: Ingenieurwissenschaften
Funding(s)
Funding Organisations
More Funding Information
JM acknowledges support by the ERC under the European Union’s Horizon 2020 research and innovation programme (grant agreement no 757983), by the International Max Planck Research School for Mathematics in the Sciences and the Evangelisches Studienwerk Villigst e.V.
Publication version
publishedVersion
Loading...
Name
s41884-022-00067-9.pdf
Size
441.05 KB
Format
Adobe PDF