Please use this identifier to cite or link to this item:
Publisher DOI: 10.1007/s41884-022-00067-9
Title: Invariance properties of the natural gradient in overparametrised systems
Language: English
Authors: van Oostrum, Jesse 
Müller, Johannes 
Ay, Nihat 
Keywords: Deep learning; Information geometry; Natural gradient; Riemannian metric
Issue Date: 27-Jun-2022
Publisher: Springer
Source: Information Geometry (in Press): (2022)
Abstract (english): 
The natural gradient field is a vector field that lives on a model equipped with a distinguished Riemannian metric, e.g. the Fisher–Rao metric, and represents the direction of steepest ascent of an objective function on the model with respect to this metric. In practice, one tries to obtain the corresponding direction on the parameter space by multiplying the ordinary gradient by the inverse of the Gram matrix associated with the metric. We refer to this vector on the parameter space as the natural parameter gradient. In this paper we study when the pushforward of the natural parameter gradient is equal to the natural gradient. Furthermore we investigate the invariance properties of the natural parameter gradient. Both questions are addressed in an overparametrised setting.
DOI: 10.15480/882.4812
ISSN: 2511-249X
Journal: Information geometry 
Institute: Data Science Foundations E-21 
Document Type: Article
Project: Projekt DEAL 
SPP 2134: Das handelnde Selbst 
Funded by: Deutsche Forschungsgemeinschaft (DFG) 
More Funding information: JM acknowledges support by the ERC under the European Union’s Horizon 2020 research and innovation programme (grant agreement no 757983), by the International Max Planck Research School for Mathematics in the Sciences and the Evangelisches Studienwerk Villigst e.V.
License: CC BY 4.0 (Attribution) CC BY 4.0 (Attribution)
Appears in Collections:Publications with fulltext

Files in This Item:
File Description SizeFormat
s41884-022-00067-9.pdfVerlags-PDF441,05 kBAdobe PDFView/Open
Show full item record

Page view(s)

checked on May 28, 2023


checked on May 28, 2023

Google ScholarTM


Note about this record

Cite this record


This item is licensed under a Creative Commons License Creative Commons