3 results
Now showing1 - 3 of 3
- Some of the metrics are blocked by yourconsent settings
Item type:Publication, Access status: with files , On dissipativity of cross-entropy loss in training ResNets — A turnpike towards architecture searchThe training of ResNets and neural ODEs can be formulated and analyzed from the perspective of optimal control. This paper proposes a dissipative formulation of the training of ResNets and neural ODEs for classification problems. Specifically, we consider a variant of the cross-entropy (label smoothing) as a loss function and as a regularization in the stage cost. Based on our dissipative formulation of the training, we prove that the training OCPs for ResNets and neural ODEs alike exhibit the turnpike phenomenon. We illustrate this finding with numerical results for the two spirals and MNIST datasets. Crucially, our training formulation ensures that the transformation of the data from input to output is achieved in the first layers. In the following layers, which constitute the turnpike, the data remains at an equilibrium state and therefore these layers do not contribute to the transformation learned. In principle, these layers can be pruned after training, resulting in a network with only the necessary number of layers thus simplifying tuning of hyperparameters.Publicationtype: Journal Article34 17 - Some of the metrics are blocked by yourconsent settings
Item type:Publication, Access status: with files , A polynomial chaos approach to stochastic LQ optimal control: error bounds and infinite-horizon results(Elsevier, 2025-02-06); ; ; ; The stochastic linear–quadratic regulator problem subject to Gaussian disturbances is well known and usually addressed via a moment-based reformulation. Here, we leverage polynomial chaos expansions, which model random variables via series expansions in a suitable L2 probability space, to tackle the non-Gaussian case. We present the optimal solutions for finite and infinite horizons and we analyze the infinite-horizon asymptotics. We show that the limit of the optimal state-input trajectory is the unique solution to a corresponding stochastic stationary optimization problem in the sense of probability measures. Moreover, we provide a constructive error analysis for finite-dimensional polynomial chaos approximations of the optimal solutions and of the optimal stationary pair in non-Gaussian settings. A numerical example illustrates our findings.Publicationtype: Journal Article32 26 - Some of the metrics are blocked by yourconsent settings
Item type:Publication, Access status: with files , On the turnpike to design of deep neural networks : explicit depth boundsIt is well-known that the training of Deep Neural Networks (DNN) can be formalized in the language of optimal control. In this context, this paper leverages classical turnpike properties of optimal control problems to attempt a quantifiable answer to the question of how many layers should be considered in a DNN. The underlying assumption is that the number of neurons per layer—i.e., the width of the DNN—is kept constant. Pursuing a different route than the classical analysis of approximation properties of sigmoidal functions, we prove explicit bounds on the required depths of DNNs based on asymptotic reachability assumptions and a dissipativity-inducing choice of the regularization terms in the training problem. Numerical results obtained for the two spiral task data set for classification indicate that the proposed constructive estimates can provide non-conservative depth bounds.Publicationtype: Journal Article31 30