Neural Network Training as an Optimal Control Problem :-An Augmented Lagrangian Approach-

Brecht Evens, Puya Latafat, Andreas Themelis, Johan Suykens, Panagiotis Patrinos

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Training of neural networks amounts to nonconvex optimization problems that are typically solved by using backpropagation and (variants of) stochastic gradient descent. In this work, we propose an alternative approach by viewing the training task as a nonlinear optimal control problem. Under this lens, backpropagation amounts to the sequential approach (single shooting) to optimal control, where the states variables have been eliminated. It is well known that single shooting may lead to ill-conditioning, and for this reason the simultaneous approach (multiple shooting) is typically preferred. Motivated by this hypothesis, an augmented Lagrangian algorithm is developed that only requires an approximate solution to the Lagrangian subproblems up to a user-defined accuracy. By applying this framework to the training of neural networks, it is shown that the inner Lagrangian subproblems are amenable to be solved using Gauss-Newton iterations. To fully exploit the structure of neural networks, the resulting linear least-squares problems are addressed by employing an approach based on forward dynamic programming. Finally, the effectiveness of our method is showcased on regression datasets.

Original languageEnglish
Title of host publication60th IEEE Conference on Decision and Control, CDC 2021
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages5136-5143
Number of pages8
ISBN (Electronic)9781665436595
DOIs
Publication statusPublished - 2021
Externally publishedYes
Event60th IEEE Conference on Decision and Control, CDC 2021 - Austin, United States
Duration: Dec 13 2021Dec 17 2021

Publication series

NameProceedings of the IEEE Conference on Decision and Control
Volume2021-December
ISSN (Print)0743-1546

Conference

Conference60th IEEE Conference on Decision and Control, CDC 2021
Country/TerritoryUnited States
CityAustin
Period12/13/2112/17/21

All Science Journal Classification (ASJC) codes

  • Control and Systems Engineering
  • Modelling and Simulation
  • Control and Optimization

Fingerprint

Dive into the research topics of 'Neural Network Training as an Optimal Control Problem :-An Augmented Lagrangian Approach-'. Together they form a unique fingerprint.

Cite this