Towards solving neural networks with optimization trajectory search

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Modern gradient based optimization methods for deep neural networks demonstrate outstanding results on image classification tasks. However, methods that do not rely on gradient feedback fail to tackle deep network optimization. In the field of evolutionary computation, applying evolutionary algorithms directly to network weights remains to be an unresolved challenge. In this paper we examine a new framework for the evolution of deep nets. Based on the empirical analysis, we propose the use of linear sub-spaces of problems to search for promising optimization trajectories in parameter space, opposed to weight evolution. We show that linear sub-spaces of loss functions are sufficiently well-behaved to allow trajectory evaluation. Furthermore, we introduce fitness measure to show that it is possible to correctly categorize trajectories according to their distance from the optimal path. As such, this work introduces an alternative approach to evolutionary optimization of deep networks.

Original languageEnglish
Title of host publicationGECCO 2019 Companion - Proceedings of the 2019 Genetic and Evolutionary Computation Conference Companion
PublisherAssociation for Computing Machinery, Inc
Pages75-76
Number of pages2
ISBN (Electronic)9781450367486
DOIs
Publication statusPublished - Jul 13 2019
Event2019 Genetic and Evolutionary Computation Conference, GECCO 2019 - Prague, Czech Republic
Duration: Jul 13 2019Jul 17 2019

Publication series

NameGECCO 2019 Companion - Proceedings of the 2019 Genetic and Evolutionary Computation Conference Companion

Conference

Conference2019 Genetic and Evolutionary Computation Conference, GECCO 2019
CountryCzech Republic
CityPrague
Period7/13/197/17/19

Fingerprint

Trajectory Optimization
Trajectories
Subspace
Neural Networks
Trajectory
Gradient
Neural networks
Network Optimization
Evolutionary Optimization
Optimal Path
Image Classification
Empirical Analysis
Evolutionary Computation
Loss Function
Evolutionary algorithms
Fitness
Optimization Methods
Parameter Space
Evolutionary Algorithms
Image classification

All Science Journal Classification (ASJC) codes

  • Artificial Intelligence
  • Theoretical Computer Science
  • Software

Cite this

Parsenadze, L. T., Vargas, D. V., & Fujita, T. (2019). Towards solving neural networks with optimization trajectory search. In GECCO 2019 Companion - Proceedings of the 2019 Genetic and Evolutionary Computation Conference Companion (pp. 75-76). (GECCO 2019 Companion - Proceedings of the 2019 Genetic and Evolutionary Computation Conference Companion). Association for Computing Machinery, Inc. https://doi.org/10.1145/3319619.3326796

Towards solving neural networks with optimization trajectory search. / Parsenadze, Lia T.; Vargas, Danilo Vasconcellos; Fujita, Toshiyuki.

GECCO 2019 Companion - Proceedings of the 2019 Genetic and Evolutionary Computation Conference Companion. Association for Computing Machinery, Inc, 2019. p. 75-76 (GECCO 2019 Companion - Proceedings of the 2019 Genetic and Evolutionary Computation Conference Companion).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Parsenadze, LT, Vargas, DV & Fujita, T 2019, Towards solving neural networks with optimization trajectory search. in GECCO 2019 Companion - Proceedings of the 2019 Genetic and Evolutionary Computation Conference Companion. GECCO 2019 Companion - Proceedings of the 2019 Genetic and Evolutionary Computation Conference Companion, Association for Computing Machinery, Inc, pp. 75-76, 2019 Genetic and Evolutionary Computation Conference, GECCO 2019, Prague, Czech Republic, 7/13/19. https://doi.org/10.1145/3319619.3326796
Parsenadze LT, Vargas DV, Fujita T. Towards solving neural networks with optimization trajectory search. In GECCO 2019 Companion - Proceedings of the 2019 Genetic and Evolutionary Computation Conference Companion. Association for Computing Machinery, Inc. 2019. p. 75-76. (GECCO 2019 Companion - Proceedings of the 2019 Genetic and Evolutionary Computation Conference Companion). https://doi.org/10.1145/3319619.3326796
Parsenadze, Lia T. ; Vargas, Danilo Vasconcellos ; Fujita, Toshiyuki. / Towards solving neural networks with optimization trajectory search. GECCO 2019 Companion - Proceedings of the 2019 Genetic and Evolutionary Computation Conference Companion. Association for Computing Machinery, Inc, 2019. pp. 75-76 (GECCO 2019 Companion - Proceedings of the 2019 Genetic and Evolutionary Computation Conference Companion).
@inproceedings{19aa0993ee0d47278bccc1305dabb283,
title = "Towards solving neural networks with optimization trajectory search",
abstract = "Modern gradient based optimization methods for deep neural networks demonstrate outstanding results on image classification tasks. However, methods that do not rely on gradient feedback fail to tackle deep network optimization. In the field of evolutionary computation, applying evolutionary algorithms directly to network weights remains to be an unresolved challenge. In this paper we examine a new framework for the evolution of deep nets. Based on the empirical analysis, we propose the use of linear sub-spaces of problems to search for promising optimization trajectories in parameter space, opposed to weight evolution. We show that linear sub-spaces of loss functions are sufficiently well-behaved to allow trajectory evaluation. Furthermore, we introduce fitness measure to show that it is possible to correctly categorize trajectories according to their distance from the optimal path. As such, this work introduces an alternative approach to evolutionary optimization of deep networks.",
author = "Parsenadze, {Lia T.} and Vargas, {Danilo Vasconcellos} and Toshiyuki Fujita",
year = "2019",
month = "7",
day = "13",
doi = "10.1145/3319619.3326796",
language = "English",
series = "GECCO 2019 Companion - Proceedings of the 2019 Genetic and Evolutionary Computation Conference Companion",
publisher = "Association for Computing Machinery, Inc",
pages = "75--76",
booktitle = "GECCO 2019 Companion - Proceedings of the 2019 Genetic and Evolutionary Computation Conference Companion",

}

TY - GEN

T1 - Towards solving neural networks with optimization trajectory search

AU - Parsenadze, Lia T.

AU - Vargas, Danilo Vasconcellos

AU - Fujita, Toshiyuki

PY - 2019/7/13

Y1 - 2019/7/13

N2 - Modern gradient based optimization methods for deep neural networks demonstrate outstanding results on image classification tasks. However, methods that do not rely on gradient feedback fail to tackle deep network optimization. In the field of evolutionary computation, applying evolutionary algorithms directly to network weights remains to be an unresolved challenge. In this paper we examine a new framework for the evolution of deep nets. Based on the empirical analysis, we propose the use of linear sub-spaces of problems to search for promising optimization trajectories in parameter space, opposed to weight evolution. We show that linear sub-spaces of loss functions are sufficiently well-behaved to allow trajectory evaluation. Furthermore, we introduce fitness measure to show that it is possible to correctly categorize trajectories according to their distance from the optimal path. As such, this work introduces an alternative approach to evolutionary optimization of deep networks.

AB - Modern gradient based optimization methods for deep neural networks demonstrate outstanding results on image classification tasks. However, methods that do not rely on gradient feedback fail to tackle deep network optimization. In the field of evolutionary computation, applying evolutionary algorithms directly to network weights remains to be an unresolved challenge. In this paper we examine a new framework for the evolution of deep nets. Based on the empirical analysis, we propose the use of linear sub-spaces of problems to search for promising optimization trajectories in parameter space, opposed to weight evolution. We show that linear sub-spaces of loss functions are sufficiently well-behaved to allow trajectory evaluation. Furthermore, we introduce fitness measure to show that it is possible to correctly categorize trajectories according to their distance from the optimal path. As such, this work introduces an alternative approach to evolutionary optimization of deep networks.

UR - http://www.scopus.com/inward/record.url?scp=85070649335&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85070649335&partnerID=8YFLogxK

U2 - 10.1145/3319619.3326796

DO - 10.1145/3319619.3326796

M3 - Conference contribution

AN - SCOPUS:85070649335

T3 - GECCO 2019 Companion - Proceedings of the 2019 Genetic and Evolutionary Computation Conference Companion

SP - 75

EP - 76

BT - GECCO 2019 Companion - Proceedings of the 2019 Genetic and Evolutionary Computation Conference Companion

PB - Association for Computing Machinery, Inc

ER -