On Fisher Information Matrix for Simple Neural Networks with Softplus Activation

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Fisher information of simple neural networks with the softplus activation function is argued. We show that, under certain conditions, FIMs of simple models have the similar interesting spectral structure as the one shown by Takeishi et al (2021) for networks with ReLU. This work helps us to understand why the FIM has such the structure.

Original languageEnglish
Title of host publication2022 IEEE International Symposium on Information Theory, ISIT 2022
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages3001-3006
Number of pages6
ISBN (Electronic)9781665421591
DOIs
Publication statusPublished - 2022
Event2022 IEEE International Symposium on Information Theory, ISIT 2022 - Espoo, Finland
Duration: Jun 26 2022Jul 1 2022

Publication series

NameIEEE International Symposium on Information Theory - Proceedings
Volume2022-June
ISSN (Print)2157-8095

Conference

Conference2022 IEEE International Symposium on Information Theory, ISIT 2022
Country/TerritoryFinland
CityEspoo
Period6/26/227/1/22

All Science Journal Classification (ASJC) codes

  • Theoretical Computer Science
  • Information Systems
  • Modelling and Simulation
  • Applied Mathematics

Fingerprint

Dive into the research topics of 'On Fisher Information Matrix for Simple Neural Networks with Softplus Activation'. Together they form a unique fingerprint.

Cite this