Fiedler regularization: Learning neural networks with graph sparsity

Conference Paper

We introduce a novel regularization approach for deep learning that incorporates and respects the underlying graphical structure of the neural network. Existing regularization methods often focus on penalizing weights in a global/uniform manner that ignores the connectivity structure of the neural network. We propose to use the Fiedler value of the neural network's underlying graph as a tool for regularization. We provide theoretical support for this approach via spectral graph theory. We show several useful properties of the Fiedler value that make it suitable for regularization. We provide an approximate, variational approach for faster computation during training. We provide an alternative formulation of this framework in the form of a structurally weighted L1 penalty, thus linking our approach to sparsity induction. We performed experiments on datasets that compare Fiedler regularization with traditional regularization methods such as Dropout and weight decay. Results demonstrate the efficacy of Fiedler regularization.

Duke Authors

Cited Authors

  • Tam, E; Dunson, D

Published Date

  • January 1, 2020

Published In

  • 37th International Conference on Machine Learning, Icml 2020

Volume / Issue

  • PartF168147-12 /

Start / End Page

  • 9288 - 9297

International Standard Book Number 13 (ISBN-13)

  • 9781713821120

Citation Source

  • Scopus