Skip to main content

Fiedler regularization: Learning neural networks with graph sparsity

Publication ,  Conference
Tam, E; Dunson, D
Published in: 37th International Conference on Machine Learning, ICML 2020
January 1, 2020

We introduce a novel regularization approach for deep learning that incorporates and respects the underlying graphical structure of the neural network. Existing regularization methods often focus on penalizing weights in a global/uniform manner that ignores the connectivity structure of the neural network. We propose to use the Fiedler value of the neural network's underlying graph as a tool for regularization. We provide theoretical support for this approach via spectral graph theory. We show several useful properties of the Fiedler value that make it suitable for regularization. We provide an approximate, variational approach for faster computation during training. We provide an alternative formulation of this framework in the form of a structurally weighted L1 penalty, thus linking our approach to sparsity induction. We performed experiments on datasets that compare Fiedler regularization with traditional regularization methods such as Dropout and weight decay. Results demonstrate the efficacy of Fiedler regularization.

Duke Scholars

Published In

37th International Conference on Machine Learning, ICML 2020

ISBN

9781713821120

Publication Date

January 1, 2020

Volume

PartF168147-12

Start / End Page

9288 / 9297
 

Citation

APA
Chicago
ICMJE
MLA
NLM
Tam, E., & Dunson, D. (2020). Fiedler regularization: Learning neural networks with graph sparsity. In 37th International Conference on Machine Learning, ICML 2020 (Vol. PartF168147-12, pp. 9288–9297).
Tam, E., and D. Dunson. “Fiedler regularization: Learning neural networks with graph sparsity.” In 37th International Conference on Machine Learning, ICML 2020, PartF168147-12:9288–97, 2020.
Tam E, Dunson D. Fiedler regularization: Learning neural networks with graph sparsity. In: 37th International Conference on Machine Learning, ICML 2020. 2020. p. 9288–97.
Tam, E., and D. Dunson. “Fiedler regularization: Learning neural networks with graph sparsity.” 37th International Conference on Machine Learning, ICML 2020, vol. PartF168147-12, 2020, pp. 9288–97.
Tam E, Dunson D. Fiedler regularization: Learning neural networks with graph sparsity. 37th International Conference on Machine Learning, ICML 2020. 2020. p. 9288–9297.

Published In

37th International Conference on Machine Learning, ICML 2020

ISBN

9781713821120

Publication Date

January 1, 2020

Volume

PartF168147-12

Start / End Page

9288 / 9297