Skip to main content

Robust Large Margin Deep Neural Networks

Publication ,  Journal Article
Sokolić, J; Giryes, R; Sapiro, G; Rodrigues, MRD
Published in: IEEE Transactions on Signal Processing
August 15, 2017

The generalization error of deep neural networks via their classification margin is studied in this paper. Our approach is based on the Jacobian matrix of a deep neural network and can be applied to networks with arbitrary nonlinearities and pooling layers, and to networks with different architectures such as feed forward networks and residual networks. Our analysis leads to the conclusion that a bounded spectral norm of the network's Jacobian matrix in the neighbourhood of the training samples is crucial for a deep neural network of arbitrary depth and width to generalize well. This is a significant improvement over the current bounds in the literature, which imply that the generalization error grows with either the width or the depth of the network. Moreover, it shows that the recently proposed batch normalization and weight normalization reparametrizations enjoy good generalization properties, and leads to a novel network regularizer based on the network's Jacobian matrix. The analysis is supported with experimental results on the MNIST, CIFAR-10, LaRED, and ImageNet datasets.

Duke Scholars

Altmetric Attention Stats
Dimensions Citation Stats

Published In

IEEE Transactions on Signal Processing

DOI

ISSN

1053-587X

Publication Date

August 15, 2017

Volume

65

Issue

16

Start / End Page

4265 / 4280

Related Subject Headings

  • Networking & Telecommunications
 

Citation

APA
Chicago
ICMJE
MLA
NLM
Sokolić, J., Giryes, R., Sapiro, G., & Rodrigues, M. R. D. (2017). Robust Large Margin Deep Neural Networks. IEEE Transactions on Signal Processing, 65(16), 4265–4280. https://doi.org/10.1109/TSP.2017.2708039
Sokolić, J., R. Giryes, G. Sapiro, and M. R. D. Rodrigues. “Robust Large Margin Deep Neural Networks.” IEEE Transactions on Signal Processing 65, no. 16 (August 15, 2017): 4265–80. https://doi.org/10.1109/TSP.2017.2708039.
Sokolić J, Giryes R, Sapiro G, Rodrigues MRD. Robust Large Margin Deep Neural Networks. IEEE Transactions on Signal Processing. 2017 Aug 15;65(16):4265–80.
Sokolić, J., et al. “Robust Large Margin Deep Neural Networks.” IEEE Transactions on Signal Processing, vol. 65, no. 16, Aug. 2017, pp. 4265–80. Scopus, doi:10.1109/TSP.2017.2708039.
Sokolić J, Giryes R, Sapiro G, Rodrigues MRD. Robust Large Margin Deep Neural Networks. IEEE Transactions on Signal Processing. 2017 Aug 15;65(16):4265–4280.

Published In

IEEE Transactions on Signal Processing

DOI

ISSN

1053-587X

Publication Date

August 15, 2017

Volume

65

Issue

16

Start / End Page

4265 / 4280

Related Subject Headings

  • Networking & Telecommunications