Skip to main content

Generalization error of deep neural networks: Role of classification margin and data structure

Publication ,  Conference
Sokolić, J; Giryes, R; Sapiro, G; Rodrigues, MRD
Published in: 2017 12th International Conference on Sampling Theory and Applications, SampTA 2017
September 1, 2017

Understanding the generalization properties of deep learning models is critical for their successful usage in many applications, especially in the regimes where the number of training samples is limited. We study the generalization properties of deep neural networks (DNNs) via the Jacobian matrix of the network. Our analysis is general to arbitrary network structures, types of non-linearities and pooling operations. We show that bounding the spectral norm of the Jacobian matrix in the network reduces the generalization error. In addition, we tie this error to the invariance in the data and the network. Experiments on the MNIST and ImageNet datasets support these findings. This short paper summarizes our generalization error theorems for DNNs and for general invariant classifiers [1], [2].

Duke Scholars

Published In

2017 12th International Conference on Sampling Theory and Applications, SampTA 2017

DOI

ISBN

9781538615652

Publication Date

September 1, 2017

Start / End Page

147 / 151
 

Citation

APA
Chicago
ICMJE
MLA
NLM
Sokolić, J., Giryes, R., Sapiro, G., & Rodrigues, M. R. D. (2017). Generalization error of deep neural networks: Role of classification margin and data structure. In 2017 12th International Conference on Sampling Theory and Applications, SampTA 2017 (pp. 147–151). https://doi.org/10.1109/SAMPTA.2017.8024476
Sokolić, J., R. Giryes, G. Sapiro, and M. R. D. Rodrigues. “Generalization error of deep neural networks: Role of classification margin and data structure.” In 2017 12th International Conference on Sampling Theory and Applications, SampTA 2017, 147–51, 2017. https://doi.org/10.1109/SAMPTA.2017.8024476.
Sokolić J, Giryes R, Sapiro G, Rodrigues MRD. Generalization error of deep neural networks: Role of classification margin and data structure. In: 2017 12th International Conference on Sampling Theory and Applications, SampTA 2017. 2017. p. 147–51.
Sokolić, J., et al. “Generalization error of deep neural networks: Role of classification margin and data structure.” 2017 12th International Conference on Sampling Theory and Applications, SampTA 2017, 2017, pp. 147–51. Scopus, doi:10.1109/SAMPTA.2017.8024476.
Sokolić J, Giryes R, Sapiro G, Rodrigues MRD. Generalization error of deep neural networks: Role of classification margin and data structure. 2017 12th International Conference on Sampling Theory and Applications, SampTA 2017. 2017. p. 147–151.

Published In

2017 12th International Conference on Sampling Theory and Applications, SampTA 2017

DOI

ISBN

9781538615652

Publication Date

September 1, 2017

Start / End Page

147 / 151