Skip to main content

Stop memorizing: A data-dependent regularization framework for intrinsic pattern learning

Publication ,  Journal Article
Zhu, W; Qiu, Q; Wang, B; Lu, J; Sapiro, G; Daubechies, I
May 18, 2018

Deep neural networks (DNNs) typically have enough capacity to fit random data by brute force even when conventional data-dependent regularizations focusing on the geometry of the features are imposed. We find out that the reason for this is the inconsistency between the enforced geometry and the standard softmax cross entropy loss. To resolve this, we propose a new framework for data-dependent DNN regularization, the Geometrically-Regularized-Self-Validating neural Networks (GRSVNet). During training, the geometry enforced on one batch of features is simultaneously validated on a separate batch using a validation loss consistent with the geometry. We study a particular case of GRSVNet, the Orthogonal-Low-rank Embedding (OLE)-GRSVNet, which is capable of producing highly discriminative features residing in orthogonal low-rank subspaces. Numerical experiments show that OLE-GRSVNet outperforms DNNs with conventional regularization when trained on real data. More importantly, unlike conventional DNNs, OLE-GRSVNet refuses to memorize random data or random labels, suggesting it only learns intrinsic patterns by reducing the memorizing capacity of the baseline DNN.

Duke Scholars

Publication Date

May 18, 2018
 

Citation

APA
Chicago
ICMJE
MLA
NLM
Zhu, W., Qiu, Q., Wang, B., Lu, J., Sapiro, G., & Daubechies, I. (2018). Stop memorizing: A data-dependent regularization framework for intrinsic pattern learning.
Zhu, Wei, Qiang Qiu, Bao Wang, Jianfeng Lu, Guillermo Sapiro, and Ingrid Daubechies. “Stop memorizing: A data-dependent regularization framework for intrinsic pattern learning,” May 18, 2018.
Zhu W, Qiu Q, Wang B, Lu J, Sapiro G, Daubechies I. Stop memorizing: A data-dependent regularization framework for intrinsic pattern learning. 2018 May 18;
Zhu W, Qiu Q, Wang B, Lu J, Sapiro G, Daubechies I. Stop memorizing: A data-dependent regularization framework for intrinsic pattern learning. 2018 May 18;

Publication Date

May 18, 2018