Explaining landscape connectivity of low-cost solutions for multilayer nets
Publication
, Journal Article
Kuditipudi, R; Wang, X; Lee, H; Zhang, Y; Li, Z; Hu, W; Arora, S; Ge, R
Published in: Advances in Neural Information Processing Systems
January 1, 2019
Mode connectivity (Garipov et al., 2018; Draxler et al., 2018) is a surprising phenomenon in the loss landscape of deep nets. Optima'at least those discovered by gradient-based optimization'turn out to be connected by simple paths on which the loss function is almost constant. Often, these paths can be chosen to be piece-wise linear, with as few as two segments. We give mathematical explanations for this phenomenon, assuming generic properties (such as dropout stability and noise stability) of well-trained deep nets, which have previously been identified as part of understanding the generalization properties of deep nets. Our explanation holds for realistic multilayer nets, and experiments are presented to verify the theory.
Duke Scholars
Published In
Advances in Neural Information Processing Systems
ISSN
1049-5258
Publication Date
January 1, 2019
Volume
32
Related Subject Headings
- 4611 Machine learning
- 1702 Cognitive Sciences
- 1701 Psychology
Citation
APA
Chicago
ICMJE
MLA
NLM
Kuditipudi, R., Wang, X., Lee, H., Zhang, Y., Li, Z., Hu, W., … Ge, R. (2019). Explaining landscape connectivity of low-cost solutions for multilayer nets. Advances in Neural Information Processing Systems, 32.
Kuditipudi, R., X. Wang, H. Lee, Y. Zhang, Z. Li, W. Hu, S. Arora, and R. Ge. “Explaining landscape connectivity of low-cost solutions for multilayer nets.” Advances in Neural Information Processing Systems 32 (January 1, 2019).
Kuditipudi R, Wang X, Lee H, Zhang Y, Li Z, Hu W, et al. Explaining landscape connectivity of low-cost solutions for multilayer nets. Advances in Neural Information Processing Systems. 2019 Jan 1;32.
Kuditipudi, R., et al. “Explaining landscape connectivity of low-cost solutions for multilayer nets.” Advances in Neural Information Processing Systems, vol. 32, Jan. 2019.
Kuditipudi R, Wang X, Lee H, Zhang Y, Li Z, Hu W, Arora S, Ge R. Explaining landscape connectivity of low-cost solutions for multilayer nets. Advances in Neural Information Processing Systems. 2019 Jan 1;32.
Published In
Advances in Neural Information Processing Systems
ISSN
1049-5258
Publication Date
January 1, 2019
Volume
32
Related Subject Headings
- 4611 Machine learning
- 1702 Cognitive Sciences
- 1701 Psychology