Skip to main content

UNDERSTANDING EDGE-OF-STABILITY TRAINING DYNAMICS WITH A MINIMALIST EXAMPLE

Publication ,  Conference
Zhu, X; Wang, Z; Wang, X; Zhou, M; Ge, R
Published in: 11th International Conference on Learning Representations, ICLR 2023
January 1, 2023

Recently, researchers observed that gradient descent for deep neural networks operates in an “edge-of-stability” (EoS) regime: the sharpness (maximum eigenvalue of the Hessian) is often larger than stability threshold 2/η (where η is the step size). Despite this, the loss oscillates and converges in the long run, and the sharpness at the end is just slightly below 2/η. While many other well-understood nonconvex objectives such as matrix factorization or two-layer networks can also converge despite large sharpness, there is often a larger gap between sharpness of the endpoint and 2/η. In this paper, we study EoS phenomenon by constructing a simple function that has the same behavior. We give rigorous analysis for its training dynamics in a large local region and explain why the final converging point has sharpness close to 2/η. Globally we observe that the training dynamics for our example have an interesting bifurcating behavior, which was also observed in the training of neural nets.

Duke Scholars

Published In

11th International Conference on Learning Representations, ICLR 2023

Publication Date

January 1, 2023
 

Citation

APA
Chicago
ICMJE
MLA
NLM
Zhu, X., Wang, Z., Wang, X., Zhou, M., & Ge, R. (2023). UNDERSTANDING EDGE-OF-STABILITY TRAINING DYNAMICS WITH A MINIMALIST EXAMPLE. In 11th International Conference on Learning Representations, ICLR 2023.
Zhu, X., Z. Wang, X. Wang, M. Zhou, and R. Ge. “UNDERSTANDING EDGE-OF-STABILITY TRAINING DYNAMICS WITH A MINIMALIST EXAMPLE.” In 11th International Conference on Learning Representations, ICLR 2023, 2023.
Zhu X, Wang Z, Wang X, Zhou M, Ge R. UNDERSTANDING EDGE-OF-STABILITY TRAINING DYNAMICS WITH A MINIMALIST EXAMPLE. In: 11th International Conference on Learning Representations, ICLR 2023. 2023.
Zhu, X., et al. “UNDERSTANDING EDGE-OF-STABILITY TRAINING DYNAMICS WITH A MINIMALIST EXAMPLE.” 11th International Conference on Learning Representations, ICLR 2023, 2023.
Zhu X, Wang Z, Wang X, Zhou M, Ge R. UNDERSTANDING EDGE-OF-STABILITY TRAINING DYNAMICS WITH A MINIMALIST EXAMPLE. 11th International Conference on Learning Representations, ICLR 2023. 2023.

Published In

11th International Conference on Learning Representations, ICLR 2023

Publication Date

January 1, 2023