Skip to main content

Guarantees for Tuning the Step Size using a Learning-to-Learn Approach

Publication ,  Conference
Wang, X; Yuan, S; Wu, C; Ge, R
Published in: Proceedings of Machine Learning Research
January 1, 2021

Choosing the right parameters for optimization algorithms is often the key to their success in practice. Solving this problem using a learning-to-learn approach-using meta-gradient descent on a meta-objective based on the trajectory that the optimizer generates-was recently shown to be effective. However, the meta-optimization problem is difficult. In particular, the meta-gradient can often explode/vanish, and the learned optimizer may not have good generalization performance if the meta-objective is not chosen carefully. In this paper we give meta-optimization guarantees for the learning-to-learn approach on a simple problem of tuning the step size for quadratic loss. Our results show that the naïve objective suffers from meta-gradient explosion/vanishing problem. Although there is a way to design the meta-objective so that the meta-gradient remains polynomially bounded, computing the meta-gradient directly using backpropagation leads to numerical issues. We also characterize when it is necessary to compute the meta-objective on a separate validation set to ensure the generalization performance of the learned optimizer. Finally, we verify our results empirically and show that a similar phenomenon appears even for more complicated learned optimizers parametrized by neural networks.

Duke Scholars

Published In

Proceedings of Machine Learning Research

EISSN

2640-3498

Publication Date

January 1, 2021

Volume

139

Start / End Page

10981 / 10990
 

Citation

APA
Chicago
ICMJE
MLA
NLM
Wang, X., Yuan, S., Wu, C., & Ge, R. (2021). Guarantees for Tuning the Step Size using a Learning-to-Learn Approach. In Proceedings of Machine Learning Research (Vol. 139, pp. 10981–10990).
Wang, X., S. Yuan, C. Wu, and R. Ge. “Guarantees for Tuning the Step Size using a Learning-to-Learn Approach.” In Proceedings of Machine Learning Research, 139:10981–90, 2021.
Wang X, Yuan S, Wu C, Ge R. Guarantees for Tuning the Step Size using a Learning-to-Learn Approach. In: Proceedings of Machine Learning Research. 2021. p. 10981–90.
Wang, X., et al. “Guarantees for Tuning the Step Size using a Learning-to-Learn Approach.” Proceedings of Machine Learning Research, vol. 139, 2021, pp. 10981–90.
Wang X, Yuan S, Wu C, Ge R. Guarantees for Tuning the Step Size using a Learning-to-Learn Approach. Proceedings of Machine Learning Research. 2021. p. 10981–10990.

Published In

Proceedings of Machine Learning Research

EISSN

2640-3498

Publication Date

January 1, 2021

Volume

139

Start / End Page

10981 / 10990