Skip to main content

Escaping from saddle points: Online stochastic gradient for tensor decomposition

Publication ,  Conference
Ge, R; Huang, F; Jin, C; Yuan, Y
Published in: Journal of Machine Learning Research
January 1, 2015

We analyze stochastic gradient descent for optimizing non-convex functions. In many cases for non-convex functions the goal is to find a reasonable local minimum, and the main concern is that gradient updates are trapped in saddle points. In this paper we identify strict saddle property for non-convex problem that allows for efficient optimization. Using this property we show that from an arbitrary starting point, stochastic gradient descent converges to a local minimum in a polynomial number of iterations. To the best of our knowledge this is the first work that gives global convergence guarantees for stochastic gradient descent on non-convex functions with exponentially many local minima and saddle points. Our analysis can be applied to orthogonal tensor decomposition, which is widely used in learning a rich class of latent variable models. We propose a new optimization formulation for the tensor decomposition problem that has strict saddle property. As a result we get the first online algorithm for orthogonal tensor decomposition with global convergence guarantee.

Duke Scholars

Published In

Journal of Machine Learning Research

EISSN

1533-7928

ISSN

1532-4435

Publication Date

January 1, 2015

Volume

40

Issue

2015

Related Subject Headings

  • Artificial Intelligence & Image Processing
  • 4905 Statistics
  • 4611 Machine learning
  • 17 Psychology and Cognitive Sciences
  • 08 Information and Computing Sciences
 

Citation

APA
Chicago
ICMJE
MLA
NLM
Ge, R., Huang, F., Jin, C., & Yuan, Y. (2015). Escaping from saddle points: Online stochastic gradient for tensor decomposition. In Journal of Machine Learning Research (Vol. 40).
Ge, R., F. Huang, C. Jin, and Y. Yuan. “Escaping from saddle points: Online stochastic gradient for tensor decomposition.” In Journal of Machine Learning Research, Vol. 40, 2015.
Ge R, Huang F, Jin C, Yuan Y. Escaping from saddle points: Online stochastic gradient for tensor decomposition. In: Journal of Machine Learning Research. 2015.
Ge, R., et al. “Escaping from saddle points: Online stochastic gradient for tensor decomposition.” Journal of Machine Learning Research, vol. 40, no. 2015, 2015.
Ge R, Huang F, Jin C, Yuan Y. Escaping from saddle points: Online stochastic gradient for tensor decomposition. Journal of Machine Learning Research. 2015.

Published In

Journal of Machine Learning Research

EISSN

1533-7928

ISSN

1532-4435

Publication Date

January 1, 2015

Volume

40

Issue

2015

Related Subject Headings

  • Artificial Intelligence & Image Processing
  • 4905 Statistics
  • 4611 Machine learning
  • 17 Psychology and Cognitive Sciences
  • 08 Information and Computing Sciences