Understanding Deflation Process in Over-parametrized Tensor Decomposition

Conference Paper

In this paper we study the training dynamics for gradient flow on over-parametrized tensor decomposition problems. Empirically, such training process often first fits larger components and then discovers smaller components, which is similar to a tensor deflation process that is commonly used in tensor decomposition algorithms. We prove that for orthogonally decomposable tensor, a slightly modified version of gradient flow would follow a tensor deflation process and recover all the tensor components. Our proof suggests that for orthogonal tensors, gradient flow dynamics works similarly as greedy low-rank learning in the matrix setting, which is a first step towards understanding the implicit regularization effect of over-parametrized models for low-rank tensors.

Duke Authors

Cited Authors

  • Ge, R; Ren, Y; Wang, X; Zhou, M

Published Date

  • January 1, 2021

Published In

Volume / Issue

  • 2 /

Start / End Page

  • 1299 - 1311

International Standard Serial Number (ISSN)

  • 1049-5258

International Standard Book Number 13 (ISBN-13)

  • 9781713845393

Citation Source

  • Scopus