Learning two-layer neural networks with symmetric inputs

Conference Paper

We give a new algorithm for learning a two-layer neural network under a general class of input distributions. Assuming there is a ground-truth two-layer network y = Aσ(Wx) + ξ, where A, W are weight matrices, ξ represents noise, and the number of neurons in the hidden layer is no larger than the input or output, our algorithm is guaranteed to recover the parameters A, W of the ground-truth network. The only requirement on the input x is that it is symmetric, which still allows highly complicated and structured input. Our algorithm is based on the method-of-moments framework and extends several results in tensor decompositions. We use spectral algorithms to avoid the complicated non-convex optimization in learning neural networks. Experiments show that our algorithm can robustly learn the ground-truth neural network with a small number of samples for many symmetric input distributions.

Duke Authors

Cited Authors

  • Ge, R; Li, Z; Kuditipudi, R; Wang, X

Published Date

  • January 1, 2019

Published In

  • 7th International Conference on Learning Representations, Iclr 2019

Citation Source

  • Scopus