Neural collapse under cross-entropy loss

Journal Article (Journal Article)

We consider the variational problem of cross-entropy loss with n feature vectors on a unit hypersphere in Rd. We prove that when d≥n−1, the global minimum is given by the simplex equiangular tight frame, which justifies the neural collapse behavior. We also prove that, as n→∞ with fixed d, the minimizing points will distribute uniformly on the hypersphere and show a connection with the frame potential of Benedetto & Fickus.

Full Text

Duke Authors

Cited Authors

  • Lu, J; Steinerberger, S

Published Date

  • July 1, 2022

Published In

Volume / Issue

  • 59 /

Start / End Page

  • 224 - 241

Electronic International Standard Serial Number (EISSN)

  • 1096-603X

International Standard Serial Number (ISSN)

  • 1063-5203

Digital Object Identifier (DOI)

  • 10.1016/j.acha.2021.12.011

Citation Source

  • Scopus