Neural collapse under cross-entropy loss
Publication
, Journal Article
Lu, J; Steinerberger, S
Published in: Applied and Computational Harmonic Analysis
July 1, 2022
We consider the variational problem of cross-entropy loss with n feature vectors on a unit hypersphere in Rd. We prove that when d≥n−1, the global minimum is given by the simplex equiangular tight frame, which justifies the neural collapse behavior. We also prove that, as n→∞ with fixed d, the minimizing points will distribute uniformly on the hypersphere and show a connection with the frame potential of Benedetto & Fickus.
Duke Scholars
Published In
Applied and Computational Harmonic Analysis
DOI
EISSN
1096-603X
ISSN
1063-5203
Publication Date
July 1, 2022
Volume
59
Start / End Page
224 / 241
Related Subject Headings
- Numerical & Computational Mathematics
- 4904 Pure mathematics
- 4901 Applied mathematics
- 0103 Numerical and Computational Mathematics
- 0102 Applied Mathematics
- 0101 Pure Mathematics
Citation
APA
Chicago
ICMJE
MLA
NLM
Lu, J., & Steinerberger, S. (2022). Neural collapse under cross-entropy loss. Applied and Computational Harmonic Analysis, 59, 224–241. https://doi.org/10.1016/j.acha.2021.12.011
Lu, J., and S. Steinerberger. “Neural collapse under cross-entropy loss.” Applied and Computational Harmonic Analysis 59 (July 1, 2022): 224–41. https://doi.org/10.1016/j.acha.2021.12.011.
Lu J, Steinerberger S. Neural collapse under cross-entropy loss. Applied and Computational Harmonic Analysis. 2022 Jul 1;59:224–41.
Lu, J., and S. Steinerberger. “Neural collapse under cross-entropy loss.” Applied and Computational Harmonic Analysis, vol. 59, July 2022, pp. 224–41. Scopus, doi:10.1016/j.acha.2021.12.011.
Lu J, Steinerberger S. Neural collapse under cross-entropy loss. Applied and Computational Harmonic Analysis. 2022 Jul 1;59:224–241.
Published In
Applied and Computational Harmonic Analysis
DOI
EISSN
1096-603X
ISSN
1063-5203
Publication Date
July 1, 2022
Volume
59
Start / End Page
224 / 241
Related Subject Headings
- Numerical & Computational Mathematics
- 4904 Pure mathematics
- 4901 Applied mathematics
- 0103 Numerical and Computational Mathematics
- 0102 Applied Mathematics
- 0101 Pure Mathematics