Skip to main content
Journal cover image

Nonlinear Approximation and (Deep) ReLU Networks

Publication ,  Journal Article
Daubechies, I; DeVore, R; Foucart, S; Hanin, B; Petrova, G
Published in: Constructive Approximation
February 1, 2022

This article is concerned with the approximation and expressive powers of deep neural networks. This is an active research area currently producing many interesting papers. The results most commonly found in the literature prove that neural networks approximate functions with classical smoothness to the same accuracy as classical linear methods of approximation, e.g., approximation by polynomials or by piecewise polynomials on prescribed partitions. However, approximation by neural networks depending on n parameters is a form of nonlinear approximation and as such should be compared with other nonlinear methods such as variable knot splines or n-term approximation from dictionaries. The performance of neural networks in targeted applications such as machine learning indicate that they actually possess even greater approximation power than these traditional methods of nonlinear approximation. The main results of this article prove that this is indeed the case. This is done by exhibiting large classes of functions which can be efficiently captured by neural networks where classical nonlinear methods fall short of the task. The present article purposefully limits itself to studying the approximation of univariate functions by ReLU networks. Many generalizations to functions of several variables and other activation functions can be envisioned. However, even in this simplest of settings considered here, a theory that completely quantifies the approximation power of neural networks is still lacking.

Duke Scholars

Published In

Constructive Approximation

DOI

EISSN

1432-0940

ISSN

0176-4276

Publication Date

February 1, 2022

Volume

55

Issue

1

Start / End Page

127 / 172

Related Subject Headings

  • Numerical & Computational Mathematics
  • 4904 Pure mathematics
  • 4901 Applied mathematics
  • 0104 Statistics
  • 0103 Numerical and Computational Mathematics
  • 0102 Applied Mathematics
 

Citation

APA
Chicago
ICMJE
MLA
NLM
Daubechies, I., DeVore, R., Foucart, S., Hanin, B., & Petrova, G. (2022). Nonlinear Approximation and (Deep) ReLU Networks. Constructive Approximation, 55(1), 127–172. https://doi.org/10.1007/s00365-021-09548-z
Daubechies, I., R. DeVore, S. Foucart, B. Hanin, and G. Petrova. “Nonlinear Approximation and (Deep) ReLU Networks.” Constructive Approximation 55, no. 1 (February 1, 2022): 127–72. https://doi.org/10.1007/s00365-021-09548-z.
Daubechies I, DeVore R, Foucart S, Hanin B, Petrova G. Nonlinear Approximation and (Deep) ReLU Networks. Constructive Approximation. 2022 Feb 1;55(1):127–72.
Daubechies, I., et al. “Nonlinear Approximation and (Deep) ReLU Networks.” Constructive Approximation, vol. 55, no. 1, Feb. 2022, pp. 127–72. Scopus, doi:10.1007/s00365-021-09548-z.
Daubechies I, DeVore R, Foucart S, Hanin B, Petrova G. Nonlinear Approximation and (Deep) ReLU Networks. Constructive Approximation. 2022 Feb 1;55(1):127–172.
Journal cover image

Published In

Constructive Approximation

DOI

EISSN

1432-0940

ISSN

0176-4276

Publication Date

February 1, 2022

Volume

55

Issue

1

Start / End Page

127 / 172

Related Subject Headings

  • Numerical & Computational Mathematics
  • 4904 Pure mathematics
  • 4901 Applied mathematics
  • 0104 Statistics
  • 0103 Numerical and Computational Mathematics
  • 0102 Applied Mathematics