Skip to main content

A Penalized Method for the Predictive Limit of Learning

Publication ,  Conference
Ding, J; Diao, E; Zhou, J; Tarokh, V
Published in: ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings
September 10, 2018

Machine learning systems learn from and make predictions by building models from observed data. Because large models tend to overfit while small models tend to underfit for a given fixed dataset, a critical challenge is to select an appropriate model (e.g. set of variables/features). Model selection aims to strike a balance between the goodness of fit and model complexity, and thus to gain reliable predictive power. In this paper, we study a penalized model selection technique that asymptotically achieves the optimal expected prediction loss (referred to as the limit of learning) offered by a set of candidate models. We prove that the proposed procedure is both statistically efficient in the sense that it asymptotically approaches the limit of learning, and computationally efficient in the sense that it can be much faster than cross validation methods. Our theory applies for a wide variety of model classes, loss functions, and high dimensions (in the sense that the models' complexity can grow with data size). We released a python package with our proposed method for general usage like logistic regression and neural networks.

Duke Scholars

Published In

ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings

DOI

ISSN

1520-6149

ISBN

9781538646588

Publication Date

September 10, 2018

Volume

2018-April

Start / End Page

4414 / 4418
 

Citation

APA
Chicago
ICMJE
MLA
NLM
Ding, J., Diao, E., Zhou, J., & Tarokh, V. (2018). A Penalized Method for the Predictive Limit of Learning. In ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings (Vol. 2018-April, pp. 4414–4418). https://doi.org/10.1109/ICASSP.2018.8461832
Ding, J., E. Diao, J. Zhou, and V. Tarokh. “A Penalized Method for the Predictive Limit of Learning.” In ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings, 2018-April:4414–18, 2018. https://doi.org/10.1109/ICASSP.2018.8461832.
Ding J, Diao E, Zhou J, Tarokh V. A Penalized Method for the Predictive Limit of Learning. In: ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings. 2018. p. 4414–8.
Ding, J., et al. “A Penalized Method for the Predictive Limit of Learning.” ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings, vol. 2018-April, 2018, pp. 4414–18. Scopus, doi:10.1109/ICASSP.2018.8461832.
Ding J, Diao E, Zhou J, Tarokh V. A Penalized Method for the Predictive Limit of Learning. ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings. 2018. p. 4414–4418.

Published In

ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings

DOI

ISSN

1520-6149

ISBN

9781538646588

Publication Date

September 10, 2018

Volume

2018-April

Start / End Page

4414 / 4418