Skip to main content

Sparse multinomial logistic regression: fast algorithms and generalization bounds.

Publication ,  Journal Article
Krishnapuram, B; Carin, L; Figueiredo, MAT; Hartemink, AJ
Published in: IEEE transactions on pattern analysis and machine intelligence
June 2005

Recently developed methods for learning sparse classifiers are among the state-of-the-art in supervised learning. These methods learn classifiers that incorporate weighted sums of basis functions with sparsity-promoting priors encouraging the weight estimates to be either significantly large or exactly zero. From a learning-theoretic perspective, these methods control the capacity of the learned classifier by minimizing the number of basis functions used, resulting in better generalization. This paper presents three contributions related to learning sparse classifiers. First, we introduce a true multiclass formulation based on multinomial logistic regression. Second, by combining a bound optimization approach with a component-wise update procedure, we derive fast exact algorithms for learning sparse multiclass classifiers that scale favorably in both the number of training samples and the feature dimensionality, making them applicable even to large data sets in high-dimensional feature spaces. To the best of our knowledge, these are the first algorithms to perform exact multinomial logistic regression with a sparsity-promoting prior. Third, we show how nontrivial generalization bounds can be derived for our classifier in the binary case. Experimental results on standard benchmark data sets attest to the accuracy, sparsity, and efficiency of the proposed methods.

Duke Scholars

Altmetric Attention Stats
Dimensions Citation Stats

Published In

IEEE transactions on pattern analysis and machine intelligence

DOI

EISSN

1939-3539

ISSN

0162-8828

Publication Date

June 2005

Volume

27

Issue

6

Start / End Page

957 / 968

Related Subject Headings

  • Regression Analysis
  • Pattern Recognition, Automated
  • Models, Statistical
  • Models, Biological
  • Information Storage and Retrieval
  • Computer Simulation
  • Cluster Analysis
  • Artificial Intelligence & Image Processing
  • Artificial Intelligence
  • Algorithms
 

Citation

APA
Chicago
ICMJE
MLA
NLM
Krishnapuram, B., Carin, L., Figueiredo, M. A. T., & Hartemink, A. J. (2005). Sparse multinomial logistic regression: fast algorithms and generalization bounds. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(6), 957–968. https://doi.org/10.1109/tpami.2005.127
Krishnapuram, Balaji, Lawrence Carin, Mário A. T. Figueiredo, and Alexander J. Hartemink. “Sparse multinomial logistic regression: fast algorithms and generalization bounds.IEEE Transactions on Pattern Analysis and Machine Intelligence 27, no. 6 (June 2005): 957–68. https://doi.org/10.1109/tpami.2005.127.
Krishnapuram B, Carin L, Figueiredo MAT, Hartemink AJ. Sparse multinomial logistic regression: fast algorithms and generalization bounds. IEEE transactions on pattern analysis and machine intelligence. 2005 Jun;27(6):957–68.
Krishnapuram, Balaji, et al. “Sparse multinomial logistic regression: fast algorithms and generalization bounds.IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 27, no. 6, June 2005, pp. 957–68. Epmc, doi:10.1109/tpami.2005.127.
Krishnapuram B, Carin L, Figueiredo MAT, Hartemink AJ. Sparse multinomial logistic regression: fast algorithms and generalization bounds. IEEE transactions on pattern analysis and machine intelligence. 2005 Jun;27(6):957–968.

Published In

IEEE transactions on pattern analysis and machine intelligence

DOI

EISSN

1939-3539

ISSN

0162-8828

Publication Date

June 2005

Volume

27

Issue

6

Start / End Page

957 / 968

Related Subject Headings

  • Regression Analysis
  • Pattern Recognition, Automated
  • Models, Statistical
  • Models, Biological
  • Information Storage and Retrieval
  • Computer Simulation
  • Cluster Analysis
  • Artificial Intelligence & Image Processing
  • Artificial Intelligence
  • Algorithms