Sparse multinomial logistic regression: fast algorithms and generalization bounds.

Published

Journal Article

Recently developed methods for learning sparse classifiers are among the state-of-the-art in supervised learning. These methods learn classifiers that incorporate weighted sums of basis functions with sparsity-promoting priors encouraging the weight estimates to be either significantly large or exactly zero. From a learning-theoretic perspective, these methods control the capacity of the learned classifier by minimizing the number of basis functions used, resulting in better generalization. This paper presents three contributions related to learning sparse classifiers. First, we introduce a true multiclass formulation based on multinomial logistic regression. Second, by combining a bound optimization approach with a component-wise update procedure, we derive fast exact algorithms for learning sparse multiclass classifiers that scale favorably in both the number of training samples and the feature dimensionality, making them applicable even to large data sets in high-dimensional feature spaces. To the best of our knowledge, these are the first algorithms to perform exact multinomial logistic regression with a sparsity-promoting prior. Third, we show how nontrivial generalization bounds can be derived for our classifier in the binary case. Experimental results on standard benchmark data sets attest to the accuracy, sparsity, and efficiency of the proposed methods.

Full Text

Duke Authors

Cited Authors

  • Krishnapuram, B; Carin, L; Figueiredo, MAT; Hartemink, AJ

Published Date

  • June 2005

Published In

Volume / Issue

  • 27 / 6

Start / End Page

  • 957 - 968

PubMed ID

  • 15943426

Pubmed Central ID

  • 15943426

Electronic International Standard Serial Number (EISSN)

  • 1939-3539

International Standard Serial Number (ISSN)

  • 0162-8828

Digital Object Identifier (DOI)

  • 10.1109/tpami.2005.127

Language

  • eng