Nonparametric Function Estimation using Overcomplete Dictionaries (with Discussion)


Book Section

We consider the nonparametric regression problem of estimating an unknown function based on noisy data. One approach to this estimation problem is to represent the function in a series expansion using a linear combination of basis functions. Overcomplete dictionaries provide a larger, but redundant collection of generating elements than a basis, however, coefficients in the expansion are no longer unique. Despite the non-uniqueness, this has the potential to lead to sparser representations by using fewer non-zero coefficients. Compound Poisson random fields and their generalization to Levy random fields are ideally suited for construction of priors on functions using these overcomplete representations for the general nonparametric regression problem, and provide a natural limiting generalization of priors for the finite dimensional version of the regression problem. While expressions for posterior modes or posterior distributions of quantities of interest are not available in closed form, the prior construction using Levy random fields permits tractable posterior simulation via a reversible jump Markov chain Monte Carlo algorithm. Efficient computation is possible because updates based on adding/deleting or updating single dictionary elements bypass the need to invert large matrices. Furthermore, because dictionary elements are only computed as needed, memory requirements scale linearly with the sample size. In comparison with other methods, the Levy random field priors provide excellent performance in terms of both mean squared error and coverage for out-of-sample predictions.

Duke Authors

Cited Authors

  • Clyde, MA; Wolpert, RL

Cited Editors

  • Bernardo, JM; Bayarri, MJ; Berger, JO; Dawid, AP; Heckerman, D; Smith, AFM; West, M

Duke Contributors

Published Date

  • 2007

Book Title

  • Bayesian Statistics 8

Start / End Page

  • 91 - 114

Published By

International Standard Book Number 13 (ISBN-13)

  • 978-0-19-921465-5