Nonparametric Function Estimation using Overcomplete Dictionaries (with Discussion)

We consider the nonparametric regression problem of estimating an unknown function based on noisy data. One approach to this estimation problem is to represent the function in a series expansion using a linear combination of basis functions. Overcomplete dictionaries provide a larger, but redundant collection of generating elements than a basis, however, coefficients in the expansion are no longer unique. Despite the non-uniqueness, this has the potential to lead to sparser representations by using fewer non-zero coefficients. Compound Poisson random fields and their generalization to Levy random fields are ideally suited for construction of priors on functions using these overcomplete representations for the general nonparametric regression problem, and provide a natural limiting generalization of priors for the finite dimensional version of the regression problem. While expressions for posterior modes or posterior distributions of quantities of interest are not available in closed form, the prior construction using Levy random fields permits tractable posterior simulation via a reversible jump Markov chain Monte Carlo algorithm. Efficient computation is possible because updates based on adding/deleting or updating single dictionary elements bypass the need to invert large matrices. Furthermore, because dictionary elements are only computed as needed, memory requirements scale linearly with the sample size. In comparison with other methods, the Levy random field priors provide excellent performance in terms of both mean squared error and coverage for out-of-sample predictions.

Duke Scholars

Author Merlise Clyde Statistical Science

Author Robert L. Wolpert Statistical Science

Contributor Merlise Clyde Statistical Science