Skip to main content

Hidden Markov Pólya Trees for High-Dimensional Distributions

Publication ,  Journal Article
Awaya, N; Ma, L
Published in: Journal of the American Statistical Association
January 1, 2024

The Pólya tree (PT) process is a general-purpose Bayesian nonparametric model that has found wide application in a range of inference problems. It has a simple analytic form and the posterior computation boils down to beta-binomial conjugate updates along a partition tree over the sample space. Recent development in PT models shows that performance of these models can be substantially improved by (i) allowing the partition tree to adapt to the structure of the underlying distributions and (ii) incorporating latent state variables that characterize local features of the underlying distributions. However, important limitations of the PT remain, including (i) the sensitivity in the posterior inference with respect to the choice of the partition tree, and (ii) the lack of scalability with respect to dimensionality of the sample space. We consider a modeling strategy for PT models that incorporates a flexible prior on the partition tree along with latent states with Markov dependency. We introduce a hybrid algorithm combining sequential Monte Carlo (SMC) and recursive message passing for posterior sampling that can scale up to 100 dimensions. While our description of the algorithm assumes a single computer environment, it has the potential to be implemented on distributed systems to further enhance the scalability. Moreover, we investigate the large sample properties of the tree structures and latent states under the posterior model. We carry out extensive numerical experiments in density estimation and two-group comparison, which show that flexible partitioning can substantially improve the performance of PT models in both inference tasks. We demonstrate an application to a mass cytometry dataset with 19 dimensions and over 200,000 observations. Supplementary Materials for this article are available online.

Duke Scholars

Published In

Journal of the American Statistical Association

DOI

EISSN

1537-274X

ISSN

0162-1459

Publication Date

January 1, 2024

Volume

119

Issue

545

Start / End Page

189 / 201

Related Subject Headings

  • Statistics & Probability
  • 4905 Statistics
  • 3802 Econometrics
  • 1603 Demography
  • 1403 Econometrics
  • 0104 Statistics
 

Citation

APA
Chicago
ICMJE
MLA
NLM
Awaya, N., & Ma, L. (2024). Hidden Markov Pólya Trees for High-Dimensional Distributions. Journal of the American Statistical Association, 119(545), 189–201. https://doi.org/10.1080/01621459.2022.2105223
Awaya, N., and L. Ma. “Hidden Markov Pólya Trees for High-Dimensional Distributions.” Journal of the American Statistical Association 119, no. 545 (January 1, 2024): 189–201. https://doi.org/10.1080/01621459.2022.2105223.
Awaya N, Ma L. Hidden Markov Pólya Trees for High-Dimensional Distributions. Journal of the American Statistical Association. 2024 Jan 1;119(545):189–201.
Awaya, N., and L. Ma. “Hidden Markov Pólya Trees for High-Dimensional Distributions.” Journal of the American Statistical Association, vol. 119, no. 545, Jan. 2024, pp. 189–201. Scopus, doi:10.1080/01621459.2022.2105223.
Awaya N, Ma L. Hidden Markov Pólya Trees for High-Dimensional Distributions. Journal of the American Statistical Association. 2024 Jan 1;119(545):189–201.

Published In

Journal of the American Statistical Association

DOI

EISSN

1537-274X

ISSN

0162-1459

Publication Date

January 1, 2024

Volume

119

Issue

545

Start / End Page

189 / 201

Related Subject Headings

  • Statistics & Probability
  • 4905 Statistics
  • 3802 Econometrics
  • 1603 Demography
  • 1403 Econometrics
  • 0104 Statistics