Dirichlet Process Mixtures of Trees with Gaussian Process Splits: A Bayesian Nonparametric Framework with Posterior Contraction Rate

Subhasish Basak; Anik Roy; Sourabh Bhattacharya

stat.MEAug 22, 2026

Dirichlet Process Mixtures of Trees with Gaussian Process Splits: A Bayesian Nonparametric Framework with Posterior Contraction Rate

Authors: Subhasish Basak, Anik Roy, Sourabh Bhattacharya

Organizations: Indian Statistical Institute

Abstract

We propose a Bayesian nonparametric mixture of regression trees with a Dirichlet process prior over tree-parameter pairs, enabling data-driven selection of ensemble size and unifying CART, BART, random forests, and boosting. A novel splitting rule driven by the posterior predictive of a Gaussian process within each terminal node generates flexible, smooth decision boundaries; remarkably, the GP density cancels exactly in the Metropolis--Hastings ratio for GROW/PRUNE moves, ensuring computational feasibility. An exact Gibbs sampler for posterior predictive inference propagates uncertainty through random tree traversal. A parallel MPI implementation distributes independent tree updates across processors, achieving adequate speedups. We prove posterior consistency at rate $n^{-1/4}$ in Hellinger distance under only continuity of the true regression function, allowing misspecification, via the identity $h(Θ)=0$ . Simulations on Friedman benchmark show near-nominal coverage (0.94 Gaussian, 0.92 Cauchy), robust to high-dimensional noise and heavy tails, outperforming BART and bagged CART. Applications to QSAR toxicity, crime, riboflavin, wheat genomics, and air quality confirm reliable credible intervals and automatic sparsity. The DP mixture offers a principled, robust, theoretically justified alternative for challenging regression with honest uncertainty quantification.

Figures & tables

Panel of Figure 1 — Figure 1: Dirichlet process mixture of trees model. The Dirichlet process generates $M$ i.i.d. tree-parameter pairs, which are combined with random mixture weights to form a sparse ensemble with a data-driven number of components. The inset illustrates a single tree component; the blue curve represents a draw $\tilde{g}$ from the GP posterior predictive used as the splitting rule at an internal node.

Method	Special case of the DP mixture
CART	$M=1$ , or all DP draws coincident (single component)
BART	Base measure $G_{0}$ supported on additive tree ensembles; appropriate mixture weights
Random Forest	$\alpha\to\infty$ , $p_{i}\approx 1/M$ , components i.i.d. from $G_{0}$
Boosting	Scaled convex combination / additive base measure / large $\alpha$ limit

Table 1: Special cases of the Dirichlet process tree mixture.

A=\min\!\left\{1,\;\frac{\Pi_{\text{struct}}(\bm{T}^{\prime})}{\Pi_{\text{struct}}(\bm{T})}\frac{f(\bm{Y}|\bm{X},\bm{T}^{\prime})}{f(\bm{Y}|\bm{X},\bm{T})}\frac{b}{b^{*}+1}\right\}

Algorithm 2 MCMC tree step (GROW/PRUNE)

Method	RMSE	Coverage	Width	CRPS	LPD
d=5 (clean)
DP tree mixture	5.248	0.940	18.249	2.956	$-3.053$
Random Forest	2.958	0.960	10.908	1.674	$-2.508$
Gradient Boosting	2.561	0.920	9.949	1.428	$-2.360$
Bagged CART	3.144	0.800	8.938	1.912	$-2.703$
BART	2.085	0.820	5.873	1.199	$-2.199$

Table 2: Predictive performance comparison for the Gaussian noise experiments. The DP mixture achieves the highest coverage among Bayesian methods.

Method	RMSE	Coverage	Width	CRPS	LPD
d=5 (clean)
DP tree mixture	111.735	0.920	22.614	19.519	$-5365.445$
Random Forest	111.026	0.920	18.312	18.967	$-284.899$
Gradient Boosting	111.043	0.900	16.845	18.766	$-336.236$
Bagged CART	111.230	0.720	12.392	19.221	$-10642.342$
BART	110.856	0.780	10.169	18.691	$-12033.924$

Table 3: Predictive performance comparison for the Cauchy noise experiments. The DP mixture clearly dominates in coverage and LPD among Bayesian methods.

Figure 2: Posterior predictive density heatmap for the DP mixture in the clean d=5 Gaussian experiment. The red points are the true test values.

Figure 3: Trace plots of the number of distinct trees $k$ and the largest mixture weights for the clean d=5 Gaussian experiment.

Figure 4: BART posterior predictive density heatmap for the clean d=5 Gaussian experiment.

Figure 5: Posterior predictive density heatmap for the DP mixture in the high-dimensional d=20 Gaussian experiment.

Figure 6: Trace plots of the number of distinct trees $k$ and the largest mixture weights for the high-dimensional d=20 Gaussian experiment.

Figure 7: BART posterior predictive density heatmap for the high-dimensional d=20 Gaussian experiment.

Figure 8: Posterior predictive density heatmap for the DP mixture in the clean d=5 Cauchy experiment.

Figure 9: Trace plots of the number of distinct trees $k$ and the largest mixture weights for the clean d=5 Cauchy experiment.

Figure 10: BART posterior predictive density heatmap for the clean d=5 Cauchy experiment.

Figure 11: Posterior predictive density heatmap for the DP mixture in the high-dimensional d=20 Cauchy experiment.

Figure 12: Trace plots of the number of distinct trees $k$ and the largest mixture weights for the high-dimensional d=20 Cauchy experiment.

Figure 13: BART posterior predictive density heatmap for the high-dimensional d=20 Cauchy experiment.

Method	RMSE	Coverage	Width	CRPS	LPD
DP tree mixture	1.8212	0.918	6.4167	1.0408	$-2.0363$
Random Forest	1.3154	0.918	4.4239	0.6880	$-1.7191$
Gradient Boosting	1.2818	0.927	4.4919	0.6813	$-1.6808$
Bagged CART	1.3274	0.827	3.2097	0.7168	$-2.1051$
BART (dbarts)	1.2765	0.582	1.5881	0.7578	$-24.0618$

Table 4: Predictive performance on the QSAR aquatic toxicity test set. The DP mixture achieves the highest coverage among Bayesian methods and offers the most reliable uncertainty quantification, albeit at a significantly higher computational cost.

Figure 14: Posterior predictive density heatmap for the DP mixture on the QSAR test set. The true test values (red points) fall within the high-density regions, confirming good calibration.

Figure 15: BART posterior predictive density heatmap on the QSAR test set. The narrow bands of high density fail to cover many true values, confirming poor coverage.

Figure 16: Trace plots of the number of distinct trees $k$ and the mixture weights for the QSAR experiment. The chain mixes efficiently, and the posterior concentrates on a sparse ensemble of 2–5 trees.

Figure 17: Histogram of the responses for the QSAR experiment exhibiting multimodality and outlyingness.

Figure 18: Histogram of the response variable (ViolentCrimesPerPop) for the Communities and Crime dataset. The distribution is highly skewed and heavy-tailed, with most communities having low crime rates and a small number of high-crime communities in the tail. This skewness explains the failure of BART and the success of the DP mixture.

Method	RMSE	Coverage	Width	CRPS	LPD
DP tree mixture	0.1969	0.963	0.9853	0.1020	0.3337
Random Forest	0.0952	0.975	0.4683	0.0491	0.8880
Gradient Boosting	0.0983	0.981	0.4442	0.0497	0.8824
Bagged CART	0.0973	0.957	0.3117	0.0420	1.3518
BART (dbarts)	0.0970	0.870	0.2133	0.0479	-3.4684

Table 5: Predictive performance on the Communities and Crime test set (162 communities).

Figure 19: Posterior predictive density heatmap for the DP mixture on the Communities and Crime test set. True values (red points) fall within high-density regions, confirming good calibration.

Figure 20: BART posterior predictive density heatmap on the Communities and Crime test set. Narrow bands of high density fail to cover many true values, confirming poor coverage.

Figure 21: Trace plots of $k$ and the mixture weights for the Communities and Crime experiment. The chain mixes efficiently, and the posterior concentrates on a sparse ensemble of 2–5 trees.

Figure 22: Histogram of the log‐transformed response variable (Riboflavin production) for the Riboflavin dataset. The distribution is unimodal with a peak around

-7

-6.5

, most observations falling in the range

[-8.5,-5.5]

, and a few extreme low‐production strains below

-9

. The negative values are due to the logarithmic transformation.

Method	RMSE	Coverage	Width	CRPS	LPD
DP tree mixture	0.6807	1.000	3.7066	0.4104	$-1.1471$
Random Forest	0.4741	1.000	2.9275	0.2864	$-0.8285$
Gradient Boosting	0.5243	1.000	3.1702	0.3126	$-0.9168$
Bagged CART	0.4551	1.000	3.0848	0.2535	$-0.8356$
BART (dbarts)	0.4434	1.000	1.9488	0.2436	$-0.6533$

Table 6: Predictive performance on the Riboflavin test set (15 observations). The DP mixture achieves near‐nominal coverage and provides the most reliable uncertainty quantification among Bayesian methods, albeit with higher RMSE.

Figure 23: Posterior predictive density heatmap for the DP mixture on the Riboflavin test set. The true test values (red points) fall within high‐density regions, confirming good calibration. The broad bands reflect the model’s honest uncertainty quantification.

Figure 24: BART posterior predictive density heatmap on the Riboflavin test set. The narrow bands of high density indicate overconfidence, despite perfect coverage in this small test set.

Figure 25: Trace plots of the number of distinct trees

k

and the mixture weights for the Riboflavin experiment. The chain mixes efficiently, and the posterior concentrates on a sparse ensemble of 1–4 trees, reflecting the model’s automatic complexity adaptation.

Figure 26: Histogram of the response variable (centered grain yield) for the Wheat genomic prediction dataset. The distribution is roughly symmetric and unimodal, centered near zero (the phenotype has been centered for genomic prediction), with most observations falling between

-2

and

2

and a few extreme values beyond

\pm 2.5

. Negative values indicate below-average yield.

Method	RMSE	Coverage	Width	CRPS	LPD
DP tree mixture	0.9571	0.975	4.1527	0.5375	$-1.3803$
Random Forest	0.8022	0.933	3.2974	0.4491	$-1.2008$
Gradient Boosting	0.7630	0.950	3.3743	0.4310	$-1.1618$
Bagged CART	0.7945	0.875	2.9497	0.4407	$-1.1924$
BART (dbarts)	0.7962	0.817	2.0454	0.4578	$-1.6870$

Table 7: Predictive performance on the Wheat genomic prediction test set (120 observations). The DP mixture achieves near‐nominal coverage and provides the most reliable uncertainty quantification among Bayesian methods, with BART exhibiting severe overconfidence.

Figure 27: Posterior predictive density heatmap for the DP mixture on the Wheat test set. The true test values (red points) fall within high‐density regions, confirming good calibration. The broad bands reflect the model’s honest uncertainty quantification.

Figure 28: BART posterior predictive density heatmap on the Wheat test set. The narrow bands of high density fail to cover many true values, confirming severe overconfidence and undercoverage.

Figure 29: Trace plots of the number of distinct trees $k$ and the mixture weights for the Wheat experiment. The chain mixes efficiently, and the posterior concentrates on a sparse ensemble of 3–7 trees, reflecting the model’s automatic complexity adaptation.

Figure 30: Histogram of the test set response variable (PM2.5) for the E403 air quality dataset (

n_{\text{test}}=250

). The distribution is right‐skewed with a long tail extending above 80

\mu

g/m 3 , reflecting occasional pollution episodes. The mean of the test set is approximately 18.8

\mu

g/m 3 , while the median is around 11.8

\mu

g/m 3 . This heavy‐tailed, heteroscedastic structure is characteristic of real‐world air pollution data and challenges methods that assume symmetric, light‐tailed errors.

Method	RMSE	Coverage	Width	CRPS	LPD
DP tree mixture	16.0651	0.808	35.4639	8.8177	$-4.5367$
Random Forest	3.7220	0.884	11.1685	2.0534	$-2.8192$
Gradient Boosting	5.0885	0.808	11.0013	2.6591	$-3.5946$
Bagged CART	3.5783	0.948	13.3813	2.0220	$-2.6411$
BART (dbarts)	5.6563	0.688	9.4918	3.1158	$-5.0528$

Table 8: Predictive performance on the air quality test set (250 hourly observations). The DP mixture achieves the highest coverage among Bayesian methods, demonstrating its robustness to model misspecification.

Figure 31: Posterior predictive density heatmap for the DP mixture on the air quality test set. The true test values (red points) generally fall within high‐density regions, confirming reasonable calibration. The broad bands reflect the model’s honest uncertainty quantification.

Figure 32: BART posterior predictive density heatmap on the air quality test set. The narrow bands of high density fail to cover many true values, confirming severe overconfidence and undercoverage.

Figure 33: Trace plots of the number of distinct trees $k$ and the mixture weights for the air quality experiment. The chain mixes efficiently, and the posterior concentrates on a sparse ensemble of 3–7 trees, reflecting the model’s automatic complexity adaptation.

Panel of Figure 34 — Figure 34: Illustration of posterior contraction. The solid blue curve is the true regression function $m_{0}$ . The grey curves represent posterior draws; their spread decreases as the sample size $n$ increases. The red dashed bands indicate the region of radius $\varepsilon_{n}$ (Hellinger distance) around the truth. As $n\to\infty$ , $\varepsilon_{n}=n^{-1/4}\to 0$ , and the posterior concentrates around $m_{0}$ .

Dirichlet Process Mixture Models

See all 8 papers

Oct 7, 2026

RoBART: Bayesian Additive Regression Trees with Tree-Specific Rotations

Bayesian additive regression trees (BART) can require many splits to approximate boundaries misaligned with the predictor axes. RoBART assigns each tree a rotation shared by all internal nodes, retaining axis-aligned splits in rotated coordinates and constant leaves. We jointly propose a Givens rotation sequence and cutpoints on the resulting grid by Metropolis-Hastings and establish reversibility with respect to the conditional posterior with leaf means integrated out. For additive functions with component-specific rotations and anisotropic Hölder smoothness, we prove posterior contraction in empirical

L_2

distance and for the noise standard deviation. Under the stated prior, design, and grid conditions, with fixed numbers of predictors, trees, and components and no more components than trees, the rate is a sum of componentwise rates determined by smoothness and the number of rotated coordinates used. We also establish a posterior contraction lower bound showing that there exist functions for which RoBART adapts to the intrinsic dimension but axis-aligned BART does not.

Jeongung Heo, Seonghyun Jeong

Yonsei University

Posterior Contraction Nonparametric Regression

Ask PDF

Jul 12, 2026

DP-Splat: Bayesian Nonparametric Complexity Control for Gaussian Splatting

3D Gaussian Splatting represents scenes as finite mixtures of anisotropic Gaussians whose number of components

K

is set by heuristic density control or user caps. Variational Bayes Gaussian Splatting (VBGS) recast splat fitting as conjugate variational inference, but

K

remains fixed. We replace the finite symmetric Dirichlet over mixture weights with a truncated stick-breaking Dirichlet-process prior -- and, as a theory-backed alternative, a sparse overfitted finite Dirichlet -- so that the number of occupied components adapts to the data while every update remains a closed-form coordinate-ascent step; a natural-gradient stochastic variant makes the per-step cost independent of the number of points. We give an exact monotonicity guarantee, a rigorous truncation-error bound correcting an anti-conservative large-

α

approximation in common use, and an honest account of what the fitted number of components estimates. Empirically: (i) the effective complexity

\hat{K}

adapts to scene complexity and recovers the true

K

within

\pm 1

on well-separated synthetic data with regime-appropriate concentration; (ii) a deconfounded comparison shows the DP prior's contribution is complexity selection, not per-component efficiency -- converged DP fits exceed single-pass fixed-

K

VBGS by +2.7 dB at matched budgets yet tie an equally converged fixed-

K

baseline, and on 3D scenes DP-Splat matches or exceeds VBGS's held-out color prediction with 5.9-7.6x fewer components; (iii) the posterior-predictive color variance is well calibrated on model-matched synthetic data; and (iv) the ordering suggested by exact-posterior asymptotics reverses under mean-field coordinate ascent: the DP prior resists over-splitting while the sparse finite mixture saturates its truncation, a gap between variational practice and posterior asymptotics documented across three orders of magnitude in

N

Aqi Dong

Embry-Riddle Aeronautical University

Bayesian Mixture Models Dirichlet Process Mixture Models Variational Inference 3D Gaussian Splatting Gaussian Mixture Models Bayesian Inference+1

Ask PDF

Sep 18, 2026

Improving the Predictive Performance of Bootstrap Aggregating by Dirichlet Resampling

We revisit Breiman's observation that reducing inter-tree correlation without weakening individual trees can improve random forests. Building on this principle, we introduce two variants: Dirichlet-Multinomial Bagging Random Forest (DM) and Dirichlet-Weighted Random Forest (DW). Both modulate sample reweighting via a concentration parameter

α>0

. We provide a simple theoretical criterion that clarifies when these variants behave indistinguishably from standard random forests, and we use it to guide a lightweight tuning strategy. In a controlled evaluation on public classification benchmarks, DM and DW are consistently competitive and often stronger than other random-forest (RF) baselines, with negligible additional runtime.

Quoc Viet Le, Joonha Park

Department of Statistics, University of Wisconsin–Madison · Department of Mathematics, University of Kansas

Random Forest

Ask PDF

Abstract

Dirichlet Process Mixture Models

Explore similar work

RoBART: Bayesian Additive Regression Trees with Tree-Specific Rotations

DP-Splat: Bayesian Nonparametric Complexity Control for Gaussian Splatting

Improving the Predictive Performance of Bootstrap Aggregating by Dirichlet Resampling