Dirichlet Process Mixtures of Trees with Gaussian Process Splits: A Bayesian Nonparametric Framework with Posterior Contraction Rate

Subhasish Basak; Anik Roy; Sourabh Bhattacharya

Figure 1: Dirichlet process mixture of trees model. The Dirichlet process generates

M

i.i.d. tree-parameter pairs, which are combined with random mixture weights to form a sparse ensemble with a data-driven number of components. The inset illustrates a single tree component; the blue curve represents a draw

\tilde{g}

from the GP posterior predictive used as the splitting rule at an internal node.

Method	Special case of the DP mixture
CART	$M=1$ , or all DP draws coincident (single component)
BART	Base measure $G_{0}$ supported on additive tree ensembles; appropriate mixture weights
Random Forest	$\alpha\to\infty$ , $p_{i}\approx 1/M$ , components i.i.d. from $G_{0}$
Boosting	Scaled convex combination / additive base measure / large $\alpha$ limit

Table 1: Special cases of the Dirichlet process tree mixture.

A=\min\!\left\{1,\;\frac{\Pi_{\text{struct}}(\bm{T}^{\prime})}{\Pi_{\text{struct}}(\bm{T})}\frac{f(\bm{Y}|\bm{X},\bm{T}^{\prime})}{f(\bm{Y}|\bm{X},\bm{T})}\frac{b}{b^{*}+1}\right\}

Algorithm 2 MCMC tree step (GROW/PRUNE)

Method	RMSE	Coverage	Width	CRPS	LPD
d=5 (clean)
DP tree mixture	5.248	0.940	18.249	2.956	$-3.053$
Random Forest	2.958	0.960	10.908	1.674	$-2.508$
Gradient Boosting	2.561	0.920	9.949	1.428	$-2.360$
Bagged CART	3.144	0.800	8.938	1.912	$-2.703$
BART	2.085	0.820	5.873	1.199	$-2.199$

Table 2: Predictive performance comparison for the Gaussian noise experiments. The DP mixture achieves the highest coverage among Bayesian methods.

Method	RMSE	Coverage	Width	CRPS	LPD
d=5 (clean)
DP tree mixture	111.735	0.920	22.614	19.519	$-5365.445$
Random Forest	111.026	0.920	18.312	18.967	$-284.899$
Gradient Boosting	111.043	0.900	16.845	18.766	$-336.236$
Bagged CART	111.230	0.720	12.392	19.221	$-10642.342$
BART	110.856	0.780	10.169	18.691	$-12033.924$

Table 3: Predictive performance comparison for the Cauchy noise experiments. The DP mixture clearly dominates in coverage and LPD among Bayesian methods.

Figure 2: Posterior predictive density heatmap for the DP mixture in the clean d=5 Gaussian experiment. The red points are the true test values.

Figure 3: Trace plots of the number of distinct trees

k

and the largest mixture weights for the clean d=5 Gaussian experiment.

Figure 4: BART posterior predictive density heatmap for the clean d=5 Gaussian experiment.

Figure 5: Posterior predictive density heatmap for the DP mixture in the high-dimensional d=20 Gaussian experiment.

Figure 6: Trace plots of the number of distinct trees

k

and the largest mixture weights for the high-dimensional d=20 Gaussian experiment.

Figure 7: BART posterior predictive density heatmap for the high-dimensional d=20 Gaussian experiment.

Figure 8: Posterior predictive density heatmap for the DP mixture in the clean d=5 Cauchy experiment.

Figure 9: Trace plots of the number of distinct trees

k

and the largest mixture weights for the clean d=5 Cauchy experiment.

Figure 10: BART posterior predictive density heatmap for the clean d=5 Cauchy experiment.

Figure 11: Posterior predictive density heatmap for the DP mixture in the high-dimensional d=20 Cauchy experiment.

Figure 12: Trace plots of the number of distinct trees

k

and the largest mixture weights for the high-dimensional d=20 Cauchy experiment.

Figure 13: BART posterior predictive density heatmap for the high-dimensional d=20 Cauchy experiment.

Method	RMSE	Coverage	Width	CRPS	LPD
DP tree mixture	1.8212	0.918	6.4167	1.0408	$-2.0363$
Random Forest	1.3154	0.918	4.4239	0.6880	$-1.7191$
Gradient Boosting	1.2818	0.927	4.4919	0.6813	$-1.6808$
Bagged CART	1.3274	0.827	3.2097	0.7168	$-2.1051$
BART (dbarts)	1.2765	0.582	1.5881	0.7578	$-24.0618$

Table 4: Predictive performance on the QSAR aquatic toxicity test set. The DP mixture achieves the highest coverage among Bayesian methods and offers the most reliable uncertainty quantification, albeit at a significantly higher computational cost.

Figure 14: Posterior predictive density heatmap for the DP mixture on the QSAR test set. The true test values (red points) fall within the high-density regions, confirming good calibration.

Figure 15: BART posterior predictive density heatmap on the QSAR test set. The narrow bands of high density fail to cover many true values, confirming poor coverage.

Figure 16: Trace plots of the number of distinct trees

k

and the mixture weights for the QSAR experiment. The chain mixes efficiently, and the posterior concentrates on a sparse ensemble of 2–5 trees.

Figure 17: Histogram of the responses for the QSAR experiment exhibiting multimodality and outlyingness.

Figure 18: Histogram of the response variable (ViolentCrimesPerPop) for the Communities and Crime dataset. The distribution is highly skewed and heavy-tailed, with most communities having low crime rates and a small number of high-crime communities in the tail. This skewness explains the failure of BART and the success of the DP mixture.

Method	RMSE	Coverage	Width	CRPS	LPD
DP tree mixture	0.1969	0.963	0.9853	0.1020	0.3337
Random Forest	0.0952	0.975	0.4683	0.0491	0.8880
Gradient Boosting	0.0983	0.981	0.4442	0.0497	0.8824
Bagged CART	0.0973	0.957	0.3117	0.0420	1.3518
BART (dbarts)	0.0970	0.870	0.2133	0.0479	-3.4684

Table 5: Predictive performance on the Communities and Crime test set (162 communities).

Figure 19: Posterior predictive density heatmap for the DP mixture on the Communities and Crime test set. True values (red points) fall within high-density regions, confirming good calibration.

Figure 20: BART posterior predictive density heatmap on the Communities and Crime test set. Narrow bands of high density fail to cover many true values, confirming poor coverage.

Figure 21: Trace plots of

k

and the mixture weights for the Communities and Crime experiment. The chain mixes efficiently, and the posterior concentrates on a sparse ensemble of 2–5 trees.

Figure 22: Histogram of the log‐transformed response variable (Riboflavin production) for the Riboflavin dataset. The distribution is unimodal with a peak around

-7

to

-6.5

, most observations falling in the range

[-8.5,-5.5]

, and a few extreme low‐production strains below

-9

. The negative values are due to the logarithmic transformation.

Method	RMSE	Coverage	Width	CRPS	LPD
DP tree mixture	0.6807	1.000	3.7066	0.4104	$-1.1471$
Random Forest	0.4741	1.000	2.9275	0.2864	$-0.8285$
Gradient Boosting	0.5243	1.000	3.1702	0.3126	$-0.9168$
Bagged CART	0.4551	1.000	3.0848	0.2535	$-0.8356$
BART (dbarts)	0.4434	1.000	1.9488	0.2436	$-0.6533$

Table 6: Predictive performance on the Riboflavin test set (15 observations). The DP mixture achieves near‐nominal coverage and provides the most reliable uncertainty quantification among Bayesian methods, albeit with higher RMSE.

Figure 23: Posterior predictive density heatmap for the DP mixture on the Riboflavin test set. The true test values (red points) fall within high‐density regions, confirming good calibration. The broad bands reflect the model’s honest uncertainty quantification.

Figure 24: BART posterior predictive density heatmap on the Riboflavin test set. The narrow bands of high density indicate overconfidence, despite perfect coverage in this small test set.

Figure 25: Trace plots of the number of distinct trees

k

and the mixture weights for the Riboflavin experiment. The chain mixes efficiently, and the posterior concentrates on a sparse ensemble of 1–4 trees, reflecting the model’s automatic complexity adaptation.

Figure 26: Histogram of the response variable (centered grain yield) for the Wheat genomic prediction dataset. The distribution is roughly symmetric and unimodal, centered near zero (the phenotype has been centered for genomic prediction), with most observations falling between

-2

and

2

and a few extreme values beyond

\pm 2.5

. Negative values indicate below-average yield.

Method	RMSE	Coverage	Width	CRPS	LPD
DP tree mixture	0.9571	0.975	4.1527	0.5375	$-1.3803$
Random Forest	0.8022	0.933	3.2974	0.4491	$-1.2008$
Gradient Boosting	0.7630	0.950	3.3743	0.4310	$-1.1618$
Bagged CART	0.7945	0.875	2.9497	0.4407	$-1.1924$
BART (dbarts)	0.7962	0.817	2.0454	0.4578	$-1.6870$

Table 7: Predictive performance on the Wheat genomic prediction test set (120 observations). The DP mixture achieves near‐nominal coverage and provides the most reliable uncertainty quantification among Bayesian methods, with BART exhibiting severe overconfidence.

Figure 27: Posterior predictive density heatmap for the DP mixture on the Wheat test set. The true test values (red points) fall within high‐density regions, confirming good calibration. The broad bands reflect the model’s honest uncertainty quantification.

Figure 28: BART posterior predictive density heatmap on the Wheat test set. The narrow bands of high density fail to cover many true values, confirming severe overconfidence and undercoverage.

Figure 29: Trace plots of the number of distinct trees

k

and the mixture weights for the Wheat experiment. The chain mixes efficiently, and the posterior concentrates on a sparse ensemble of 3–7 trees, reflecting the model’s automatic complexity adaptation.

Figure 30: Histogram of the test set response variable (PM2.5) for the E403 air quality dataset (

n_{\text{test}}=250

). The distribution is right‐skewed with a long tail extending above 80

\mu

g/m 3 , reflecting occasional pollution episodes. The mean of the test set is approximately 18.8

\mu

g/m 3 , while the median is around 11.8

\mu

g/m 3 . This heavy‐tailed, heteroscedastic structure is characteristic of real‐world air pollution data and challenges methods that assume symmetric, light‐tailed errors.

Method	RMSE	Coverage	Width	CRPS	LPD
DP tree mixture	16.0651	0.808	35.4639	8.8177	$-4.5367$
Random Forest	3.7220	0.884	11.1685	2.0534	$-2.8192$
Gradient Boosting	5.0885	0.808	11.0013	2.6591	$-3.5946$
Bagged CART	3.5783	0.948	13.3813	2.0220	$-2.6411$
BART (dbarts)	5.6563	0.688	9.4918	3.1158	$-5.0528$

Table 8: Predictive performance on the air quality test set (250 hourly observations). The DP mixture achieves the highest coverage among Bayesian methods, demonstrating its robustness to model misspecification.

Figure 31: Posterior predictive density heatmap for the DP mixture on the air quality test set. The true test values (red points) generally fall within high‐density regions, confirming reasonable calibration. The broad bands reflect the model’s honest uncertainty quantification.

Figure 32: BART posterior predictive density heatmap on the air quality test set. The narrow bands of high density fail to cover many true values, confirming severe overconfidence and undercoverage.

Figure 33: Trace plots of the number of distinct trees

k

and the mixture weights for the air quality experiment. The chain mixes efficiently, and the posterior concentrates on a sparse ensemble of 3–7 trees, reflecting the model’s automatic complexity adaptation.

Figure 34: Illustration of posterior contraction. The solid blue curve is the true regression function

m_{0}

. The grey curves represent posterior draws; their spread decreases as the sample size

n

increases. The red dashed bands indicate the region of radius

\varepsilon_{n}

(Hellinger distance) around the truth. As

n\to\infty

,

\varepsilon_{n}=n^{-1/4}\to 0

, and the posterior concentrates around

m_{0}

.

Dirichlet Process Mixtures of Trees with Gaussian Process Splits: A Bayesian Nonparametric Framework with Posterior Contraction Rate

Abstract

Figures & tables

Dirichlet Process Mixture Models

RoBART: Bayesian Additive Regression Trees with Tree-Specific Rotations

DP-Splat: Bayesian Nonparametric Complexity Control for Gaussian Splatting

Improving the Predictive Performance of Bootstrap Aggregating by Dirichlet Resampling

Abstract

Dirichlet Process Mixture Models

Explore similar work

RoBART: Bayesian Additive Regression Trees with Tree-Specific Rotations

DP-Splat: Bayesian Nonparametric Complexity Control for Gaussian Splatting

Improving the Predictive Performance of Bootstrap Aggregating by Dirichlet Resampling