Improving the Predictive Performance of Bootstrap Aggregating by Dirichlet Resampling

Quoc Viet Le; Joonha Park

stat.MLSep 18, 2026

Improving the Predictive Performance of Bootstrap Aggregating by Dirichlet Resampling

Authors: Quoc Viet Le, Joonha Park

Organizations: Department of Statistics, University of Wisconsin–Madison · Department of Mathematics, University of Kansas

Abstract

We revisit Breiman's observation that reducing inter-tree correlation without weakening individual trees can improve random forests. Building on this principle, we introduce two variants: Dirichlet-Multinomial Bagging Random Forest (DM) and Dirichlet-Weighted Random Forest (DW). Both modulate sample reweighting via a concentration parameter $α>0$ . We provide a simple theoretical criterion that clarifies when these variants behave indistinguishably from standard random forests, and we use it to guide a lightweight tuning strategy. In a controlled evaluation on public classification benchmarks, DM and DW are consistently competitive and often stronger than other random-forest (RF) baselines, with negligible additional runtime.

Figures & tables

Dataset	ET	RF-bootstrap	RF-no-bootstrap	Subsample	DM	DW
adult	0.8293 $\pm$ 0.0009	0.8491 $\pm$ 0.0011	0.8431 $\pm$ 0.0010	0.8499 $\pm$ 0.0011	0.8561 $\pm$ 0.0011 4,4	0.8567 $\pm$ 0.0011 4,4
banknote	0.9993 $\pm$ 0.0005	0.9938 $\pm$ 0.0012	0.9945 $\pm$ 0.0011	0.9942 $\pm$ 0.0006	0.9945 $\pm$ 0.0006 0,3	0.9971 $\pm$ 0.0009 3,3
breast_cancer	0.9649 $\pm$ 0.0035	0.9649 $\pm$ 0.0039	0.9667 $\pm$ 0.0025	0.9684 $\pm$ 0.0035	0.9684 $\pm$ 0.0035 0,4	0.9702 $\pm$ 0.0033 0,4
credit_g	0.7510 $\pm$ 0.0084	0.7650 $\pm$ 0.0098	0.7540 $\pm$ 0.0078	0.7615 $\pm$ 0.0080	0.7710 $\pm$ 0.0077 1,4	0.7615 $\pm$ 0.0093 0,4
digits	0.9839 $\pm$ 0.0015	0.9769 $\pm$ 0.0013	0.9781 $\pm$ 0.0017	0.9767 $\pm$ 0.0022	0.9781 $\pm$ 0.0017 0,3	0.9817 $\pm$ 0.0012 3,4
ecoli	0.8809 $\pm$ 0.0071	0.8941 $\pm$ 0.0098	0.8824 $\pm$ 0.0076	0.8985 $\pm$ 0.0080	0.9000 $\pm$ 0.0092 0,4	0.8971 $\pm$ 0.0076 0,4

Table 1: Accuracy (Mean

\pm

SE) of DM/DW vs four baselines (ET, RF-bootstrap, RF-no-bootstrap, Subsample). Superscripts for DM/DW indicate: (1) baselines significantly outperformed (10% significance level), and (2) baselines for which DM/DW is not significantly worse.

Dataset	ET	RF-bootstrap	RF-no-bootstrap	Subsample	DM	DW
adult	0.5109 $\pm$ 0.0078	0.3573 $\pm$ 0.0034	0.4893 $\pm$ 0.0069	0.3541 $\pm$ 0.0039	0.3332 $\pm$ 0.0044 4,4	0.3321 $\pm$ 0.0043 4,4
banknote	0.0106 $\pm$ 0.0028	0.0243 $\pm$ 0.0064	0.0231 $\pm$ 0.0066	0.0205 $\pm$ 0.0048	0.0199 $\pm$ 0.0047 0,3	0.0149 $\pm$ 0.0049 0,4
breast_cancer	0.1195 $\pm$ 0.0182	0.1309 $\pm$ 0.0197	0.1168 $\pm$ 0.0155	0.1186 $\pm$ 0.0170	0.1203 $\pm$ 0.0165 0,4	0.1183 $\pm$ 0.0166 0,4
credit_g	0.5874 $\pm$ 0.0410	0.5664 $\pm$ 0.0395	0.5853 $\pm$ 0.0380	0.5738 $\pm$ 0.0391	0.5601 $\pm$ 0.0387 0,4	0.5724 $\pm$ 0.0402 0,4
digits	0.0792 $\pm$ 0.0045	0.0978 $\pm$ 0.0054	0.0906 $\pm$ 0.0046	0.0987 $\pm$ 0.0066	0.0904 $\pm$ 0.0051 0,3	0.0857 $\pm$ 0.0048 1,4
ecoli	0.3245 $\pm$ 0.0269	0.3222 $\pm$ 0.0319	0.3307 $\pm$ 0.0262	0.3161 $\pm$ 0.0288	0.3100 $\pm$ 0.0301 0,4	0.3135 $\pm$ 0.0261 0,4

Table 2: Log loss (Mean

\pm

SE) of DM/DW vs four baselines

Dataset	ET	RF-bootstrap	RF-no-bootstrap	Subsample	DM	DW
adult	0.8779 $\pm$ 0.0010	0.9018 $\pm$ 0.0010	0.8870 $\pm$ 0.0011	0.9023 $\pm$ 0.0011	0.9053 $\pm$ 0.0011 4,4	0.9055 $\pm$ 0.0011 4,4
banknote	0.9992 $\pm$ 0.0004	0.9965 $\pm$ 0.0013	0.9970 $\pm$ 0.0012	0.9972 $\pm$ 0.0006	0.9972 $\pm$ 0.0006 0,3	0.9985 $\pm$ 0.0008 0,4
breast_cancer	0.9936 $\pm$ 0.0027	0.9928 $\pm$ 0.0031	0.9938 $\pm$ 0.0022	0.9943 $\pm$ 0.0028	0.9942 $\pm$ 0.0028 0,4	0.9944 $\pm$ 0.0026 0,4
credit_g	0.8446 $\pm$ 0.0145	0.8537 $\pm$ 0.0142	0.8450 $\pm$ 0.0129	0.8498 $\pm$ 0.0137	0.8577 $\pm$ 0.0138 0,4	0.8495 $\pm$ 0.0145 0,4
digits	0.9984 $\pm$ 0.0003	0.9974 $\pm$ 0.0005	0.9977 $\pm$ 0.0004	0.9973 $\pm$ 0.0007	0.9977 $\pm$ 0.0005 0,4	0.9980 $\pm$ 0.0004 0,4
glass	0.8070 $\pm$ 0.0409	0.7848 $\pm$ 0.0476	0.7706 $\pm$ 0.0598	0.7955 $\pm$ 0.0477	0.7924 $\pm$ 0.0516 0,4	0.7961 $\pm$ 0.0515 0,4

Table 3: AUROC (Mean

\pm

SE) of DM/DW vs four baselines.

Dataset	Method	Best $\alpha$	Comparable $\alpha$ Values	Count	$n$
adult	DM	0.1573	0.0133, 0.1744	3	48842
adult	DW	0.0133	0.0066, 0.0091	3	48842
banknote	DM	4.8365	0.8498, 1.1743, 1.5397, 2.3157	5	1372
banknote	DW	0.0496	0.2862, 0.8498, 4.8365	4	1372
breast_cancer	DM	5.7722	0.0570, 0.1428, 0.2972, 2.2322, 6.4756, 10.1280	7	569
breast_cancer	DW	5.7722	0.0256, 0.0570, 0.1428, 0.2972, 2.2322, 6.4756, 10.1280	8	569

Table 4: The best

\alpha

and the

\alpha

values yielding comparable accuracy for DM/DW on each dataset. The training sample size

n

is also shown.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	$n$	$C$	Imbalance	$\alpha_{\mathrm{RF}}(n,\varepsilon{=}0.01)$
Adult	48,842	2	high	0.205
Banknote	1,372	2	low	7.283
Breast Cancer (WDBC)	569	2	mid	17.542
Credit-G (German)	1,000	2	high	9.989
Digits (sklearn)	1,797	10	low	5.561
Ecoli	336	8	high	29.670

Appendix

Table 5: Dataset Statistics

method	alpha	accuracy	logloss	auroc	n	d	C
ET		0.8293±0.0009	0.5109±0.0078	0.8779±0.0010	45222	105	2
RF-bootstrap		0.8491±0.0011	0.3573±0.0034	0.9018±0.0010	45222	105	2
RF-no-bootstrap		0.8431±0.0010	0.4893±0.0077	0.8870±0.0010	45222	105	2
Subsample		0.8499±0.0011	0.3616±0.0047	0.9015±0.0011	45222	105	2
Bayesian RF	1.0	0.8436±0.0009	0.4576±0.0093	0.8894±0.0010	45222	105	2
DM	0	0.7808±0.0032	0.4286±0.0020	0.8708±0.0026	45222	105	2

Appendix

Table 6: Results for Adult

method	alpha	accuracy	logloss	auroc	n	d	C
ET		0.9993±0.0005	0.0158±0.0011	1.0000±0.0000	1372	4	2
RF-bootstrap		0.9938±0.0012	0.0309±0.0022	0.9998±0.0001	1372	4	2
RF-no-bootstrap		0.9945±0.0011	0.0240±0.0020	0.9998±0.0001	1372	4	2
Subsample		0.9942±0.0006	0.0304±0.0022	0.9997±0.0001	1372	4	2
Bayesian RF	1.0	0.9945±0.0011	0.0229±0.0019	0.9999±0.0001	1372	4	2
DM	0.0007	0.8898±0.0069	0.4070±0.0065	0.9691±0.0027	1372	4	2

Appendix

Table 7: Results for Banknote

method	alpha	accuracy	logloss	auroc	n	d	C
ET		0.9649±0.0035	0.0982±0.0061	0.9941±0.0016	569	30	2
RF-bootstrap		0.9649±0.0039	0.2130±0.0507	0.9893±0.0033	569	30	2
RF-no-bootstrap		0.9667±0.0025	0.1541±0.0431	0.9913±0.0027	569	30	2
Subsample		0.9684±0.0035	0.1304±0.0305	0.9922±0.0024	569	30	2
Bayesian RF	1.0	0.9719±0.0037	0.1278±0.0345	0.9921±0.0028	569	30	2
DM	0.0012	0.9298±0.0070	0.3449±0.0057	0.9892±0.0024	569	30	2

Appendix

Table 8: Results for Breast Cancer

method	alpha	accuracy	logloss	auroc	n	d	C
ET		0.7510±0.0084	0.5024±0.0106	0.7741±0.0128	1000	61	2
RF-bootstrap		0.7650±0.0098	0.4941±0.0099	0.7868±0.0147	1000	61	2
RF-no-bootstrap		0.7540±0.0078	0.4953±0.0104	0.7820±0.0136	1000	61	2
Subsample		0.7615±0.0080	0.4908±0.0092	0.7910±0.0137	1000	61	2
Bayesian RF	1.0	0.7665±0.0080	0.4921±0.0093	0.7902±0.0141	1000	61	2
DM	0.0005	0.7000±0.0000	0.5931±0.0031	0.6860±0.0206	1000	61	2

Appendix

Table 9: Results for Credit German

method	alpha	accuracy	logloss	auroc	n	d	C
ET		0.9839±0.0015	0.2840±0.0025	0.9996±0.0001	1797	64	10
RF-bootstrap		0.9769±0.0013	0.3098±0.0031	0.9994±0.0001	1797	64	10
RF-no-bootstrap		0.9781±0.0017	0.2588±0.0033	0.9995±0.0001	1797	64	10
Subsample		0.9767±0.0022	0.3039±0.0025	0.9995±0.0001	1797	64	10
Bayesian RF	1.0	0.9797±0.0021	0.2699±0.0029	0.9996±0.0001	1797	64	10
DM	0	0.1692±0.0183	2.2434±0.0044	0.7839±0.0081	1797	64	10

Appendix

Table 10: Results for Digits

method	alpha	accuracy	logloss	n	d	C
ET		0.8809±0.0071	0.3900±0.0168	336	7	8
RF-bootstrap		0.8941±0.0098	0.4288±0.0550	336	7	8
RF-no-bootstrap		0.8824±0.0076	0.5224±0.0866	336	7	8
Bayesian RF	1.0	0.8897±0.0077	0.4656±0.0646	336	7	8
Subsample		0.8985±0.0080	0.5175±0.0839	336	7	8
DM	0.0236	0.8544±0.0083	0.5448±0.0083	336	7	8

Appendix

Table 11: Results for Ecoli

method	alpha	accuracy	logloss	auroc	n	d	C
ET		0.7907±0.0125	0.6105±0.0215	0.9591±0.0044	214	9	6
RF-bootstrap		0.7767±0.0131	0.6565±0.0285	0.9492±0.0050	214	9	6
RF-no-bootstrap		0.7698±0.0203	0.7119±0.0993	0.9486±0.0057	214	9	6
Subsample		0.7837±0.0155	0.6526±0.0299	0.9475±0.0054	214	9	6
Bayesian RF	1.0	0.7907±0.0177	0.6154±0.0320	0.9525±0.0055	214	9	6
DM	0.0012	0.4186±0.0222	1.3903±0.0066	0.8357±0.0116	214	9	6

Appendix

Table 12: Results for Glass

method	alpha	accuracy	logloss	auroc	n	d	C
ET		0.9507±0.0057	0.1632±0.0072	0.9917±0.0019	351	34	2
RF-bootstrap		0.9423±0.0065	0.1830±0.0110	0.9828±0.0036	351	34	2
RF-no-bootstrap		0.9380±0.0079	0.1639±0.0125	0.9847±0.0034	351	34	2
Subsample		0.9451±0.0074	0.1773±0.0103	0.9840±0.0033	351	34	2
Bayesian RF	1.0	0.9451±0.0074	0.1723±0.0109	0.9849±0.0034	351	34	2
DM	0.0389	0.9380±0.0082	0.2895±0.0075	0.9735±0.0048	351	34	2

Appendix

Table 13: Results for Ionosphere

method	alpha	accuracy	logloss	auroc	n	d	C
ET		0.9600±0.0097	0.0928±0.0123	0.9966±0.0017	150	4	3
RF-bootstrap		0.9500±0.0124	0.1160±0.0233	0.9950±0.0027	150	4	3
RF-no-bootstrap		0.9433±0.0071	0.2316±0.1249	0.9924±0.0038	150	4	3
Subsample		0.9500±0.0124	0.1172±0.0229	0.9949±0.0025	150	4	3
Bayesian RF	1.0	0.9533±0.0102	0.1157±0.0245	0.9951±0.0021	150	4	3
DM	0.0481	0.9633±0.0126	0.1509±0.0128	0.9968±0.0015	150	4	3

Appendix

Table 14: Results for Iris

method	alpha	accuracy	logloss	auroc	n	d	C
ET		0.9723±0.0008	0.2714±0.0014	0.9997±0.0000	20000	16	26
RF-bootstrap		0.9644±0.0009	0.2761±0.0021	0.9996±0.0000	20000	16	26
RF-no-bootstrap		0.9679±0.0009	0.2272±0.0016	0.9996±0.0000	20000	16	26
Subsample		0.9644±0.0009	0.2733±0.0021	0.9995±0.0000	20000	16	26
Bayesian RF	1.0	0.9673±0.0008	0.2358±0.0016	0.9996±0.0000	20000	16	26
DM	0.0013	0.7991±0.0022	1.4427±0.0036	0.9814±0.0003	20000	16	26

Appendix

Table 15: Results for Letter

method	alpha	accuracy	logloss	auroc	n	d	C
ET		0.8807±0.0020	0.2991±0.0026	0.9398±0.0013	19020	10	2
RF-bootstrap		0.8835±0.0016	0.2914±0.0020	0.9402±0.0012	19020	10	2
RF-no-bootstrap		0.8831±0.0015	0.2932±0.0033	0.9393±0.0012	19020	10	2
Subsample		0.8835±0.0018	0.2935±0.0024	0.9404±0.0012	19020	10	2
Bayesian RF	1.0	0.8836±0.0014	0.2914±0.0023	0.9406±0.0011	19020	10	2
DM	0	0.6806±0.0039	0.5723±0.0022	0.8238±0.0064	19020	10	2

Appendix

Table 16: Results for Magic04

method	alpha	accuracy	logloss	auroc	n	d	C
ET		0.9861±0.0010	0.2065±0.0011	0.9997±0.0000	5620	64	10
RF-bootstrap		0.9844±0.0010	0.2159±0.0018	0.9997±0.0000	5620	64	10
RF-no-bootstrap		0.9861±0.0010	0.1822±0.0015	0.9997±0.0000	5620	64	10
Subsample		0.9846±0.0011	0.2129±0.0015	0.9996±0.0000	5620	64	10
Bayesian RF	1.0	0.9843±0.0013	0.1902±0.0014	0.9997±0.0000	5620	64	10
DM	0	0.4194±0.0243	2.0544±0.0049	0.9030±0.0028	5620	64	10

Appendix

Table 17: Results for Optdigits

method	alpha	accuracy	logloss	auroc	n	d	C
ET		0.9709±0.0012	0.2461±0.0134	0.9777±0.0014	5473	10	5
RF-bootstrap		0.9728±0.0012	0.1610±0.0140	0.9881±0.0010	5473	10	5
RF-no-bootstrap		0.9692±0.0014	0.2775±0.0183	0.9757±0.0021	5473	10	5
Subsample		0.9722±0.0011	0.1648±0.0151	0.9888±0.0012	5473	10	5
Bayesian RF	1.0	0.9701±0.0015	0.2616±0.0152	0.9773±0.0016	5473	10	5
DM	0	0.8977±0.0000	0.4578±0.0228	0.8478±0.0149	5473	10	5

Appendix

Table 18: Results for Page Blocks

method	alpha	accuracy	logloss	auroc	n	d	C
ET		0.9935±0.0003	0.0796±0.0039	0.9998±0.0001	10992	16	10
RF-bootstrap		0.9918±0.0005	0.0874±0.0044	0.9997±0.0001	10992	16	10
RF-no-bootstrap		0.9928±0.0005	0.0730±0.0046	0.9997±0.0001	10992	16	10
Subsample		0.9918±0.0004	0.0885±0.0049	0.9997±0.0001	10992	16	10
Bayesian RF	1.0	0.9933±0.0004	0.0775±0.0049	0.9997±0.0001	10992	16	10
DM	0.0001	0.7895±0.0097	1.3476±0.0078	0.9758±0.0009	10992	16	10

Appendix

Table 19: Results for Pendigits

method	alpha	accuracy	logloss	auroc	n	d	C
ET		0.9182±0.0022	0.2190±0.0075	0.9702±0.0016	5404	5	2
RF-bootstrap		0.9127±0.0022	0.2588±0.0128	0.9634±0.0018	5404	5	2
RF-no-bootstrap		0.9131±0.0021	0.2452±0.0118	0.9668±0.0016	5404	5	2
Subsample		0.9119±0.0026	0.2531±0.0104	0.9631±0.0019	5404	5	2
Bayesian RF	1.0	0.9169±0.0022	0.2403±0.0116	0.9676±0.0016	5404	5	2
DM	0.0002	0.7340±0.0030	0.4718±0.0044	0.8364±0.0041	5404	5	2

Appendix

Table 20: Results for Phoneme

Random Forest

Random forests are ensembles of randomized decision trees used for prediction, feature analysis, and causal or structured-data modeling.

See all 31 papers

Decision-Path Patterns as Tree Reliability Signals: Path-based Adaptive Weighting for Random Forest Classification
May 20, 2026Youngjoon ParkEnsemble Learning Binary Classification
Independent Researcher
Distributional Split Criteria for Random Forests: Extensions, Shrinkage, and the Robustness of Mean Splitting
Jul 26, 2026Silas KoemenMaximum Mean Discrepancy Random Forest
Minimax Rates and Spectral Distillation for Tree Ensembles
May 12, 2026Binh Duc Vu, David S. WatsonModel Compression Gradient-Boosted Decision Trees
King’s College London

May 20, 2026J/K move · Enter open · S save

Decision-Path Patterns as Tree Reliability Signals: Path-based Adaptive Weighting for Random Forest Classification

Youngjoon Park

Independent Researcher

Ensemble Learning Binary Classification Random Forest

Open paper AskPDF arXiv

Random forests construct each tree with a different, randomised representation of the feature space. Their uniform voting cannot correct errors in regions where trees with incorrect representations probabilistically outnumber correct ones, even when the ensemble collectively holds enough correct information - a reducible error that this paper addresses. We propose using the structural pattern of each tree's decision path as an instance-adaptive reliability signal to identify and differentially weight the more reliable trees. At inference, a random forest reaches its prediction through the root-to-leaf path the sample traverses in each tree, so path-level reliability offers a finer granularity than tree-level weighting can access. We show that this signal reflects the actual reliability of each tree's decision, and that using it yields a statistically significant accuracy improvement over RF on 36 binary classification benchmarks (Wilcoxon p < 0.0001). Class-recall regression - the typical failure mode of RF correction methods - is measured: zero minority-recall regressions and a single majority-recall regression at the 0.2 pp threshold, indicating bias reduction rather than a class trade-off. We further quantify the reducible error accessible to the method from the fitted RF alone; this estimate correlates strongly with per-dataset gain (Pearson r = +0.840, p < 0.0001). On the qualifying group it identifies, the method delivers a mean +0.99 pp accuracy improvement with strict wins on every dataset (7/0/0); an optional amplification mechanism further raises this to +1.48 pp.

Abstract

Random Forest

Explore similar work

Decision-Path Patterns as Tree Reliability Signals: Path-based Adaptive Weighting for Random Forest Classification

Distributional Split Criteria for Random Forests: Extensions, Shrinkage, and the Robustness of Mean Splitting

Minimax Rates and Spectral Distillation for Tree Ensembles