Just add noise: Debiasing tree-based variable importance in mixed data

Jiahe Li; Omar Melikechi

stat.MLSep 12, 2026

Just add noise: Debiasing tree-based variable importance in mixed data

Authors: Jiahe Li, Omar Melikechi

Organizations: Department of Statistical Science, Duke University

Abstract

Variable importance scores from tree-based methods such as random forests favor continuous predictors over categorical ones. We present a theoretical analysis of this bias and propose a simple remedy: add a small amount of noise to each categorical predictor. The correction is demonstrated on a variety of simulated and real-world datasets and combined with integrated path stability selection to perform variable selection with false discovery control for mixed data.

Figures & tables

Figure 1. Adding noise yields equal numbers of candidate splits . Five samples are drawn from the independent pair $(X,Z)$ , where $X\sim\mathcal{N}(0,1)$ and $Z\sim\text{Bernoulli}(1/2)$ . ( Left ) The 5 samples of $X$ admit 4 splits (vertical lines), while the $Z$ samples only admit 1 (horizontal line). ( Right ) The $Z$ samples admit 4 splits after adding noise.

Figure 2. Mixed data bias. MDI scores from CART, plotted as a function of $n$ for the model $Y=X+Z+\varepsilon$ , where $X\sim N(0,1)$ , $Z=\pm 1$ with equal probability, and $\varepsilon\sim N(0,1)$ are mutually independent. Although $\operatorname{Var}(X)=\operatorname{Var}(Z)=1$ , normalized MDIs for $X$ and $Z$ converge to $2/3$ and $1/3$ , respectively, in accordance with Theorem 2.4 . After adding noise to $Z$ , both scores converge to $1/2$ . Scores are averaged over 200 trials; error bars show 95% Monte Carlo confidence intervals for the mean.

	Independent covariates		Correlated covariates
Method	Linear	Nonlinear	Linear	Nonlinear
RF	0.80 (0.05)	0.80 (0.05)	0.71 (0.05)	0.72 (0.05)
RF-one-time	0.94 (0.05)	0.89 (0.07)	0.86 (0.07)	0.86 (0.06)
RF-per-tree	0.96 (0.04)	0.92 (0.05)	0.88 (0.06)	0.90 (0.05)
UFI	0.92 (0.05)	0.87 (0.07)	0.83 (0.06)	0.83 (0.07)
CForest	0.92 (0.05)	0.82 (0.07)	0.84 (0.08)	0.82 (0.08)

Table 1. Feature-ranking performance . Mean AUC over

100

trials, with the standard deviation in parentheses; the best AUC in each setting is in bold (higher is better).

Figure 3. IPSS with different MDI scores . TPR (top row) and FDR (bottom row) against the target FDR level for IPSS with MDI scores from random forests (RF), one-time jittered random forests (RF-one-time), UFI, XGBoost (XGB), and one-time jittered XGBoost (XGB-one-time). Columns correspond to the four simulation designs. The dashed diagonal in the FDR plots marks exact FDR calibration; methods lying at or below this line successfully control FDR.

Figure 4. Rank of pathologic stage over 50 random seeds . Rank of pathologic stage among the $425$ bladder cancer predictors of 18-month survival over $50$ random seeds.

Dataset	$n$	$p$	$q$	RF	RF-one-time	XGB	XGB-one-time
QSAR Biodegradation	1055	19	22	14.7 (14.7)	0.0 (0.0)	0.6 (0.5)	0.2 (0.1)
Cylinder Banding	540	13	22	10.7 (9.5)	0.0 (0.0)	0.0 (0.0)	0.0 (0.0)
Hypothyroid	3163	7	18	9.8 (7.0)	0.2 (0.0)	2.7 (2.7)	0.3 (0.0)
German Credit	1000	3	17	6.7 (3.0)	0.0 (0.0)	0.5 (0.4)	0.0 (0.0)
Hepatitis	155	6	13	6.0 (6.0)	0.0 (0.0)	0.1 (0.1)	0.0 (0.0)
SAheart	462	8	1	0.0 (0.0)	0.0 (0.0)	0.0 (0.0)	0.0 (0.0)

Table 2. Number of synthetic null predictors selected . For each dataset (

n

samples,

p

continuous and

q

categorical predictors), the mean number of synthetic null predictors selected by IPSS with each base selector at target FDR

\alpha=0.1

over

10

draws, with the number that are continuous in parentheses.

Figure S1 . ROC curves . Panels correspond to the independent-linear, independent-nonlinear, correlated-linear, and correlated-nonlinear simulation designs. Curves closer to the upper-left corner indicate better ranking of active predictors above inactive predictors; the dashed diagonal represents random ranking.

$n$	RF	RF-one-time	RF-per-tree	UFI	CForest
250	0.3	0.4	0.4	0.9	88.0
500	0.6	0.9	0.9	2.4	181.9
1000	1.2	1.9	2.0	7.0	362.8
2000	1.8	2.8	2.9	13.8	–
4000	3.9	6.3	6.5	49.3	–
8000	9.0	14.8	14.9	184.2	–

Table S1 . Runtime as a function of sample size . Mean runtime in seconds over five trials, with

p=50

and matched tree-specific parameters; trial-to-trial variation was negligible. CForest was run only up to

n=1000

; dashes indicate settings where its runtime made the comparison impractical.

Step	Removed	Remaining
Patients with clinical and microRNA data	—	409
Undetermined 18-month survival status	100	309
Stage I or unrecorded stage	3	306

Table S2 . Cohort construction . Complete-case exclusions applied in order to the

409

patients with both clinical and microRNA data.

Figure S2 . Selections by predictor type . One panel per dataset (continuous and categorical predictor counts in the title); within each, one horizontal bar per base selector (RF-1T and XGB-1T are the one-time jittered selectors) whose length is the number of predictors selected by IPSS at a nominal FDR level of $0.1$ , split by the predictor’s number of distinct values, with continuous predictors counted in the top bin ( $>20$ ). RF selects the most predictors on every dataset except SAheart and is dominated by continuous and many-category predictors.

Figure S3 . Ranking distributions of synthetic nulls . One panel per dataset; within each, one row per importance method and one column per rank position (rank $1$ is most important). Observed predictors are grey; synthetic null predictors are coloured by their number of distinct values, from yellow (binary) to red ( $20$ or more, and all continuous nulls), averaged over $50$ seeds. Each dataset is augmented with one permuted null per observed predictor, so half of every row is null; the dashed line marks the midpoint. Under standard RF the nulls reach into the top half in order of cardinality; one-time jittering and UFI push them below the midpoint.

Dependence and signal	Method	$\delta=10^{-6}$	$\delta=10^{-4}$	$\delta=10^{-2}$	$\delta=10^{-1}$
Independent–linear	RF-one-time	0.945 (0.039)	0.944 (0.041)	0.944 (0.040)	0.944 (0.040)
	XGB-one-time	0.843 (0.066)	0.842 (0.062)	0.838 (0.064)	0.838 (0.064)
Independent–nonlinear	RF-one-time	0.891 (0.059)	0.891 (0.058)	0.890 (0.058)	0.890 (0.058)
	XGB-one-time	0.789 (0.076)	0.777 (0.076)	0.774 (0.074)	0.774 (0.074)
Correlated–linear	RF-one-time	0.852 (0.067)	0.857 (0.067)	0.858 (0.067)	0.858 (0.067)
	XGB-one-time	0.784 (0.085)	0.775 (0.081)	0.773 (0.081)	0.773 (0.081)

Table S3 . Feature-recovery AUC under different jitter strengths . Entries are the mean (standard deviation) across

100

trials.

Figure S4 . Sensitivity of selection to jitter strength. Average AUC in each of the four simulation settings across different jitter strengths, $\delta$ . Points are the mean AUC over $100$ trials; error bars are one standard deviation.

Figure S5 . Jittering has little effect on predictive accuracy . Distribution over Penn Machine Learning Benchmark (PMLB) datasets of the error ratio between jittered and unjittered ensembles, for random forests (top) and XGBoost (bottom). The dashed gray line at an error ratio of one marks no change; values below one indicate that jittering reduces error, while values above indicate an increase.

Random Forest

Random forests are ensembles of randomized decision trees used for prediction, feature analysis, and causal or structured-data modeling.

See all 34 papers

Jun 9, 2026

Correcting Variable Importance Scored by Random Forests

Variable importance produced by Random Forests (RF) is used widely in statistical data analysis, and has played an important role in a variety of tasks such as assisting model interpretation, model selection and diagnosis, and cost-bounded learning etc. However, the calculation of variable importance in RF does not take into account of the correlations among variables, and variables that are correlated to many other variables tend to receive a lower importance index or being completely masked (i.e., with an importance index near zero) by other strongly correlated variables. To prevent influence from unwanted correlated variables in calculating variable importance, we propose to group variables by their conditional correlations (conditional on the response variable). We explore two computationally efficient options, with one grouping variables individually, and then separates the variable of interest from all correlated variables, while the other uses clustering to group variables according to their pair-wise conditional correlations. Our experiments show that both lead to sensible corrections to the importance of variables.

Guancheng Zhou, Haiping Xu, Jason Liu +1

Mathematics and Data Science University of Massachusetts, Dartmouth, MA · Computer and Information Science University of Massachusetts, Dartmouth, MA · The Rivers School, Weston, MA

Feature Attribution Random Forest

Ask PDF

Apr 13, 2026

Trustworthy Feature Importance Avoids Unrestricted Permutations

Feature importance methods using unrestricted permutations are flawed due to extrapolation errors; such errors appear in all non-trivial variable importance approaches. We propose three new approaches: conditional model reliance and Knockoffs with Gaussian transformation, and restricted ALE plot designs. Theoretical and numerical results show our strategies reduce/eliminate extrapolation.

Emanuele Borgonovo, Francesco Cappelli, Xuefei Lu +2

Department of Decision Sciences and Bocconi Institute for Data Science and Analytics, Bocconi University, Milan, Italy · Department of Decision Science, Bocconi University, Milan, Italy · SKEMA Business School, Université Côte d’Azur, France +2

Feature Attribution Perturbation-Based Feature Attribution

Ask PDF

Sep 18, 2026

Improving the Predictive Performance of Bootstrap Aggregating by Dirichlet Resampling

We revisit Breiman's observation that reducing inter-tree correlation without weakening individual trees can improve random forests. Building on this principle, we introduce two variants: Dirichlet-Multinomial Bagging Random Forest (DM) and Dirichlet-Weighted Random Forest (DW). Both modulate sample reweighting via a concentration parameter

α>0

. We provide a simple theoretical criterion that clarifies when these variants behave indistinguishably from standard random forests, and we use it to guide a lightweight tuning strategy. In a controlled evaluation on public classification benchmarks, DM and DW are consistently competitive and often stronger than other random-forest (RF) baselines, with negligible additional runtime.

Quoc Viet Le, Joonha Park

Department of Statistics, University of Wisconsin–Madison · Department of Mathematics, University of Kansas

Random Forest

Ask PDF

Abstract

Random Forest

Explore similar work

Correcting Variable Importance Scored by Random Forests

Trustworthy Feature Importance Avoids Unrestricted Permutations

Improving the Predictive Performance of Bootstrap Aggregating by Dirichlet Resampling