Just add noise: Debiasing tree-based variable importance in mixed data

Jiahe Li; Omar Melikechi

Figure 1. Adding noise yields equal numbers of candidate splits . Five samples are drawn from the independent pair

(X,Z)

, where

X\sim\mathcal{N}(0,1)

and

Z\sim\text{Bernoulli}(1/2)

. ( Left ) The 5 samples of

X

admit 4 splits (vertical lines), while the

Z

samples only admit 1 (horizontal line). ( Right ) The

Z

samples admit 4 splits after adding noise.

Figure 2. Mixed data bias. MDI scores from CART, plotted as a function of

n

for the model

Y=X+Z+\varepsilon

, where

X\sim N(0,1)

,

Z=\pm 1

with equal probability, and

\varepsilon\sim N(0,1)

are mutually independent. Although

\operatorname{Var}(X)=\operatorname{Var}(Z)=1

, normalized MDIs for

X

and

Z

converge to

2/3

and

1/3

, respectively, in accordance with Theorem 2.4 . After adding noise to

Z

, both scores converge to

1/2

. Scores are averaged over 200 trials; error bars show 95% Monte Carlo confidence intervals for the mean.

	Independent covariates		Correlated covariates
Method	Linear	Nonlinear	Linear	Nonlinear
RF	0.80 (0.05)	0.80 (0.05)	0.71 (0.05)	0.72 (0.05)
RF-one-time	0.94 (0.05)	0.89 (0.07)	0.86 (0.07)	0.86 (0.06)
RF-per-tree	0.96 (0.04)	0.92 (0.05)	0.88 (0.06)	0.90 (0.05)
UFI	0.92 (0.05)	0.87 (0.07)	0.83 (0.06)	0.83 (0.07)
CForest	0.92 (0.05)	0.82 (0.07)	0.84 (0.08)	0.82 (0.08)

Table 1. Feature-ranking performance . Mean AUC over

100

trials, with the standard deviation in parentheses; the best AUC in each setting is in bold (higher is better).

Figure 3. IPSS with different MDI scores . TPR (top row) and FDR (bottom row) against the target FDR level for IPSS with MDI scores from random forests (RF), one-time jittered random forests (RF-one-time), UFI, XGBoost (XGB), and one-time jittered XGBoost (XGB-one-time). Columns correspond to the four simulation designs. The dashed diagonal in the FDR plots marks exact FDR calibration; methods lying at or below this line successfully control FDR.

Figure 4. Rank of pathologic stage over 50 random seeds . Rank of pathologic stage among the

425

bladder cancer predictors of 18-month survival over

50

random seeds.

Dataset	$n$	$p$	$q$	RF	RF-one-time	XGB	XGB-one-time
QSAR Biodegradation	1055	19	22	14.7 (14.7)	0.0 (0.0)	0.6 (0.5)	0.2 (0.1)
Cylinder Banding	540	13	22	10.7 (9.5)	0.0 (0.0)	0.0 (0.0)	0.0 (0.0)
Hypothyroid	3163	7	18	9.8 (7.0)	0.2 (0.0)	2.7 (2.7)	0.3 (0.0)
German Credit	1000	3	17	6.7 (3.0)	0.0 (0.0)	0.5 (0.4)	0.0 (0.0)
Hepatitis	155	6	13	6.0 (6.0)	0.0 (0.0)	0.1 (0.1)	0.0 (0.0)
SAheart	462	8	1	0.0 (0.0)	0.0 (0.0)	0.0 (0.0)	0.0 (0.0)

Table 2. Number of synthetic null predictors selected . For each dataset (

n

samples,

p

continuous and

q

categorical predictors), the mean number of synthetic null predictors selected by IPSS with each base selector at target FDR

\alpha=0.1

over

10

draws, with the number that are continuous in parentheses.

Figure S1 . ROC curves . Panels correspond to the independent-linear, independent-nonlinear, correlated-linear, and correlated-nonlinear simulation designs. Curves closer to the upper-left corner indicate better ranking of active predictors above inactive predictors; the dashed diagonal represents random ranking.

$n$	RF	RF-one-time	RF-per-tree	UFI	CForest
250	0.3	0.4	0.4	0.9	88.0
500	0.6	0.9	0.9	2.4	181.9
1000	1.2	1.9	2.0	7.0	362.8
2000	1.8	2.8	2.9	13.8	–
4000	3.9	6.3	6.5	49.3	–
8000	9.0	14.8	14.9	184.2	–

Table S1 . Runtime as a function of sample size . Mean runtime in seconds over five trials, with

p=50

and matched tree-specific parameters; trial-to-trial variation was negligible. CForest was run only up to

n=1000

; dashes indicate settings where its runtime made the comparison impractical.

Step	Removed	Remaining
Patients with clinical and microRNA data	—	409
Undetermined 18-month survival status	100	309
Stage I or unrecorded stage	3	306

Table S2 . Cohort construction . Complete-case exclusions applied in order to the

409

patients with both clinical and microRNA data.

Figure S2 . Selections by predictor type . One panel per dataset (continuous and categorical predictor counts in the title); within each, one horizontal bar per base selector (RF-1T and XGB-1T are the one-time jittered selectors) whose length is the number of predictors selected by IPSS at a nominal FDR level of

0.1

, split by the predictor’s number of distinct values, with continuous predictors counted in the top bin (

>20

). RF selects the most predictors on every dataset except SAheart and is dominated by continuous and many-category predictors.

Figure S3 . Ranking distributions of synthetic nulls . One panel per dataset; within each, one row per importance method and one column per rank position (rank

1

is most important). Observed predictors are grey; synthetic null predictors are coloured by their number of distinct values, from yellow (binary) to red (

20

or more, and all continuous nulls), averaged over

50

seeds. Each dataset is augmented with one permuted null per observed predictor, so half of every row is null; the dashed line marks the midpoint. Under standard RF the nulls reach into the top half in order of cardinality; one-time jittering and UFI push them below the midpoint.

Dependence and signal	Method	$\delta=10^{-6}$	$\delta=10^{-4}$	$\delta=10^{-2}$	$\delta=10^{-1}$
Independent–linear	RF-one-time	0.945 (0.039)	0.944 (0.041)	0.944 (0.040)	0.944 (0.040)
	XGB-one-time	0.843 (0.066)	0.842 (0.062)	0.838 (0.064)	0.838 (0.064)
Independent–nonlinear	RF-one-time	0.891 (0.059)	0.891 (0.058)	0.890 (0.058)	0.890 (0.058)
	XGB-one-time	0.789 (0.076)	0.777 (0.076)	0.774 (0.074)	0.774 (0.074)
Correlated–linear	RF-one-time	0.852 (0.067)	0.857 (0.067)	0.858 (0.067)	0.858 (0.067)
	XGB-one-time	0.784 (0.085)	0.775 (0.081)	0.773 (0.081)	0.773 (0.081)

Table S3 . Feature-recovery AUC under different jitter strengths . Entries are the mean (standard deviation) across

100

trials.

Figure S4 . Sensitivity of selection to jitter strength. Average AUC in each of the four simulation settings across different jitter strengths,

\delta

. Points are the mean AUC over

100

trials; error bars are one standard deviation.

Figure S5 . Jittering has little effect on predictive accuracy . Distribution over Penn Machine Learning Benchmark (PMLB) datasets of the error ratio between jittered and unjittered ensembles, for random forests (top) and XGBoost (bottom). The dashed gray line at an error ratio of one marks no change; values below one indicate that jittering reduces error, while values above indicate an increase.

Just add noise: Debiasing tree-based variable importance in mixed data

Abstract

Figures & tables

Random Forest

Correcting Variable Importance Scored by Random Forests

Trustworthy Feature Importance Avoids Unrestricted Permutations

Improving the Predictive Performance of Bootstrap Aggregating by Dirichlet Resampling

Abstract

Random Forest

Explore similar work

Correcting Variable Importance Scored by Random Forests

Trustworthy Feature Importance Avoids Unrestricted Permutations

Improving the Predictive Performance of Bootstrap Aggregating by Dirichlet Resampling