First-Mover Bias in Gradient Boosting Explanations: Mechanism, Detection, and Resolution

Drake Caraker; Bryan Arnold; David Rhoads

cs.LGMar 22, 2026

First-Mover Bias in Gradient Boosting Explanations: Mechanism, Detection, and Resolution

Authors: Drake Caraker, Bryan Arnold, David Rhoads

Organizations: Independent Researchers

Abstract

We identify first-mover bias -- path-dependent concentration of SHAP feature importance from sequential residual fitting in gradient boosting -- as a mechanistic contributor to attribution instability under multicollinearity. Scaling up a single model amplifies this effect: a Large Single Model matching our method's total tree count produces the poorest attribution reproducibility of any approach tested. We show that model independence largely neutralizes first-mover bias. Both DASH (Diversified Aggregation of SHAP) and simple seed-averaging (Stochastic Retrain) restore stability by breaking the sequential dependency chain. At rho=0.9, both achieve stability ~0.977, while Single Best degrades to 0.958 and LSM to 0.938. On Breast Cancer, DASH improves stability from 0.376 to 0.925 (+0.549), outperforming Stochastic Retrain by +0.063. Under nonlinear DGPs, the advantage emerges at rho>=0.7. DASH provides two diagnostic tools -- the Feature Stability Index and Importance-Stability Plot -- that detect first-mover bias without ground truth. A crossed ANOVA with formal F-statistics confirms the mechanism: DASH shifts variance from model-dominated (40.6%) to data-dominated (73.6%). Software at https://github.com/DrakeCaraker/dash-shap

Figures & tables

Figure 1: Direct isolation of the

T

-scaling prediction. Concentration within a correlated group (max/sum importance,

\rho{=}0.9

, group of 5 features with true importance

0.20

each) as a function of sequential tree count

T

. Single sequential model (red squares): concentration grows monotonically with

T

, confirming the first-mover prediction. Independent ensemble of

M

models averaged at matched total tree count (green circles): concentration remains flat regardless of per-model depth. The divergence isolates residual dependency as the operative mechanism. Mean

\pm

SD across 50 repetitions.

Panel of Figure 2 — Figure 2: The DASH five-stage pipeline. Models are trained independently (Stage 1), filtered for quality (Stage 2), selected for diversity (Stage 3), explained via TreeSHAP and averaged (Stage 4), and audited for stability (Stage 5).

Parameter	Values
max_depth	$\{3,4,5,6,8,10,12\}$
learning_rate	$\{0.01,0.03,0.05,0.1,0.2,0.3\}$
colsample_bytree	$\{0.1,0.15,0.2,0.25,0.3,0.4,0.5\}$
subsample	$\{0.5,0.6,0.7,0.8,0.9,1.0\}$
reg_alpha	$\{0,0.01,0.1,1.0,5.0,10.0\}$
reg_lambda	$\{0,0.01,0.1,1.0,5.0,10.0\}$

Table 1: Hyperparameter search space for population generation.

	Method	Description
Dependent	Single Best	Best of 30 hyperparameter-tuned models (standard practice)
	Single Best ( $M$ )	Best of $M{=}200$ models (training-budget-matched to DASH )
	Large Single Model	One XGBoost with $\sim$ 15K trees, low colsample_bytree
	LSM (Tuned)	Grid search over max_depth , learning_rate
	Ensemble SHAP	Single 2000-tree ensemble ( colsample_bytree =0.8)
Independent	Stochastic Retrain	$K$ models, different seeds, same hyperparameters

Table 2: Methods compared in the benchmark. All use XGBoost as the base learner.

$\rho$	Method	Stability ( $\pm$ SE)	Accuracy	Equity (CV $\downarrow$ )	RMSE
0.0	Single Best	$.972\pm.001$	.985	.163	.611
	Large Single Model	$.955\pm.002$	.976	.170	.772
	Stoch. Retrain	$\mathbf{.975}\pm.001$	.987	.169	.581
	DASH (MaxMin)	$.973\pm.001$	.985	.153	.606
0.5	Single Best	$.974\pm.001$	.987	.179	.620
	Large Single Model	$.967\pm.001$	.983	.187	.768

Table 3: The mechanism experiment: DASH vs. Single Best vs. Large Single Model across correlation levels (50 repetitions per

\rho

). Both DASH and LSM use the same low colsample_bytree ; the critical contrast for our hypothesis is that DASH ’s models are trained independently while LSM trains trees sequentially. Bold indicates best per metric per

\rho

level. The four principal methods are shown; all nine are compared at

\rho=0.9

in Table 4 .

Figure 3: First-mover bias visualized: per-feature importance within a correlated group ( $\rho=0.9$ , true importance $=0.40$ each). Single Best and LSM concentrate on an arbitrary feature; DASH distributes proportionally. Averaged over 5 repetitions; error bars show $\pm 1$ SD. The $\rho_{S}$ values in panel titles denote Spearman correlation with ground truth for this group (not the within-group feature correlation $\rho$ ).

	Method	Stability ( $\pm$ SE)	Accuracy	Equity (CV $\downarrow$ )	$K_{\text{eff}}$	Time (s)
Dependent	Single Best	$.958\pm.002$	.978	.232	1	0 43.5
	Single Best †	$.964\pm.001$	.981	.214	1	248.8
	Large Single Model	$.938\pm.002$	.968	.258	1	00 6.4
	LSM (Tuned)	$.948\pm.002$	.973	.271	1	0 88.9
	Ensemble SHAP	$.956\pm.001$	.977	.236	1	—
Independent	Stochastic Retrain	$.977\pm.002$	.988	.182	30	233.5

Table 4: All methods at

\rho=0.9

(50 repetitions), grouped by whether they achieve model independence. Bold indicates best per metric. † Matched training budget (

M{=}200

models trained). Timing entries marked — share infrastructure with other methods and were not independently measured. 95% BCa bootstrap CIs on stability: Single Best

[0.952,0.963]

, DASH

[0.975,0.978]

(non-overlapping).

Method	Source	SS	df	$F$	$p$
Single Best	Data	0.588	6	10.3	$1.2\times 10^{-6}$
	Model	0.636	6	11.2	$5.1\times 10^{-7}$
	Residual	0.341	36	—	—
DASH (MaxMin)	Data	0.405	6	43.4	$4.9\times 10^{-15}$
	Model	0.089	6	0 9.5	$2.8\times 10^{-6}$
	Residual	0.056	36	—	—

Table 5: Two-way ANOVA for the

7\times 7

crossed variance decomposition (

\rho=0.9

). Sum of squares are aggregated across

P=50

features;

F

-statistics use the residual (interaction + noise) as the error term. 3 3 3 Because each cell contains one

P

-dimensional importance vector and the SS are summed across features, these

F

-tests are an aggregate summary rather than a per-feature multivariate test. Per-feature ANOVA tables are available in the code repository. Both data and model effects are significant for both methods; the critical contrast is that DASH shifts from model-dominated (SB) to data-dominated variance.

$\rho$	Comparison	Metric	$p_{\text{HB}}$	Cohen’s $d$
0.7	DASH vs SB	Accuracy	0.031	$+$ 0.98
0.7	DASH vs LSM	Accuracy	0.002	$+$ 1.97
0.7	DASH vs SB	Equity	$<$ 0.001	$-$ 1.03
0.7	DASH vs LSM	Equity	$<$ 0.001	$-$ 1.78
0.9	DASH vs SB	Accuracy	0.010	$+$ 1.59
0.9	DASH vs LSM	Accuracy	$<$ 0.001	$+$ 3.42

Table 6: Wilcoxon signed-rank tests with Holm–Bonferroni step-down correction (50 paired repetitions). Bold

p

-values are significant at adjusted

\alpha=0.05

. Selected comparisons shown from 26 total tests; full results available in the code repository.

Figure 4: Stability, accuracy, and equity as a function of within-group correlation $\rho$ (linear DGP, 50 repetitions per level). Independent methods ( DASH , Stochastic Retrain) are flat across correlation levels; dependent methods (Single Best, LSM) degrade monotonically. All seven methods shown; Table 3 reports the four principal methods in detail, Table 4 compares all at $\rho=0.9$ .

$\rho$	Mean FSI (signal)	Mean FSI (noise)	Ratio	$\beta$ Spearman
0.0	0.285	0.905	0.31	$-$ 0.995
0.5	0.337	1.253	0.27	$-$ 0.995
0.7	0.362	1.424	0.25	$-$ 0.995
0.9	0.420	1.636	0.26	$-$ 0.994
0.95	0.461	1.843	0.25	$-$ 0.991

Table 7: FSI validation on synthetic data (50 repetitions per

\rho

). Mean FSI for signal features (

\beta_{g}>0

) vs. noise features (

\beta_{g}=0

\beta

Spearman is the rank correlation between FSI and ground-truth importance magnitude across all 50 features.

Panel of Figure 5 — Figure 5: Diagnostic outputs on Breast Cancer. (a) The IS Plot identifies robust drivers (Quadrant I, green) and collinear cluster members (Quadrant II, red) without access to the correlation matrix. (b) The local disagreement map shows consensus SHAP values $\pm 1$ SD across the ensemble; wide error bars indicate model-dependent attributions.

Dataset	Method	Stability ( $\pm$ SE)	$\Delta$ Stab.	RMSE
Breast Cancer	Single Best	$.376\pm.043$	—	—
	Single Best ( $M{=}200$ ) †	$.339\pm.037$	$-0.037$	—
	Stochastic Retrain	$.862\pm.010$	$+0.486$	—
	Random Selection	$.919\pm.004$	$+0.543$	—
	Random Forest	$.922\pm.004$	$+0.546$	—
	DASH (MaxMin)	$\mathbf{.925}\pm.004$	$+0.549$	—

Table 8: Real-world dataset results (50 repetitions each unless noted). Stability reported with

\pm

SE from BCa bootstrap. Bold indicates best. Breast Cancer is a classification task (RMSE not applicable). † Matched training budget (

M{=}200

models trained).

\Delta

Stab. is relative to standard Single Best. All sourced from 50-rep SageMaker run (ml.g5.16xlarge).

Stability
$\rho$	SB	LSM	LSM-T	SR	DASH
0.0	0.931	0.913	0.929	0.935	0.933
0.5	0.850	0.837	0.846	0.860	0.857
0.7	0.845	0.825	0.850	0.853	0.858
0.9	0.811	0.778	0.825	0.857	0.887
0.95	0.801	0.755	0.791	0.859	0.884

Table 9: Nonlinear DGP: stability and equity across correlation levels (50 repetitions per

\rho

level). SB = Single Best, LSM = Large Single Model, LSM-T = LSM (Tuned), SR = Stochastic Retrain. Five methods with distinct training structures are shown; Random Selection and Naive Top-

N

(which performed similarly to DASH in the linear regime) are omitted for space and are available in the code repository. Bold indicates best per

\rho

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

$\varepsilon$	Models Passing	$K_{\text{eff}}$	Stability	Accuracy	Equity
0.03	9.3	4.0 $\pm$ 1.4	0.9759	0.9876	0.183
0.05	21.0	6.6 $\pm$ 1.9	0.9764	0.9879	0.178
0.08	46.1	11.9 $\pm$ 3.0	0.9767	0.9881	0.175
0.10	63.0	16.1 $\pm$ 3.6	0.9774	0.9884	0.173

Appendix

Table 10: Epsilon sensitivity at

\rho=0.9

(50 repetitions). Performance is robust across a

3\times

range.

Figure 6: Ablation sensitivity: stability as a function of each DASH hyperparameter at three correlation levels ( $\rho\in\{0.0,0.9,0.95\}$ ). Top left : Population size $M$ —stability is effectively invariant ( $\Delta<0.005$ across $M\in\{50,100,200,500\}$ ). Top right : Selected models $K$ —saturates at $K\approx 20$ . Bottom left : Filter threshold $\varepsilon$ —robust across a $3\times$ range ( $0.03$ – $0.10$ ). Bottom right : Diversity threshold $\delta$ —sensitive above $0.05$ ; the default $\delta=0.05$ is at the elbow.

Method	Models	SHAP Evals	Per-rep (s)	Ratio
Large Single Model	1	1	00 6.4	$0.1\times$
Single Best	30	1	0 43.5	$1.0\times$
LSM (Tuned)	1	1	0 88.9	$2.0\times$
DASH (MaxMin)	200	$K_{\text{eff}}$	140.3	$3.2\times$
Stochastic Retrain	30	30	233.5	$5.4\times$
Single Best ( $M{=}200$ )	200	1	248.8	$5.7\times$

Appendix

Table 11: Computational cost at

\rho=0.9

(50 repetitions). Wall-clock times are hardware-dependent (Apple M-series, single thread) and reported for relative comparison; absolute times on the SageMaker ml.g5.16xlarge instance are

\sim

\times

higher due to nthread=1 per-rep parallelism.

Method	Stability	Bias ( $f_{0}$ )	Passive leak ( $f_{1}$ )
Single Best	1.000	0.068	0.068
Stochastic Retrain	1.000	0.074	0.074
DASH (MaxMin)	1.000	0.089	0.089
Large Single Model	1.000	0.084	0.084

Appendix

Table 12: Asymmetric causal DGP at

\rho=0.9

, 50 repetitions. “Passive leak” is mean attribution to the causally inert feature

f_{1}

(ground truth: 0.0). Lower is better for both bias and passive leak. All methods achieve stability = 1.000 (deterministic DGP).

SHAP Feature Attribution

SHAP (SHapley Additive exPlanations) feature attribution assigns model predictions to input features using Shapley values from cooperative game theory.

See all 36 papers

The Attribution Impossibility: No Feature Ranking Is Faithful, Stable, and Complete Under Collinearity
Apr 8, 2026Drake Caraker, Bryan Arnold, David RhoadsFeature Attribution SHAP Feature Attribution
Independent Researchers
RoSHAP: A Distributional Framework and Robust Metric for Stable Feature Attribution
May 14, 2026Lanxin Xiang, Liang Shi, Youhui Ye +3Feature Attribution Shapley Value Attribution
Department of Statistics Virginia Tech · Department of Statistics Virginia Tech Transportation Institute · Department of Computer Science Virginia Tech
qshap: Fast Shapley Decomposition of $R^2$ for Gradient-Boosted Trees
Aug 25, 2026Zhongli Jiang, Min Zhang, Dabao ZhangFeature Attribution Shapley Value Attribution
University of California, Irvine

Apr 8, 2026J/K move · Enter open · S save

The Attribution Impossibility: No Feature Ranking Is Faithful, Stable, and Complete Under Collinearity

Drake Caraker, Bryan Arnold, David Rhoads

Independent Researchers

Feature Attribution SHAP Feature Attribution

Open paper AskPDF arXiv

No feature ranking can be simultaneously faithful, stable, and complete when features are collinear. For collinear pairs, ranking reduces to a coin flip. We prove this impossibility, quantify it for four model classes, resolve it via ensemble averaging (DASH), and machine-verify it with 305 Lean 4 theorems. We characterize the complete attribution design space: exactly two families of methods exist -- faithful-complete methods (unstable, with rankings that flip up to 50% of the time) and ensemble methods like DASH (stable, reporting ties for symmetric features) -- and no method lies outside this dichotomy. The impossibility is quantitative: the attribution ratio diverges as 1/(1-rho^2) for gradient boosting, is infinite for Lasso, and converges for random forests. DASH (Diversified Aggregation of SHAP) is provably Pareto-optimal among unbiased aggregations, achieving the Cramer-Rao variance bound with a tight ensemble size formula. In a survey of 77 public datasets, 68% exhibit attribution instability. Switching to conditional SHAP does not escape the impossibility when features have equal causal effects. The framework includes practical diagnostics -- a Z-test workflow and single-model screening tool -- and has direct consequences for fairness auditing: SHAP-based proxy discrimination audits are provably unreliable under collinearity. The design space theorem, diagnostics, and impossibility are mechanically verified in Lean 4 (305 theorems from 16 axioms, 0 sorry) -- to our knowledge, the first formally verified impossibility in explainable AI.

Abstract

SHAP Feature Attribution

Explore similar work

The Attribution Impossibility: No Feature Ranking Is Faithful, Stable, and Complete Under Collinearity

RoSHAP: A Distributional Framework and Robust Metric for Stable Feature Attribution

qshap: Fast Shapley Decomposition of R2R^2R2 for Gradient-Boosted Trees

qshap: Fast Shapley Decomposition of $R^2$ for Gradient-Boosted Trees