Seeing Through Distractions: Context-Stable Attribution via the Least Core

Sai Ganesh Nagarajan; Toshinori Yamauchi; Hiroshi Kera

cs.CVOct 26, 2026

Seeing Through Distractions: Context-Stable Attribution via the Least Core

Authors: Sai Ganesh Nagarajan, Toshinori Yamauchi, Hiroshi Kera

Organizations: University of Southern Denmark, IMADA, National Institute of Informatics · Chiba University, Graduate School of Informatics

Abstract

Shapley-value-based explanations are widely used for feature attribution in machine learning. However, in computer vision, a feature may reduce the target-class score when added to the complete genuine image, even when the predicted label remains unchanged. We formalize such features as contextual distractors and investigate their attributions under semivalues and the least core. Under a weak average marginal monotonicity assumption, we derive quantitative semivalue--least-core gap bounds and sufficient conditions based on the probability of sampling smaller contexts. We further show that shifting semivalue weights toward larger contexts decreases the signed gap. Finally, controlled image-classification experiments and evaluations on Salient ImageNet provide strong empirical support for our theoretical findings.

Figures & tables

Figure 1: Controlled marker-bias setup. A red marker is added to all dog images and a subset of cat images, while bird images remain unmarked. This creates a dog-associated cue that may act as a contextual distractor for cat predictions. In Section 4 , we observe that while Shapley by nature, averages contributions across contexts, the least core incorporates stability constraints that account for the marker’s effect on the complete genuine image and the predicted label.

Figure 2: Analysis of the controlled marker-bias experiment. (a) Example images with synthetic markers. (b) For 100 cat-class images, the x-axis shows the normalized cat-class logit $v(T)$ of masked images, where coalitions $T$ are sampled from non-marker regions. The y-axis shows the logit change caused by adding the marker region $D$ , i.e., $v(T\cup D)-v(T)$ . (c) Comparison of Shapley and least-core scores for the marker region over the same 100 cat-class images. Each point corresponds to one image, and the dashed line indicates equal attribution by the two methods.

Figure 3: Representative examples of contextual distractors. Input + D: the original input, with the distractor region $D$ outlined in yellow. Shapley and Least-Core: patch-attribution heatmaps, where red and blue indicate positive and negative attribution, respectively; $A_{D}$ gives the total attribution assigned to the patches in $D$ . Context dependence: the marginal contribution $\Delta_{D}(T)=v(T\cup D)-v(T)$ plotted against the target-class logit $v(T)$ obtained from the surrounding context $T$ . Logit change in strong context: a sampled strong coalition $T$ before and after adding $D$ , reporting the resulting target-class logit $v(T\cup D)$ and its change $\Delta_{D}$ from $v(T)$ .

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

	Grad-CAM	Grad-CAM++	Shapley-MC	Least-Core
$\epsilon$ (mean $\pm$ std)	$0.6990\pm 0.1445$	$0.6899\pm 0.1397$	$0.6851\pm 0.1637$	$\mathbf{0.5822\pm 0.1389}$

Appendix

Table 1: Epsilon values on 100 ImageNet test images. Lower is better.

Figure 4: Controlled marker-bias experiment. Left: representative inputs. The marker is correlated with the dog class and can therefore mislead cat predictions. Middle and right: for both ResNet-50 and ViT-B/16, the marginal contribution $\Delta_{D}(T)=v(T\cup D)-v(T)$ decreases with the strength of the non-marker context. The marker helps in weak contexts but suppresses the cat logit in strong contexts.

Figure 5: Per-image marker attribution for ResNet-50 (left) and ViT-B/16 (right). We compare the Shapley and Banzhaf ( $p=0.5$ ) scores with the corresponding least-core scores. Each point represents one retained image, and the dashed line indicates equal attribution; points above the line receive more attribution from the semivalue than from the least-core.

Model	Least-core	Shapley	Gap	Banzhaf ( $p=0.5$ )	Gap
ResNet-50	0.041	0.220	0.179	0.076	0.035
ViT-B/16	-0.221	0.419	0.639	0.355	0.576

Appendix

Table 2: Mean marker-patch attribution. Each Gap column reports the corresponding semivalue minus the least-core attribution; a positive value indicates that the semivalue assigns more credit to the marker.

Figure 6: Effect of context-size weighting on the group attribution to the marker region $D$ . The left two panels show ResNet-50 and the right two show ViT-B/16. Within each pair, the first panel varies $p$ for the weighted-Banzhaf family, and the second shifts the Beta-Shapley parameters from small-context weighting $(4,1)$ to large-context weighting $(1,4)$ . The solid blue curves show the group semivalue obtained at each parameter setting. The dotted green and dashed red lines show the independently estimated group Shapley baseline and the least-core attribution, respectively.

Model	Region	Outcome	Shapley	Core	Gap	S $>$ C
ResNet-50	Contextual distractor	$A_{D}$	2.338	1.262	1.076	82.5%
		$A_{\mathrm{mask}}$	1.528	0.873	0.655	78.8%
	Random-SAM control	$A_{D}$	0.614	0.838	$-0.224$	33.1%
		$A_{\mathrm{mask}}$	0.434	0.579	$-0.144$	33.1%
ViT-B/16	Contextual distractor	$A_{D}$	2.177	1.645	0.532	80.6%
		$A_{\mathrm{mask}}$	1.435	1.112	0.323	76.9%

Appendix

Table 3: Shapley and sampled egalitarian least-core attribution for contextual distractors and prediction-active random-SAM controls. Gap is Shapley minus Core, and S

>

C is the fraction of regions for which the Shapley attribution exceeds the Core attribution.

Feature Attribution

Feature attribution estimates which input features or internal components contribute to a model’s predictions.

See all 234 papers

cc-Shapley: Measuring Multivariate Feature Importance Needs Causal Context
Feb 23, 2026Jörg Martin, Stefan HaufeFeature Attribution Shapley Value Attribution
Physikalisch-Technische Bundesanstalt, Berlin, Germany · Technische Universität Berlin, Berlin, Germany · Charité – Universitätsmedizin Berlin, Berlin, Germany
RelShap: Relationally Consistent Shapley Explanations
Aug 11, 2026Seungeun Lee, Joao Fonseca, Julia StoyanovichFeature Attribution Shapley Value Attribution
New York University, New York, USA · INESC-ID, Lisbon, Portugal
RoSHAP: A Distributional Framework and Robust Metric for Stable Feature Attribution
May 14, 2026Lanxin Xiang, Liang Shi, Youhui Ye +3Feature Attribution Shapley Value Attribution
Department of Statistics Virginia Tech · Department of Statistics Virginia Tech Transportation Institute · Department of Computer Science Virginia Tech

Feb 23, 2026J/K move · Enter open · S save

cc-Shapley: Measuring Multivariate Feature Importance Needs Causal Context

Jörg Martin, Stefan Haufe

Physikalisch-Technische Bundesanstalt, Berlin, Germany · Technische Universität Berlin, Berlin, Germany · Charité – Universitätsmedizin Berlin, Berlin, Germany

Feature Attribution Shapley Value Attribution Causal Attribution Spurious Correlation Causal Inference

Open paper AskPDF arXiv

Explainable artificial intelligence promises to yield insights into relevant features, thereby enabling humans to examine and scrutinize machine learning models or even facilitating scientific discovery. Considering the widespread technique of Shapley values, we find that purely data-driven operationalization of multivariate feature importance is unsuitable for such purposes. Even for simple problems with two features, spurious associations due to collider bias and suppression arise from considering one feature only in the observational context of the other, which can lead to misinterpretations. Causal knowledge about the data-generating process is required to identify and correct such misleading feature attributions. We propose cc-Shapley (causal context Shapley), an interventional modification of conventional observational Shapley values leveraging knowledge of the data's causal structure, thereby analyzing the relevance of a feature in the causal context of the remaining features. We show theoretically that this eradicates spurious association induced by collider bias. We compare the behavior of Shapley and cc-Shapley values on various, synthetic, and real-world datasets. We observe nullification or reversal of associations compared to univariate feature importance when moving from observational to cc-Shapley.

Abstract

Feature Attribution

Explore similar work

cc-Shapley: Measuring Multivariate Feature Importance Needs Causal Context

RelShap: Relationally Consistent Shapley Explanations

RoSHAP: A Distributional Framework and Robust Metric for Stable Feature Attribution