Spectral-Spatial Interpretation

Haotian Ma; Ruqi Yang; Philip Townsend

cs.LGOct 15, 2026

Spectral-Spatial Interpretation

Authors: Haotian Ma, Ruqi Yang, Philip Townsend

Organizations: Department of Computer Sciences University of Wisconsin–Madison · Department of Forest and Wildlife Ecology University of Wisconsin–Madison

Abstract

Post-hoc image explanations usually show where a model looks using spatial heatmaps or indicate which channels matter using global channel scores. These separate views are limiting for multispectral data because channel relevance is often spatially localized and physically meaningful. We introduce Spectral-Spatial Interpretation (SSI), a full-grid attribution method over position-channel players (g,c). For each spatial position, SSI fixes the observed surrounding context, defines a conditional game over the channels at that position, and applies the Shapley value to this local game. The result is a full spectral-spatial attribution matrix rather than a single heatmap or a global channel ranking. SSI has a clear local interpretation. It satisfies Shapley fairness within each position, and aggregating its channel attributions exactly recovers the position-level contextual contrast obtained by replacing that position with the baseline. SSI also differs from context-averaged Owen-style hierarchical accounting, which uses the same position-channel grouping but averages over external spatial coalitions. On HLS Burn Scar, SSI outperforms an MC-Owen estimator across retained attribution metrics while using only about 6% of its evaluation budget. Across CUB-200, EuroSAT, and HLS Burn Scar under both per-channel mean and zero baselines, SSI achieves the best joint deletion and insertion AUC over dense, flat, and rank-1 baselines, yields the strongest spectral hit rates and channel deletion, and leads spatial localization on both annotated datasets. Code is available at https://github.com/AtheavenMa/Spectral-Spatial-Interpretation.

Figures & tables

Figure 1: Spectral channels carry spatially distinct physical information. On a Painted Bunting (a), the red belly, green back, and blue head each align with a different RGB channel; on a Sentinel-2 scene (b), water, forest, and built-up areas align with B3, B8A, and B12. SSI is designed to recover this joint spectral-spatial structure.

Figure 2: Overview of SSI. For each spatial position

g

, SSI fixes the observed surrounding context and allocates the local contribution among channels, producing a full joint attribution matrix

\Phi(g,c)

for spatial and spectral interpretation.

Per-channel mean baseline
	CUB-200			EuroSAT			HLS Burn Scar
Method	Joint $\downarrow$	Rk1 $\downarrow$	Ins. $\uparrow$	Joint $\downarrow$	Rk1 $\downarrow$	Ins. $\uparrow$	Joint $\downarrow$	Rk1 $\downarrow$	Ins. $\uparrow$
SSI	.0088 $\pm$ .001	.0104 $\pm$ .001	.7132 $\pm$ .019	.1397 $\pm$ .020	.1682 $\pm$ .021	.7319 $\pm$ .023	.1545 $\pm$ .008	.1657 $\pm$ .010	.6638 $\pm$ .008
IG	.0112 $\pm$ .001	.0126 $\pm$ .001	.2895 $\pm$ .022	.2097 $\pm$ .030	.2267 $\pm$ .030	.6621 $\pm$ .029	.2160 $\pm$ .011	.2723 $\pm$ .015	.6291 $\pm$ .009
DeepSHAP	.0112 $\pm$ .001	.0127 $\pm$ .001	.2853 $\pm$ .022	.2078 $\pm$ .029	.2255 $\pm$ .031	.6614 $\pm$ .028	.2149 $\pm$ .011	.2704 $\pm$ .016	.6321 $\pm$ .009
KernelSHAP	.0178 $\pm$ .001	.0222 $\pm$ .002	.3481 $\pm$ .019	.2434 $\pm$ .027	.2824 $\pm$ .028	.5767 $\pm$ .030	.2096 $\pm$ .012	.2558 $\pm$ .016	.5500 $\pm$ .008

Table 1: Joint faithfulness. Deletion AUC (lower is better) and insertion AUC (higher is better). Values are mean

\pm

bootstrap std. “Joint” and “Rank-1” evaluate position-channel deletion; “Ins.” evaluates grid-sum insertion. Best scores per column are in bold . Upper block: per-channel mean baseline; lower block: zero baseline.

Figure 3: Spatial and spectral aggregation views. Panels (c) and (f) show spatial localization using mIoU at different coverage thresholds. Panels (b) and (e) report spectral Hit@K, while panels (a) and (d) show channel deletion AUC (lower is better).

Figure 4: Qualitative examples of spectral-spatial attribution. Representative CUB-200, HLS Burn Scar, and EuroSAT examples showing channel-aggregated maps and selected channel-level attributions. SSI illustrates localized channel evidence that is not captured by a single heatmap or a global channel ranking.

	Deletion $\downarrow$				Spatial mIoU $\uparrow$			Hit Rate $\uparrow$
Method	Joint	Rk1	Ch.	Ins. $\uparrow$	@5%	@15%	@25%	$K{=}1$	$K{=}3$	$K{=}5$
SSI	.155 $\pm$ .008	.166 $\pm$ .010	.218 $\pm$ .009	.664 $\pm$ .008	.311 $\pm$ .012	.387 $\pm$ .014	.332 $\pm$ .017	.317 $\pm$ .007	.603 $\pm$ .016	.887 $\pm$ .016
MC-Owen	.173 $\pm$ .010	.254 $\pm$ .017	.261 $\pm$ .017	.627 $\pm$ .009	.299 $\pm$ .018	.254 $\pm$ .017	.224 $\pm$ .017	.287 $\pm$ .011	.460 $\pm$ .017	.730 $\pm$ .013

Table 2: Observed-context SSI vs. context-averaged MC-Owen on HLS Burn Scar. Under the forward-pass counting convention of Appendix B.3 , MC-Owen with

M{=}200

block-respecting permutations uses up to

76{,}800

evaluations, and SSI uses up to

G\cdot K(C{+}1){=}4{,}480

evaluations. Grid-sum insertion ranks positions by channel-summed attribution. Values are mean

\pm

bootstrap std.

	Deletion $\downarrow$				Spatial mIoU $\uparrow$			Hit Rate $\uparrow$
Method	Joint	Rk1	Ch.	Ins. $\uparrow$	@5%	@15%	@25%	$K{=}1$	$K{=}3$	$K{=}5$
SSI	.155 $\pm$ .008	.166 $\pm$ .010	.218 $\pm$ .009	.664 $\pm$ .008	.311 $\pm$ .012	.387 $\pm$ .014	.332 $\pm$ .017	.317 $\pm$ .007	.603 $\pm$ .016	.887 $\pm$ .016
Expected Gradients	.185 $\pm$ .010	.238 $\pm$ .015	.275 $\pm$ .017	.637 $\pm$ .010	.292 $\pm$ .016	.284 $\pm$ .016	.236 $\pm$ .018	.287 $\pm$ .005	.510 $\pm$ .020	.720 $\pm$ .012
Guided IG	.185 $\pm$ .011	.244 $\pm$ .016	.280 $\pm$ .017	.630 $\pm$ .011	.296 $\pm$ .016	.283 $\pm$ .015	.233 $\pm$ .018	.293 $\pm$ .005	.500 $\pm$ .019	.703 $\pm$ .010
FullGrad	.278 $\pm$ .014	.277 $\pm$ .014	.581 $\pm$ .013	.543 $\pm$ .015	.131 $\pm$ .018	.135 $\pm$ .015	.128 $\pm$ .011	.000 $\pm$ .000	.333 $\pm$ .000	.667 $\pm$ .000

Table 3: Additional gradient-based baselines on HLS Burn Scar. All methods are evaluated using the same position-channel player space. Spatial mIoU is reported at top-5%, top-15%, and top-25% coverage. Values are mean

\pm

reported variability. Best scores per column are in bold .

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Formula	CUB-200	EuroSAT	HLS
		$(16{\times}16,\,C{=}3)$	$(8{\times}8,\,C{=}13)$	$(8{\times}8,\,C{=}6)$
IG	$300$	300	300	300
DeepSHAP	$1{,}000$	1,000	1,000	1,000
KernelSHAP	$2\|\mathcal{P}\|{+}2048$	3,584	3,712	2,816
RISE	$8000{+}C$	8,003	8,013	8,006
SSI	$G\cdot\min(2^{C},\,K(C{+}1))$	2,048	8,960	4,480

Appendix

Table 4: Forward-pass evaluations per image.

|\mathcal{P}|{=}G{\times}C

is the total number of position-channel players. SSI evaluations are reported under a worst-case convention that does not assume cached endpoint states; cached implementations further reduce SSI’s cost. MC-Owen is shown at

M{=}200

block-respecting permutations.

Method	ROAD LeRF $\uparrow$	ROAD MoRF $\downarrow$
SSI	.657 $\pm$ .010	.1143 $\pm$ .0058
MC-Owen	.629 $\pm$ .009	.1275 $\pm$ .0058
RISE	.624 $\pm$ .010	.1293 $\pm$ .0067
IG	.623 $\pm$ .010	.1316 $\pm$ .0058
DeepSHAP	.622 $\pm$ .009	.1325 $\pm$ .0057
Expected Gradients	.620 $\pm$ .010	.1354 $\pm$ .0060

Appendix

Table 5: ROAD evaluation on HLS Burn Scar. LeRF AUC (higher is better) and MoRF AUC (lower is better). Values are mean

\pm

reported variability, as in the author rebuttal. Best scores per column are in bold .

Method	Joint.Del $\downarrow$	Rk1.Del $\downarrow$	GridSum.Ins $\uparrow$
SSI	.1544 $\pm$ .0082	.1588 $\pm$ .0097	.6310 $\pm$ .0077
IG	.2499 $\pm$ .0159	.2792 $\pm$ .0171	.5739 $\pm$ .0089
DeepSHAP	.2481 $\pm$ .0145	.2793 $\pm$ .0176	.5835 $\pm$ .0089
KernelSHAP	.2129 $\pm$ .0149	.2534 $\pm$ .0179	.5651 $\pm$ .0080
RISE	.4320 $\pm$ .0188	.4330 $\pm$ .0190	.5327 $\pm$ .0070

Appendix

Table 6: Faithfulness test on HLS Burn Scar with

4{\times}4

grid. Values are mean

\pm

bootstrap std. Best in bold .

Method	Joint.Del $\downarrow$	Rk1.Del $\downarrow$	GridSum.Ins $\uparrow$
SSI	.0239 $\pm$ .0011	.0264 $\pm$ .0013	.7267 $\pm$ .0111
IG	.0257 $\pm$ .0012	.0284 $\pm$ .0015	.6515 $\pm$ .0149
DeepSHAP	.0259 $\pm$ .0012	.0279 $\pm$ .0013	.6511 $\pm$ .0146
KernelSHAP	.0263 $\pm$ .0012	.0285 $\pm$ .0013	.6900 $\pm$ .0124
RISE	.0329 $\pm$ .0017	.0329 $\pm$ .0015	.7109 $\pm$ .0113

Appendix

Table 7: Faithfulness test on CUB-200 with

4{\times}4

grid. Values are mean

\pm

bootstrap std. Best in bold .

Method	Joint.Del $\downarrow$	Rk1.Del $\downarrow$	GridSum.Ins $\uparrow$
SSI	.0123 $\pm$ .0007	.0154 $\pm$ .0010	.7567 $\pm$ .0116
IG	.0152 $\pm$ .0008	.0169 $\pm$ .0010	.5215 $\pm$ .0213
DeepSHAP	.0151 $\pm$ .0008	.0170 $\pm$ .0010	.5204 $\pm$ .0221
KernelSHAP	.0169 $\pm$ .0008	.0240 $\pm$ .0016	.5688 $\pm$ .0188
RISE	.0225 $\pm$ .0013	.0225 $\pm$ .0012	.7425 $\pm$ .0118

Appendix

Table 8: Faithfulness test on CUB-200 with

8{\times}8

grid. Values are mean

\pm

bootstrap std. Best in bold .

Method	Joint.Del $\downarrow$	Rk1.Del $\downarrow$	GridSum.Ins $\uparrow$
SSI	.1295 $\pm$ .0158	.1425 $\pm$ .0155	.7096 $\pm$ .0195
IG	.1957 $\pm$ .0233	.1745 $\pm$ .0217	.6731 $\pm$ .0224
DeepSHAP	.1963 $\pm$ .0234	.1750 $\pm$ .0217	.6730 $\pm$ .0223
KernelSHAP	.1416 $\pm$ .0199	.1458 $\pm$ .0181	.6432 $\pm$ .0240
RISE	.2116 $\pm$ .0272	.2120 $\pm$ .0263	.7007 $\pm$ .0188

Appendix

Table 9: Faithfulness test on EuroSAT with

4{\times}4

grid. Values are mean

\pm

bootstrap std. Best in bold .

Method	@0.1	@0.2	@0.3	@0.4	@0.5	Avg.
SSI	.2208 $\pm$ .0102	.3103 $\pm$ .0080	.3592 $\pm$ .0082	.3893 $\pm$ .0102	.4036 $\pm$ .0118	.3366 $\pm$ .0044
IG	.2148 $\pm$ .0083	.2926 $\pm$ .0053	.3228 $\pm$ .0054	.3342 $\pm$ .0079	.3361 $\pm$ .0099	.3001 $\pm$ .0034
DeepSHAP	.2150 $\pm$ .0079	.2929 $\pm$ .0050	.3220 $\pm$ .0052	.3333 $\pm$ .0074	.3362 $\pm$ .0101	.2999 $\pm$ .0033
KernelSHAP	.1061 $\pm$ .0024	.1787 $\pm$ .0030	.2267 $\pm$ .0044	.2674 $\pm$ .0071	.2970 $\pm$ .0095	.2152 $\pm$ .0026
RISE	.2099 $\pm$ .0090	.2888 $\pm$ .0074	.3293 $\pm$ .0072	.3529 $\pm$ .0086	.3671 $\pm$ .0103	.3096 $\pm$ .0038

Appendix

Table 10: CUB-200 spatial localization. mIoU at top-coverage levels. Values are mean

\pm

bootstrap std. Higher is better. Best scores are in bold .

Method	@0.05	@0.10	@0.15	@0.20	@0.25	Avg.
SSI	.3109 $\pm$ .0113	.3886 $\pm$ .0131	.3874 $\pm$ .0137	.3587 $\pm$ .0159	.3321 $\pm$ .0155	.3555 $\pm$ .0065
IG	.2722 $\pm$ .0175	.2699 $\pm$ .0162	.2552 $\pm$ .0162	.2396 $\pm$ .0168	.2245 $\pm$ .0174	.2523 $\pm$ .0075
DeepSHAP	.2850 $\pm$ .0177	.2776 $\pm$ .0157	.2583 $\pm$ .0157	.2441 $\pm$ .0178	.2270 $\pm$ .0174	.2584 $\pm$ .0076
KernelSHAP	.0809 $\pm$ .0102	.0932 $\pm$ .0101	.1075 $\pm$ .0115	.1183 $\pm$ .0115	.1228 $\pm$ .0127	.1045 $\pm$ .0050
RISE	.1589 $\pm$ .0108	.1913 $\pm$ .0116	.2145 $\pm$ .0119	.2346 $\pm$ .0113	.2585 $\pm$ .0131	.2116 $\pm$ .0048

Appendix

Table 11: HLS Burn Scar spatial localization. mIoU at top-coverage levels. Values are mean

\pm

bootstrap std. Higher is better. Best scores are in bold .

Method	Ch.Del. $\downarrow$	Hit@1	Hit@2	Hit@3	Hit@4	Hit@5
SSI	.2180 $\pm$ .0094	.3167 $\pm$ .0068	.4333 $\pm$ .0158	.6033 $\pm$ .0159	.7567 $\pm$ .0193	.8867 $\pm$ .0163
IG	.2724 $\pm$ .0178	.3000 $\pm$ .0092	.3667 $\pm$ .0148	.4200 $\pm$ .0154	.5767 $\pm$ .0148	.7600 $\pm$ .0148
DeepSHAP	.2723 $\pm$ .0174	.3000 $\pm$ .0103	.3667 $\pm$ .0145	.4233 $\pm$ .0160	.5767 $\pm$ .0147	.7600 $\pm$ .0151
KernelSHAP	.2568 $\pm$ .0161	.2833 $\pm$ .0120	.3567 $\pm$ .0126	.4633 $\pm$ .0163	.6067 $\pm$ .0167	.7533 $\pm$ .0145
RISE	.3166 $\pm$ .0204	.2333 $\pm$ .0153	.3067 $\pm$ .0200	.5000 $\pm$ .0192	.5533 $\pm$ .0159	.6700 $\pm$ .0034

Appendix

Table 12: HLS Burn Scar spectral aggregation. Lower channel deletion scores and higher hit rates are better. Values are mean

\pm

bootstrap std. Best scores are in bold .

Method	Ch.Del. $\downarrow$	Hit@1	Hit@2	Hit@3	Hit@4	Hit@5
SSI	.1039 $\pm$ .0165	.1768 $\pm$ .0231	.3472 $\pm$ .0236	.4545 $\pm$ .0323	.5328 $\pm$ .0289	.5960 $\pm$ .0264
IG	.1426 $\pm$ .0231	.0258 $\pm$ .0115	.2220 $\pm$ .0283	.3061 $\pm$ .0313	.3881 $\pm$ .0322	.4624 $\pm$ .0335
DeepSHAP	.1437 $\pm$ .0241	.0258 $\pm$ .0113	.2220 $\pm$ .0291	.3061 $\pm$ .0306	.3843 $\pm$ .0341	.4573 $\pm$ .0358
KernelSHAP	.1347 $\pm$ .0209	.0879 $\pm$ .0205	.2207 $\pm$ .0298	.3295 $\pm$ .0330	.4232 $\pm$ .0335	.5394 $\pm$ .0366
RISE	.1387 $\pm$ .0207	.0788 $\pm$ .0181	.1876 $\pm$ .0241	.2629 $\pm$ .0288	.3311 $\pm$ .0270	.4258 $\pm$ .0242

Appendix

Table 13: EuroSAT spectral aggregation. Lower channel deletion scores and higher hit rates are better. Values are mean

\pm

bootstrap std. Best scores are in bold .

Feature Attribution

Feature attribution estimates which input features or internal components contribute to a model’s predictions.

See all 234 papers

May 19, 2026

Spectral Integrated Gradients for Coarse-to-Fine Feature Attribution

Integrated Gradients (IG) is a widely adopted feature attribution method that satisfies desirable axiomatic properties. However, the choice of integration path significantly affects the quality of attributions, and the standard straight-line path introduces all input features simultaneously, often accumulating noisy gradients along the way. To address this limitation, we propose Spectral Integrated Gradients, which constructs integration paths based on singular value decomposition (SVD) of the baseline-to-input difference. By progressively activating singular components from largest to smallest, SIG introduces global structure before fine-grained details, naturally following a coarse-to-fine progression. Through extensive evaluation across diverse image classification datasets, we demonstrate that SIG produces cleaner attribution maps with reduced noise and achieves improved quantitative performance compared to existing path-based attribution methods. Our code is available at https://github.com/leekwoon/sig/.

Soyeon Kim, Seongwoo Lim, Kyowoon Lee +1

Korea Advanced Institute of Science and Technology Seongnam, Gyeonggi-do, Republic of Korea · INEEJI Corp. Seongnam, Gyeonggi-do, Republic of Korea

Gradient-Based Attribution Integrated Gradients Feature Attribution

Ask PDF

Jun 17, 2024

CHAOSMINING: Benchmarking Post-Hoc Attribution with Sparse Informative Features in High Dimensions

Post-hoc attribution is widely used to identify important model inputs, but evaluating whether these attributions identify truly informative features is difficult because real datasets rarely provide reliable ground truth. We introduce a multimodal benchmark containing symbolic tabular, vision, and audio tasks with known informative feature sets. In the main benchmark conditions, informative variables, spatial regions, or channels occupy fixed input coordinates while the remaining inputs provide irrelevant or distracting information. We use the benchmark to study how attribution quality depends on predictive performance, irrelevant-feature burden and structure, model configuration, and attribution mechanism, while separately measuring identification, stability, and computational cost. In most symbolic-data sweeps, informative-set identification co-varies with predictive performance, while the relative ordering of attribution methods remains largely stable. Across modalities, no method dominates all architectures and conditions, and greater attribution complexity does not consistently improve identification. Simple gradient attribution is often competitive at lower computational cost, while the vision and audio results show that architecture and the form of irrelevant content materially affect attribution quality.

Ge Shi, Fangyi Liu, Ziwen Kan +1

Department of Computer Science University of California, Davis

Gradient-Based Attribution Feature Attribution Explainability Evaluation

Ask PDF

Sep 22, 2026

MorphoSHAP: Rethinking the Unit of Attribution in Explanation for Deep Visual Models

Visual attribution methods typically explain predictions using pixels, superpixels, or regular patches. These representations can localize important regions, but provide limited information about their structure. We introduce MorphoSHAP, a model-agnostic post-hoc method that instead uses morphological shapes as the players of a Shapley attribution game. Using the Tree of Shapes, each shape is described by its scale, geometry, and signed contribution, providing explanations of where the evidence lies, what type of structure carries it, and how strongly it affects the prediction. This shared morphological vocabulary enables spatial, textual, and global class-level explanations beyond image-specific heatmaps. To the best of our knowledge, MorphoSHAP is the first SHAP-based image attribution framework to combine these different forms of explanation. Across five diverse datasets and three architectures, MorphoSHAP achieves strong insertion/deletion performance and outperforms competing attribution methods on several benchmarks. Finally, a user study shows that MorphoSHAP provides explanations that are easy to use and are preferred over standard attribution baselines.

Anirudh Prabhakaran, Alexandre Rocchi, Gianni Franchi

AMIAD, Pˆole Recherche, Palaiseau

Feature Attribution SHAP Feature Attribution Explainable Image Classification Explainability Evaluation

Ask PDF

Abstract

Feature Attribution

Explore similar work

Spectral Integrated Gradients for Coarse-to-Fine Feature Attribution

CHAOSMINING: Benchmarking Post-Hoc Attribution with Sparse Informative Features in High Dimensions

MorphoSHAP: Rethinking the Unit of Attribution in Explanation for Deep Visual Models