READ: Reading the Selective Scan for Faithful Video Attribution

Magdalena Trędowicz; Piotr Kubaty; Arkadiusz Lewicki; Jacek Tabor; Łukasz Struski

cs.CVOct 9, 2026

READ: Reading the Selective Scan for Faithful Video Attribution

Authors: Magdalena Trędowicz, Piotr Kubaty, Arkadiusz Lewicki, Jacek Tabor, Łukasz Struski

Organizations: Jagiellonian University, Kraków, Poland · University of Information Technology and Management in Rzeszów, Poland

Abstract

Selective state-space models are an efficient approach to long-sequence video modeling. However, their recurrent computation makes it difficult to determine which video tokens drive a prediction. Existing attribution methods often require additional backward passes or repeated model evaluations, making them substantially more expensive than the forward inference they aim to explain. In this work, we introduce READ, a parameter-free attribution method for selective state-space models that unrolls the recurrence at the position read by the classifier. READ traces the contribution accumulated in the recurrent state back to the video tokens that contribute to the final prediction. By anchoring attribution at the classifier's readout, it produces a single spatiotemporal relevance map from which frame- and region-level relevance can be obtained directly. Despite requiring only one forward pass, READ achieves competitive faithfulness and localisation compared with substantially more expensive attribution methods. Experimental results show that READ avoids the systematic relevance biases of existing methods while recovering meaningful temporal and class-specific structure across datasets and backbones. More broadly, our results show that the recurrence underlying selective state-space models is not only an efficient mechanism for inference, but also a direct source of interpretable evidence for their predictions.

Figures & tables

Figure 2: Overview of READ. Top: the frozen VideoMamba backbone. The class token ( cls ) closes the backward scan, making its final state the classifier readout. Bottom: READ decomposes this readout in a single forward pass. Unrolling the selective scan gives an exact decomposition into per-token contributions. Each contribution is projected onto the target class through the readout and the block-wise class direction. The resulting scores are aggregated across blocks. We correct the systematic offset introduced by the causal convolution, account for dominant channel contributions, and suppress cancellations across blocks. The resulting signed scores form a single spatiotemporal relevance map and temporal profile. Red denotes evidence for the target class, blue evidence against it. Example shows frames of a Kinetics-400 clip.

Method	Fwd.	Bwd.	Time (ms) $\downarrow$	Mem. (GB) $\downarrow$	Aux. mem.
Layer-CAM	1	1	$194$	$5.67$	$\mathcal{O}(N_{L}L)$
IntGrad	16	16	$2\,017$	$5.47$	$\mathcal{O}(N_{L}L)$
MambaLRP	1	1	$19\,807$	$6.08$	$\mathcal{O}(N_{L}L)$
MambaLRP ∗	1	1	$143$	$7.01$	$\mathcal{O}(N_{L}L)$
READ (ours)	1	0	$\mathbf{125}$	$3.47$	$\mathcal{O}(L)$
READ (ours) ∗	1	0	$172$	$\mathbf{0.06}$	$\mathcal{O}(L)$

Table 1: Computational cost of attribution. Measurements for the VideoMamba-M backbone on a single RTX 4090 with batch size 1, averaged over 25 runs after warm-up. Time and memory report the additional cost of computing an attribution map beyond a standard classification forward pass, which takes 37,ms and 0.51,GB. Fwd./Bwd. denotes the number of forward and backward passes required by each method. The ∗ variants use optimized implementations: fused selective-scan kernels for MambaLRP and streaming accumulation for READ.

Method	peak in step $\uparrow$			mass in step $\uparrow$			tIoU $\uparrow$
	Ti	S	M	Ti	S	M	Ti	S	M
Centre	0.073	0.073	0.073	—	—	—	0.106	0.106	0.106
Random	0.503	0.443	0.520	0.501	0.500	0.500	0.181	0.173	0.173
Layer-CAM	0.370	0.177	0.203	0.454	0.344	0.377	0.173	0.160	0.162
IntGrad	0.470	0.460	0.487	0.500	0.502	0.500	0.175	0.178	0.181
MambaLRP	0.503	0.473	0.470	0.565	0.550	0.557	0.223	0.219	0.215

Table 2: Temporal localisation on COIN. Results on 300 test clips for each VideoMamba backbone. Ti, S, and M denote the Tiny, Small, and Medium backbones, respectively. Peak in step is the fraction of clips for which the most-attributed frame falls within the annotated procedure step; Mass in step is the fraction of positive attribution mass within the step; and tIoU is the temporal intersection-over-union between the top-

25\%

attributed frames and the annotated step. Chance denotes the expected value for a uniformly distributed temporal attribution. Bold indicates the best attribution method in each column.

	Temporal $\uparrow$			Spatial $\uparrow$			Spatio-temporal $\uparrow$
Method	SSv2	K-400	COIN	SSv2	K-400	COIN	SSv2	K-400	COIN
Centre	$-0.010$	$-0.010$	$-0.003$	$0.101$	$0.053$	$0.009$	$0.100$	$0.053$	$0.006$
Random	$-0.025$	$-0.024$	$-0.018$	$-0.031$	$-0.029$	$-0.032$	$-0.031$	$-0.029$	$-0.032$
Layer-CAM	$0.095$	$0.049$	$0.063$	$0.281$	$0.241$	$0.209$	$0.302$	$0.258$	$0.243$
IntGrad	$0.081$	$0.074$	$0.138$	$0.205$	$0.236$	$0.250$	$0.217$	$0.245$	$0.265$
MambaLRP	$\mathbf{0.184}$	$0.139$	$0.253$	$\mathbf{0.451}$	$\mathbf{0.536}$	$\mathbf{0.564}$	$\mathbf{0.474}$	$\mathbf{0.547}$	$\mathbf{0.610}$

Table 3: Faithfulness at VideoMamba-S: the insertion

-

MoRF AUC gap,

\mathrm{AUC}_{\mathrm{Ins}}-\mathrm{AUC}_{\mathrm{MoRF}}

, at the temporal, spatial and spatio-temporal levels (higher is better). Bold marks the best result per column. The grey rows are controls that see neither the model nor the target class.

Figure 3: Attribution concentration and spatial-temporal coherence. Left: cumulative attribution mass as a function of the fraction of spatio-temporal cells, averaged over 120 Kinetics-400 clips. Curves further to the left indicate that the attribution mass is concentrated on fewer cells. The dashed horizontal line marks $50\%$ attribution mass. Right: attribution maps for the same clip at a $\frac{1}{16}$ attribution budget, shown every second frame. READ concentrates attribution on fewer frames and more spatially compact regions than the compared attribution method, while both methods select the same number of cells. Full quantitative results across SSv2, Kinetics-400, and COIN are reported in Sec. E.2 .

Figure 4: Qualitative comparison on Kinetics-400 , class “playing tennis”, correctly classified by VideoMamba-S. The nine of the $16$ frames (index above each column); rows show the input, READ, three baselines. One diverging colormap for all maps (red: evidence for the predicted class; blue: against).

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

	Something-Something v2			Kinetics-400
Frozen from $A$	corr. $A$	corr. $B$	$=A$ (%)	corr. $A$	corr. $B$	$=A$ (%)
all four	$0.986$	$0.112$	$92$	$0.975$	$0.055$	$92$
$z$ only	$0.977$	$0.123$	$88$	$0.946$	$0.071$	$92$
$\Delta$ , $z$	$0.985$	$0.118$	$92$	$0.973$	$0.059$	$98$
$\Delta$ , $B$ , $C$ (no $z$ )	$0.157$	$0.261$	$2$	$0.075$	$0.324$	$0$
$\Delta$ only	$0.143$	$0.351$	$2$	$0.005$	$0.371$	$0$

Appendix

Table 4: Transplanting fields between clips. The content of clip

B

is processed with the listed fields frozen from clip

A

. corr.

A

and corr.

B

are the correlations of the resulting logits with those of each source clip, and

=A

is the percentage of pairs whose predicted class is

A

’s;

40

pairs per dataset. Freezing

z

alone transfers the prediction. Freezing the selective dynamics without it does not. The grey row is the null control and must follow

B

exactly, which it does.

direction only, no post-processing
	Something-Something v2			Kinetics-400
Direction	temporal	spatial	joint	temporal	spatial	joint
endpoint projection	$0.170$ (+6.4)	$0.356$ (-5.0)	$0.383$ (-4.2)	$0.127$ (-9.6)	$0.399$ (-16.7)	$0.419$ (-17.0)
path-integrated	$0.174$ (+8.0)	$0.364$ (+0.4)	$0.393$ (+2.7)	$0.128$ (-9.7)	$0.426$ (-4.7)	$0.443$ (-6.3)
full estimator
classifier row	$0.156$ (+0.7)	$0.348$ (-11.1)	$0.373$ (-11.3)	$0.130$ (-9.8)	$0.396$ (-21.1)	$0.414$ (-22.0)

Appendix

Table 5: Ablation of the readout direction. Insertion–MoRF gap under the mask protocol; higher is better. The upper block omits post-processing. The lower block uses the full estimator and varies only the readout direction.

	Something-Something v2						Kinetics-400
	mask			prune			mask			prune
Configuration	tm	sp	jo	tm	sp	jo	tm	sp	jo	tm	sp	jo
$\kappa=96$ , $\beta=0.5$	$\mathbf{+9.7}$	$\mathbf{+3.7}$	$\mathbf{+6.4}$	$\mathbf{+5.9}$	$-2.3$	$-4.1$	$\mathbf{+4.4}$	$\mathbf{+2.7}$	$+0.1$	$+2.0$	$-6.6$	$-8.6$
$\kappa=96$ , $\beta=1$	$\mathbf{+9.1}$	$\mathbf{+5.0}$	$\mathbf{+7.8}$	$\mathbf{+5.5}$	$-1.2$	$-0.8$	$\mathbf{+5.1}$	$\mathbf{+5.2}$	$\mathbf{+2.9}$	$\mathbf{+3.2}$	$-3.1$	$-5.9$
$\kappa=128$ , $\beta=0.5$	$\mathbf{+10.1}$	$\mathbf{+4.2}$	$\mathbf{+6.1}$	$\mathbf{+6.3}$	$-2.5$	$-3.7$	$\mathbf{+3.9}$	$\mathbf{+2.9}$	$+0.7$	$+1.5$	$-5.9$	$-8.4$
$\kappa=128$ , $\beta=1$	$\mathbf{+8.7}$	$\mathbf{+4.5}$	$\mathbf{+7.0}$	$\mathbf{+6.1}$	$-1.2$	$-1.0$	$\mathbf{+4.8}$	$\mathbf{+5.2}$	$\mathbf{+3.5}$	$\mathbf{+3.7}$	$-2.5$	$-4.4$

Appendix

Table 6: Ablation of the channel budget

\kappa

and cancellation exponent

\beta

. Standardised paired differences relative to

\kappa=96

\beta=2

; positive values favour the variant. Bold marks values

\geq 2

and grey marks values

\leq-2

. The reference row is omitted. TM, SP, and JO denote temporal, spatial, and joint evaluation.

	SSv2		K-400		COIN
	Mass@ $\tfrac{1}{2}$ $\downarrow$	Contig. $\uparrow$	Mass@ $\tfrac{1}{2}$ $\downarrow$	Contig. $\uparrow$	Mass@ $\tfrac{1}{2}$ $\downarrow$	Contig. $\uparrow$
Centre	$0.364$	$1.00$	$0.364$	$1.00$	$0.364$	$1.00$
Random	$0.293$	$0.21$	$0.293$	$0.21$	$0.293$	$0.21$
Layer-CAM	$0.176$	$0.77$	$0.173$	$0.72$	$0.173$	$0.73$
IntGrad	$0.108$	$0.51$	$0.122$	$0.45$	$0.113$	$0.48$
MambaLRP	$0.066$	$0.87$	$0.086$	$0.81$	$0.071$	$0.85$

Appendix

Table 7: Attribution concentration and contiguity across datasets. Bold marks the best value among attribution methods. Grey rows denote content-independent controls.

Figure 5: Average faithfulness ranks across backbone sizes. Results aggregate 54 evaluation settings; lower ranks are better. Bars connect methods whose average ranks are not separated by the Nemenyi test at

\alpha=0.05

Panel of Figure 6 — Figure 6: Joint insertion and deletion curves across datasets and backbone sizes. Results use the mask protocol, with all methods evaluated on the same clips within each panel.

Figure 7: Classification accuracy under frame budgets. Top-1 accuracy with

k

selected frames retained and the remaining frames replaced by the clip mean. Dotted lines indicate full-clip accuracy. Uniform sampling selects evenly spaced frames.

Figure 8: Insertion curves on Something-Something v2. Mean target-class probability over 2000 clips under the mask protocol, at the temporal, spatial, and joint levels. Higher curves indicate better performance.

Figure 9: Deletion (MoRF) curves on Something-Something v2. The clips, protocol, and evaluation levels match Fig. 8 . Lower curves indicate better performance.

Figure 10: Average faithfulness ranks for VideoMamba-S. Results aggregate 18 evaluation settings; lower ranks are better. Bars connect methods whose average ranks are not separated by the Nemenyi test at

\alpha=0.05

	Localisation $\uparrow$			Sanity $\downarrow$
Method	Pointing	ST-IoU	Energy	Rand. wts.	Class sens.
Centre	$0.70$	$0.32$	—	$1.00$	$1.00$
Random	$0.44$	$0.19$	$0.49$	$1.00$	$1.00$
Layer-CAM	$0.66$	$0.29$	$0.68$	$0.06$	$0.71$
IntGrad	$0.70$	$0.24$	$0.64$	$\mathbf{0.01}$	$\mathbf{0.32}$
MambaLRP	$\mathbf{0.90}$	$\mathbf{0.35}$	$0.81$	$0.05$	$0.70$

Appendix

Table 8: Localisation and sanity checks on Something-Something v2 with VideoMamba-S. All methods are evaluated on the same 300 clips. Higher is better for localisation. Sanity checks report the absolute correlation with maps obtained after randomising all model weights ( rand. wts. ) and the correlation between maps for the predicted and runner-up classes ( class sens. ); lower is better for both. Bold marks the best attribution method in each column, excluding the content-independent controls. Dashes indicate quantities not measured.

	Pointing $\uparrow$			ST-IoU $\uparrow$			Energy $\uparrow$
Method	Ti	S	M	Ti	S	M	Ti	S	M
Centre	$0.70$	$0.70$	$0.70$	$0.32$	$0.32$	$0.32$	—	—	—
Random	$0.44$	$0.44$	$0.44$	$0.19$	$0.19$	$0.19$	$0.49$	$0.49$	$0.49$
Layer-CAM	$0.66$	$0.66$	$0.64$	$0.29$	$0.29$	$0.28$	$0.72$	$0.68$	$0.67$
IntGrad	$0.74$	$0.70$	$0.72$	$0.24$	$0.24$	$0.24$	$0.67$	$0.64$	$0.64$
MambaLRP	$\mathbf{0.89}$	$\mathbf{0.90}$	$\mathbf{0.90}$	$\mathbf{0.36}$	$\mathbf{0.35}$	$\mathbf{0.36}$	$0.81$	$0.81$	$0.83$

Appendix

Table 9: Spatial localisation across backbone sizes on Something-Something v2. Higher is better for all metrics. Bold marks the best attribution method per column, excluding the content-independent controls.

		sanity $\downarrow$			class $\to 0$
Dataset	Backbone	Layer-CAM	MambaLRP	READ	Layer-CAM	MambaLRP	READ
SSv2	Ti	$0.039$	$0.045$	$\mathbf{0.019}$	$0.746$	$-0.068$	$\mathbf{0.001}$
	S	$\mathbf{0.041}$	$0.048$	$0.055$	$0.698$	$-0.086$	$\mathbf{-0.028}$
	M	$0.080$	$0.047$	$\mathbf{0.027}$	$0.726$	$-0.116$	$\mathbf{0.005}$
Kinetics-400	Ti	$0.052$	$0.042$	$\mathbf{0.014}$	$0.751$	$\mathbf{-0.035}$	$-0.052$
	S	$\mathbf{0.058}$	$0.067$	$0.074$	$0.720$	$\mathbf{0.015}$	$-0.025$

Appendix

Table 10: Sanity and class specificity across datasets and backbones. Bold marks the lowest absolute correlation after weight randomisation ( Sanity ) and the correlation closest to zero for a uniformly random target class ( Class ) within each row.

Dataset	Backbone	chance	Layer-CAM	IntGrad	MambaLRP	READ (ours)
SSv2	Ti	6.2	29.0	17.7	23.3	3.3
	S	6.2	89.3	10.7	35.7	6.7
	M	6.2	89.0	18.3	28.0	8.3
Kinetics-400	Ti	6.2	36.7	26.0	44.3	12.3
	S	6.2	86.7	14.7	67.0	28.0
	M	6.2	87.3	25.3	55.0	26.3

Appendix

Table 11: First-frame concentration across datasets and backbones. Percentage of clips whose temporal relevance peaks at frame 0. Chance denotes the uniform reference

1/T

. Bold marks values below the threshold defined in the text.

Figure 11: Faithfulness and computational cost on Something-Something v2. The insertion–MoRF gap is plotted against runtime (top) and peak memory (bottom), with logarithmic cost axes. Each point represents a method and backbone pair; marker size indicates the backbone. Dashed lines show the empirical Pareto frontier.

Backbone	Method	Fwd.	Bwd.	total	aux.	peak	aux.	Auxiliary memory
				time (ms) $\downarrow$		memory (GB) $\downarrow$
Ti	Layer-CAM	1	1	$91$	$70$	$1.55$	$1.41$	$\mathcal{O}\!\left(N_{L}L\right)$
	IntGrad	16	16	$781$	$760$	$1.50$	$1.36$	$\mathcal{O}\!\left(N_{L}L\right)$
	MambaLRP	1	1	$14\,969$	$14\,947$	$1.67$	$1.53$	$\mathcal{O}\!\left(N_{L}L\right)$
	MambaLRP ∗	1	1	$83$	$62$	$1.88$	$1.74$	$\mathcal{O}\!\left(N_{L}L\right)$
	READ (ours)	1	0	$\mathbf{60}$	$\mathbf{38}$	$1.04$	$0.90$	$\mathcal{O}\!\left(L\right)$

Appendix

Table 12: Computational cost of one attribution map across backbone sizes. Auxiliary costs are relative to a classification-only forward pass. Stars denote fused MambaLRP and streaming READ implementations. Bold marks the lowest cost per backbone among attribution methods.

Panel of Figure 12 — Figure 12: READ explanations for two Kinetics-400 clips of “playing violin”. Both clips are correctly classified by VideoMamba-S. Each panel shows seven of the 16 input frames (top) and the corresponding attribution maps (bottom). Red and blue indicate positive and negative relevance, respectively, on a colour scale symmetric about zero.

Panel of Figure 13 — Figure 13: Something-Something v2. Classes, top to bottom: Throwing something , Opening something . Rows within each panel: input, READ, MambaLRP, Layer-CAM, RISE and the static centre control. The bottom strip is the temporal profile $\tau_{t}$ . All maps share one diverging colormap.

Panel of Figure 14 — Figure 14: Kinetics-400. Classes, top to bottom: swimming backstroke , cleaning shoes . Layout as in Fig. 13 .

Panel of Figure 15 — Figure 15: COIN. Classes, top to bottom: making a salmon , using a sewing machine . Layout as in Fig. 13 ; clips here have $32$ frames.

Feature Attribution

Feature attribution estimates which input features or internal components contribute to a model’s predictions.

See all 233 papers

Sufficient Frame Subsets for Video Understanding
Oct 18, 2026Lahiru WijayasinghaNeural Network Interpretability Video Understanding
Independent Researcher
A Good Initialization is All You Need for Faithful Visual Attribution
Jul 7, 2026Zihan Gu, Jiayu Wang, Hua Zhang +1Feature Attribution Perturbation-Based Feature Attribution
Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences
STAC: Selective Spatiotemporal Aggregation and Compression for Video Reasoning Segmentation
Jul 3, 2026Syed Ariff Syed Hesham, Yun Liu, Guolei Sun +4Video Reasoning Reasoning Segmentation
School of EEE, Nanyang Technological University, Singapore · Institute for Infocomm Research, A*STAR, Singapore · VCIP, CS, Nankai University, Tianjin, China +4

Oct 18, 2026J/K move · Enter open · S save

Sufficient Frame Subsets for Video Understanding

Lahiru Wijayasingha

Independent Researcher

Neural Network Interpretability Video Understanding Temporal Video Understanding Video Action Recognition

Open paper AskPDF arXiv

Understanding when during a video a model makes its critical observations is essential for interpreting video-understanding systems. However, existing attribution methods often spread evidence diffusely across near-duplicate frames. We introduce Sufficient Frame Subsets: compact sets of frames in which all other frames are masked, yet each subset independently preserves the model's prediction. We evaluate our method on two complementary benchmarks, UCF101 and Something-Something v2, using both CNN- and transformer-based architectures. Our results show that the proposed algorithm identifies highly compact and effective frame subsets for explaining model decisions. We also find that models capable of relying on a greater number of sufficient frame subsets tend to perform better, possibly because they are more robust, consistent with prior research. Finally, the extracted frame subsets help us analyze model failure cases, reveal which aspects of a video the model has learned to emphasize, and identify opportunities for improving future training procedures.

Abstract

Feature Attribution

Explore similar work

Sufficient Frame Subsets for Video Understanding

A Good Initialization is All You Need for Faithful Visual Attribution

STAC: Selective Spatiotemporal Aggregation and Compression for Video Reasoning Segmentation