READ: Reading the Selective Scan for Faithful Video Attribution

Understanding when during a video a model makes its critical observations is essential for interpreting video-understanding systems. However, existing attribution methods often spread evidence diffusely across near-duplicate frames. We introduce Sufficient Frame Subsets: compact sets of frames in which all other frames are masked, yet each subset independently preserves the model's prediction. We evaluate our method on two complementary benchmarks, UCF101 and Something-Something v2, using both CNN- and transformer-based architectures. Our results show that the proposed algorithm identifies highly compact and effective frame subsets for explaining model decisions. We also find that models capable of relying on a greater number of sufficient frame subsets tend to perform better, possibly because they are more robust, consistent with prior research. Finally, the extracted frame subsets help us analyze model failure cases, reveal which aspects of a video the model has learned to emphasize, and identify opportunities for improving future training procedures.

Faithful visual attribution identifies which image regions support a model prediction. Search-based perturbation methods lead the insertion--deletion faithfulness frontier by masking regions and measuring score changes, but they usually output a complete ordering of all regions. Many applications, especially MLLM attribution and repair, only need a compact top-$k$ evidence mask. We study this mask-first attribution problem. An exactly $k$-region mask is combinatorial: useful evidence can depend on interactions among fine regions. Coarse grouping can stabilize early search but aggregates redundant content, whereas one-step scoring can miss high-value combinations. We introduce two forward-only methods. CoPAIR uses a PhaseWin--Greedy gap diagnosis to construct coarse singleton/pair candidates that warm-start full-ordering search. TRACE directly searches fixed-cardinality fine-region masks with cross-entropy sampling, elite retention, and distribution updates, with a finite-budget recovery analysis. The resulting evidence set can be returned as a compact attribution mask or used to initialize Greedy or PhaseWin when a complete ranking is required. Across ImageNet classification with CLIP ViT-L/14, CLIP RN101, and ResNet-101, our initialized search methods establish a new state-of-the-art frontier for faithful full-ordering attribution under inclusive forward-call accounting. On POPE and RePOPE with Qwen2.5-VL-3B-Instruct and LLaVA-v1.5-7B, TRACE+Greedy gives the strongest search-based MLLM attribution results. Direct TRACE masks further achieve single-point RePOPE repair rates of $94.44\%$ and $96.00\%$, showing that compact evidence masks can be actionable attribution outputs, not merely prefixes of full rankings.

Video reasoning segmentation demands pixel-accurate object tracking across hundreds of frames under complex natural language queries, producing dense spatiotemporal tokens whose quadratic self-attention cost makes long-video processing prohibitive. Existing methods address this through token compression, yet typically operate on encoder features lacking temporal context, constraining selection before content redundancy can be reliably assessed. Informed compression requires contextual awareness, but acquiring that awareness at full resolution incurs the same quadratic cost compression aims to reduce. State-space models resolve this constraint, as their linear recurrence selectively conditions each token on temporal context at $\mathcal{O}(T)$ cost, producing representations where content redundancy becomes assessable. Building on this, Selective SpatioTemporal Aggregation and Compression (STAC) enriches features via decoupled bidirectional spatial and causal temporal scanning, leveraging recurrence-derived redundancy for hierarchical compression with adaptive thresholds optimised with segmentation objective. STAC achieves 85% token reduction and 1.8$\times$ speedup while surpassing compression-free baselines on reasoning segmentation benchmarks in a zero-shot streaming-compatible setting. Code is available here.