cs.CVNov 8, 2026

Disentangling Where and When:Factored Spatio-Temporal Explanations for Video Action Recognition

Authors: Bhavan Vasu, Giuseppe Raffa, Prasad Tadepalli

Organizations: Oregon State University · Intel Labs

Abstract

Post-hoc explanation methods for video action recognition typically conflate spatial appearance and temporal dynamics into a single saliency or importance map. We propose a perturbation-based framework that disentangles these contributions into a spatial appearance mask (MaM_a) and a temporal motion mask (MmM_m). The two masks are jointly optimized against a frozen action recognition model through factored perturbation architecture, with nearest frame replacement to identify important temporal dynamics. A separation penalty discourages the appearance mask from spending budget on frames that the motion mask has already discarded. A ReLU-based area constraint penalizes explanations larger than an assigned budget. To evaluate these explanations, we introduce a separation protocol that sweeps spatial pixels and temporal frames independently, backed by proofs characterizing the proposed method. Across EPIC-Kitchens-55, EGTEA Gaze+, and Something-Something V2, our method attains the lowest deletion AUC against leading baselines while producing masks roughly five times sparser.

Figures & tables

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. TraCE: Trajectory-Aware Concepts for Explainable Video Understanding

    Nov 4, 2026Wooil Lee, Junpyo Hong, Jongseo Lee +3Video UnderstandingConcept-Based Explanations

  2. TEMPURA: Temporal Event Masked Prediction and Understanding for Reasoning in Action

    May 2, 2025Jen-Hao Cheng, Yi-Hao Peng, Huapeng Zhou +11Temporal Action SegmentationTemporal Video Grounding