Post-hoc explanation methods for video action recognition typically conflate spatial appearance and temporal dynamics into a single saliency or importance map. We propose a perturbation-based framework that disentangles these contributions into a spatial appearance mask (Ma) and a temporal motion mask (Mm). The two masks are jointly optimized against a frozen action recognition model through factored perturbation architecture, with nearest frame replacement to identify important temporal dynamics. A separation penalty discourages the appearance mask from spending budget on frames that the motion mask has already discarded. A ReLU-based area constraint penalizes explanations larger than an assigned budget. To evaluate these explanations, we introduce a separation protocol that sweeps spatial pixels and temporal frames independently, backed by proofs characterizing the proposed method. Across EPIC-Kitchens-55, EGTEA Gaze+, and Something-Something V2, our method attains the lowest deletion AUC against leading baselines while producing masks roughly five times sparser.
Figures & tables
Figure 1: Factored explanations separate appearance (noun: plate) evidence, where the model looks , from motion evidence (verb: put-down), when the model acts. Standard saliency instead yields a single unified map that conflates appearance and motion.
Insertion AUC
Deletion AUC
Fidelity+
Fidelity −
Mask size
Method
App ↑
Mot ↑
App ↓
Mot ↓
App ↑
Mot ↑
App ↑
Mot ↑
Ma↓
Mm↓
EGTEA (N = 400), TSM/TimeSformer
EP + Mm
.72 /.65
.79/.71
.41 / .24
.77/ .67
.87/.85
.87/.80
.14/ .40
.11/.17
50.5 / 50.7
61.6 / 61.6
EP
.70/.65
–
.43 / .27
–
.91 / .96
–
.12/ .33
–
50.6 / 50.8
–
STEP + Mm
.77 / .72
.84 / .82
.46 / .37
.79/.74
.97 / .97
.96 / .96
.43 / .72
.12/.20
60.6/69.0
81.8/83.3
STEP
.70/ .73
–
.51/.41
–
.99 / .99
–
.28 / .57
–
56.4/65.4
–
Table 1: Explanation quality comparison. Each cell reports TSM/TimeSformer. App/Mot denote appearance/motion; Ins/Del denote insertion/deletion AUC. F+ and F− are fidelity metrics; Ma , Mm are active pixel/frame fractions. Color intensity within each dataset/model column maps to relative performance (darker = better, computed by min–max normalization, bucketed into 6 levels). Paired significance tests and standard deviations are reported in Appendix B.7 .
Figure 2: Qualitative figure showing (a) Input video clip; (b) Motion mask Mm ; (c) The video clip with the winning nearest-frame replacement that resulted in Mm ; (d) Appearance mask Ma produced by our approach, STEP and EP. Highlighted regions in Ma show the top 15% of important regions.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Qualitative comparison of explanation methods on a single Epic-Kitchens clip for action put-cream and wash-knife . Each column is one of the 8 uniformly sampled frames. Row 1 (original): input frames. Row 2 (STEP [ 15 ] ): spatial mask Ma learned by STEP; the highlighted region is shown sharp while the complement is replaced by a Gaussian blur ( 31×31 , σ=20 ), matching the perturbation seen by the model. Row 3 (Extremal Perturbation [ 5 ] ): same rendering for the EP spatial mask. Row 4 (Ours): our disentangled spatial mask under the same blur-blend rendering; frame borders are blue when the temporal mask Mm>0.5 (model-relevant frame) and gray when frozen (nearest-active-frame replacement applied). Values above each frame give the soft Mm weight.
Figure 4: Cumulative ablation on a frozen TSM trained on EGTEA Gaze+. Each bar adds a component on top of the previous configuration.
Precision
Ins. (5%)
Ins. (60%)
Change
Del. (5%)
Del. (60%)
Change
0.10
0.282
0.328
+16.4%
0.268
0.236
−11.8%
0.30
0.293
0.396
+35.5%
0.258
0.228
−11.7%
0.50
0.303
0.444
+46.6%
0.252
0.224
−11.1%
0.70
0.311
0.484
+55.5%
0.244
0.223
−8.8%
0.90
0.314
0.505
+60.8%
0.242
0.218
−9.9%
Appendix
Table 2: Insertion vs. deletion AUC under a 12× increase in mask size at fixed precision.
Metric
Full
w/o semantic split
Δ
App. Del. AUC ↓
0.128
0.152
+0.024
Mot. Del. AUC ↓
0.234
0.285
+0.051
App. F−↑
0.323
0.295
−0.028
Mot. F−↑
0.656
0.616
−0.040
App. Ins. AUC ↑
0.630
0.673
+0.043
Mot. Ins. AUC ↑
0.703
0.778
+0.075
Appendix
Table 3: Effect of removing the semantic split. Δ = (w/o split) − (full).
Mask
Intended gradient
Cross-axis gradient
Leak ratio (mean / median)
Ma
∥∂Lnoun/∂Ma∥=1.45×10−2
∥∂Lverb/∂Ma∥=1.38×10−2
0.88 / 0.85
Mm
∥∂Lverb/∂Mm∥=6.50×10−1
∥∂Lnoun/∂Mm∥=3.20×10−1
5.31 / 1.23
Appendix
Table 4: Gradient magnitudes at convergence. Ratios are computed per clip and then aggregated, so they are not the ratio of the column means.
Metric
With flow ( λF=1.312 )
Without flow ( λF=0 )
Mot. Ins. AUC ↑
0.334
0.186
Mot. Del. AUC ↓
0.152
0.096
Mot. F+↑
0.465
0.175
Mot. F−↑
0.613
0.647
Mm size (%)
67.1
37.7
r(Mm,flow)
0.37
-0.212
Appendix
Table 5: Flow-alignment ablation on 50 SSv2 clips (TSM). r : Pearson correlation between Mm and per-frame flow magnitude.
Dataset
Method
App. Ins. ↑
Mot. Ins. ↑
App. Del. ↓
Mot. Del. ↓
SSv2
EP
0.321±0.002
-
0.143±0.001
-
EP + Mm
0.266±0.003
0.402±0.007
0.085±0.003
0.399±0.007
STEP
0.347±0.003
-
0.172±0.001
-
STEP + Mm
0.357±0.009
0.488±0.001
0.147±0.005
0.444±0.012
Unified Mask
0.409±0.008
-
0.186±0.005
-
Ours
0.318±0.007
0.25±0.003
0.11±0.004
0.112±0.002
Appendix
Table 6: Mean ± std over three seeds (TSM) on the SSV2 dataset.
Dataset
Baseline
App. Ins.
Mot. Ins.
App. Del.
Mot. Del.
SSv2
EP + Mm
+0.052∗p=2.3×10−3
−0.146∗p=3.6×10−9
−0.028∗p=3.6×10−5
+0.286∗p=2.1×10−28
EP
−0.020p=0.30
–
+0.002p=0.51
–
STEP + Mm
−0.039∗p=8.6×10−3
−0.232∗p=6.7×10−19
+0.034∗p=0.033
+0.331∗p=8.0×10−32
STEP
−0.029∗p=0.021
–
+0.059∗p=1.9×10−4
–
Unified Mask
−0.091∗p=4.4×10−6
–
+0.072∗p=5.2×10−15
–
EGTEA
EP + Mm
+0.110∗p=3×10−22
+0.106∗p=7×10−15
+0.060∗p=6×10−13
+0.368∗p=3×10−51
Appendix
Table 7: Paired comparison of Ours against each baseline (TSM). Each cell gives the mean per-item difference, oriented so that positive values favor our method, and the Wilcoxon signed-rank p -value; ∗ : p<0.05 . Items are clips for SSv2 ( n=200 ) and EGTEA ( n=400 ), and videos for EPIC-55 ( n=89 – 93 ; per-segment scores averaged within each video and over seeds). Motion metrics are omitted for baselines without a motion mask, as in Table 1 . Differences are computed on the seed-averaged per-item scores used for this test, so they can differ slightly from differences of the rounded means in Table 1 .
Predicted verb
Clips
Mean active Mm frames
Mean verb confidence
Wash
5
0.80
0.895
Cut
10
1.40
0.849
Move around
4
1.50
0.776
Close
3
1.67
0.875
Appendix
Table 8: Verb classes ranked by few active motion frames (of 8) and high confidence (TSM).
Figure 5: Figure showing how hyperparameters (λA,λM,λS,λD) were picked with a held out validation set using a combination of grid search and Bayesian optimization on Optuna for minimizing deletion AUC and maximizing Insertion AUC across 100 trials on the EGTEA Gaze+ dataset with the TSM model.
(a) Loss weights and target areas (per dataset)
EPIC-Kitchens-55
EGTEA Gaze+
Something-Something V2
λA (M a area)
13.85
0.3841
0.133
λM (M m area)
9.44
0.786
0.263
λS (L K smoothness)
0.143
0.0107
0.0877
λD (disentanglement)
7.026
38.88
38.88
λF (flow alignment)
N/A
N/A
1.312
Appendix
Table 9: Optimization hyperparameters for our disentangled mask (Ma,Mm) . Loss weights (λA,λM,λS,λD,λF) were tuned on a held-out split via Bayesian optimization. Optimizer settings, schedule, and temperature anneal are identical across datasets and backbones.
Dataset
Param.
Value
App. Ins. ↑
Mot. Ins. ↑
App. Del. ↓
Mot. Del. ↓
EGTEA
va
0.26
.699
.730
.193
.270
0.40
.696
.729
.196
.267
0.53 †
.700
.730
.192
.270
0.66
.694
.729
.197
.270
0.79
.696
.731
.198
.270
vm
0.31
.695
.727
.162
.267
Appendix
Table 10: Hyperparameter sensitivity (TSM; 20 clips × 3 seeds, mean AUC). One hyperparameter is varied at a time; † marks the tuned value used for all other rows. This study uses its own clip subset, so absolute values are not directly comparable to Table 1 .
Video understanding models often rely on motion, yet their explanations can obscure which motion sources support a prediction. We identify motion-source under-specification as a limitation of existing video XAI: region-centric explanations can be source-ambiguous, while motion vocabularies restricted to single-person pose sequences are source-incomplete. We propose TraCE (Trajectory-Aware Concepts for Explainable Video Understanding), a concept-based framework that discovers source-flexible motion concepts by clustering optical-flow trajectories from training videos. These concepts represent dynamic evidence from people, objects, and interactions, while object and scene concepts account for static context. The resulting typed trajectory/object/scene interface supports class prediction for action recognition and question-conditioned concept matching for video question answering (VQA). We evaluate TraCE across five random seeds on six benchmarks: NTU RGB+D Mutual, Chiral SSv2, UCF-101, KTH, TOMATO, and STAR. Relative to non-interpretable baselines, TraCE maintains competitive task performance, with an average accuracy drop of 1.0 points for action recognition and a nominal gain of 0.4 points for VQA. Motion perturbation and concept-removal tests provide complementary evidence of directional semantic alignment and decision faithfulness, while sample-level interventions illustrate targeted model debugging. Our project page is available at https://leewooil.github.io/TraCE_project_page/.
Wooil Lee, Junpyo Hong, Jongseo Lee +3
Kyung Hee University · University of California San Diego
Not every moment in a video is equally useful for recognizing an action. A few decisive moments often carry the evidence: the instant an object moves, the peak of a dive, or one object placed on another. Existing video datasets rarely provide large-scale, action-conditioned labels for which short segments carry that evidence. This work asks whether vision-language models can identify meaningful moments at scale and whether those labels improve action recognition. We introduce Meaningful Moments (MM), a corpus of about 4.58 million VLM-generated per-segment importance scores for nearly 500,000 class-labeled videos spanning Something-Something v2 (SSv2), Kinetics-400 (K400), and Diving-48 across 622 action classes. We freeze the action classifier and vary only temporal sampling. An alpha-parameterized comparison of importance-led selection against an importance-inverted control separates gains from the VLM signal from gains caused by variable-density sampling alone. Across all three datasets, VLM-derived importance is informative: at the hard-cut endpoint, keeping VLM-important segments outperforms keeping VLM-designated filler. Usefulness depends on class structure. On Diving-48, importance-led fast-forwarding preserves full-video recognition while the inverted control collapses by roughly 30 percentage points. On K400, hard-cut selection beats uniform sampling by 1.45 percentage points, rising to 2.05 under threshold selection. On SSv2, the labels carry signal but do not recover full-video performance. Direct scoring matches greedy removal within its noise floor at roughly 25 times fewer oracle calls, and a four-VLM cross-oracle study shows consistent downstream trends. We release the importance labels and evaluation protocol under CC-BY-4.0; SSv2 class strings remain under CC-BY-NC-4.0.
Understanding causal event relationships and achieving fine-grained temporal grounding in videos remain challenging for vision-language models (VLMs). We propose TEMPURA (Temporal Event Masked Prediction and Understanding for Reasoning in Action), a two-stage training framework that enhances the video temporal understanding of VLMs. Inspired by infilling techniques in language modeling, TEMPURA first performs masked event prediction, learning to reconstruct missing events and generate step-by-step causal explanations from dense event annotations. It then learns video segmentation and dense captioning, decomposing videos into non-overlapping events with detailed, timestamp-aligned descriptions. We train TEMPURA on VER, our large-scale dataset of 500K videos annotated with temporally aligned event descriptions and structured reasoning steps. Experiments on video temporal grounding and highlight detection benchmarks show that TEMPURA substantially improves strong base VLMs across model families and scales, confirming that combining event-level reasoning with fine-grained temporal segmentation is an effective recipe for video temporal understanding.
Jen-Hao Cheng, Yi-Hao Peng, Huapeng Zhou +11
University of Washington · Carnegie Mellon University · National Yang Ming Chiao Tung University +1