Post-hoc explanation methods for video action recognition typically conflate spatial appearance and temporal dynamics into a single saliency or importance map. We propose a perturbation-based framework that disentangles these contributions into a spatial appearance mask (Ma) and a temporal motion mask (Mm). The two masks are jointly optimized against a frozen action recognition model through factored perturbation architecture, with nearest frame replacement to identify important temporal dynamics. A separation penalty discourages the appearance mask from spending budget on frames that the motion mask has already discarded. A ReLU-based area constraint penalizes explanations larger than an assigned budget. To evaluate these explanations, we introduce a separation protocol that sweeps spatial pixels and temporal frames independently, backed by proofs characterizing the proposed method. Across EPIC-Kitchens-55, EGTEA Gaze+, and Something-Something V2, our method attains the lowest deletion AUC against leading baselines while producing masks roughly five times sparser.
Figures & tables
Figure 1: Factored explanations separate appearance (noun: plate) evidence, where the model looks , from motion evidence (verb: put-down), when the model acts. Standard saliency instead yields a single unified map that conflates appearance and motion.
Insertion AUC
Deletion AUC
Fidelity+
Fidelity −
Mask size
Method
App ↑
Mot ↑
App ↓
Mot ↓
App ↑
Mot ↑
App ↑
Mot ↑
Ma↓
Mm↓
EGTEA (N = 400), TSM/TimeSformer
EP + Mm
.72 /.65
.79/.71
.41 / .24
.77/ .67
.87/.85
.87/.80
.14/ .40
.11/.17
50.5 / 50.7
61.6 / 61.6
EP
.70/.65
–
.43 / .27
–
.91 / .96
–
.12/ .33
–
50.6 / 50.8
–
STEP + Mm
.77 / .72
.84 / .82
.46 / .37
.79/.74
.97 / .97
.96 / .96
.43 / .72
.12/.20
60.6/69.0
81.8/83.3
STEP
.70/ .73
–
.51/.41
–
.99 / .99
–
.28 / .57
–
56.4/65.4
–
Table 1: Explanation quality comparison. Each cell reports TSM/TimeSformer. App/Mot denote appearance/motion; Ins/Del denote insertion/deletion AUC. F+ and F− are fidelity metrics; Ma , Mm are active pixel/frame fractions. Color intensity within each dataset/model column maps to relative performance (darker = better, computed by min–max normalization, bucketed into 6 levels). Paired significance tests and standard deviations are reported in Appendix B.7 .
Figure 2: Qualitative figure showing (a) Input video clip; (b) Motion mask Mm ; (c) The video clip with the winning nearest-frame replacement that resulted in Mm ; (d) Appearance mask Ma produced by our approach, STEP and EP. Highlighted regions in Ma show the top 15% of important regions.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Qualitative comparison of explanation methods on a single Epic-Kitchens clip for action put-cream and wash-knife . Each column is one of the 8 uniformly sampled frames. Row 1 (original): input frames. Row 2 (STEP [ 15 ] ): spatial mask Ma learned by STEP; the highlighted region is shown sharp while the complement is replaced by a Gaussian blur ( 31×31 , σ=20 ), matching the perturbation seen by the model. Row 3 (Extremal Perturbation [ 5 ] ): same rendering for the EP spatial mask. Row 4 (Ours): our disentangled spatial mask under the same blur-blend rendering; frame borders are blue when the temporal mask Mm>0.5 (model-relevant frame) and gray when frozen (nearest-active-frame replacement applied). Values above each frame give the soft Mm weight.
Figure 4: Cumulative ablation on a frozen TSM trained on EGTEA Gaze+. Each bar adds a component on top of the previous configuration.
Precision
Ins. (5%)
Ins. (60%)
Change
Del. (5%)
Del. (60%)
Change
0.10
0.282
0.328
+16.4%
0.268
0.236
−11.8%
0.30
0.293
0.396
+35.5%
0.258
0.228
−11.7%
0.50
0.303
0.444
+46.6%
0.252
0.224
−11.1%
0.70
0.311
0.484
+55.5%
0.244
0.223
−8.8%
0.90
0.314
0.505
+60.8%
0.242
0.218
−9.9%
Appendix
Table 2: Insertion vs. deletion AUC under a 12× increase in mask size at fixed precision.
Metric
Full
w/o semantic split
Δ
App. Del. AUC ↓
0.128
0.152
+0.024
Mot. Del. AUC ↓
0.234
0.285
+0.051
App. F−↑
0.323
0.295
−0.028
Mot. F−↑
0.656
0.616
−0.040
App. Ins. AUC ↑
0.630
0.673
+0.043
Mot. Ins. AUC ↑
0.703
0.778
+0.075
Appendix
Table 3: Effect of removing the semantic split. Δ = (w/o split) − (full).
Mask
Intended gradient
Cross-axis gradient
Leak ratio (mean / median)
Ma
∥∂Lnoun/∂Ma∥=1.45×10−2
∥∂Lverb/∂Ma∥=1.38×10−2
0.88 / 0.85
Mm
∥∂Lverb/∂Mm∥=6.50×10−1
∥∂Lnoun/∂Mm∥=3.20×10−1
5.31 / 1.23
Appendix
Table 4: Gradient magnitudes at convergence. Ratios are computed per clip and then aggregated, so they are not the ratio of the column means.
Metric
With flow ( λF=1.312 )
Without flow ( λF=0 )
Mot. Ins. AUC ↑
0.334
0.186
Mot. Del. AUC ↓
0.152
0.096
Mot. F+↑
0.465
0.175
Mot. F−↑
0.613
0.647
Mm size (%)
67.1
37.7
r(Mm,flow)
0.37
-0.212
Appendix
Table 5: Flow-alignment ablation on 50 SSv2 clips (TSM). r : Pearson correlation between Mm and per-frame flow magnitude.
Dataset
Method
App. Ins. ↑
Mot. Ins. ↑
App. Del. ↓
Mot. Del. ↓
SSv2
EP
0.321±0.002
-
0.143±0.001
-
EP + Mm
0.266±0.003
0.402±0.007
0.085±0.003
0.399±0.007
STEP
0.347±0.003
-
0.172±0.001
-
STEP + Mm
0.357±0.009
0.488±0.001
0.147±0.005
0.444±0.012
Unified Mask
0.409±0.008
-
0.186±0.005
-
Ours
0.318±0.007
0.25±0.003
0.11±0.004
0.112±0.002
Appendix
Table 6: Mean ± std over three seeds (TSM) on the SSV2 dataset.
Dataset
Baseline
App. Ins.
Mot. Ins.
App. Del.
Mot. Del.
SSv2
EP + Mm
+0.052∗p=2.3×10−3
−0.146∗p=3.6×10−9
−0.028∗p=3.6×10−5
+0.286∗p=2.1×10−28
EP
−0.020p=0.30
–
+0.002p=0.51
–
STEP + Mm
−0.039∗p=8.6×10−3
−0.232∗p=6.7×10−19
+0.034∗p=0.033
+0.331∗p=8.0×10−32
STEP
−0.029∗p=0.021
–
+0.059∗p=1.9×10−4
–
Unified Mask
−0.091∗p=4.4×10−6
–
+0.072∗p=5.2×10−15
–
EGTEA
EP + Mm
+0.110∗p=3×10−22
+0.106∗p=7×10−15
+0.060∗p=6×10−13
+0.368∗p=3×10−51
Appendix
Table 7: Paired comparison of Ours against each baseline (TSM). Each cell gives the mean per-item difference, oriented so that positive values favor our method, and the Wilcoxon signed-rank p -value; ∗ : p<0.05 . Items are clips for SSv2 ( n=200 ) and EGTEA ( n=400 ), and videos for EPIC-55 ( n=89 – 93 ; per-segment scores averaged within each video and over seeds). Motion metrics are omitted for baselines without a motion mask, as in Table 1 . Differences are computed on the seed-averaged per-item scores used for this test, so they can differ slightly from differences of the rounded means in Table 1 .
Predicted verb
Clips
Mean active Mm frames
Mean verb confidence
Wash
5
0.80
0.895
Cut
10
1.40
0.849
Move around
4
1.50
0.776
Close
3
1.67
0.875
Appendix
Table 8: Verb classes ranked by few active motion frames (of 8) and high confidence (TSM).
Figure 5: Figure showing how hyperparameters (λA,λM,λS,λD) were picked with a held out validation set using a combination of grid search and Bayesian optimization on Optuna for minimizing deletion AUC and maximizing Insertion AUC across 100 trials on the EGTEA Gaze+ dataset with the TSM model.
(a) Loss weights and target areas (per dataset)
EPIC-Kitchens-55
EGTEA Gaze+
Something-Something V2
λA (M a area)
13.85
0.3841
0.133
λM (M m area)
9.44
0.786
0.263
λS (L K smoothness)
0.143
0.0107
0.0877
λD (disentanglement)
7.026
38.88
38.88
λF (flow alignment)
N/A
N/A
1.312
Appendix
Table 9: Optimization hyperparameters for our disentangled mask (Ma,Mm) . Loss weights (λA,λM,λS,λD,λF) were tuned on a held-out split via Bayesian optimization. Optimizer settings, schedule, and temperature anneal are identical across datasets and backbones.
Dataset
Param.
Value
App. Ins. ↑
Mot. Ins. ↑
App. Del. ↓
Mot. Del. ↓
EGTEA
va
0.26
.699
.730
.193
.270
0.40
.696
.729
.196
.267
0.53 †
.700
.730
.192
.270
0.66
.694
.729
.197
.270
0.79
.696
.731
.198
.270
vm
0.31
.695
.727
.162
.267
Appendix
Table 10: Hyperparameter sensitivity (TSM; 20 clips × 3 seeds, mean AUC). One hyperparameter is varied at a time; † marks the tuned value used for all other rows. This study uses its own clip subset, so absolute values are not directly comparable to Table 1 .