Temporal Forgery Localization (TFL) aims to precisely identify manipulated segments in video and audio, offering strong interpretability for security and forensics. While recent State Space Models (SSMs) show promise in precise temporal reasoning, their use in TFL is hindered by ambiguous boundaries, sparse forgeries, and limited long-range modeling. We propose DeformTrace, which enhances SSMs with deformable dynamics and relay mechanisms to address these challenges. Specifically, Deformable Self-SSM (DS-SSM) introduces dynamic receptive fields into SSMs for precise temporal localization. To further enhance its capacity for temporal reasoning and mitigate long-range decay, a Relay Token Mechanism is integrated into DS-SSM. Besides, Deformable Cross-SSM (DC-SSM) partitions the global state space into query-specific subspaces, reducing non-forgery information accumulation and boosting sensitivity to sparse forgeries. These components are integrated into a hybrid architecture that combines the global modeling of Transformers with the efficiency of SSMs. Extensive experiments show that DeformTrace achieves state-of-the-art performance with fewer parameters, faster inference, and stronger robustness.
Figures & tables
Figure 1: Overview of our main contributions (we take the video sequence for illustration). (a) Vanilla SSM; (b) Deformable Self-SSM with learnable temporal offsets for flexible local sampling; (c) Visualization of hidden attention, where relay tokens expand receptive fields and maintain long-range dependencies; (d) Deformable Cross-SSM enables cross-sequence interactions by allowing each query token to partition the global state space into subspaces.
Figure 2: Illustration of the overall scheme of DeformTrace. Built on TadTR ( Liu et al. 2022 ) , DeformTrace integrates a multi-scale audio-visual feature extraction module, a deformable encoder for temporal modeling, and a deformable decoder for forgery localization and video-level classification. By incorporating deformable self- and cross-SSM modules, it combines Mamba’s efficient state updates with Transformer’s global modeling. Relay tokens with enhanced and cooperation losses help preserve long-range information dependencies during self-scanning.
Type
Method
Venue
Modality
mAP(%)
mAR(%)
0.5
0.75
0.95
Avg.
100
50
20
10
Avg.
Anchor- based
BMN
ICCV’19
V
24.0
7.60
0.10
10.6
53.3
41.2
31.6
26.9
38.3
BMN(I3D)
ICCV’19
V
10.6
1.70
0.00
4.10
48.5
44.4
37.1
31.6
40.4
Anchor- free
AGT
Arxiv’21
V
17.9
9.40
0.10
9.10
43.2
34.2
24.6
16.7
29.7
ActionFormer
ECCV’22
V
95.3
90.2
23.7
69.7
88.4
89.6
90.3
90.4
89.7
TriDet
CVPR’23
V
96.3
86.8
23.6
68.9
91.0
90.4
90.0
88.7
90.0
Table 1: Temporal forgery localization results on LAV-DF ( Cai et al. 2023 ) benchmark. The modality denotes the input type: V for visual, A for audio and AV for both. The best results are bolded , and second-best underlined .
Type
Method
Venue
Modality
mAP(%)
mAR(%)
0.5
0.75
0.9
0.95
Avg.
50
30
20
10
5
Avg.
Anchor- free
Meso4
WIFS’18
V
9.86
6.05
2.22
0.59
4.68
38.9
38.9
38.8
36.5
26.9
36.0
MesoInception4
WIFS’18
V
8.50
5.16
1.89
0.50
4.01
39.3
39.2
39.0
35.8
24.6
35.6
EfficientViT
ICIAP’22
V
14.7
2.42
0.13
0.01
4.32
27.1
27.0
26.4
23.9
20.3
24.9
TriDet+VideoMAEv2
CVPR’23
V
21.7
5.83
0.54
0.06
7.03
20.3
20.2
20.1
19.5
18.2
19.7
TriDet+InternVideo
CVPR’23
V
29.7
9.02
0.79
0.09
9.89
24.1
24.1
24.0
23.5
22.6
23.6
Table 2: Temporal forgery localization results on AV-Deepfake1M ( Cai et al. 2024 ) benchmark. The modality denotes the input type: V for visual, A for audio and AV for both. The best results are bolded , and second-best underlined .
Method
Total[M]
Train[M]
FLOPs[G]
Time[ms]
BA-TFD
5.5
5.5
948.1
605
BA-TFD+
152.9
152.9
218.2
681
UMMAFormer
165.9
49.72
1563.9
857
FullFormer (Ours)
116.7
19.2
233.7
126
DeformTrace (Ours)
118.3
20.8
212.4
104
Table 3: Model size and computation comparison of AV TFL methods. Total , Train , and Time denote total parameters, trainable parameters, and per-video inference time.
Figure 3: Robustness evaluation under various compression and degradation scenarios. The experiments include 6 visual distortions (Block-wise, Color Contrast, Gaussian Noise (video), Gaussian Blur, JPEG Compression and Video Compression) and 4 audio distortions (Gaussian Noise (audio), Reverberation, Pitch Shift and Audio Compression). In the figure, colors denote methods; solid lines show mAP across intensities, dashed lines show mAP on clean videos. DeformTrace achieves the highest mAP on clean videos and demonstrates strong robustness across various distortion scenarios.
#
DS-SSM
DC-SSM
Lenh
Lcoop
mAP
mAR
AUC
Vanilla SSM
1
✗
✗
✗
✗
41.2
68.7
90.3
Deformable SSM
2
✓
✗
✗
✗
44.4
73.2
95.7
3
✗
✓
✗
✗
47.8
75.7
97.3
4
✓
✓
✗
✗
49.8
78.2
98.1
Table 4: Ablation study results on AV-Deepfake1M.
Figure 4: (a) Ablation study on the number of relay tokens (avg. video length: 9s). (b) Performance vs. video duration.