Temporal Forgery Localization (TFL) aims to precisely identify manipulated segments within videos or audio streams, providing interpretable evidence for multimedia forensics and security. While most existing TFL methods rely on dense frame-level labels in a fully supervised manner, Weakly Supervised TFL (WS-TFL) reduces labeling cost by learning only from binary video-level labels. However, current WS-TFL approaches suffer from mismatched training and inference objectives, limited supervision from binary labels, gradient blockage caused by non-differentiable top-k aggregation, and the absence of explicit modeling of inter-proposal relationships. To address these issues, we propose GEM-TFL (Graph-based EM-powered Temporal Forgery Localization), a two-phase classification-regression framework that effectively bridges the supervision gap between training and inference. Built upon this foundation, (1) we enhance weak supervision by reformulating binary labels into multi-dimensional latent attributes through an EM-based optimization process; (2) we introduce a training-free temporal consistency refinement that realigns frame-level predictions for smoother temporal dynamics; and (3) we design a graph-based proposal refinement module that models temporal-semantic relationships among proposals for globally consistent confidence estimation. Extensive experiments on benchmark datasets demonstrate that GEM-TFL achieves more accurate and robust temporal forgery localization, substantially narrowing the gap with fully supervised methods.
Figures & tables
Figure 1: Comparison between prior and our WS-TFL pipelines. (a) Prior methods [ 40 , 39 ] : trained with clip-level labels but required for temporal localization, leading to noisy proposals due to the training–inference mismatch. (b) Ours: a two-phase design aligning classification and regression to refine coarse predictions for precise boundary localization.
Figure 2: Overview of the classification phase. (a) Label Attribute Decomposition: The feature enhancement module aligns and fuses audio–visual features, after which the attention and attribute branches produce frame-level attention and attribute predictions optimized through EM to capture diverse forgery patterns. (b) Temporal Consistency Refinement: The non-differentiable top- k operation blocks gradient flow between the attention and attribute branches, causing inconsistent temporal responses. To address this, frame-level attribute predictions are alternately projected onto the row constraint (attention-weighted alignment between frame- and clip-level predictions) and the column constraint (categorical distribution). The refined attribute predictions are then used to generate initial pseudo proposals. (c) Graph-based Proposal Refinement: Proposals are mapped into a unified space, where a proposal graph is constructed and confidence values are diffused to obtain fusion weights. These are integrated and thresholded at zero to produce the final pseudo proposals, merging fragmented proposals (e.g., p1 , p2 , p3 ) into continuous ones (e.g., pfuse ) from a global and relational perspective.
Supervision
Method
Modality
mAP@IoU(%)
mAR@Proposals(%)
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Avg.
20
10
5
2
Avg.
Fully
AFormer ECCV’22 [ 44 ]
V
98.0
97.7
97.3
96.8
96.3
95.5
94.6
96.6
99.2
99.0
98.4
96.0
98.1
TriDet CVPR’23 [ 31 ]
V
95.0
94.7
94.4
93.8
93.2
92.2
90.7
93.4
97.1
96.9
96.3
93.7
96.0
UFormer MM’23 [ 46 ]
AV
97.7
97.6
97.4
97.1
96.7
96.0
94.9
96.8
98.6
98.5
98.2
95.1
97.6
MFMS MM’24 [ 47 ]
AV
98.0
97.9
97.8
97.6
97.3
96.7
95.8
97.3
98.9
98.9
98.6
95.6
98.0
Weakly
CoLA CVPR’21 [ 43 ]
V
31.3
25.9
19.4
13.4
8.6
5.2
2.7
15.2
41.2
41.2
40.8
37.8
40.3
Table 1: Temporal forgery localization results of fully and weakly supervised methods on LAV-DF [ 5 ] . ’UFormer’ and ’PFormer’ denote UMMAFormer [ 46 ] and PseudoFormer [ 22 ] . ’V’, ’A’, and ’AV’ indicate visual, audio, and audio–visual modalities. Fully supervised methods use forged-segment timestamps as supervision, while weakly supervised ones use binary clip-level labels, except WMMT [ 39 ] , which adopts a quadruple signal (both real, both fake, visual fake only, and audio fake only). The best mAP/mAR are highlighted in red , and the second-best in blue .
Figure 3: Comparison between prior and our WS-TFL pipelines.
Supervision
Method
Modality
mAP@IoU(%)
mAR@Proposals(%)
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Avg.
20
10
5
2
Avg.
Fully
AFormer ECCV’22 [ 44 ]
V
67.3
67.3
67.2
67.2
67.0
66.7
65.5
66.9
83.1
82.9
82.6
78.7
81.8
TriDet CVPR’23 [ 31 ]
V
55.7
55.6
55.4
55.1
54.7
53.8
51.7
54.6
74.9
74.2
72.9
68.0
72.5
UFormer MM’23 [ 46 ]
AV
91.8
91.7
91.5
91.3
91.0
90.4
89.0
90.9
95.0
94.6
93.9
89.5
93.3
MFMS MM’24 [ 47 ]
AV
94.7
94.6
94.6
94.5
94.3
93.9
92.5
94.1
96.7
96.4
96.0
92.0
95.3
Weakly
CoLA CVPR’21 [ 43 ]
V
3.22
1.09
0.39
0.14
0.05
0.02
0.01
0.70
20.7
20.0
16.1
8.17
16.2
Table 2: Temporal forgery localization results of fully and weakly supervised methods on the AV-Deepfake1M [ 4 ] benchmark.
#
LAD
TCR
GPR
LP
mAP (0.1:0.7)
mAR (0.1:0.7)
Param [M]
1
✗
✗
✗
✗
5.12
35.6
17.1
2
✓
✗
✗
✗
23.8 ↑ 18.7
44.3 ↑ 8.7
17.2
3
✓
✓
✗
✗
27.3 ↑ 22.2
46.7 ↑ 11.1
17.2
4
✓
✓
✓
✗
31.9 ↑ 26.8
51.2 ↑ 15.6
17.2
5
✓
✓
✓
✓
42.7 ↑ 37.6
59.8 ↑ 24.2
62.2
Table 3: Ablation study on the components. Performance gains relative to the base model (row #1) are indicated as subscripts.
Figure 4: T-SNE visualizations of classification features under different latent forgery attribute d m . Overlaps between attributes are highlighted in gray. “binary” denotes the setting without EM optimization, where the model directly performs binary classification.
Method
mAP@IoU(%)
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Avg.
CoLA
1.35
0.13
0.03
0.01
0.01
0.01
0.01
0.22
LOCO
0.27
0.04
0.01
0.00
0.00
0.00
0.00
0.05
MDP
15.5
9.10
7.76
4.01
2.33
1.83
0.42
5.85
PFormer
26.3
22.7
16.6
12.7
9.28
6.44
2.15
13.7
WMMT
21.7
18.5
15.7
13.1
10.2
7.39
4.45
13.0
Table 4: Cross-dataset generalization performance. Models are trained on AV-Deepfake1M [ 4 ] and tested on LAV-DF [ 5 ] .
Current temporal forgery localization (TFL) approaches typically rely on temporal boundary regression or continuous frame-level anomaly detection paradigms to derive candidate forgery proposals. However, they suffer not only from feature granularity misalignment but also from costly computation. To address these issues, we propose word-anchored temporal forgery localization (WAFL), a novel paradigm that shifts the TFL task from temporal regression and continuous localization to discrete word-level binary classification. Specifically, we first analyze the essence of temporal forgeries and identify the minimum meaningful forgery units, word tokens, and then align data preprocessing with the natural linguistic boundaries of speech. To adapt powerful pre-trained foundation backbones for feature extraction, we introduce the forensic feature realignment (FFR) module, mapping representations from the pre-trained semantic space to a discriminative forensic manifold. This allows subsequent lightweight linear classifiers to efficiently perform binary classification and accomplish the TFL task. Furthermore, to overcome the extreme class imbalance inherent to forgery detection, we design the artifact-centric asymmetric (ACA) loss, which breaks the standard precision-recall trade-off by dynamically suppressing overwhelming authentic gradients while asymmetrically prioritizing subtle forensic artifacts. Extensive experiments demonstrate that WAFL significantly outperforms state-of-the-art approaches in localization performance under both in- and cross-dataset settings, while requiring substantially fewer learnable parameters and operating at high computational efficiency.
Tianyi Wang, Xi Shao, Harry Cheng +2
National University of Singapore · Nanjing University of Posts and Telecommunications · Qilu University of Technology (Shandong Academy of Sciences)
Driven by Artificial Intelligence-Generated Content (AIGC), the authenticity of audio-visual content is facing severe challenges. Temporal Forgery Localization (TFL) aims to precisely identify manipulated segments within untrimmed sequences. However, existing methods are limited by CNNs' local receptive fields or Transformers' quadratic complexity, while emerging linear models often struggle to balance global authentic context compression with local abrupt forgery perception. To address this, we propose MG-RWKV, a multi-granularity framework that leverages the data-dependent state evolution of RWKV to achieve efficient full-sequence processing with O(T) complexity. Our framework features three core innovations: (1) a Bidirectional RWKV architecture that captures bidirectional temporal contexts without quadratic overhead; (2) a Multi-Granularity Mixture of Experts (MG-MoE) that performs dynamic routing over explicit temporal receptive fields, adaptively selecting granularities based on forgery duration to significantly enhance decision interpretability; and (3) Cross-Granularity Consistency (CGC), which aligns adjacent feature pyramid levels through hierarchical scale-wise pairing and spatial boundary-aware weighting, effectively reducing false positives in authentic regions. Extensive experiments on Lav-DF, TVIL, and Psynd datasets demonstrate that MG-RWKV achieves state-of-the-art performance with low computational cost.
Jingchen Ni, Cangjin Yu, Dan Jiang +6
1Tsinghua University, China · 2Soochow University, China
The rapid proliferation of artificial intelligence-generated content necessitates reliable multimodal forensics. Beyond video-level binary classification, precisely localizing sparsely distributed forged segments in long-form videos remains a critical challenge. This task is particularly difficult when manipulations are subtly embedded and cross-modal signals are weak and temporally diffuse. To address these challenges, we propose EVAS, an end-to-end multimodal framework for temporal forgery localization. At its core, a Multi-Stage Audio-Visual Synergy mechanism facilitates progressive cross-modal interaction to learn deep multimodal forensic representations and capture high-order semantic traces of sparse manipulations. Furthermore, we introduce a Boundary-Aware Refinement strategy to achieve steered boundary calibration. By incorporating invalid-frame masking, this strategy suppresses ambiguous regions and sharpens transition predictions. We adopt a decoupled training paradigm with auxiliary heads to disentangle representation learning from inference objectives, enhancing model generalization and stability. Additionally, a lightweight HourglassFFN is incorporated to reduce computational overhead. Extensive experiments demonstrate that EVAS achieves state-of-the-art average localization accuracy and average recall across three benchmark datasets, validating its effectiveness for fine-grained temporal forgery localization.