ActivityForensics: A Comprehensive Benchmark for Localizing Manipulated Activity in Videos
Authors: Peijun Bao, Anwei Luo, Gang Pan, Alex C. Kot, Xudong Jiang
Organizations: College of Computer Science and Technology, Zhejiang University · School of Electrical and Electronic Engineering, Nanyang Technological University · School of Computing and Artificial Intelligence, Jiangxi University of Finance and Economics · Jiangxi Provincial Key Laboratory of Multimedia Intelligent Processing · Faculty of Engineering, Shenzhen MSU-BIT University · VinUniversity
Temporal forgery localization aims to temporally identify manipulated segments in videos. Most existing benchmarks focus on appearance-level forgeries, such as face swapping and object removal. However, recent advances in video generation have driven the emergence of activity-level forgeries that modify human actions to distort event semantics, resulting in highly deceptive forgeries that critically undermine media authenticity and public trust. To overcome this issue, we introduce ActivityForensics, the first large-scale benchmark for localizing manipulated activity in videos. It contains over 6K forged video segments that are seamlessly blended into the video context, rendering high visual consistency that makes them almost indistinguishable from authentic content to the human eye. We further propose Temporal Artifact Diffuser (TADiff), a simple yet effective baseline that exposes artifact cues through a diffusion-based feature regularizer. Based on ActivityForensics, we introduce comprehensive evaluation protocols covering intra-domain, cross-domain, and open-world settings, and benchmark a wide range of state-of-the-art forgery localizers to facilitate future research. The dataset and code are available at https://activityforensics.github.io.
Figures & tables
Figure 1: a) Existing datasets for temporal forgery localization mainly focus on appearance-level forgeries such as object removal and face manipulation. b) Driven by the remarkable advances in video generation and editing in recent years, however, activity-level forgeries have become increasingly prevalent and pose significant risks to media integrity and societal trust. c) To address this emerging threat, we present ActivityForensics, the first dataset for localizing manipulated activities in videos.
Figure 2: Overview of grounding-assisted data generation pipeline. 1) We leverage video captioning and temporal grounding to obtain activity descriptions and localize their corresponding temporal segments. 2) Subsequently, grounded segments and manipulated descriptions are harnessed as conditioning signals to automatically perform activity manipulations. 3) The manipulated segments are finally seamlessly merged into the rest of the video, while remaining visually consistent across both tampered and authentic regions. The green bounding boxes indicate the original regions, while the red ones correspond to the manipulated regions.
Figure 3: Statistics of the ActivityForensics dataset. a) Histogram of forgery-segment counts across manipulation methods, where vidu is used only for evaluation. b) Distribution of manipulated segment durations. c) Distribution of the ratio between manipulated segment duration and overall video duration.
Category
Method
Guidance type
Video Generation
Wan [ 33 ]
Text driven
Scifi [ 9 ]
Frame Interpolation
FCVG [ 44 ]
Pose driven
Vidu [ 1 ]
Commercial API
Video Editing
VACE [ 15 ]
Text driven
LTX [ 11 ]
Text driven
Table 1: Summary of manipulation methods in ActivityForensics.
Figure 4: Overview of Temporal Artifact Diffuser (TADiff). Different from action localization that relies on high-level semantics for event understanding, manipulated activity localization requires sensitivity to subtle temporal and visual artifacts. To this end, TADiff injects stochastic perturbations into the temporal feature space of ActionFormer to suppress semantic bias, and then amplifies artifact cues via iterative denoising, composed of Feature-wise Linear Modulation (FiLM) and Denoising Diffusion Implicit Model (DDIM) updates.
Table 2: Quantitative comparisons under intra-domain and open-world settings. Each section reports Average Precision (AP) at multiple tIoU thresholds and Average Recall (AR) at various proposal counts. Orange numbers indicate improvements over the ActionFormer baseline on which our TADiff is built.
Table 3: Quantitative comparisons under cross-domain scenarios. Orange numbers illustrate gains over the ActionFormer baseline.
Domain
noise
denoise
avg AP
avg AR
Intra-Domain
✗
✗
70.67
74.31
✓
✗
70.38
74.01
✗
✓
73.52
76.22
✓
✓
75.05
77.15
Open-World
✗
✗
77.82
83.31
✓
✗
79.75
84.82
Table 4: Module ablation studies under the intra-domain scenario.
Figure 5: Impact of denoising step number.
Figure 6: Qualitative comparison of the baseline and TADiff. The darker yellow rectangle represents the ground-truth forgery segments, while the darker blue one denotes the model’s prediction.
Figure 7: t-SNE visualization of features without and with Temporal Artifact Diffuser (TADiff). The Fisher discriminant score increases from 1.74 to 2.64 after introducing TADiff, which reflects better inter-class separability and reduced intra-class variance in the learned feature space.
Rapid advancements in video diffusion models and temporal editing tools have enabled the generation of highly realistic human-centric videos, posing unprecedented challenges to digital content forensics. Existing benchmarks primarily focus on either face-swapping or global text-to-video synthesis, overlooking the crucial dimensions of human-object or human-human interactions and multi-modal alignment. To address these limitations, we introduce HumanForge, a unified, large-scale, and multi-paradigm human-centric video forgery dataset. To construct and annotate this dataset without labor-intensive manual labeling or hallucinated monolithic prompts, we propose Gen2Anno, a modular active multi-agent pipeline built on LangGraph. Gen2Anno coordinates six specialized agents-ranging from source profiling to MoE-based reference analysis and closed-loop forensic verification-to generate over 18K high-fidelity video segments and produce structured, contrastive omni-annotations containing binary decisions, fine-grained artifact categories, and spatio-temporal localization. Extensive benchmarks using state-of-the-art traditional detectors and Large Multimodal Models (LMMs) demonstrate the significant challenges of zero-shot generalization and fine-grained reasoning on HumanForge. Code and dataset will be publicly released.
Wenbo Xu, Zhimin Chen, Xiaojie Liang +2
School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou 510006, China
Rapid advances in AI-generated video (AIGV) have increased the risks posed by deceptive video manipulation. Unlike fully synthetic videos, manipulated videos retain most source content and alter only localized regions, making forensic analysis particularly challenging. Existing video forgery research faces two limitations in both data and methodology: (1) High-quality datasets and benchmarks tailored for manipulated videos remain scarce. (2) Multimodal large language models (MLLMs) extend forgery analysis beyond binary classification but struggle to use low-level forensic cues and provide precise pixel-level grounding. Specifically, we introduce ManiVid, a unified forensic analysis task covering forgery detection, artifact grounding, and anomaly explanation for manipulated videos. We construct ManiVid-38K, the first dataset to combine paired, open-vocabulary localized manipulations of general videos with authenticity labels, forgery masks, and anomaly explanations. It comprises about 19K manually verified real-fake video pairs, mostly at 1080P resolution, generated under 2 paradigms with 15 powerful generation models. We sample 1K pairs for ManiVidBench, balanced across six manipulation types and generation models for fair evaluation. We further propose ManiVidLens, a unified framework for explainable video forgery analysis. Its Forensic Evidence Router supplies shared low-level forensic evidence for multimodal reasoning and video segmentation. Its Prompt Distill Module converts grounding states into semantic and geometric prompts and distills spatial priors for mask decoding and full-video propagation. ManiVidLens achieves relative gains over the strongest comparison methods in artifact grounding (+21.1% mIoU; +21.3% J&F) and anomaly explanation (+131.3% ROUGE-L; +9.9% CSS). Its forgery detection remains comparable to dedicated classifiers (0.914 Acc; 0.913 F1).
Hengrui Kang, Zhonghao Yan, Yuxuan Yang +7
Shanghai Jiao Tong University · Shanghai AI Laboratory · Beijing University of Posts and Telecommunications +3
The rapid proliferation of artificial intelligence-generated content necessitates reliable multimodal forensics. Beyond video-level binary classification, precisely localizing sparsely distributed forged segments in long-form videos remains a critical challenge. This task is particularly difficult when manipulations are subtly embedded and cross-modal signals are weak and temporally diffuse. To address these challenges, we propose EVAS, an end-to-end multimodal framework for temporal forgery localization. At its core, a Multi-Stage Audio-Visual Synergy mechanism facilitates progressive cross-modal interaction to learn deep multimodal forensic representations and capture high-order semantic traces of sparse manipulations. Furthermore, we introduce a Boundary-Aware Refinement strategy to achieve steered boundary calibration. By incorporating invalid-frame masking, this strategy suppresses ambiguous regions and sharpens transition predictions. We adopt a decoupled training paradigm with auxiliary heads to disentangle representation learning from inference objectives, enhancing model generalization and stability. Additionally, a lightweight HourglassFFN is incorporated to reduce computational overhead. Extensive experiments demonstrate that EVAS achieves state-of-the-art average localization accuracy and average recall across three benchmark datasets, validating its effectiveness for fine-grained temporal forgery localization.