cs.CVSep 25, 2026
SaveManiVid: Unified and Explainable Forensic Analysis of Manipulated Videos
Organizations: Shanghai Jiao Tong University · Shanghai AI Laboratory · Beijing University of Posts and Telecommunications · Fudan University ·
Abstract
Rapid advances in AI-generated video (AIGV) have increased the risks posed by deceptive video manipulation. Unlike fully synthetic videos, manipulated videos retain most source content and alter only localized regions, making forensic analysis particularly challenging. Existing video forgery research faces two limitations in both data and methodology: (1) High-quality datasets and benchmarks tailored for manipulated videos remain scarce. (2) Multimodal large language models (MLLMs) extend forgery analysis beyond binary classification but struggle to use low-level forensic cues and provide precise pixel-level grounding. Specifically, we introduce ManiVid, a unified forensic analysis task covering forgery detection, artifact grounding, and anomaly explanation for manipulated videos. We construct ManiVid-38K, the first dataset to combine paired, open-vocabulary localized manipulations of general videos with authenticity labels, forgery masks, and anomaly explanations. It comprises about 19K manually verified real-fake video pairs, mostly at 1080P resolution, generated under 2 paradigms with 15 powerful generation models. We sample 1K pairs for ManiVidBench, balanced across six manipulation types and generation models for fair evaluation. We further propose ManiVidLens, a unified framework for explainable video forgery analysis. Its Forensic Evidence Router supplies shared low-level forensic evidence for multimodal reasoning and video segmentation. Its Prompt Distill Module converts grounding states into semantic and geometric prompts and distills spatial priors for mask decoding and full-video propagation. ManiVidLens achieves relative gains over the strongest comparison methods in artifact grounding (+21.1% mIoU; +21.3% J&F) and anomaly explanation (+131.3% ROUGE-L; +9.9% CSS). Its forgery detection remains comparable to dedicated classifiers (0.914 Acc; 0.913 F1).
Figures & tables
Figure 1: Overview of ManiVid. ManiVid unifies forgery detection, artifact grounding, and anomaly explanation. The ManiVid-38K dataset provides paired, open-vocabulary manipulations across six types with authenticity labels, pixel-level masks, and anomaly explanations. ManiVidBench provides 2K videos for evaluation, and ManiVidLens offers an end-to-end solution.
Figure 2: Dataset Curation. ManiVid collects high-quality real videos from three public datasets and uses the MLLM assistant to generate open-vocabulary , reasonable , and feasible manipulation instructions. Videos are then manipulated by diverse models, followed by the auto reflection and human verification. ( ) denotes the number of generation models used.
| Dataset | Domain | Video Generation | Comprehensive Annotation | Scale | |||||||
| Task | Model | Paradigm | V.P. | O.V.I | Label | Mask | Explanation | H.V. | |||
| Forgery Type: Fully Synthetic (Directly applying T2V and I2V models) | |||||||||||
| VidProM ( Wang and Yang, 2024 ) | General | - | 4 | T2V | ✗ | ✔ | ✔ | ✗ | ✗ | ✗ | 6.69M |
| GenVideo ( Chen et al., 2026 ) | General | - | 20 | T2V, I2V | ✗ | ✗ | ✔ | ✗ | ✗ | ✗ | 2.31M |
| AEGIS ( Li et al., 2025 ) | General | - | 10 | T2V, I2V | ✗ | ✗ | ✔ | ✗ | ✔ | ✔ | 10470 |
| GenVideoBench ( Ni et al., 2026 ) | General | - | 11 | T2V, I2V | ✔ | ✔ | ✔ | ✗ | ✗ | ✗ | 6.78M |
Table 1: Comparison with Existing Video Forgery Datasets. V.P. denotes the availability of semantic-aligned real-fake video pairs; O.V.I. denotes the release of open-vocabulary instructions; H.V. denotes human verification. Datasets that cannot guarantee source videos are real are excluded .
Figure 3: Model Architecture. (a) ManiVidLens unifies forgery detection, artifact grounding, and anomaly explanation. (b) The Forensic Evidence Router (FER) builds a compact forensic evidence bank and routes it to the MLLM and SAM2. (c) The Prompt Distill Module (PDM) converts MLLM grounding-token states into SAM2 prompts. Dashed arrows mark training-only paths.
Figure 4: Qualitative Comparison of ManiVidLens with Different Models. We visualize two representative manipulation cases: (Left) adding a knife and (Right) changing the refrigerator material to steel. Across both cases, ManiVidLens demonstrates consistently better forensic analysis than LEGION (Image-based), Sa2VA (Video-based), and GPT-5.6 Sol (General MLLM).
| Method | Focus | ManiVidBench | Avg. | ||||||||||||
| Addition | Removal | Replacement | Color | Material | Portrait | ||||||||||
| mIoU | J&F | mIoU | J&F | mIoU | J&F | mIoU | J&F | mIoU | J&F | mIoU | J&F | mIoU | J&F | ||
| Image Expert Models (3) | |||||||||||||||
| LEGION ( Kang et al., 2025 ) | Synth. | 0.343 | 0.352 | 0.091 | 0.102 | 0.385 | 0.392 | 0.323 | 0.346 | 0.360 | 0.370 | 0.192 | 0.213 | 0.312 | 0.324 |
| FakeShield ( Xu et al., 2025 ) | Mani. | 0.365 | 0.374 | 0.068 | 0.073 | 0.332 | 0.333 | 0.429 | 0.430 | 0.373 | 0.371 | 0.161 | 0.153 | 0.318 | 0.320 |
| SIDA ( Huang et al., 2025a ) | Both | 0.013 | 0.113 | 0.004 | 0.020 | 0.010 | 0.019 | 0.020 | 0.042 | 0.009 | 0.025 | 0.002 | 0.004 | 0.011 | 0.041 |
Table 2: Artifact Grounding Performance. Synth. and Mani. denote fully synthetic and manipulated content, respectively. marks frame-wise approximations on 30 uniformly sampled frames; marks native box outputs evaluated with shared SAM2 propagation over every video frame.
Table 4: Forgery Detection Performance. Synth. and Mani. denote fully synthetic and manipulated content. Acc and F1 denote video-level accuracy and F1; Avg. covers all 2,000 videos. Image expert models use majority voting over 30 uniformly sampled frames at a threshold.
Appendix figures & tables39 assets
Supplementary material from the paper’s appendix.
Appendix
| Dimension | Full Synthesis | Manipulation |
| Source Video Reference | None or partial | Complete |
| Generation Paradigm | Direct T2V / I2V | V2V / Conditional I2V |
| Modification Scope | Entire video | Localized regions |
| Real–Fake Video Pair | Unavailable | Naturally paired |
| Semantic Alignment | Low | High |
| Visual Alignment | Low | High |
Table 7: Scope Comparison between Full Synthesis and Manipulation.
| Source Dataset | OpenVid-1M | InternVid | OpenVE-3M | Total |
| Sampled Num | 222391 | 5882 | 15000 | 243273 |
| + Deduplication | 139697 | 3974 | 9292 | 152963 |
| + Quality Filter | 35643 | 2959 | 2506 | 41108 |
Table 8: Data Statistics During Collection. The number of source videos retained at different stages of the collection process.
| Content Category | Definition | Examples |
| Human | The video is centered on one or more humans. The person(s) usually occupy a large portion of the frame or are the clear narrative focus. The background is typically simple and does not form a complete environment. Notes: (1) If humans appear but are small and the environment is the main focus, consider Indoor or Outdoor instead. (2) If the video mainly conveys a time-continuous activity or process where “what is happening” is more important than “who the person is”, consider Event . | Reporter, Police Officer, Businessman, Farmer, … |
| Animal | The video is centered on one or more animals, which occupy a large portion of the frame or are the narrative focus. Notes: (1) If animals are small or only decorative and the environment dominates, use Indoor or Outdoor . (2) If the video mainly presents a continuous activity performed by the animals, consider Event . | Cat, Dog, Bird, Horse, Cow, Sheep, … |
| Object | The video focuses on one or a few inanimate objects. The background is simple and does not form a complete environment. Notes: (1) Hands or partial human bodies may appear, but the object itself is the core subject. (2) If the video emphasizes a multi-step, time-continuous process, consider Event . | Knife, Fork, Food, Tool, Cup, Toy, Laptop, … |
| Indoor | The video primarily presents a complete and visually rich indoor environment (e.g., rooms, offices, shops). No single subject dominates, or subjects are small and secondary. Notes: (1) Use this when the spatial layout, interior structure, and overall atmosphere are the main information. | Room, Office, Shop, Gym, Hospital, … |
| Outdoor | The video primarily presents a complete and visually rich outdoor environment or scenery (e.g., natural landscapes, city streets, building exteriors, aerial views). No single subject dominates, or subjects are small and secondary. Notes: (1) Even if people or animals appear, choose Outdoor as long as they are minor and the scenery is the main focus. | Natural Landscape, City Street, Aerial View, … |
| Event | The video is centered on a time-continuous activity or event. The key information lies in what is happening over time rather than the appearance of a specific person, animal, object, or environment. Notes: (1) It includes any multi-step actions with clear temporal progression. (2) The main subject may be human, animal, or object, but the activity itself is the narrative core. | Cooking Process, Ceremony, Sports Game, … |
Table 9: Content Category Definitions and Examples.
Table 10: Content Categorization Prompt.
Table 11: Video Recaption Prompt.
| Manipulation Task | Definition | Example |
| Addition | Integrate a new, semantically relevant object into the scene. | Add a red backpack onto the empty bench in <video> . |
| Removal | Erase a specific object with clear boundaries and inpaint the background naturally. | Remove the blue suitcase from <video> , seamlessly inpainting the revealed pavement. |
| Replacement | Swap an object with a fine-grained variant or a logically related entity. | Replace the white sedan in <video> with a red SUV, preserving the original motion trajectory. |
| Color Alteration | Change the color of a specific object while preserving its original texture and shape. | Change the color of the woman’s jacket in <video> to blue, maintaining its original fabric texture. |
| Material Modification | Alter the physical material properties. | Change the material of the rotating vase in <video> into polished metal, maintaining its original motion. |
| Portrait Editing | Modify facial features, expressions, or skin tones. ( Note : Only applicable to close-up or medium shots of people). | Modify the facial features of the woman at the center of <video> to exhibit a smiling expression, keeping others consistent. |
Table 12: Manipulation Task Definitions and Examples.
| Paradigm | V2V Model | (Stage 1) Image Editing Model | (Stage 2) Canny-guided I2V Model |
|---|---|---|---|
| Direct V2V (53.80%) | Kling-O1 (5.80%) Kling-V3-Omni (2.65%) Runway Aleph (8.48%) Wan 2.2-Video-Edit (10.43%) VideoCoF (21.36%) VACE (4.39%) VideoPainter (0.69%) | – | – |
| Nano Banana 2 (13.92%) Seedream 5.0 (14.16%) GPT-Image-1.5 (0.39%) FLUX.2-dev-32B (5.75%) FLUX.2-klein-9B (3.34%) Qwen-Image (8.64%) Indirect I2V (46.20%) | Nano Banana 2 (13.92%) Seedream 5.0 (14.16%) GPT-Image-1.5 (0.39%) FLUX.2-dev-32B (5.75%) FLUX.2-klein-9B (3.34%) Qwen-Image (8.64%) – | Nano Banana 2 (13.92%) | Kling-V3-Omni (1.77%) Wan 2.2 (5.54%) LTX-2 (6.60%) |
| Seedream 5.0 (14.16%) | Kling-V3-Omni (1.81%) Wan 2.2 (7.72%) LTX-2 (4.63%) | ||
| GPT-Image-1.5 (0.39%) | Kling-V3-Omni (0.00%) Wan 2.2 (0.39%) LTX-2 (0.00%) | ||
| FLUX.2-dev-32B (5.75%) | Kling-V3-Omni (1.14%) Wan 2.2 (1.27%) LTX-2 (3.34%) | ||
| FLUX.2-klein-9B (3.34%) | Kling-V3-Omni (0.00%) Wan 2.2 (1.41%) LTX-2 (1.93%) |
Table 14: Detailed Distribution of Manipulation Paradigms and Models in ManiVid-38K.
Figure 5: Manipulation Examples of Seven Direct V2V Models. The first four rows are closed source models, while the last three rows are open source models. Zoom in for a better view.
Figure 6: Manipulation Examples of three indirect I2V Models. The first row is Kling-V3-Omni (closed source), while the last two rows are open source models. Zoom in for a better view.
Table 15: Automatic Reflection Prompt.
| Anomaly Perspective | Definition |
| Boundary Coherence | Imperfect transitions between the manipulated region and the original footage, including overly smoothed or unnaturally sharp boundary edges, as well as texture discrepancies in surface noise, resolution, or material quality relative to the surrounding unedited areas. |
| Perspective and Proportion | Geometric spatial errors where the manipulated content violates the camera perspective, exhibits focal-length-related distortions, or shows unnatural scaling and structural proportions relative to its immediate environment. |
| Contextual Inconsistency | Semantic or logical anomalies where the manipulated content defies the established scene context, including physically impossible object interactions, gravity-defying kinematics, or elements that fundamentally contradict surrounding real-world physics. |
| Color and Lighting | Mismatches in illumination, shadow trajectories, or color-space distribution between the manipulated content and the global physical environment. |
| Texture Discontinuity | Sudden degradation in surface quality, excessive smoothing, or unnatural loss of high-frequency details within the manipulated region. |
| Temporal Flicker | Unstable pixel generation or erratic high-frequency variations of static visual features across consecutive frames, typically manifested as localized shimmering or micro-jitter in textures that should remain structurally rigid. |
Table 16: Definitions of Anomaly Explanation Perspectives.
Figure 7: Additional statistical analyses of ManiVid-38K.
Figure 8: Examples of Addition and Removal in ManiVid-38K. Zoom in for a better view.
Figure 9: Examples of Replacement and Color Alteration in ManiVid-38K. Zoom in for a better view.
Figure 10: Examples of Material Modification and Portrait Editing in ManiVid-38K. Zoom in for a better view.
Figure 11: Statistical Analysis of Our ManiVidBench.
| Task protocol | Instruction objective and supervised assistant response format |
| Classification | Determine the manipulation status and one applicable manipulation type. The response contains Manipulation status: followed by Manipulation type: . |
| Explanation | Describe the relevant visual, physical, and temporal anomalies. The response contains exactly one <think>...</think> reasoning block. |
| Segmentation | Locate the manipulated region and describe its target briefly. A positive response contains the target fields and exactly one marker. |
Table 18: Task-specific generation protocols used by ManiVidLens. Each row summarizes one instruction objective and its exact assistant response structure.
Table 19: Task Prompts Used in the ManiVidLens Framework. Separate prompts are the default; the All-in-One prompt is used only for the 3-task ablation.
Figure 12: Training and Inference Pipelines of ManiVidLens. (a) Stage 1 learns task-specific responses and authenticity routing. Stage 2 adds box-prompt distillation and sampled-frame mask supervision, with PDM reading teacher-forced token states. (b) Inference constructs SAM2 prompts from generated token states and propagates masks across the full video using only video and task inputs. GT denotes ground-truth responses, boxes, and masks used for supervision.
| Training aspect | Stage 1 implementation details | Stage 2 implementation details |
| Stage length | One complete training epoch | One complete training epoch |
| Task setup | Three separate task instances with shared-loss scaling | Three separate task instances with shared-loss scaling |
| Batch ratio | Balanced original-to-manipulated ratio of | Manipulation-focused original-to-manipulated ratio of |
| Shared weights | Language and classification weights are and | Language and classification weights are and |
| Grounding weights | Every grounding-loss weight remains zero | , , and |
| Class weight | The classifier uses unit positive-class weight | The classifier uses one-third positive-class weight |
Table 20: Two-stage training implementation for the current full profile. The table reports the launcher defaults and stage-specific optimization settings.
| Task | Metric | Direct V2V (52.3%) | Indirect I2V (47.7%) | Overall | ||||||||||
| Kling-O1 | Kling-V3 Omni | Runway Aleph | Wan 2.2 Video-Edit | VideoCoF | VACE | VideoPainter | Avg. | Kling-V3 Omni | Wan 2.2 | LTX-2 | Avg. | |||
| Grounding | mIoU | 0.435 | 0.419 | 0.452 | 0.553 | 0.317 | 0.396 | 0.465 | 0.413 | 0.292 | 0.383 | 0.440 | 0.398 | 0.407 |
| J&F | 0.450 | 0.436 | 0.466 | 0.574 | 0.333 | 0.414 | 0.469 | 0.429 | 0.303 | 0.402 | 0.452 | 0.413 | 0.422 | |
| Explanation | R-L | 0.533 | 0.557 | 0.605 | 0.621 | 0.553 | 0.596 | 0.607 | 0.567 | 0.615 | 0.608 | 0.600 | 0.605 | 0.583 |
| CSS | 0.686 | 0.740 | 0.828 | 0.855 | 0.722 | 0.796 | 0.789 | 0.750 | 0.835 | 0.826 | 0.811 | 0.821 | 0.780 | |
| Detection | Acc | 0.751 | 0.941 | 0.972 | 0.976 | 0.850 | 0.974 | 1.000 | 0.868 | 0.980 | 0.967 | 0.982 | 0.975 | 0.914 |
Table 21: ManiVidLens Performance by Generation Model. ( ) indicate each generation paradigm’s share of ManiVidBench. Indirect I2V groups follow the final video generator, pooling over first-frame editors. Avg. aggregates each paradigm, while Overall matches the main tables under the evaluation protocol in Appendix E.1.3 .
Figure 13: Confusion Matrix for Seven-Class Manipulation Attribution.
| Setting | ManiVidBench | Avg. | ||||||||||||
| Addition | Removal | Replacement | Color | Material | Portrait | |||||||||
| mIoU | J&F | mIoU | J&F | mIoU | J&F | mIoU | J&F | mIoU | J&F | mIoU | J&F | mIoU | J&F | |
| Ablation 1: Number of Sampled Frames | ||||||||||||||
| 3 Frames | 0.411 | 0.432 | 0.118 | 0.118 | 0.366 | 0.382 | 0.395 | 0.417 | 0.450 | 0.456 | 0.278 | 0.286 | 0.354 | 0.368 |
| 5 Frames | 0.444 | 0.467 | 0.135 | 0.137 | 0.421 | 0.439 | 0.487 | 0.507 | 0.526 | 0.532 | 0.290 | 0.306 | 0.407 | 0.422 |
| 7 Frames | 0.508 | 0.533 | 0.114 | 0.120 | 0.440 | 0.460 | 0.452 | 0.474 | 0.483 | 0.494 | 0.353 | 0.370 | 0.413 | 0.431 |
Table 22: Category-wise Ablation for Artifact Grounding. Column groups follow the six manipulation categories, with benchmark-level scores in Avg. Metrics are mIoU and J&F ( ).
| Setting | ManiVidBench | Avg. | ||||||||||||
| Addition | Removal | Replacement | Color | Material | Portrait | |||||||||
| R-L | CSS | R-L | CSS | R-L | CSS | R-L | CSS | R-L | CSS | R-L | CSS | R-L | CSS | |
| Ablation 1: Number of Sampled Frames | ||||||||||||||
| 3 Frames | 0.587 | 0.791 | 0.574 | 0.761 | 0.571 | 0.756 | 0.575 | 0.756 | 0.578 | 0.781 | 0.544 | 0.695 | 0.575 | 0.763 |
| 5 Frames | 0.603 | 0.813 | 0.576 | 0.773 | 0.581 | 0.777 | 0.582 | 0.772 | 0.587 | 0.795 | 0.545 | 0.699 | 0.583 | 0.780 |
| 7 Frames | 0.613 | 0.831 | 0.591 | 0.796 | 0.593 | 0.797 | 0.592 | 0.801 | 0.603 | 0.817 | 0.555 | 0.721 | 0.596 | 0.803 |
Table 23: Category-wise Ablation for Anomaly Explanation. Column groups follow the six manipulation categories in the main results, with benchmark-level scores in Avg. Metrics are ROUGE-L (R-L) and Cosine Similarity Score (CSS) ( ).
| Setting | ManiVidBench | Avg. | ||||||||||||
| Addition | Removal | Replacement | Color | Material | Portrait | |||||||||
| Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 | |
| Ablation 1: Number of Sampled Frames | ||||||||||||||
| 3 Frames | 0.953 | 0.953 | 0.920 | 0.920 | 0.885 | 0.885 | 0.906 | 0.906 | 0.912 | 0.912 | 0.805 | 0.803 | 0.906 | 0.905 |
| 5 Frames | 0.966 | 0.966 | 0.943 | 0.943 | 0.868 | 0.868 | 0.917 | 0.917 | 0.939 | 0.939 | 0.839 | 0.839 | 0.914 | 0.913 |
| 7 Frames | 0.971 | 0.971 | 0.958 | 0.958 | 0.899 | 0.899 | 0.930 | 0.930 | 0.931 | 0.931 | 0.822 | 0.821 | 0.926 | 0.926 |
Table 24: Category-wise Ablation for Forgery Detection. Column groups follow the six manipulation categories in the main results, with benchmark-level scores in Avg. Metrics are video-level Accuracy (Acc) and F1 ( ) for original and manipulated videos.
| Prompt Design | ManiVidBench | Avg. | ||||||||||||
| Addition | Removal | Replacement | Color | Material | Portrait | |||||||||
| Task Mode: Classification | ||||||||||||||
| Metrics | Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 |
| All-in-one | 0.838 | 0.833 | 0.779 | 0.768 | 0.777 | 0.766 | 0.758 | 0.744 | 0.790 | 0.782 | 0.627 | 0.567 | 0.778 | 0.767 |
| Separate (Ours) | 0.966 | 0.966 | 0.943 | 0.943 | 0.868 | 0.868 | 0.917 | 0.917 | 0.939 | 0.939 | 0.839 | 0.839 | 0.914 | 0.913 |
| Task Mode: Explanation | ||||||||||||||
Table 25: Category-wise Ablation on Prompt Design. All-in-one and separate prompts are compared across six manipulation categories and three tasks. Avg. reports benchmark-level scores. Higher metric values ( ) indicate better performance on all three tasks.
| Setting | Grounding | Explanation | Detection | ||||
| mIoU | J&F | R-L | CSS | Acc | F1 | ||
| ManiVidLens | 0.407 | 0.422 | 0.583 | 0.780 | 0.914 | 0.913 | |
| H.264 Compression | CRF 23 | 0.384 (-5.5%) | 0.400 (-5.2%) | 0.562 (-3.7%) | 0.735 (-5.8%) | 0.882 (-3.4%) | 0.882 (-3.5%) |
| CRF 40 | 0.246 (-39.5%) | 0.255 (-39.5%) | 0.514 (-11.8%) | 0.638 (-18.2%) | 0.591 (-35.3%) | 0.535 (-41.5%) | |
| Downsampling | 0.75 | 0.399 (-2.0%) | 0.416 (-1.4%) | 0.577 (-1.2%) | 0.775 (-0.6%) | 0.895 (-2.0%) | 0.895 (-2.0%) |
| 0.50 | 0.400 (-1.7%) | 0.415 (-1.7%) | 0.567 (-2.7%) | 0.764 (-2.0%) | 0.859 (-6.0%) | 0.859 (-6.0%) | |
Table 26: Robustness Evaluation under Different Video Degradations. Grounding, explanation, and detection scores ( ) are evaluated under four types of video degradation. Parenthesized values show signed percentage changes relative to the undegraded ManiVidLens baseline, with negative values indicating performance degradation for the corresponding metric.
Figure 14: A ManiVidLens Case on an Addition Sample.
Figure 15: A ManiVidLens Case on an Addition Sample.
Figure 16: A ManiVidLens Case on a Removal Sample.
Figure 17: A ManiVidLens Case on a Removal Sample.
Figure 18: A ManiVidLens Case on a Replacement Sample.
Figure 19: A ManiVidLens Case on a Replacement Sample.
Figure 20: A ManiVidLens Case on a Color Alteration Sample.
Figure 21: A ManiVidLens Case on a Color Alteration Sample.
Figure 22: A ManiVidLens Case on a Material Modification Sample.
Figure 23: A ManiVidLens Case on a Material Modification Sample.
Figure 24: A ManiVidLens Case on a Portrait Editing Sample.
Figure 25: A ManiVidLens Case on a Portrait Editing Sample.