Beyond Static Artifacts: A Forensic Benchmark for Video Deepfake Reasoning in Vision Language Models
Organizations: Institute of Artificial Intelligence, China Telecom (TeleAI) · Peking University · Fudan University
Abstract
Current Vision-Language Models (VLMs) for deepfake detection excel at identifying spatial artifacts but overlook a critical dimension: temporal inconsistencies in video forgeries. Adapting VLMs to reason about these dynamic cues remains a distinct challenge. To bridge this gap, we propose Forensic Answer-Questioning (FAQ), a large-scale benchmark that formulates temporal deepfake analysis as a multiple-choice task. FAQ introduces a three-level hierarchy to progressively evaluate and equip VLMs with forensic capabilities: (1) Facial Perception, testing the ability to identify static visual artifacts; (2) Temporal Deepfake Grounding, requiring the localization of dynamic forgery artifacts across frames; and (3) Forensic Reasoning, challenging models to synthesize evidence for final authenticity verdicts. We evaluate a range of VLMs on FAQ and generate a corresponding instruction-tuning set, FAQ-IT. Extensive experiments show that models fine-tuned on FAQ-IT achieve advanced performance on both in-domain and cross-dataset detection benchmarks. Ablation studies further validate the impact of our key design choices, confirming that FAQ is the driving force behind the temporal reasoning capabilities of these VLMs.
Figures & tables
| Benchmark | MCQ | #Sample | Annotation | #Task Types | Trace |
| Image-based Benchmarks | |||||
| GenImage [ 47 ] | - | 2,681,167 | A | 1 | - |
| DDVQA [ 46 ] | - | 14,784 | H | 1 | - |
| VLFFD [ 36 ] | - | 25,663 | A+H | 1 | - |
| MMTD-Set [ 43 ] | - | 34,470 | A | 2 | - |
| SID-Set desc [ 14 ] | - | 3,000 | A | 2 | - |
| Model | Level 1 | Level 2 | Level 3 | Average |
|---|---|---|---|---|
| Proprietary Large Vision Language Models | ||||
| GPT-4o | ||||
| Gemini-2.5-Flash | ||||
| Open-sourced Large Vision Language Models | ||||
| InternVL-Chat-V1-2 | ||||
| LLaVA-1.5-13b | ||||
| Model | Condition | Level 1 | Level 2 | Level 3 | Average |
|---|---|---|---|---|---|
| Qwen2.5-VL | 24.1 | 23.8 | 16.8 | 21.6 | |
| 31.3 ↑7.2 | 21.9 ↓1.9 | 17.9 ↑1.1 | 23.9 ↑2.3 | ||
| 89.9 ↑65.8 | 41.4 ↑17.6 | 25.8 ↑9.0 | 52.4 ↑30.8 | ||
| LLaVA-NeXT | 40.0 | 29.0 | 21.2 | 30.3 | |
| 49.2 ↑9.2 | 28.8 ↓0.2 | 23.3 ↑2.1 | 33.8 ↑3.5 | ||
| 88.8 ↑48.8 | 45.8 ↑16.8 | 26.5 ↑5.3 | 53.7 ↑23.4 |
| Model | Condition | FS | NT | F2F | DF | FSh | Avg |
|---|---|---|---|---|---|---|---|
| In-domain MCQ ACC (%) | |||||||
| Qwen2.5-VL | 9.7 | 7.4 | 13.5 | 9.6 | 7.7 | 9.6 | |
| 10.5 ↑0.8 | 13.3 ↑5.8 | 12.8 ↓0.7 | 10.4 ↑0.8 | 10.7 ↑3.0 | 11.5 ↑1.9 | ||
| 45.9 ↑36.2 | 46.7 ↑39.3 | 24.4 ↑10.9 | 45.3 ↑35.7 | 45.3 ↑37.6 | 41.5 ↑31.9 | ||
| LLaVA-NeXT | 24.7 | 23.0 | 19.7 | 21.6 | 24.2 | 22.6 | |
| 27.2 ↑2.5 | 20.8 ↓2.2 | 25.0 ↑5.3 | 22.4 ↑0.8 | 27.9 ↑3.7 | 24.7 ↑2.1 | ||
| Model | Com. | FS | NT | F2F | DF | FSh |
|---|---|---|---|---|---|---|
| Qwen2.5-VL FAQ-IT | c23 | |||||
| c40 | ||||||
| original | 27.2 | |||||
| LLaVA-NeXT FAQ-IT | c23 | 49.3 | 46.9 | |||
| c40 | ||||||
| original | 51.9 | 50.9 |
| Model | Condition | Edges | Color | Light | Texture | Pixel | Shapes |
|---|---|---|---|---|---|---|---|
| Qwen2.5-VL | 11.3 | 8.4 | 4.5 | 8.0 | 13.2 | 9.0 | |
| 13.5 | 9.1 | 7.9 | 10.9 | 14.3 | 12.1 | ||
| 52.9 | 45.9 | 46.2 | 43.1 | 50.2 | 44.3 | ||
| LLaVA-NeXT | 9.8 | 11.1 | 14.9 | 16.6 | 10.7 | 11.4 | |
| 12.8 | 13.9 | 18.8 | 16.4 | 14.3 | 13.4 | ||
| 45.7 | 42.1 | 47.9 | 42.1 | 43.9 | 46.8 |
| Training Setup | CDF | DFo | WDF | ||||
|---|---|---|---|---|---|---|---|
| Model | Training Data | Acc | F1 | Acc | F1 | Acc | F1 |
| Qwen2.5-VL | 17.9 | 13.8 | 14.5 | 11.1 | 10.8 | 8.9 | |
| 49.1 | 59.1 | 54.7 | 57.9 | 40.9 | 54.8 | ||
| 73.3 | 71.0 | 78.6 | 70.1 | 65.9 | 59.3 | ||
| LLaVA-NeXT | 12.3 | 10.2 | 10.5 | 9.5 | 8.7 | 7.6 | |
| 47.2 | 49.0 | 52.3 | 54.8 | 39.8 | 49.6 | ||
| Exp 1: Robustness | Exp 2: Strategy | Exp 3: Open-Ended QA | |||
|---|---|---|---|---|---|
| Cond. | Acc | Strategy | Acc | Model | Score |
| Original | 54.4% | Baseline | 30.3% | Random | 0.0% |
| Bright. | 52.8\%_{\textbf{{\color[rgb]{1,0,0}-1.6}}} | Multi-Stage | 35.6\%_{\text{{{\color[rgb]{0,0.3906,0}+5.3}}}} | GPT-4o | 13.2% |
| Compr. | 49.8\%_{\textbf{{\color[rgb]{1,0,0}-4.6}}} | Lvl 3 Only | 37.8\%_{\text{{{\color[rgb]{0,0.3906,0}+7.5}}}} | Baseline | 18.5% |
| Blur | 46.2\%_{\textbf{{{\color[rgb]{1,0,0}-8.2}}}} | Mixed | \mathbf{54.4\%_{\text{{{\color[rgb]{0,0.3906,0}+24.1}}}}} | Ours | |
| Overall Benchmark Accuracy (%) | ||||||
|---|---|---|---|---|---|---|
| Model | Data Ratio | Level 1 | Level 2 | Level 3 | Average | |
| Qwen2.5-VL | 100% (Raw) | 90.3 | 40.1 | 23.5 | 51.3 | |
| 90% (Ours) | 89.9 | 41.4 | 25.8 | 52.4 | ||
| 80% | 89.7 | 41.0 | 24.9 | 51.9 | ||
| 70% | 88.1 | 39.5 | 23.2 | 50.3 | ||
| 60% | 85.4 | 36.8 | 20.1 | 47.4 | ||
| Configuration | Value |
|---|---|
| Video Input Preprocessing | |
| Max Pixels per Video | |
| Max Frames per Video | |
| Input Video Frame Resolution | |
| Training / Fine-tuning Configuration | |
| Total Training Epochs | |