Deepfake detection models often generate natural-language explanations, yet their reasoning is frequently ungrounded in visual evidence, limiting reliability. Existing evaluations measure classification accuracy but overlook reasoning fidelity. We propose DeepfakeJudge, a framework for scalable reasoning supervision and evaluation, that integrates an out-of-distribution benchmark containing recent generative and editing forgeries, a human-annotated subset with visual reasoning labels, and a suite of evaluation models, that specialize in evaluating reasoning rationales without the need for explicit ground truth reasoning rationales. The Judge is optimized through a bootstrapped generator-evaluator process that scales human feedback into structured reasoning supervision and supports both pointwise and pairwise evaluation. On the proposed meta-evaluation benchmark, our reasoning-bootstrapped model achieves an accuracy of 96.2%, outperforming 30x larger baselines. The reasoning judge attains very high correlation with human ratings and 98.9% percent pairwise agreement on the human-annotated meta-evaluation subset. These results establish reasoning fidelity as a quantifiable dimension of deepfake detection and demonstrate scalable supervision for interpretable deepfake reasoning. Our user study shows that participants preferred the reasonings generated by our framework 70% of the time, in terms of faithfulness, groundedness, and usefulness, compared to those produced by other models and datasets. All of our datasets, models, and codebase are open-sourced.
Figures & tables
Figure 1: Comparison of reasoning rationales from SIDA, Qwen-3-VL-235B, Gemini-2.5-Flash, and Human Annotation in our proposed DeepfakeJudge-Reason. Red and green indicate incorrect and correct flags respectively. Our human annotations provide dense, localized, and accurate reasoning.
Figure 2: Existing metrics (ROUGE, METEOR, BERTScore) fail to capture reasoning quality. DeepFakeJudge directly evaluates the image, providing both a reasoning quality score and a rationale.
Figure 3: a) Data generation process for OOD-dataset generation b) Data Distribution for DeepfakeJudge-Detect/Reason splits. i) Shows the distribution of top-10 classes in the real-subset, ii) shows the distribution of generation models, and iii) shows the label-wise distribution.
Figure 4: Overview of the DeepFakeJudge bootstrapping pipeline. Step 1: Generating gold standard reasoning rationales using the in-domain human annotated dataset (Section 3.1 ) Step 2: The generator creates reasoning responses for each image–label pair across five rating levels. Step 3: The evaluator provides feedback and re-scores the responses until alignment is achieved. Step 4: All accepted responses are paraphrased to create stylistically diverse but semantically consistent data.
Figure 5: Comparison of BERTScore and BLEU of candidate ratings against the gold standard ratings. Our data bootstrapping method generates reasonings of continuously degrading quality.
Figure 6: Data distribution for DeepfakeJudge-Meta dataset.
Type
Model
Real Acc
Real F1
Fake Acc
Fake F1
Overall Acc
Overall F1
Closed
Gemini-2.5-Flash
96.6
73.7
34.5
50.0
65.5
61.9
ChatGPT-4o-mini
95.8
70.2
22.7
35.8
59.3
53.0
Open
InternVL3.5-1B-HF
47.8
63.0
47.8
11.4
47.8
37.2
Qwen3-VL-2B-Instruct
49.8
44.7
49.8
54.0
49.8
49.3
Qwen-3-VL-8B-Instruct
50.4
23.7
50.4
63.2
50.4
43.5
Google-Gemma-12B
57.7
57.4
49.4
54.0
66.1
60.8
Table 1: Comparison of SOTA open-sourced, closed-source, reasoning, and deepfake models on the DeepfakeJudge-Detect datasets.
Type
Model
B-1
B-2
B-3
R-1
R-2
R-L
METEOR
BERT
DFJ-3B ( ↑ )
Closed
Gemini-2.5-Flash
0.05
0.02
0.02
0.30
0.05
0.17
0.17
0.60
3.17
ChatGPT-4o-mini
0.01
0.01
0.01
0.15
0.01
0.08
0.05
0.35
2.83
Open
InternVL3.5-1B-HF
0.05
0.02
0.02
0.27
0.05
0.17
0.15
0.56
2.44
Qwen-3-VL-2B-Instruct
0.07
0.02
0.02
0.31
0.04
0.18
0.15
0.59
2.36
Qwen-3-VL-4B-Instruct
0.01
0.01
0.01
0.20
0.01
0.11
0.12
0.56
2.93
Qwen-3-VL-8B-Instruct
0.01
0.01
0.01
0.16
0.01
0.10
0.09
0.53
2.51
Table 2: Evaluation of models on the DeepfakeJudge-Reason. BLEU ( B-1–3 ), ROUGE ( R-1–L ), METEOR, and BERTScore are normalized to [0, 1]. DeepfakeJudge-3B (DFJ-3B) provides more consistent comparative ratings on a 1–5 scale, whereas other metrics yield inconclusive results. For all of the above metrics, higher score means better performance.
Type
Model
DeepfakeJudge-Meta
DeepfakeJudge-Meta-Human
RMSE ( ↓ )
MSE ( ↓ )
s ( ↑ )
p ( ↑ )
RMSE ( ↓ )
MSE ( ↓ )
s ( ↑ )
p ( ↑ )
Closed
Gemini-Flash-2.5
1.09
1.20
0.82
0.83
1.11
1.24
0.85
0.83
GPT-4o-Mini
0.78
0.60
0.87
0.87
0.81
0.66
0.88
0.86
Open
Qwen-3-VL-2B-Instruct
1.60
2.55
0.60
0.61
1.41
1.98
0.69
0.72
Qwen-3-VL-4B-Instruct
1.59
2.53
0.57
0.58
1.53
2.34
0.60
0.61
Qwen-3-VL-8B-Instruct
1.19
1.41
0.76
0.75
1.28
1.63
0.79
0.76
Table 3: Comparison of regression and correlation metrics across DeepfakeJudge-Meta and DeepfakeJudge-Meta-Human for pointwise evaluation. Arrows indicate desired direction of improvement (↑ higher is better, ↓ lower is better).
Type
Model
DFJM
DFJMH
Closed
Gemini-Flash-2.5
91.7
94.2
GPT-4o-Mini
90.3
89.8
Open
Qwen-3-VL-2B-Instruct
74.8
65.1
Qwen-3-VL-4B-Instruct
75.8
72.7
Qwen-3-VL-8B-Instruct
86.0
88.6
Qwen-3-VL-30B-Instruct
91.3
96.3
Table 4: Pairwise accuracy results for models on the DeepfakeJudge-Meta and DeepfakeJudge-Meta-Human datasets.
Figure 7: Overview of the human annotation framework. a) Shows the user dashboard, b) shows an example of the annotation process. The user selects a flag, draws the corresponding bounding box over the image, and then writes a referring expression.
Table 7: Example of a real image and corresponding degraded ratings produced by our bootstrapping process.
Table 8: Example of an edited image and corresponding degraded ratings produced by our bootstrapping process.
Table 9: Example of a fake image and corresponding degraded ratings produced by our bootstrapping process.
Table 10: Example of an edited image and corresponding degraded ratings produced by our bootstrapping process.
Table 11: Example of fake image and corresponding degraded ratings produced by our bootstrapping process..
Table 12: Example of a real image and corresponding degraded ratings produced by our bootstrapping process..
Qualitative Examples of Paraphrased Gold-Standard Ratings.
(Image Label: Edited)
Variant
Reasoning
Gold
The man’s face and head area exhibit unnatural texture, appearing synthetic and inconsistent with typical skin and hair details. Additionally, the motorcycle seat shows a dip as if supporting weight, yet the man is seated further forward, not making contact with that indentation. These discrepancies in texture and physical contact indicate manipulation in these areas. The rest of the image, including the motorcycle, background, and rider’s clothing, appears visually consistent and realistic.
Para-1
The man’s face texture looks somewhat unnatural, with an inconsistent pattern that suggests synthetic alteration. Also, the motorcycle seat shows a clear dip as if bearing weight, but the man is seated slightly forward, not pressing down on that area. These visual clues suggest the face region was edited. The rest of the scene, including the motorcycle and background, seems consistent and natural.
Para-2
The texture of the man’s face appears somewhat artificial, exhibiting an inconsistent pattern that indicates possible synthetic modification. Additionally, the motorcycle seat shows a noticeable dip as if someone is applying pressure, yet the man is positioned slightly forward, not affecting that area. These visual indicators imply that the face region has been altered. The remainder of the scene, including the motorcycle and background, appears coherent and natural.
Table 13: Comparison of a fake image and the corresponding paraphrased and Gold-standard reasoning produced by our bootstrapping process.
(Image Label: Fake)
Variant
Reasoning
Gold
Two animals blend confusingly into the surrounding foliage, lacking clear form or recognition, with edges smudged against leaves and branches. The front leaves exhibit inconsistent lighting compared to the overall scene, appearing artificially bright with an uneven glow that doesn’t match the ambient light direction. Shadows under the animals are absent, resulting in unnatural floating effect and weak grounding. The perspective appears compressed in the lower-right, making the background look too close and flattened, disrupting spatial depth cues. These issues with object blending, lighting disparity, missing shadows, and distorted depth strongly indicate the image is AI-generated.
Para-1
The image shows two animals poorly defined and blending into the dense foliage, with unclear boundaries that reduce recognizability. The lighting on the leaves in the foreground is uneven and doesn’t align well with the rest of the scene, suggesting inconsistent illumination. Notably, shadows beneath the animals are missing, causing them to appear as if floating above the ground. The perspective in the lower-right section looks unnaturally compressed, making the background seem too near and flattening the spatial depth. These subtle inconsistencies in object clarity, lighting, shadows, and perspective point toward an artificially generated image.
Para-2
The image depicts two animals that are not clearly defined and are blending into the surrounding foliage, resulting in indistinct edges that hinder identification. The lighting on the foreground leaves is inconsistent and does not match the overall scene, indicating uneven illumination. Additionally, the shadows beneath the animals are absent, making them appear as if they are floating. The perspective in the lower-right area appears unnaturally compressed, causing the background to seem too close and flattening the depth of the scene. These subtle discrepancies in object definition, lighting, shadows, and perspective suggest that the image is artificially generated.
Para-3
This image features two animals that are poorly defined and merge with the dense foliage, leading to unclear boundaries that make recognition difficult. The lighting on the leaves in the foreground is uneven and does not correspond well with the rest of the scene, suggesting inconsistency in illumination. Notably, the absence of shadows under the animals creates an illusion of them floating above the ground. The perspective in the lower-right section appears unnaturally squished, making the background look too close and diminishing spatial depth. These minor inconsistencies in object clarity, lighting, shadows, and perspective indicate that the image is likely artificially generated.
Table 14: Comparison of a fake image and the corresponding paraphrased and Gold-standard reasoning produced by our bootstrapping process.
(Image Label: Fake)
Variant
Reasoning
Gold
The horse appears to be missing its left hind leg, resulting in a strange, incomplete posture. Its ears and face display inconsistent shapes and edges that do not align naturally, lending a distorted appearance. Despite the reflective surface of the water beneath, the horse’s reflection is entirely absent, which breaks expected visual continuity. Additionally, the horse seems to be walking directly on the water’s surface rather than on solid ground, creating an unrealistic effect of suspension. These geometric inconsistencies, lack of proper reflections, and object incompleteness strongly suggest that the image was AI-generated.
Para-1
The horse’s left hind leg is notably absent, causing an unnatural pose. The ears and face show some irregularities in shape that suggest digital alteration. The reflection on the water is missing where it should appear, breaking the expected visual consistency. Also, the horse appears to stand on water rather than solid ground, which is unrealistic. These signs of incomplete objects and missing reflections strongly indicate AI generation.
Para-2
The left hind leg of the horse is conspicuously missing, resulting in an awkward stance. Additionally, the shape of the ears and face presents inconsistencies that imply digital manipulation. The expected reflection on the water is absent, disrupting the visual coherence. Furthermore, the horse appears to be walking on water, which is implausible. These indicators of incomplete elements and absent reflections strongly suggest AI generation.
Para-3
The horse’s left hind leg is clearly missing, leading to an unnatural posture. There are also discrepancies in the shape of the ears and face that hint at digital alterations. The reflection on the water is lacking, which undermines the visual consistency. Moreover, the horse seems to be standing on water instead of solid ground, which is unrealistic. These signs of incomplete objects and missing reflections strongly indicate AI generation.
Table 15: Comparison of a fake image and the corresponding paraphrased and Gold-standard reasoning produced by our bootstrapping process.
Table 18: Performance of DeepfakeJudge-3B on the DeepfakeJudge-Meta dataset with and without paraphrasing.
Model
Paraphrasing
DFJM (%)
DeepfakeJudge-3B
✓
94.4
DeepfakeJudge-3B
✗
89.1
Table 19: Pairwise accuracy of DeepfakeJudge-3B on the DeepfakeJudge-Meta dataset with and without paraphrasing.
Image
Ground Truth Reasoning (from DDVQA)
Evaluation Response
Score
The image looks fake. It is an image with manipulated face regions. The person’s mouth looks a bit fake because overly small mouth underage individuals. The person’s skin looks a bit fake because the skin is overly smooth and lacks of details, such as skin texture the person has mismatched bangs.
This response correctly identifies the manipulated face regions and mentions the smooth skin, which aligns with the ground truth label. However, it incorrectly suggests the person is underage, which is not supported by the image.
3
The image looks fake. It looks like a computer-generated artist styled image for animation, carton or films. The person’s eyes look very fake because the eyes look unnaturally asymmetric. The person has mismatched bangs. The person’s nose looks very fake because the nose lookes curved.
This response correctly identifies the image as fake, noting asymmetry and mismatched features. However, it lacks specificity about the style and suggests the person is animated, which is not entirely accurate.
3
The image looks fake. It looks like an ai-generated image although with human appearance but with unrealistic texture or lighting. The person’s eyes look a bit fake because the eyes look unnaturally asymmetric. The eyes looks too rigid and lifeless to convey expresssion. The person has mismatched bangs. Mismatched mustache. The lighting in the image is inconsistent, unrealistic shadows. The person’s nose looks very fake because the nose lookes curved. The nose is placed at the wrong place on the face.
Fully accurate, complete, and well-grounded. The response precisely explain the reason behind the correct classification with no notable errors or omissions.
5
The image looks fake. It is an image with manipulated face regions. The person’s skin looks very fake because the skin is overly smooth and lacks of details, such as skin texture there are stains/flaws on the person’s face there are areas looking unnaturally brighter there area of looking unnaturally darker the person’s nose looks a bit fake because the nose do not align properly with other facial features. The person’s eyebrows look a bit fake because the eyebrows do not match the face’s structure.
This response accurately identifies the manipulation of facial features, noting the lack of natural skin texture and the unnatural brightness and darkness. However, it could have been more specific about the alignment issues with the nose and eyebrows.
4
Table 20: Rationales and scores generated by DeepfakeJudge-3B on the ground truth reasonings of the test-set of DD-VQA. DeepfakeJudge-3B correctly identifies where the reasoning is ungrounded and superficial, and gives a rating accordingly.
Multimodal large language models (MLLMs) can explain deepfake verdicts in natural language, but such explanations are not necessarily visually grounded in the visual evidence underlying the prediction. A model may describe plausible artifacts inferred from language priors rather than from image evidence. Existing grounding methods improve visual reliance through decoding or attention interventions, but they generally strengthen grounding over the entire image, making them ill-suited for forensic artifacts that are subtle, spatially localized, and image-dependent. We propose Look Before You Judge, a training-free framework that formulates explainable deepfake detection as a sequential evidence acquisition process. Instead of directly predicting image authenticity from holistic visual reasoning, our framework first identifies image-specific candidate evidence regions by contrasting the MLLM's decoder-to-visual attention between an original image and its Gaussian-blurred counterpart. The identified regions are then inspected individually, and the resulting local evidence is integrated with the global image context before reaching a final verdict. The framework operates without manipulation masks, external forensic models, or parameter updates, making it directly applicable to off-the-shelf MLLMs. Across five open-source MLLMs on TriDF and MMTD-Set, our framework improves detection accuracy by up to 12.8%, reduces CHAIR by up to 33.4% and hallucination rate by up to 21.3%, and outperforms representative training-free decoding and attention methods.
Chia-Ling Chen, Yu-Ting Ta, Jian-Yu Jiang-Lin +8
National Taiwan University · National Tsing Hua University · National Yang Ming Chiao Tung University
As deepfake detection models increasingly produce natural language explanations, their reasoning often remains weakly grounded in visual artifacts, limiting reliability and user trust. Existing benchmarks mainly evaluate classification accuracy, overlooking whether explanations reflect the actual manipulations. This gap hinders progress toward deployable, explainable deepfake detection systems. To this end, we introduce XPlainVerse, a large-scale benchmark designed for joint deepfake detection and human-centered explanation. XPlainVerse comprises one million real and manipulated images, pairing authentic images from five established sources with forgeries generated by twelve off-the-shelf image editing and synthesis models. We further propose a multi-stage filtering pipeline, Edit-Check, to verify if manipulations satisfy their intended edits, enabling reliable reasoning supervision at scale. Beyond dataset scale, XPlainVerse provides two complementary explanation styles: technical explanations for expert analysis and simplified explanations optimized for non-technical users. To evaluate explanation quality beyond surface similarity, we propose novel metrics, EntityScore and EvidenceScore, that measure reasoning fidelity by checking whether explanations correctly identify manipulated entities and visual evidence. Human annotations on 2,000 explanation pairs validate our dataset quality against human judgment. We believe XPlainVerse will establish grounded explanation quality as a measurable dimension of deepfake detection and support scalable research on trustworthy, interpretable models.
Abhijeet Narang, Kartik Kuckreja, Shreya Ghosh +3
Monash University · MBZUAI · The University of Queensland
Recent advances in generative image models have made many manipulated images highly realistic, raising the need for detectors that are not only accurate but also able to provide visual evidence for their decisions. In this paper, we present our solution to the Explainable Deepfake Detection Challenge [2] on the XPlainVerse dataset [1], where systems are required to predict whether an image is real or fake and generate both complex and simple explanations grounded in visible forensic cues. Our method follows a modular detection-and-explanation design. For the real/fake decision, we build a multi-backbone detector that combines several DINOv3 models with Mesorch manipulation-localization features, bringing together pretrained visual representations, DCT-aware cues, and multi-scale forensic information. To inject explanation evidence into the detector, we use a Grounding-DINO-based pseudo-mask generation pipeline that converts local artifact descriptions from training explanations into weak patch- level supervision for an Artifact Evidence Map. We further introduce a local patch-level contrastive objective that separates artifact and authenticity evidence in the detector feature space without requiring paired images or pixel-level manipulation masks. For language output, we use class-conditional Qwen3-VL models to generate complex explanations for fake and real predictions, followed by a GRPO-optimized text simplification model. The proposed methods were trained and evaluated on the challenge subset of XPlainVerse. On the full test split, our submission achieves 0.9349 detection accuracy, 0.5571 explanation score, and a 0.7456 final challenge score.
Artem Filippov, Aleksandr Gushchin, Kirill Koltsov +2
Lomonosov Moscow State University Moscow, Russia · MSU Institute for Artificial Intelligence Moscow, Russia