cs.CVFeb 23, 2026
SavePixels Don't Lie (But Your Detector Might): Bootstrapping MLLM-as-a-Judge for Trustworthy Deepfake Detection and Reasoning Supervision
Organizations: MBZUAI · Monash University *
Abstract
Deepfake detection models often generate natural-language explanations, yet their reasoning is frequently ungrounded in visual evidence, limiting reliability. Existing evaluations measure classification accuracy but overlook reasoning fidelity. We propose DeepfakeJudge, a framework for scalable reasoning supervision and evaluation, that integrates an out-of-distribution benchmark containing recent generative and editing forgeries, a human-annotated subset with visual reasoning labels, and a suite of evaluation models, that specialize in evaluating reasoning rationales without the need for explicit ground truth reasoning rationales. The Judge is optimized through a bootstrapped generator-evaluator process that scales human feedback into structured reasoning supervision and supports both pointwise and pairwise evaluation. On the proposed meta-evaluation benchmark, our reasoning-bootstrapped model achieves an accuracy of 96.2%, outperforming 30x larger baselines. The reasoning judge attains very high correlation with human ratings and 98.9% percent pairwise agreement on the human-annotated meta-evaluation subset. These results establish reasoning fidelity as a quantifiable dimension of deepfake detection and demonstrate scalable supervision for interpretable deepfake reasoning. Our user study shows that participants preferred the reasonings generated by our framework 70% of the time, in terms of faithfulness, groundedness, and usefulness, compared to those produced by other models and datasets. All of our datasets, models, and codebase are open-sourced.
Figures & tables
Figure 1: Comparison of reasoning rationales from SIDA, Qwen-3-VL-235B, Gemini-2.5-Flash, and Human Annotation in our proposed DeepfakeJudge-Reason. Red and green indicate incorrect and correct flags respectively. Our human annotations provide dense, localized, and accurate reasoning.
Figure 2: Existing metrics (ROUGE, METEOR, BERTScore) fail to capture reasoning quality. DeepFakeJudge directly evaluates the image, providing both a reasoning quality score and a rationale.
Figure 3: a) Data generation process for OOD-dataset generation b) Data Distribution for DeepfakeJudge-Detect/Reason splits. i) Shows the distribution of top-10 classes in the real-subset, ii) shows the distribution of generation models, and iii) shows the label-wise distribution.
Figure 4: Overview of the DeepFakeJudge bootstrapping pipeline. Step 1: Generating gold standard reasoning rationales using the in-domain human annotated dataset (Section 3.1 ) Step 2: The generator creates reasoning responses for each image–label pair across five rating levels. Step 3: The evaluator provides feedback and re-scores the responses until alignment is achieved. Step 4: All accepted responses are paraphrased to create stylistically diverse but semantically consistent data.
Figure 5: Comparison of BERTScore and BLEU of candidate ratings against the gold standard ratings. Our data bootstrapping method generates reasonings of continuously degrading quality.
Figure 6: Data distribution for DeepfakeJudge-Meta dataset.
| Type | Model | Real Acc | Real F1 | Fake Acc | Fake F1 | Overall Acc | Overall F1 |
| Closed | Gemini-2.5-Flash | 96.6 | 73.7 | 34.5 | 50.0 | 65.5 | 61.9 |
| ChatGPT-4o-mini | 95.8 | 70.2 | 22.7 | 35.8 | 59.3 | 53.0 | |
| Open | InternVL3.5-1B-HF | 47.8 | 63.0 | 47.8 | 11.4 | 47.8 | 37.2 |
| Qwen3-VL-2B-Instruct | 49.8 | 44.7 | 49.8 | 54.0 | 49.8 | 49.3 | |
| Qwen-3-VL-8B-Instruct | 50.4 | 23.7 | 50.4 | 63.2 | 50.4 | 43.5 | |
| Google-Gemma-12B | 57.7 | 57.4 | 49.4 | 54.0 | 66.1 | 60.8 |
Table 1: Comparison of SOTA open-sourced, closed-source, reasoning, and deepfake models on the DeepfakeJudge-Detect datasets.
| Type | Model | B-1 | B-2 | B-3 | R-1 | R-2 | R-L | METEOR | BERT | DFJ-3B ( ) |
| Closed | Gemini-2.5-Flash | 0.05 | 0.02 | 0.02 | 0.30 | 0.05 | 0.17 | 0.17 | 0.60 | 3.17 |
| ChatGPT-4o-mini | 0.01 | 0.01 | 0.01 | 0.15 | 0.01 | 0.08 | 0.05 | 0.35 | 2.83 | |
| Open | InternVL3.5-1B-HF | 0.05 | 0.02 | 0.02 | 0.27 | 0.05 | 0.17 | 0.15 | 0.56 | 2.44 |
| Qwen-3-VL-2B-Instruct | 0.07 | 0.02 | 0.02 | 0.31 | 0.04 | 0.18 | 0.15 | 0.59 | 2.36 | |
| Qwen-3-VL-4B-Instruct | 0.01 | 0.01 | 0.01 | 0.20 | 0.01 | 0.11 | 0.12 | 0.56 | 2.93 | |
| Qwen-3-VL-8B-Instruct | 0.01 | 0.01 | 0.01 | 0.16 | 0.01 | 0.10 | 0.09 | 0.53 | 2.51 |
Table 2: Evaluation of models on the DeepfakeJudge-Reason. BLEU ( B-1–3 ), ROUGE ( R-1–L ), METEOR, and BERTScore are normalized to [0, 1]. DeepfakeJudge-3B (DFJ-3B) provides more consistent comparative ratings on a 1–5 scale, whereas other metrics yield inconclusive results. For all of the above metrics, higher score means better performance.
| Type | Model | DeepfakeJudge-Meta | DeepfakeJudge-Meta-Human | ||||||
|---|---|---|---|---|---|---|---|---|---|
| RMSE ( ) | MSE ( ) | s ( ) | p ( ) | RMSE ( ) | MSE ( ) | s ( ) | p ( ) | ||
| Closed | Gemini-Flash-2.5 | 1.09 | 1.20 | 0.82 | 0.83 | 1.11 | 1.24 | 0.85 | 0.83 |
| GPT-4o-Mini | 0.78 | 0.60 | 0.87 | 0.87 | 0.81 | 0.66 | 0.88 | 0.86 | |
| Open | Qwen-3-VL-2B-Instruct | 1.60 | 2.55 | 0.60 | 0.61 | 1.41 | 1.98 | 0.69 | 0.72 |
| Qwen-3-VL-4B-Instruct | 1.59 | 2.53 | 0.57 | 0.58 | 1.53 | 2.34 | 0.60 | 0.61 | |
| Qwen-3-VL-8B-Instruct | 1.19 | 1.41 | 0.76 | 0.75 | 1.28 | 1.63 | 0.79 | 0.76 | |
Table 3: Comparison of regression and correlation metrics across DeepfakeJudge-Meta and DeepfakeJudge-Meta-Human for pointwise evaluation. Arrows indicate desired direction of improvement (↑ higher is better, ↓ lower is better).
| Type | Model | DFJM | DFJMH |
| Closed | Gemini-Flash-2.5 | 91.7 | 94.2 |
| GPT-4o-Mini | 90.3 | 89.8 | |
| Open | Qwen-3-VL-2B-Instruct | 74.8 | 65.1 |
| Qwen-3-VL-4B-Instruct | 75.8 | 72.7 | |
| Qwen-3-VL-8B-Instruct | 86.0 | 88.6 | |
| Qwen-3-VL-30B-Instruct | 91.3 | 96.3 |
Table 4: Pairwise accuracy results for models on the DeepfakeJudge-Meta and DeepfakeJudge-Meta-Human datasets.
Figure 7: Overview of the human annotation framework. a) Shows the user dashboard, b) shows an example of the annotation process. The user selects a flag, draws the corresponding bounding box over the image, and then writes a referring expression.
Table 7: Example of a real image and corresponding degraded ratings produced by our bootstrapping process.
Table 8: Example of an edited image and corresponding degraded ratings produced by our bootstrapping process.
Table 9: Example of a fake image and corresponding degraded ratings produced by our bootstrapping process.
Table 10: Example of an edited image and corresponding degraded ratings produced by our bootstrapping process.
Table 11: Example of fake image and corresponding degraded ratings produced by our bootstrapping process..
Table 12: Example of a real image and corresponding degraded ratings produced by our bootstrapping process..
| Qualitative Examples of Paraphrased Gold-Standard Ratings. | |
| (Image Label: Edited) | |
| Variant | Reasoning |
| Gold | The man’s face and head area exhibit unnatural texture, appearing synthetic and inconsistent with typical skin and hair details. Additionally, the motorcycle seat shows a dip as if supporting weight, yet the man is seated further forward, not making contact with that indentation. These discrepancies in texture and physical contact indicate manipulation in these areas. The rest of the image, including the motorcycle, background, and rider’s clothing, appears visually consistent and realistic. |
| Para-1 | The man’s face texture looks somewhat unnatural, with an inconsistent pattern that suggests synthetic alteration. Also, the motorcycle seat shows a clear dip as if bearing weight, but the man is seated slightly forward, not pressing down on that area. These visual clues suggest the face region was edited. The rest of the scene, including the motorcycle and background, seems consistent and natural. |
| Para-2 | The texture of the man’s face appears somewhat artificial, exhibiting an inconsistent pattern that indicates possible synthetic modification. Additionally, the motorcycle seat shows a noticeable dip as if someone is applying pressure, yet the man is positioned slightly forward, not affecting that area. These visual indicators imply that the face region has been altered. The remainder of the scene, including the motorcycle and background, appears coherent and natural. |
Table 13: Comparison of a fake image and the corresponding paraphrased and Gold-standard reasoning produced by our bootstrapping process.
| (Image Label: Fake) | |
| Variant | Reasoning |
| Gold | Two animals blend confusingly into the surrounding foliage, lacking clear form or recognition, with edges smudged against leaves and branches. The front leaves exhibit inconsistent lighting compared to the overall scene, appearing artificially bright with an uneven glow that doesn’t match the ambient light direction. Shadows under the animals are absent, resulting in unnatural floating effect and weak grounding. The perspective appears compressed in the lower-right, making the background look too close and flattened, disrupting spatial depth cues. These issues with object blending, lighting disparity, missing shadows, and distorted depth strongly indicate the image is AI-generated. |
| Para-1 | The image shows two animals poorly defined and blending into the dense foliage, with unclear boundaries that reduce recognizability. The lighting on the leaves in the foreground is uneven and doesn’t align well with the rest of the scene, suggesting inconsistent illumination. Notably, shadows beneath the animals are missing, causing them to appear as if floating above the ground. The perspective in the lower-right section looks unnaturally compressed, making the background seem too near and flattening the spatial depth. These subtle inconsistencies in object clarity, lighting, shadows, and perspective point toward an artificially generated image. |
| Para-2 | The image depicts two animals that are not clearly defined and are blending into the surrounding foliage, resulting in indistinct edges that hinder identification. The lighting on the foreground leaves is inconsistent and does not match the overall scene, indicating uneven illumination. Additionally, the shadows beneath the animals are absent, making them appear as if they are floating. The perspective in the lower-right area appears unnaturally compressed, causing the background to seem too close and flattening the depth of the scene. These subtle discrepancies in object definition, lighting, shadows, and perspective suggest that the image is artificially generated. |
| Para-3 | This image features two animals that are poorly defined and merge with the dense foliage, leading to unclear boundaries that make recognition difficult. The lighting on the leaves in the foreground is uneven and does not correspond well with the rest of the scene, suggesting inconsistency in illumination. Notably, the absence of shadows under the animals creates an illusion of them floating above the ground. The perspective in the lower-right section appears unnaturally squished, making the background look too close and diminishing spatial depth. These minor inconsistencies in object clarity, lighting, shadows, and perspective indicate that the image is likely artificially generated. |
Table 14: Comparison of a fake image and the corresponding paraphrased and Gold-standard reasoning produced by our bootstrapping process.
| (Image Label: Fake) | |
| Variant | Reasoning |
| Gold | The horse appears to be missing its left hind leg, resulting in a strange, incomplete posture. Its ears and face display inconsistent shapes and edges that do not align naturally, lending a distorted appearance. Despite the reflective surface of the water beneath, the horse’s reflection is entirely absent, which breaks expected visual continuity. Additionally, the horse seems to be walking directly on the water’s surface rather than on solid ground, creating an unrealistic effect of suspension. These geometric inconsistencies, lack of proper reflections, and object incompleteness strongly suggest that the image was AI-generated. |
| Para-1 | The horse’s left hind leg is notably absent, causing an unnatural pose. The ears and face show some irregularities in shape that suggest digital alteration. The reflection on the water is missing where it should appear, breaking the expected visual consistency. Also, the horse appears to stand on water rather than solid ground, which is unrealistic. These signs of incomplete objects and missing reflections strongly indicate AI generation. |
| Para-2 | The left hind leg of the horse is conspicuously missing, resulting in an awkward stance. Additionally, the shape of the ears and face presents inconsistencies that imply digital manipulation. The expected reflection on the water is absent, disrupting the visual coherence. Furthermore, the horse appears to be walking on water, which is implausible. These indicators of incomplete elements and absent reflections strongly suggest AI generation. |
| Para-3 | The horse’s left hind leg is clearly missing, leading to an unnatural posture. There are also discrepancies in the shape of the ears and face that hint at digital alterations. The reflection on the water is lacking, which undermines the visual consistency. Moreover, the horse seems to be standing on water instead of solid ground, which is unrealistic. These signs of incomplete objects and missing reflections strongly indicate AI generation. |
Table 15: Comparison of a fake image and the corresponding paraphrased and Gold-standard reasoning produced by our bootstrapping process.
| Model | Para | RMSE ( ) | MSE ( ) | s ( ) | p ( ) |
|---|---|---|---|---|---|
| DeepfakeJudge-3B | ✓ | 0.69 | 0.48 | 0.92 | 0.92 |
| DeepfakeJudge-3B | ✗ | 1.39 | 1.93 | 0.60 | 0.62 |
Table 18: Performance of DeepfakeJudge-3B on the DeepfakeJudge-Meta dataset with and without paraphrasing.
| Model | Paraphrasing | DFJM (%) |
|---|---|---|
| DeepfakeJudge-3B | ✓ | 94.4 |
| DeepfakeJudge-3B | ✗ | 89.1 |
Table 19: Pairwise accuracy of DeepfakeJudge-3B on the DeepfakeJudge-Meta dataset with and without paraphrasing.
| Image | Ground Truth Reasoning (from DDVQA) | Evaluation Response | Score |
|---|---|---|---|
| The image looks fake. It is an image with manipulated face regions. The person’s mouth looks a bit fake because overly small mouth underage individuals. The person’s skin looks a bit fake because the skin is overly smooth and lacks of details, such as skin texture the person has mismatched bangs. | This response correctly identifies the manipulated face regions and mentions the smooth skin, which aligns with the ground truth label. However, it incorrectly suggests the person is underage, which is not supported by the image. | 3 | |
| The image looks fake. It looks like a computer-generated artist styled image for animation, carton or films. The person’s eyes look very fake because the eyes look unnaturally asymmetric. The person has mismatched bangs. The person’s nose looks very fake because the nose lookes curved. | This response correctly identifies the image as fake, noting asymmetry and mismatched features. However, it lacks specificity about the style and suggests the person is animated, which is not entirely accurate. | 3 | |
| The image looks fake. It looks like an ai-generated image although with human appearance but with unrealistic texture or lighting. The person’s eyes look a bit fake because the eyes look unnaturally asymmetric. The eyes looks too rigid and lifeless to convey expresssion. The person has mismatched bangs. Mismatched mustache. The lighting in the image is inconsistent, unrealistic shadows. The person’s nose looks very fake because the nose lookes curved. The nose is placed at the wrong place on the face. | Fully accurate, complete, and well-grounded. The response precisely explain the reason behind the correct classification with no notable errors or omissions. | 5 | |
| The image looks fake. It is an image with manipulated face regions. The person’s skin looks very fake because the skin is overly smooth and lacks of details, such as skin texture there are stains/flaws on the person’s face there are areas looking unnaturally brighter there area of looking unnaturally darker the person’s nose looks a bit fake because the nose do not align properly with other facial features. The person’s eyebrows look a bit fake because the eyebrows do not match the face’s structure. | This response accurately identifies the manipulation of facial features, noting the lack of natural skin texture and the unnatural brightness and darkness. However, it could have been more specific about the alignment issues with the nose and eyebrows. | 4 |
Table 20: Rationales and scores generated by DeepfakeJudge-3B on the ground truth reasonings of the test-set of DD-VQA. DeepfakeJudge-3B correctly identifies where the reasoning is ungrounded and superficial, and gives a rating accordingly.
More in this topic
Deepfake Detection
Deepfake detection identifies manipulated or AI-generated media to support trustworthy communication, identity verification, and forensic analysis.
See all 233 papers