cs.CVMar 20, 2026
SaveFREAK: A Fine-grained Hallucination Evaluation Benchmark for Advanced MLLMs
Organizations: Wangxuan Institute of Computer Technology, Peking University · State Key Laboratory of General Artificial Intelligence
Abstract
Multimodal Large Language Models (MLLMs) suffer from hallucinations. Existing hallucination evaluation benchmarks are often limited by over-simplified tasks leading to saturated metrics, or insufficient diversity that fails to adequately assess the hallucination extent in state-of-the-art multimodal models. To address this gap, we propose FREAK, a comprehensive multimodal benchmark designed for fine-grained hallucination assessment in MLLMs. Through high-quality photorealistic images featuring fine-grained counter-commonsense edits, FREAK innovatively evaluates hallucination phenomena in detailed visual perception of MLLMs. Extensive experiments on FREAK show severe hallucination issues in SOTA models regarding detailed visual perception. To enable deeper investigation, we curate a controlled subset to indirectly evaluate the model's ability to perceive target detailed information. Through systematic evaluation of prevailing Chain-of-Thought (CoT) prompting techniques within this task, we reveal critical insights regarding hallucination patterns and model reasoning processes.
Figures & tables
Table 1: Comparison between FREAK and other AI-generated benchmarks. FREAK shows uniqueness because of the photorealistic images, fine-grained and diverse counter-commonsense(CCS) content, which strongly challenges SOTA models.
Figure 1: Generation pipeline of FREAK, including a three-step generation paradigm. We first prompt LLMs to generate CCS content for later stages, next we use image genration model and editing model to generate CCS image. Finally we filter and verify the generated data to form FREAK benchmark. In this example, we exhibit the generation process of “Traffic Light” . The generated image shows the positions of the red and green lights swapped, incorrectly with red at the bottom and green at the top, which violates both commonsense and the traffic regulations of various countries.
Figure 2: Overview of FREAK. Items in FREAK can be categorized into six subtasks, each comprising questions that are straightforward for human solvers. The right panel shows representative examples for each subtask. Notably, certain questions are assigned to more than one subtask.
| Accuracy ( ) | HalluRate ( ) | |||||||||||||
| Dete. | Count. | Analy. | Attr. | OCR | Pos. | Overall | Dete. | Count. | Analy. | Attr. | OCR | Pos. | Overall | |
| Human Baseline | 86.93 | 88.65 | 83.44 | 83.92 | 94.24 | 88.08 | 86.71 | 7.19 | 6.76 | 10.94 | 5.22 | 4.32 | 6.22 | 6.95 |
| Non-Reasoning Models | ||||||||||||||
| GPT-4.1 | 50.25 | 19.45 | 33.26 | 48.24 | 38.82 | 39.37 | 42.01 | 36.62 | 46.20 | 49.23 | 38.41 | 40.61 | 49.58 | 44.54 |
| InternVL3-78B | 43.03 | 20.12 | 29.65 | 48.49 | 45.15 | 38.41 | 39.32 | 44.11 | 47.75 | 54.91 | 38.74 | 38.59 | 52.65 | 48.76 |
| InternVL3-38B | 40.06 | 17.84 | 28.09 | 46.46 | 46.81 | 37.21 | 37.24 | 44.99 | 48.20 | 57.15 | 40.36 | 33.76 | 46.68 | 48.79 |
Table 2: Evaluation results of SOTA MLLMs, which are outperformed by human experts with wide margins. The highest model performance in each column is highlighted in green, and the second-highest is highlighted in blue. The reasoning process is enabled if the MLLM is capable.
| Model | Size | Accuracy ( ) | HalluRate ( ) | ||
|---|---|---|---|---|---|
| Direct | CoT | Direct | CoT | ||
| GPT-4.1 | - | 42.01 | 40.66( 1.45 ) | 45.43 | 46.30 ( 2.65 ) |
| InternVL3-78B | 78B | 39.32 | 33.91( 5.41 ) | 48.76 | 52.83 ( 4.07 ) |
| InternVL3-38B | 38B | 37.24 | 36.40( 0.84 ) | 48.79 | 49.00( 0.21 ) |
| Qwen2.5-VL-72B | 72B | 39.39 | 33.39( 6.00 ) | 46.82 | 50.95 ( 4.13 ) |
| Qwen2.5-VL-32B | 32B | 34.65 | 29.82( 4.83 ) | 49.66 | 54.52( 4.86 ) |
Table 4: Performance of different models under two response modes: (1) direct answer generation; (2) reasoning before final output. For non-reasoning models, we employ CoT prompting, while for reasoning models, we activate their reasoning mode. For o3, we use ”low” and ”high” respectively in reasoning effort parameters of OpenAI API.
Figure 5: Clustering results showing the evolution of ground-truth probabilities during the reasoning processes. The legend indicates the representative count for each clustered curve. The green curve demonstrates a thought process distinct from conventional MLLMs. Evaluation is conducted on multiple-choice questions with cyclic permutation, where each question is repeated six times.
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
| Choice | Precision | Recall | F1-score | Support |
|---|---|---|---|---|
| Choice A | 0.83 | 0.86 | 0.84 | 340 |
| Choice B | 0.86 | 0.86 | 0.86 | 335 |
| Choice C | 0.87 | 0.83 | 0.85 | 335 |
| Accuracy | 0.85 | 0.85 | 0.85 | 1010 |
| Macro Avg | 0.64 | 0.64 | 0.64 | 1010 |
| Weighted Avg | 0.86 | 0.85 | 0.85 | 1010 |
Table 5: Human performance metrics on the multiple-choice task. We randomly shuffled the order of the answer choices and ensured that the number of questions with A, B, and C as the correct answer was balanced. Option D was fixed as: ‘Correct answer is not listed.’
| Model | Size | Accuracy | Consist. Acc. | Precision | Recall | F1 |
|---|---|---|---|---|---|---|
| o3 | - | 43.98 | 32.40 | 45.08 | 43.98 | 44.52 |
| Gemini2.5 flash | - | 41.00 | 25.10 | 42.84 | 41.00 | 41.88 |
| Gemini2.5 pro | - | 42.73 | 33.00 | 45.24 | 42.73 | 43.95 |
| o4-mini | - | 41.58 | 27.60 | 42.92 | 41.58 | 42.24 |
| GPT 4.1 | - | 43.78 | 31.50 | 45.51 | 43.78 | 44.45 |
| Claude4 sonnet | - | 30.30 | 18.50 | 30.60 | 30.30 | 30.42 |
Table 6: More evaluation results on FREAK, including Accuracy , Consistency Accuracy , Weighted-Precision , Weighted-Recall , and Weighted-F1 . It should be noted that due to the use of cyclic permutation, the support for options A, B, and C is 2000 each, whereas the support for option D is 0. After excluding option D, the weighted metrics are equivalent to the macro-averaged metrics.
| Model | Precision | Recall | F1 | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Op. A | Op. B | Op. C | Op. A | Op. B | Op. C | Op. A | Op. B | Op. C | |
| o3 | 44.47 | 45.28 | 45.49 | 43.20 | 44.35 | 44.40 | 43.82 | 44.81 | 44.94 |
| o4-mini | 42.58 | 43.35 | 42.82 | 41.45 | 41.70 | 41.60 | 42.01 | 42.51 | 42.20 |
| GPT 4.1 | 44.69 | 46.15 | 45.68 | 50.95 | 40.45 | 39.95 | 47.62 | 43.11 | 42.62 |
| Gemini2.5 pro | 44.46 | 45.85 | 45.42 | 42.50 | 42.55 | 43.15 | 43.46 | 44.14 | 44.26 |
| Gemini2.5 flash | 42.96 | 43.19 | 42.36 | 43.50 | 39.00 | 40.50 | 43.23 | 40.99 | 41.41 |
Table 7: The Precision, Recall, and F1 scores evaluation results of various models across the three categories of options A, B, and C. Some models exhibit noticeable variations in performance across different options, indicating the presence of certain option order biases under our task design.
| Model | Consistency Rate | P/N Consistency Rate | Sample Size |
| Gemini2.5 pro | 87.74 | 87.74 | 106 |
| Gemini2.5 flash | 91.00 | 95.00 | 100 |
| o3 | 86.54 | 94.23 | 104 |
| GPT 4.1 | 94.12 | 95.10 | 102 |
| InternVL3-78B | 88.24 | 92.16 | 102 |
| InternVL3-38B | 92.00 | 96.00 | 100 |
Table 8: The statistical results of sampling for consistency between the LLM and human evaluators. Consistency Rate refers to the proportion of instances where the LLM’s evaluation results align with those of human assessors. The P/N Consistency Rate consolidates Commonsense Error and Other Error into a single category, considering only two types of judgments before calculating the consistency proportion. We use GPT-5-mini(2025-08-07) as the judge model in this study.
Figure 6: Prompt used for LLM-assisted option selection. For models that generated reasoning but failed to output the final choice in the required format, we used advanced LLMs to make the selection based on their reasoning traces.
Figure 7: Proportion of different type of instances in subset. TP : Correct before (with normal image) and after substitution; TN : Correct before, but incorrect after; FP : Incorrect before, but correct after; FN : Incorrect both before and after.
| Model | SeedEdit 3.0 | Nano Banana | Seedream 4.0 | |||
|---|---|---|---|---|---|---|
| Accuracy | Consitency | Accuracy | Consitency | Accuracy | Consitency | |
| Gemini-2.5-flash | 46.23 | - | 44.78 | 83.18 | 47.54 | 79.99 |
| o4-mini | 45.86 | - | 44.20 | 83.04 | 50.14 | 83.48 |
| Qwen2.5-VL-32B | 38.26 | - | 38.12 | 82.32 | 39.71 | 86.96 |
Table 9: Ablation study on editing models. We use different image editing models and evaluate three MLLMs with CCS images generated by these editing models. Comprehensive results indicates that using of SeedEdit 3.0 doesn’t introduce structure bias.
| Model | Size | Acc(D) | Acc(v1) | Acc(v2) |
|---|---|---|---|---|
| InternVL3-38B | 38B | 42.60 | 40.28 | 36.74 |
| InternVL3-14B | 14B | 36.03 | 35.13 | 28.42 |
| Qwen2.5-VL-32B | 32B | 36.03 | 29.95 | 29.50 |
| Qwen2.5-VL-7B | 7B | 36.08 | 32.98 | 33.32 |
| Phi4 multimodal | 6B | 37.55 | 29.13 | 25.08 |
Table 10: Result of CoT prompt ablation experiment. Acc(D) : Accuracy using direct prompt (Fig. 10 ); Acc(v1) : Accuracy using CoT prompt v1 (Fig. 11 ); Acc(v2) : Accuracy using CoT prompt v2 (Fig. 13 ).
Figure 8: Prompt used for CCS content generation
Figure 9: Prompt used for image generation.
Figure 10: Prompt used evaluation.
Figure 11: Prompt used for CoT evaluation. Results are listed in Table 9
Figure 12: Prompt used for Figure 5 , This prompt delete the example to avoid fixed thinking pattern.
Figure 13: Another CoT prompt that used in previous work ( Kojima et al., 2023 ) . We use this prompt to eliminate potential performance degradation caused by the prompt in Figure 11 itself.
Figure 14: Prompt used for LLM-assisted option selection. For models that generated reasoning but failed to output the final choice in the required format, we used advanced LLMs to make the selection based on their reasoning traces.
Figure 15: Prompt used for LLM’s judgment.
Figure 16: An Detection example of error case on InternVL3-38B
Figure 17: An Detection example of error case on InternVL3-38B
Figure 18: An Analysis example of error case on InternVL3-38B
Figure 19: An Attribute example of error case on InternVL3-38B
Figure 20: An OCR example of error case on InternVL3-38B
Figure 21: An Analysis example of error case on InternVL3-38B
Figure 22: An Analysis example of error case on Kimi-VL-A3B-Thinking. The model merges self-reflective reasoning pattern. This pattern appears when the model realize the image’s content conflicts with it’s parametric knowledge.
Figure 23: An Attrirubte example of error case on Kimi-VL-A3B-Thinking. The model merges self-reflective reasoning pattern. This pattern appears when the model realize the image’s content conflicts with it’s parametric knowledge.
Figure 24: An Attrirubte example of error case on Kimi-VL-A3B-Thinking. The model merges self-reflective reasoning pattern. This pattern appears when the model realize the image’s content conflicts with it’s parametric knowledge.
Figure 25: The overall experimental results of the probe experiment, where all curves are clustered from the original samples. The red curve represents the probability evolution in successful cases, the blue curve corresponds to error cases, and the green curve captures a specific thinking pattern observed in the reasoning model.