Multimodal large language models (MLLMs) remain prone to visual hallucinations when correct responses depend on subtle visual details. One line of post-training methods constructs visual contrasts through image retrieval, generation, or editing to improve fine-grained perception. However, these contrasts are not built around the model's own hallucinations, making it difficult to ensure that supervision targets its actual perceptual confusions. Another line suppresses hallucinated outputs through response correction or preference learning under fixed image conditions, but leaves the design of supervision signals on the image-input side underexplored. Inspired by how humans correct misconceptions, we introduce DreamRealm, a post-training framework that explores diverse latent hallucinated responses, minimally edits original images to make them valid, and trains the model to assign each response to its supporting image. Our data construction pipeline, RealmForgePipe, mines observed and latent hallucinations from a target model and materializes them through image editing. Using hallucinations mined from Qwen3.5-9B, we construct the Realized Hallucination Dataset (RealmSet). Our training objective, RealmMatch, constructs a likelihood score matrix across image--response combinations and combines supervised learning with bidirectional contrastive losses to reinforce correct matches and suppress mismatches. Compared with the original Qwen3.5-9B model, DreamRealm improves performance across six visual perception and hallucination benchmarks, including gains of 7.26 percentage points in HallusionBench accuracy and 11.34 percentage points in MMVP pair accuracy. Together with ablation studies on data construction and training methods, these results demonstrate that hallucination assignment is a promising paradigm for improving visual perception and mitigating hallucinations.
Figures & tables
Figure 1: Motivation of DreamRealm. Existing post-training methods correct hallucinated responses, while supervision on the visual conditions that would make them valid remains underexplored. Inspired by how humans correct misconceptions, DreamRealm materializes these conditions and teaches the model to match hallucinated responses to their supporting images.
Figure 2: Overview of DreamRealm. Top: RealmForgePipe mines hallucinated responses through high-temperature sampling and key-attribute enumeration, selects candidates through likelihood ranking and Top-K filtering, and edits images to make the selected hallucinated responses valid. Bottom: RealmMatch constructs an image–response score matrix from original images with their ground-truth answers and edited images with their corresponding hallucinated responses. Through SFT on matched image–response pairs and bidirectional contrastive learning, it assigns hallucinated responses to their supporting images, learning the correspondence between responses and visual conditions.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Hallucination mining results from RealmForgePipe on four examples involving actions and spatial relations. Each example shows the original image, question, ground-truth answer, and selected mined responses ranked by likelihood. Percentages denote candidate-relative weight ρi(e) within the full error candidate pool, rather than calibrated answer probabilities; displayed values are not renormalized over the shown subset. Responses are shortened for readability.
Figure 4: Additional hallucination mining examples covering counting, clock reading, color recognition, and chart analysis. Selected candidates are shown with their candidate-relative weights from the original mining runs, without renormalization over the displayed subset. These weights are not calibrated answer probabilities. Chart responses are shortened for readability.
Figure 5: Hallucination-guided image editing examples from RealmForgePipe (1/2). From top to bottom: actions, illumination, spatial relations, mirror arrangements, object counts, and scene structure. Each row pairs the original image and ground-truth answer (yellow) with edited images and their corresponding hallucinated responses (cyan). Red text highlights the response attributes targeted by editing. Responses are shortened for readability.
Figure 6: Hallucination-guided image editing examples from RealmForgePipe (2/2). From top to bottom: local object replacement, object–container relations, food content, clock time, surface patterns, and color attributes. Together with Figure 5 , the two pages show 12 original-image families and 54 edited images. Yellow and cyan labels denote original and edited image–response pairs; red text highlights the targeted attributes.
Data
Count
Mined hallucinated responses
114,982
Candidates entering image editing
96,320
Successfully generated edited images
91,986
VLM blind-check approvals
43,169
Human-approved edited pairs
28,960
Appendix
Table 1: Data construction summary for RealmForgePipe.
Category
Synthesis
Generated
BlindCheck
Human
Final pass rate (%)
count
56,518
53,900
24,752
14,552
25.75
other
7,980
7,488
3,246
3,245
40.66
count_by_attribute
6,503
6,296
2,269
1,940
29.83
shape
5,450
5,298
2,008
1,227
22.51
color
5,831
5,657
2,763
2,429
41.66
chart_describe
2,705
2,633
2,027
1,242
45.91
Appendix
Table 2: Category-wise data construction counts and final pass rates. Synthesis denotes candidates entering image editing; Generated denotes successfully generated edited images; BlindCheck denotes records marked BlindCheck OK ; Human denotes human-approved edited pairs. Final pass rate is Human / Synthesis.
Figure 7: A recorded editing example. Enlarging the red cube makes the target response Large valid for the same question.
Hyperparameter
Value
GPUs
8 NVIDIA H100
Training epochs
3
Peak learning rate
5×10−6
Learning-rate schedule
Cosine
Warmup ratio
0.03
Weight decay
0
Appendix
Table 3: Training hyperparameters for the Qwen3.5-9B RealmMatch configuration. Batch units are complete question-centered families, rather than individual image–response combinations.
Model / Method
Overall ↑
Original ↑
HG-Edited ↑
Qwen3.5-9B (Raw)
64.80
66.44
62.17
+ DreamRealm
68.30
70.76
64.36
Qwen2.5-VL-7B (Raw)
48.40
51.23
43.86
+ DreamRealm
59.66
61.99
55.93
Gemma-4-E4B-it (Raw)
39.54
40.90
37.36
+ DreamRealm
50.54
53.09
46.46
Appendix
Table 4: In-distribution accuracy on the 5,000-example held-out test set.
Figure 8: Compact qualitative comparisons (1/2), cases 1–16. Green and red identify model answers, not independent correctness judgments. Case 14 shows only the differing final angle in the formula. IDs beginning with human_ denote edited inputs.
Figure 9: Compact qualitative comparisons (2/2), cases 17–32. Reflected vehicles in case 22 include the car carrying the dog. All responses are condensed from the recorded outputs.
Method
Backbone
Setup
VCD [ 13 ]
LLaVA-1.5-7B
Inference-time decoding
CoCI-CAP [ 17 ]
LLaVA-OneVision (Qwen2-7B)
Inference-time decoding
RLAIF-V [ 42 ]
LLaVA-1.5-7B
Public checkpoint
SymMPO [ 16 ]
LLaVA-1.5-7B
Public checkpoint
mDPO [ 34 ]
Bunny (Llama-3-8B)
Reproduced
VC-STaR [ 19 ]
Qwen2.5-VL-7B
Reproduced
Appendix
Table 5: Backbone and setup for each Block I method.
Large vision-language models (LVLMs) have demonstrated remarkable capabilities in multimodal understanding. However, they remain prone to hallucinations, generating responses that are inconsistent with the visual evidence. Existing mitigation methods largely address language-prior bias or cross-modal imbalance, while progressive visual degradation across perception and memory remains underexplored. In this work, we propose Saliency-Driven Perceptual Realignment (SDPR), a training-free framework that mitigates the degradation of visual awareness throughout inference. Specifically, we first introduce saliency-driven attention redistribution to release attention hijacked by non-semantic sink tokens, thereby recovering critical visual evidence. Second, we identify spatial distortion in the KV cache and propose saliency-driven cache alignment to preserve query-relevant visual features during generation. Finally, we introduce prior-constrained contrastive decoding to penalize unfaithful predictions induced by dominant language priors. Our proposed SDPR is robust against hallucinations due to its holistic alignment of visual awareness across the entire generative trajectory. Extensive experiments across diverse LVLM architectures show that SDPR outperforms state-of-the-art methods on both hallucination and general-purpose benchmarks, requiring no additional training and incurring minimal runtime overhead. The code is available blue{here}.
Pengxu Chen, Yao Zhu, Guangming Zhu +4
Xidian University Xi’an, China · Tsinghua University Beijing, China · Shanghai Road Transport Development Center Shanghai, China +1
Existing research on object hallucination in multimodal large language models (MLLMs) predominantly attributes the problem to language priors such as over-reliance on textual co-occurrence statistics. We challenge this view by presenting quantitative evidence for a complementary, under-explored cause: visual-origin hallucination, where hallucinations arise from incorrect visual feature extraction and misalignment between image and text embeddings. Through cosine similarity analysis and Smooth Grad-CAM entropy measurements, we show that hallucinated samples exhibit systematically lower image-text similarity (average 0.158 vs. -0.122) and inverted attention patterns, where attention is dispersed when the target object is present but wrongly concentrated when it is absent. Guided by this diagnosis, we propose Adversarial Contrastive Fine-Tuning (ACFT). ACFT uses an Adversarial Hallucination Attribute Flipping (AHAF) procedure, involving minimal, targeted adversarial perturbations that flip an image's hallucination attribute, to construct perfectly aligned positive-negative pairs, which are then used for contrastive fine-tuning. AHAF simultaneously serves as a diagnostic probe, revealing that MLLM visual representations lie dangerously close to hallucination decision boundaries. Requiring only 0.9% of the COCO dataset and adding zero inference overhead, ACFT achieves state-of-the-art performance on POPE, MME, and four description-level hallucination benchmarks across LLaVA, MiniGPT-4, and Qwen2.5-VL. Code is available at https://github.com/zxp555/ACFT_MM
Despite impressive progress in capabilities of large vision-language models (LVLMs), these systems remain vulnerable to hallucinations, i.e., outputs that are not grounded in the visual input. Prior work has attributed hallucinations in LVLMs to factors such as limitations of the vision backbone or the dominance of the language component, yet the relative importance of these factors remains unclear. To resolve this ambiguity, We propose HalluScope, a benchmark to better understand the extent to which different factors induce hallucinations. Our analysis indicates that hallucinations largely stem from excessive reliance on textual priors and background knowledge, especially information introduced through textual instructions. To mitigate hallucinations induced by textual instruction priors, we propose HalluVL-DPO, a framework for fine-tuning off-the-shelf LVLMs towards more visually grounded responses. HalluVL-DPO leverages preference optimization using a curated training dataset that we construct, guiding the model to prefer grounded responses over hallucinated ones. We demonstrate that our optimized model effectively mitigates the targeted hallucination failure mode, while preserving or improving performance on other hallucination benchmarks and visual capability evaluations. To support reproducibility and further research, we will publicly release our evaluation benchmark, preference training dataset, and code at https://pegah-kh.github.io/projects/prompts-override-vision/ .
Pegah Khayatan, Jayneel Parekh, Arnaud Dapogny +3
1ISIR, Sorbonne Universit´e, Paris, France · 2Valeo.ai, Paris, France