Vision-language models remain susceptible to visual hallucinations, often mixing grounded information with unsupported claims in the same response. Response-level reinforcement learning assigns a shared advantage to all tokens in a response, causing negative updates to suppress correct information alongside unsupported claims. To reduce hallucination while preserving useful visual information, we introduce SARGO (Self-guided Advantage-Routed Grounding Optimization), a post-training framework for selective reinforcement and suppression through on-policy self-guidance. Direct and reciprocal weighting gives better-supported tokens stronger reinforcement on positive updates and weaker penalties on negative updates. Agreement between overall and hallucination-specific advantages selects when to apply this guidance. At the response level, a reference-calibrated reward couples factuality with the retention and expansion of grounded information, discouraging lower hallucination scores obtained through omission. Experiments on Qwen3-VL-4B, Qwen3-VL-8B, and Gemma3-4B demonstrate that SARGO outperforms the evaluated mitigation baselines across Obj-Hal, AMBER, THRONE, and SHR evaluations. SARGO achieves these gains while preserving useful visual information and general multimodal capabilities.
Figures & tables
Figure 1: Object hallucination and information retention on Obj-Hal. Comparison of SARGO and baselines on (a) Qwen3-VL-4B, (b) Qwen3-VL-8B, and (c) Gemma3-4B. Lower CHAIR i and higher object recall (ObjRec) indicate better performance. SARGO achieves the lowest CHAIR i across all three backbones while retaining high object recall.
Figure 2: Overview of SARGO. The frozen initial policy supplies reference responses for reward calibration, while the current policy scores its own rollouts under ordinary and privileged contexts to obtain token-level grounding support. Agreement between overall and hallucination-specific advantages selects when self-guidance applies. Direct and reciprocal weighting modulate the relative strength of token-level reinforcement and suppression, illustrated by bar magnitude. Neutral routes retain uniform token weights.
Model
Method
Obj-Hal
AMBER
THRONE
GPT-4-assisted
CHAIR s↓
CHAIR i↓
Recall ↑
CHAIR ↓
HalRate ↓
Cover ↑
F0.5 @5 ↑
F0.5 @9 ↑
SHR ↓
Qwen3-VL-4B
Vanilla
55.0
10.3
75.7
8.0
60.8
74.2
74.2
81.2
34.5
OPSD
55.7
10.1
74.8
7.6
60.3
74.4
73.8
81.2
33.2
GRPO
45.3
10.3
75.3
7.2
46.4
71.5
76.5
86.0
25.2
OPSD+GRPO
54.3
11.9
76.3
9.6
60.2
73.5
74.9
83.5
30.6
mDPO
48.3
9.0
73.0
6.4
44.1
68.8
76.2
85.2
30.8
Table 1: Comparison of SARGO and baseline methods on hallucination benchmarks using Qwen3-VL-4B, Qwen3-VL-8B, and Gemma3-4B. THRONE reports classwise F0.5 at majority (@5) and unanimous (@9) agreement. All scores are percentages. Bold and underlined values indicate the best and second-best results within each backbone, respectively.
Figure 3: Effects of token weighting and advantage routing on Qwen3-VL-4B. All variants share the same response reward. No weighting uses uniform token weights. Positive only and Negative only apply teacher weighting in one branch, retaining uniform weights in the other. RL-only routing uses the overall advantage alone to select the weighting branch. Full denotes SARGO.
Method
Obj-Hal
AMBER
THRONE
CHAIR s↓
CHAIR i↓
Recall ↑
CHAIR ↓
HalRate ↓
Cover ↑
F0.5 @5 ↑
F0.5 @9 ↑
SARGO
34.7
5.8
75.9
3.9
36.7
76.2
82.3
88.0
w/o response quality
11.0
3.0
72.7
1.7
6.8
58.0
82.9
87.8
w/o hallucination
51.0
10.4
79.9
5.6
49.5
77.2
79.4
85.8
w/o coverage
30.7
5.4
68.3
4.1
35.3
62.3
81.0
86.4
w/o coverage gate
35.0
6.0
70.4
4.1
37.5
69.8
80.8
86.5
Table 2: Ablation study of SARGO’s response reward on Qwen3-VL-4B. Each variant removes one component from the full reward.
Figure 4: General multimodal performance on Qwen3-VL-4B and Qwen3-VL-8B. Comparison on (a) MME total score and (b) MMStar accuracy (%). SARGO improves both metrics over Vanilla at both scales.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Teacher prompt for on-policy self-guidance. Image-specific object labels, scene cues, and reference captions provide grounding context for scoring the current policy’s sampled responses.
Qwen3-VL-4B
Qwen3-VL-8B
Method
MME ↑
MMStar ↑
MME ↑
MMStar ↑
Vanilla
2321
55.4
2386
60.3
OPSD
2337
55.2
2368
60.2
GRPO
2340
55.0
2392
61.1
OPSD+GRPO
2228
53.9
2374
60.1
mDPO
2301
54.3
2346
56.8
Appendix
Table 3: General multimodal results on MME and MMStar. MME reports total scores and MMStar reports accuracy (%). Bold and underlined values denote the best and second-best results within each backbone, respectively.
Figure 6: Visualization results comparing OPSD, GRPO, SymMPO, and SARGO. Hallucinations are marked in red .
Figure 7: Visualization results comparing OPSD, GRPO, SymMPO, and SARGO. Hallucinations are marked in red .
Figure 8: Visualization results comparing OPSD, GRPO, SymMPO, and SARGO. Hallucinations are marked in red .
Large vision-language models (VLMs) frequently suffer from hallucinations, generating content that is inconsistent with visual inputs. Existing methods typically address this problem through post-hoc filtering, additional training objectives, or external verification, but they do not intervene during the decoding process when hallucinations arise. In this work, we introduce SAGE, a Sink-Aware Grounded Decoding framework that mitigates hallucinations by dynamically modulating self-attention during generation. Hallucinations are strongly correlated with attention sink tokens - punctuation or function tokens that accumulate disproportionate attention despite carrying limited semantic content. SAGE leverages these tokens as anchors to monitor grounding reliability in real time. At each sink trigger, the method extracts semantic concepts from the generated sequence, estimates their visual grounding using both self-attention maps and gradient-based attribution, and measures their spatial agreement. Based on this signal, self-attention distributions are adaptively sharpened or broadened to reinforce grounded regions or suppress unreliable ones. Extensive experiments across diverse hallucination benchmarks demonstrate that SAGE consistently outperforms existing decoding strategies, achieving substantial reductions in hallucination while preserving descriptive coverage, without requiring model retraining or architectural modifications. Our method achieves an average relative improvement of 10.65% on MSCOCO and 7.19% on AMBER across diverse VLM architectures, demonstrating consistent gains in hallucination mitigation.
Large vision-language models (LVLMs) have demonstrated remarkable capabilities in multimodal understanding. However, they remain prone to hallucinations, generating responses that are inconsistent with the visual evidence. Existing mitigation methods largely address language-prior bias or cross-modal imbalance, while progressive visual degradation across perception and memory remains underexplored. In this work, we propose Saliency-Driven Perceptual Realignment (SDPR), a training-free framework that mitigates the degradation of visual awareness throughout inference. Specifically, we first introduce saliency-driven attention redistribution to release attention hijacked by non-semantic sink tokens, thereby recovering critical visual evidence. Second, we identify spatial distortion in the KV cache and propose saliency-driven cache alignment to preserve query-relevant visual features during generation. Finally, we introduce prior-constrained contrastive decoding to penalize unfaithful predictions induced by dominant language priors. Our proposed SDPR is robust against hallucinations due to its holistic alignment of visual awareness across the entire generative trajectory. Extensive experiments across diverse LVLM architectures show that SDPR outperforms state-of-the-art methods on both hallucination and general-purpose benchmarks, requiring no additional training and incurring minimal runtime overhead. The code is available blue{here}.
Pengxu Chen, Yao Zhu, Guangming Zhu +4
Xidian University Xi’an, China · Tsinghua University Beijing, China · Shanghai Road Transport Development Center Shanghai, China +1
Large vision-language models (LVLMs) often hallucinate objects that are not present in the input image, largely because visual grounding weakens as decoding progresses. Existing inference-time mitigation methods modify logits or hidden states throughout generation, but they suffer from three key limitations: they lack an explicit grounding objective, intervene even when the model is already well-grounded, and use fixed correction strengths that do not adapt to the severity of grounding failure. We propose BRACS (Barrier-Regulated Adaptive Closed-form Steering), a training-free steering framework that addresses these issues through barrier-regulated adaptive closed-form steering. BRACS monitors the model's own attention to measure visual grounding and applies corrections to the hidden states only when grounding deteriorates. The corrective update is computed analytically in closed form, requiring no training of auxiliary networks or model retraining. Experiments on LLaVA-1.5-7B and Qwen-VL-Chat show that BRACS consistently outperforms prior methods on hallucination benchmarks, reducing CHAIRs by 9.4 points and improving POPE F1 by 2.7 points, while matching or improving performance on four general multimodal benchmarks. BRACS also remains efficient, operating at 80% of greedy decoding throughput and achieving 1.3 times higher speed on average than the baselines.