SARGO: Agreement-Gated Reinforcement and Suppression via On-Policy Self-Guidance for Visual Hallucination Mitigation
Organizations: Tsinghua Shenzhen International Graduate School, Tsinghua University · Harbin Institute of Technology, Shenzhen
Abstract
Vision-language models remain susceptible to visual hallucinations, often mixing grounded information with unsupported claims in the same response. Response-level reinforcement learning assigns a shared advantage to all tokens in a response, causing negative updates to suppress correct information alongside unsupported claims. To reduce hallucination while preserving useful visual information, we introduce SARGO (Self-guided Advantage-Routed Grounding Optimization), a post-training framework for selective reinforcement and suppression through on-policy self-guidance. Direct and reciprocal weighting gives better-supported tokens stronger reinforcement on positive updates and weaker penalties on negative updates. Agreement between overall and hallucination-specific advantages selects when to apply this guidance. At the response level, a reference-calibrated reward couples factuality with the retention and expansion of grounded information, discouraging lower hallucination scores obtained through omission. Experiments on Qwen3-VL-4B, Qwen3-VL-8B, and Gemma3-4B demonstrate that SARGO outperforms the evaluated mitigation baselines across Obj-Hal, AMBER, THRONE, and SHR evaluations. SARGO achieves these gains while preserving useful visual information and general multimodal capabilities.
Figures & tables
| Model | Method | Obj-Hal | AMBER | THRONE | GPT-4-assisted | |||||
|---|---|---|---|---|---|---|---|---|---|---|
| CHAIR | CHAIR | Recall | CHAIR | HalRate | Cover | @5 | @9 | SHR | ||
| Qwen3-VL-4B | Vanilla | 55.0 | 10.3 | 75.7 | 8.0 | 60.8 | 74.2 | 74.2 | 81.2 | 34.5 |
| OPSD | 55.7 | 10.1 | 74.8 | 7.6 | 60.3 | 74.4 | 73.8 | 81.2 | 33.2 | |
| GRPO | 45.3 | 10.3 | 75.3 | 7.2 | 46.4 | 71.5 | 76.5 | 86.0 | 25.2 | |
| OPSD+GRPO | 54.3 | 11.9 | 76.3 | 9.6 | 60.2 | 73.5 | 74.9 | 83.5 | 30.6 | |
| mDPO | 48.3 | 9.0 | 73.0 | 6.4 | 44.1 | 68.8 | 76.2 | 85.2 | 30.8 | |
| Method | Obj-Hal | AMBER | THRONE | |||||
|---|---|---|---|---|---|---|---|---|
| CHAIR | CHAIR | Recall | CHAIR | HalRate | Cover | @5 | @9 | |
| SARGO | 34.7 | 5.8 | 75.9 | 3.9 | 36.7 | 76.2 | 82.3 | 88.0 |
| w/o response quality | 11.0 | 3.0 | 72.7 | 1.7 | 6.8 | 58.0 | 82.9 | 87.8 |
| w/o hallucination | 51.0 | 10.4 | 79.9 | 5.6 | 49.5 | 77.2 | 79.4 | 85.8 |
| w/o coverage | 30.7 | 5.4 | 68.3 | 4.1 | 35.3 | 62.3 | 81.0 | 86.4 |
| w/o coverage gate | 35.0 | 6.0 | 70.4 | 4.1 | 37.5 | 69.8 | 80.8 | 86.5 |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Qwen3-VL-4B | Qwen3-VL-8B | |||
|---|---|---|---|---|
| Method | MME | MMStar | MME | MMStar |
| Vanilla | 2321 | 55.4 | 2386 | 60.3 |
| OPSD | 2337 | 55.2 | 2368 | 60.2 |
| GRPO | 2340 | 55.0 | 2392 | 61.1 |
| OPSD+GRPO | 2228 | 53.9 | 2374 | 60.1 |
| mDPO | 2301 | 54.3 | 2346 | 56.8 |
VLM Hallucination Mitigation
Vision-language model hallucination mitigation keeps generated claims grounded in visual evidence.
See all 243 papers