cs.AINov 19, 2026

When Safety Rationales Hallucinate: Diagnosing and Grounding Multimodal Safety Guards

Authors: Yingshuo Wang, Weixiang Zhao, Guancheng Li, Jiahe Guo, Yu Zhang, Yanyan Zhao

Organizations: Harbin Institute of Technology

Abstract

Multimodal safety guards increasingly generate rationales alongside safety predictions, yet whether these rationales are reliable remains largely unexplored. We systematically study rationale hallucination from two perspectives: visual rationale hallucination, where rationales contain unsupported visual facts, and safety attribution hallucination, where grounded content is assigned an incorrect safety interpretation. Across rationale-generation methods, multiple vision-language backbones, and safety datasets, we find that rationale hallucination is widespread and does not consistently diminish with higher classification accuracy or increased model scale. Further analysis of the Think-with-Image paradigm shows that hallucination is concentrated in fine-grained attributes and relations, while visual tools are often invoked infrequently or without sufficient spatial focus. Based on these findings, we propose Tool-Contrastive Grounding Reinforcement Learning (TCG-RL), which rewards tool use only when it improves rationale grounding over a tool-free reference. TCG-RL substantially reduces rationale hallucination while maintaining competitive safety prediction performance. By revealing the gap between prediction correctness and rationale reliability, our study motivates a shift in multimodal safety guarding from outcome-only evaluation toward safety decisions that are both accurate and verifiably grounded.

Figures & tables

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Seeing Through the Chain: Understanding and Mitigating Hallucinations in Multimodal Large Reasoning Models

    Feb 3, 2026Hao Fang, Jinyu Li, Jiawei Kong +4Hallucination MitigationMultimodal Reasoning

  2. ThinkingGuard: Decoding Implicit Hazards via Step-by-Step Risk Attribution in Multimodal Large Language Models

    Sep 29, 2026Ruochen Zhang, Yao Huang, Yitong Sun +5Multimodal ReasoningLLM Safety