cs.SDNov 19, 2026

From Attention to Semantics: Detecting and Mitigating Sound Event Hallucinations in Large Audio-Language Models

Authors: Tianyi Sun, Minghao Lv, Chunyang Yu, Shenyue Wang, Mengyue Wu

Organizations: Shanghai Jiao Tong University · Guangdong OPPO Mobile Telecommunications Corp., Ltd.

Abstract

Large Audio-Language Models (LALMs) excel at diverse audio tasks, yet remain prone to hallucinating sound events absent from the input. Prior work links such hallucinations to insufficient audio attention. However, we find that real and hallucinated events can receive comparable audio attention mass while attending to different audio positions. To examine whether the attended representations support the generated events, we adapt LatentLens to retrieve neighboring contextualized text representations. The retrieved contexts reveal semantic support for real events, but a semantic mismatch between hallucinated events and their attended audio representations. Directly masking the corresponding audio regions can suppress hallucinated events at the cost of real content. We are motivated by these findings to introduce an explainable and training-free detect-then-mitigate framework for effectively hallucination detection and mitigation that aggregates semantic support and applies differential audio masking to protect regions associated with events predicted to be supported. Extensive experiments on three LALMs and two datasets demonstrate that our method significantly improves hallucination detection and hallucination mitigation performance.

Figures & tables

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Audio Hallucination Attacks: Probing the Reliability of Large Audio Language Models

    Mar 31, 2026Ashish Seth, Sonal Kumar, Ramaneswaran Selvakumar +5Audio-Language Model EvaluationLLM Reliability

  2. MISHAP-Bench: A Hallucination Benchmark for Large Audio-Language Models

    Sep 27, 2026Zhi Wen Soi, Giulio Segalini, Jian-Jia Chen +1Audio-Language Model EvaluationAudio-Language Model Hallucination Detection

  3. Silence is Golden: Mitigating Hallucinations in Large Audio-Language Models via Layer-Weighted Vector Steering

    Oct 14, 2025Tsung-En Lin, Kuan-Yi Lee, Hung-Yi LeeAudio QAHallucination in Language Models