Audio Hallucination Attacks: Probing the Reliability of Large Audio Language Models

Ashish Seth; Sonal Kumar; Ramaneswaran Selvakumar; Nishit Anand; Utkarsh Tyagi; Prem Seetharaman; Ramani Duraiswami; Dinesh Manocha

cs.SDMar 31, 2026

Audio Hallucination Attacks: Probing the Reliability of Large Audio Language Models

Authors: Ashish Seth, Sonal Kumar, Ramaneswaran Selvakumar, Nishit Anand, Utkarsh Tyagi, Prem Seetharaman, Ramani Duraiswami, Dinesh Manocha

Organizations: University of Maryland, College Park · Adobe Research

Abstract

Large Audio Language Models (LALMs) achieve strong performance on audio-language tasks; however, their reliability in real-world settings remains underexplored. We introduce Audio Hallucination Attacks (AHA), an attack suite called AHA-Eval, comprising 6.5K QA pairs designed to test whether LALMs genuinely ground their responses in the audio input. AHA targets two attack surfaces: (i) query-based attacks, which exploit question structure to induce hallucinations about absent sounds, and (ii) audio-based attacks, which inject synthetic speech describing non-existent events into the audio stream. Evaluating state-of-the-art LALMs, including Audio Flamingo 3 and Gemini 3 Pro, we observe high attack success rates of 95.35% and 79.65%, respectively, revealing a reliability gap that is hidden by standard benchmark performance. To mitigate this, we propose a 120K QA post-alignment dataset, AHA-Guard, which successfully reduces attack success rates by up to 49%.

Figures & tables

Figure 1: Explicit Vs. Implicit Queries. \raisebox{-1pt} {1}⃝ Given an audio clip of ocean waves with no seagulls present, Gemini 3 Pro [ 1 ] correctly rejects an explicit query about the sound’s existence, \raisebox{-1pt} {2}⃝ yet when posed an implicit query that presupposes the sound, the model bypasses the crucial grounding step and produces a confident but hallucinated response.

Figure 2: Overview of the AHA Data Curation and Attack Generation Pipeline. \raisebox{-1pt} {1}⃝ Data Filtering: We filter audio clips from AudioCaps [ 17 ] , Clotho [ 18 ] , and MusicCaps [ 19 ] using LLM-based caption consistency checks, resulting in 8K verified audio–caption pairs. \raisebox{-1pt} {2}⃝ Hallucinated Sound Generation: For each clip, we use Gemini 3 Pro [ 1 ] to generate counterfactual sound events, including two adversarial (contextually plausible) and two random (out-of-context) sounds. \raisebox{-1pt} {3}⃝ QA Construction: We leverage each hallucinated sound to generate both explicit attacks (directly querying the presence of the sound) and implicit attacks (presupposing its existence). \raisebox{-1pt} {4}⃝ Audio Manipulation: Beyond prompt-based attacks, we prepend TTS-synthesized utterances referencing the hallucinated sounds to the original audio, creating acoustically grounded false cues. \raisebox{-1pt} {5}⃝ AHA Outputs: Our pipeline produces AHA-Eval (6.5K attack pairs) and AHA-Guard (120K DPO preference pairs) (All the prompt used can be found in the supplementary material).

Open-Source Models
Models	AS	Random (ASR % $\downarrow$ )		Adversarial (ASR % $\downarrow$ )
Models	AS	Expl.	Impl.	Expl.	Impl.
R1-AQA [ 24 ]	T	71.68	79.97	80.14	77.63
R1-AQA [ 24 ]	A	10.52	96.49	10.47	93.22
Qwen 2.5-Omni [ 23 ]	T	23.58	68.74	28.32	79.19
Qwen 2.5-Omni [ 23 ]	A	68.87	59.59	94.57	96.71

Table 1: ASR of LALMs on AHA-Eval. We compare the attack success rate (ASR) of LALMs on our AHAs with random and adversarial sound events, using both explicit (Expl.) and implicit (Impl.) queries across text (T) and audio (A) attack spaces (AS).

Models	AS	Random (ASR % $\downarrow$ )		Adversarial (ASR % $\downarrow$ )
Models	AS	Expl.	Impl.	Expl.	Impl.
Baseline	T	23.58	68.74	28.32	79.19
Baseline	A	68.87	59.59	94.57	96.71
+ CoT [ 21 ]	T	13.56	82.90	21.59	86.87
+ CoT [ 21 ]	A	79.38	72.16	94.19	97.29
+ DPO [ 22 ]	T	13.88	39.01	33.16	40.24

Table 2: Comparing hallucination mitigation methods. We evaluate several mitigation techniques on AHA-Eval, including test-time methods such as Chain-of-Thought (CoT) and train-time approaches such as DPO.

Figure 3: Qualitative Example. An example of a multi-turn conversation with Gemini 3 Pro, where the model, when exposed to a hallucinated audio track, hallucinates and describes non-existent sounds. The error further propagates when the model is asked specific questions about the hallucinated sound.

Figure 4: Investigating the cause of hallucinations. (Top) Attention to audio during token generation. (Bottom) Confidence of hallucinated “Yes” responses.

Audio-Language Model Evaluation

Audio-language model evaluation tests whether systems understand speech, music, and environmental sound reliably, fairly, and with evidence grounded in audio.

See all 195 papers

HalluAudio: A Comprehensive Benchmark for Hallucination Detection in Large Audio-Language Models
Apr 21, 2026Feiyu Zhao, Yiming Chen, Wenhuan Lu +3Audio-Language Model Evaluation Audio-Language Model Hallucination Detection
College of Intelligence and Computing, Tianjin University, China · ASUS Intelligent Cloud Services, Singapore
MISHAP-Bench: A Hallucination Benchmark for Large Audio-Language Models
Sep 27, 2026Zhi Wen Soi, Giulio Segalini, Jian-Jia Chen +1Audio-Language Model Evaluation Audio-Language Model Hallucination Detection
University of Neuchâtel · TU Dortmund University · RWTH Aachen University +1
Hijacking Large Audio-Language Models via Context-Agnostic and Imperceptible Auditory Prompt Injection
Apr 16, 2026Meng Chen, Kun Wang, Li Lu +2Audio-Language Models LLM Security
The State Key Laboratory of Blockchain and Data Security, Zhejiang University · Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security · National University of Singapore +1

Apr 21, 2026J/K move · Enter open · S save

HalluAudio: A Comprehensive Benchmark for Hallucination Detection in Large Audio-Language Models

Feiyu Zhao, Yiming Chen, Wenhuan Lu, Daipeng Zhang+2

College of Intelligence and Computing, Tianjin University, China · ASUS Intelligent Cloud Services, Singapore

Audio-Language Model Evaluation Audio-Language Model Hallucination Detection Hallucination in Language Models Audio Understanding Audio-Language Models Multimodal Hallucination

Open paper AskPDF arXiv

Large Audio-Language Models (LALMs) have recently achieved strong performance across various audio-centric tasks. However, hallucination, where models generate responses that are semantically incorrect or acoustically unsupported, remains largely underexplored in the audio domain. Existing hallucination benchmarks mainly focus on text or vision, while the few audio-oriented studies are limited in scale, modality coverage, and diagnostic depth. We therefore introduce HalluAudio, the first large-scale benchmark for evaluating hallucinations across speech, environmental sound, and music. HalluAudio comprises over 5K human-verified QA pairs and spans diverse task types, including binary judgments, multi-choice reasoning, attribute verification, and open-ended QA. To systematically induce hallucinations, we design adversarial prompts and mixed-audio conditions. Beyond accuracy, our evaluation protocol measures hallucination rate, yes/no bias, error-type analysis, and refusal rate, enabling a fine-grained analysis of LALM failure modes. We benchmark a broad range of open-source and proprietary models, providing the first large-scale comparison across speech, sound, and music. Our results reveal significant deficiencies in acoustic grounding, temporal reasoning, and music attribute understanding, underscoring the need for reliable and robust LALMs.

Abstract

Audio-Language Model Evaluation

Explore similar work

HalluAudio: A Comprehensive Benchmark for Hallucination Detection in Large Audio-Language Models

MISHAP-Bench: A Hallucination Benchmark for Large Audio-Language Models

Hijacking Large Audio-Language Models via Context-Agnostic and Imperceptible Auditory Prompt Injection