A Sensitivity Analysis of Multi-Event Audio Grounding in Audio LLMs

Taehan Lee; Jaehan Jung; Hyukjun Lee

cs.SDMar 4, 2026

A Sensitivity Analysis of Multi-Event Audio Grounding in Audio LLMs

Authors: Taehan Lee, Jaehan Jung, Hyukjun Lee

Organizations: Sogang University, South Korea

Abstract

Audio LLMs have shown a strong ability to understand audio samples, yet their reliability in complex acoustic scenes remains under-explored. Unlike prior work limited to small scale or less controlled query construction, we present a large-scale evaluation of event grounding and false alarms as auditory scene complexity increases. Using 71K AudioCapsV2 clips, we extract normalized (source, attribute) events and build two query types: present-event queries for ground-truth detection and absent-event queries to probe hallucinations, using similarity-filtered negative sampling in an audio-aligned text embedding space. We evaluate four SOTA Audio LLMs with 12 prompt variants over 500K yes/no queries per model. Across models, increasing event count consistently lowers true-positive rate and raises false-positive rate, while prompts induce a strong trade-off between the two. Our confidence analysis shows that models become more uncertain on multi-event audio, revealing room for improvement.

Figures & tables

Figure 1: Overall pipeline of audio event extraction.

Figure 2:

\operatorname{erank}

distribution of audio embeddings from EAT (left) and SSLAM (right).

# events	1	2	3	4	5
# samples	25.8%	49.3%	20.5%	3.8%	0.5%

Table 1: Distribution of audio samples by the number of events.

$x_{A}$	( water , splashing )		( none , vibrating )
$x_{B}$	( rain , hitting )	( oil , sizzling )	( saw , running )	( insect , buzzing )
ReCLAP	0.59 / #16	0.49 / #25	0.22 / #95	0.17 / #128
T5	0.36 / #37	-0.01 / #230	-0.11 / #538	0.04 / #167
Qwen3	0.33 / #40	-0.01 / #234	-0.04 / #321	0.14 / #141

Table 2: Cosine similarity

\operatorname{sim}(f(x_{A}),f(x_{B}))

and rank of

x_{B}

relative to

x_{A}

among all events, denoted #.

Figure 3: Visualization of the event embedding space on principal component axes. Blue dots indicate all other events.

Question Templates
$q_{1}$	In this audio, is there any { $x$ }?
$q_{2}$	Can you hear a sound of { $x$ }?
$q_{3}$	Does this audio contain the sound of { $x$ }?
$q_{4}$	Is there any occurrence of { $x$ }?
Response Instructions
$r_{1}$	Respond using only one word: yes or no.

Table 3: Question templates and response instructions used in prompts for evaluating the existence of

x

in an audio sample.

Panel of Figure 4 — Figure 4: True Positive Rate (green) on present-event detection and False Positive Rate (red) on absent-event detection, plotted against the number of events across audio LLMs. Solid ( - ), dashed ( $--$ ) and dotted ( $..$ ) lines indicate the average across prompts, the prompt with the highest TPR / lowest FPR, and the prompt with the lowest TPR / highest FPR, respectively.

Figure 5: Normalized output token probability distribution of Audio LLMs. Yes / No scores are at the left / right axis, respectively.

	$\text{Qwen}_{30B}^{3}$	$\text{Qwen}_{7B}^{2.5}$	$\text{Qwen}_{3B}^{2.5}$	$\text{AF}_{7B}^{3}$
$\text{PD}_{h}$	$q_{2}r_{2}/91.8$	$q_{1}r_{1}/90.3$	$q_{1}r_{1}/90.9$	$q_{1}r_{3}/94.2$
$\text{PD}_{l}$	$q_{4}r_{3}/86.4$	$q_{3}r_{3}/84.6$	$q_{4}r_{2}/79.3$	$q_{3}r_{3}/75.8$
$\text{AD}_{l}$	$q_{4}r_{3}/5.6$	$q_{3}r_{2}/3.6$	$q_{4}r_{2}/3.9$	$q_{3}r_{1}/3.4$
$\text{AD}_{h}$	$q_{2}r_{2}/11.5$	$q_{1}r_{1}/9.1$	$q_{1}r_{1}/11.2$	$q_{4}r_{1}/21.0$
$\tau_{\text{bias}}$	$-0.90$	$-0.66$	$-0.87$	$-0.72$
$r$	$-0.949$	$-0.791$	$-0.730$	$-0.848$

Table 4: Prompts and their corresponding performance (%).

\text{PD}_{h}

\text{PD}_{l}

denote the prompts with the highest / lowest TPR on present-event detection.

\text{AD}_{l}

\text{AD}_{h}

denote the prompts with the lowest / highest FPR on absent-event detection.

Audio-Language Model Evaluation

Audio-language model evaluation tests whether systems understand speech, music, and environmental sound reliably, fairly, and with evidence grounded in audio.

See all 206 papers

Tracing Audio Grounding and Answer Selection in Audio LLMs
Sep 4, 2026Hyebin Cho, Suho Yoo, Jihoo Jung +1Audio-Language Models Audio QA
Korea Advanced Institute of Science and Technology, South Korea
SpotSound: Enhancing Large Audio-Language Models with Fine-Grained Temporal Grounding
Apr 14, 2026Luoyi Sun, Xiao Zhou, Zeqian Li +3Temporal Audio Grounding Audio-Language Models
Zhejiang University Hangzhou, Zhejiang, China · SAI, Shanghai Jiao Tong University Shanghai, China
Does Location Matter? A Black-Box Causal Audit Reveals Model-Dependent Audio Grounding in LALM Hallucination
Oct 9, 2026Mohammad Khalooei, Mohammad SabokrouAudio-Language Models Audio-Language Model Hallucination Detection
Sharif University of Technology, Iran · Amirkabir University of Technology, Iran · New Uzbekistan University, Uzbekistan +1

Sep 4, 2026J/K move · Enter open · S save

Tracing Audio Grounding and Answer Selection in Audio LLMs

Hyebin Cho, Suho Yoo, Jihoo Jung, Joon Son Chung

Korea Advanced Institute of Science and Technology, South Korea

Audio-Language Models Audio QA Mechanistic Interpretability

Open paper AskPDF arXiv

Audio Large Language Models (Audio LLMs) have advanced in audio understanding, yet they can still predict the answer by reasoning from textual cues or linguistic priors rather than the provided audio. A common remedy is to train models on data whose answers cannot be inferred from text alone. This approach can improve performance, but what changes within the model remains unclear. In this paper, we ask what must happen inside the model for the audio to actually determine the answer. Our findings are threefold. (1) Replacing the audio with silence or unrelated audio causes substantially larger performance degradation in the trained model than in the pretrained model. (2) Acoustic information most strongly shapes the model's representations of the answer choices in early-to-middle layers, while training mainly increases the influence of audio information on the final prediction in middle-to-late layers. (3) The weights learned during training have their largest impact in specific layer bands. Together, these results provide a mechanistic account of how training strengthens the use of acoustic evidence in Audio LLMs.

Abstract

Audio-Language Model Evaluation

Explore similar work

Tracing Audio Grounding and Answer Selection in Audio LLMs

SpotSound: Enhancing Large Audio-Language Models with Fine-Grained Temporal Grounding

Does Location Matter? A Black-Box Causal Audit Reveals Model-Dependent Audio Grounding in LALM Hallucination