Broadening Uncertainty Estimation for Audio Question Answering Across Methods, Formats, and Inputs

Aaron Isidore Grace; Weiran Wang

cs.SDSep 24, 2026

Broadening Uncertainty Estimation for Audio Question Answering Across Methods, Formats, and Inputs

Authors: Aaron Isidore Grace, Weiran Wang

Organizations: David R. Cheriton School of Computer Science, University of Waterloo, Waterloo, ON, Canada · Department of Computer Science, University of Iowa, Iowa City, IA, USA

Abstract

Audio-language models can produce confident answers unsupported by the audio, motivating uncertainty estimates that identify unreliable responses. We compare probability-based, sampling-based, self-verification, evidential, and contrastive measures across four open-weight models and five audio QA benchmarks. In multiple-choice evaluation, first-token measures are strongest overall, with top-1 probability achieving a mean AUROC of .740, compared with .708 for ten-sample discrete semantic entropy, while requiring no additional model calls. Across four benchmarks, shifting from multiple-choice to open-ended evaluation lowers mean accuracy from 57.6% to 36.6%, yet uncertainty remains predictive of errors: semantic entropy, maximum token entropy, and semantic agreement achieve mean AUROCs of .697, .694, and .693, respectively. To test whether uncertainty reflects the evidence available to answer the question, we perform input ablations that remove either the audio or the question. Across top-1 confidence, entropy, and sampling-based measures, removing audio reduces error-detection AUROC by .101 on average, compared with .010 when removing the question. Together, these results establish efficient uncertainty baselines and show that uncertainty in audio-language models depends substantially more on available audio evidence than on question text.

Figures & tables

			Single-pass			Sampling	Contrastive			Self-verification
Benchmark	Model	Acc.	TOP1	$H_{\mathrm{all}}$	$\kappa$	DSE	UE	Sup	L2	P(IK)	P(aud)
AQUA-Bench	AF3	55.5	.685	.660	.642	.646	.665	.596	.564	.487	.829
	Phi-4-MM	31.2	.510	.499	.392	.504	.514	.536	.470	.338	.270
	Qwen2-A	39.0	.730	.734	.693	.690	.734	.746	.685	.659	.575
	Qwen2.5-O	73.9	.729	.692	.693	.703	.702	.507	.413	.458	.910
MMAU	AF3	74.6	.860	.851	.842	.823	.856	.651	.492	.604	.623

Table 1: Multiple-choice accuracy (%) and uncertainty AUROC across models and benchmarks. Bold indicates the best AUROC in each row.

			Token uncertainty			Seq. likelihood		Sampling		Contrastive		Self-verification
Benchmark	Model	Match	$\mathit{TOP1}_{\mathit{MIN}}$	$H_{\max}$	$H_{\mathrm{first}}$	NLL	PPL	SemEnt	SemAgr	UE	CWO	P(IK)	P(aud)
AQUA	AF3	78.8	.750	.743	.703	.754	.730	.730	.758	.666	.376	.489	.378
	Phi-4-MM	45.7	.763	.733	.728	.782	.778	.759	.759	.718	.242	.281	.281
	Qwen2-A	32.5	.715	.776	.762	.732	.716	.771	.755	.785	.486	.747	.639
	Qwen2.5-O	80.5	.657	.643	.636	.677	.671	.648	.650	.551	.351	.416	.718
MMAR	AF3	31.5	.692	.687	.586	.678	.621	.697	.720	.597	.522	.510	.520

Table 2: Open-ended evaluation: judge-match accuracy (%) and uncertainty AUROC. Bold indicates the best AUROC in each row.

Dataset	Mean score	AUROC	Mean score	AUROC	$\Delta$	AUROC	$\Delta$	AUROC	$\Delta$	AUROC	$\Delta$	AUROC
	Canonical				No question				No audio
	$H_{\mathrm{all}}$		DSE		$H_{\mathrm{all}}$		DSE		$H_{\mathrm{all}}$		DSE
AQUA-Bench	.068	.646	.610	.636	$+.004$	.667	$+.036$	.649	$+.018$	.630	$+.148$	.613
MMAU	.049	.821	.460	.786	$+.010$	.786	$+.096$	.756	$+.016$	.728	$+.139$	.699
MMAR	.063	.700	.588	.679	$+.003$	.707	$+.025$	.684	$+.012$	.606	$+.100$	.575
MMSU	.057	.785	.537	.753	$+.004$	.756	$+.029$	.729	$+.015$	.659	$+.120$	.638

Table 3: Multiple-choice input ablation, averaged over four models. Mean score is the mean uncertainty value under full input,

\Delta

its mean change under ablation (ablated minus canonical). Bold marks the best AUROC in each condition.

Explore similar work

Walking Through Uncertainty: An Empirical Study of Uncertainty Estimation for Audio-Aware Large Language Models
Apr 28, 2026Chun-Yi Kuan, Wei-Ping Huang, Hung-yi LeeLarge Language Model Uncertainty Large Audio Language Models
Graduate Institute of Communication Engineering, National Taiwan University, Taiwan · Artificial Intelligence Center of Research Excellence (AI-CoRE), National Taiwan University, Taiwan
Learning from Audio-Dependency Errors: Data Curation Strategies Based on Model Confusion Patterns in Audio Question Answering
Jun 20, 2026Hyeonuk NamLarge Audio Language Models Audio Understanding
Independent Researcher · Seoul, South Korea
ORCA: Open-ended Response Correctness Assessment for Audio Question Answering
Nov 28, 2025Šimon Sedláček, Sara Barahona, Bolaji Yusuf +9Large Audio Language Models Open-Ended Responses
AUDIAS, Universidad Autónoma de Madrid, Spain · University of Buenos Aires, Argentina · Tufts University, USA +1

Apr 28, 2026, eess.ASJ/K move · Enter open · S save

Walking Through Uncertainty: An Empirical Study of Uncertainty Estimation for Audio-Aware Large Language Models

Chun-Yi Kuan, Wei-Ping Huang, Hung-yi Lee

Graduate Institute of Communication Engineering, National Taiwan University, Taiwan · Artificial Intelligence Center of Research Excellence (AI-CoRE), National Taiwan University, Taiwan

Large Language Model Uncertainty Large Audio Language Models Uncertainty Semantic Entropy Reasoning Benchmark

Open paper AskPDF arXiv

Recent audio-aware large language models (ALLMs) have demonstrated strong capabilities across diverse audio understanding and reasoning tasks, but they still frequently produce hallucinated or overly confident outputs. While uncertainty estimation has been extensively studied in text-only LLMs, it remains largely unexplored for ALLMs, where audio-conditioned generation introduces additional challenges such as perceptual ambiguity and cross-modal grounding. In this work, we present the first systematic empirical study of uncertainty estimation in ALLMs. We benchmark five representative methods, including predictive entropy, length-normalized entropy, semantic entropy, discrete semantic entropy, and P(True), across multiple models and diverse evaluation settings spanning general audio understanding, reasoning, hallucination detection, and unanswerable question answering. Our results reveal two key findings. First, semantic-level and verification-based methods consistently outperform token-level baselines on general audio reasoning benchmarks. Second, on trustworthiness-oriented benchmarks, the relative effectiveness of uncertainty methods becomes notably more model- and benchmark-dependent, indicating that conclusions drawn from general reasoning settings do not straightforwardly transfer to hallucination and unanswerable-question scenarios. We further explore uncertainty-based adaptive inference as a potential downstream application. We hope this study provides a foundation for future research on reliable, uncertainty-aware audio-language systems.

Abstract

Explore similar work

Walking Through Uncertainty: An Empirical Study of Uncertainty Estimation for Audio-Aware Large Language Models

Learning from Audio-Dependency Errors: Data Curation Strategies Based on Model Confusion Patterns in Audio Question Answering

ORCA: Open-ended Response Correctness Assessment for Audio Question Answering