cs.SDSep 3, 2026

Knowing When Not to Answer: Pseudo-Ensembles for Abstention in Music Audio-Language Models

Authors: Aanya Maheshwari, Vatsal Raina

Organizations: Jumeirah College, Dubai · Apta AI, Spark AI Research

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Music audio-language models are evaluated almost entirely by accuracy on multiple-choice questions. This protocol forces the model to commit to an option, so a lucky guess looks the same as real musical understanding. What is missing is a way to tell when the model does not know the answer, so that it can abstain instead of guessing. The usual solution, an ensemble of independently trained models, is far too expensive here, which leaves the entropy of a single predictive distribution as the only available confidence signal. We instead build pseudo-ensembles from one pretrained model by perturbing its input in ways that cannot change the correct answer, then averaging the resulting distributions over the options. Our main construction simply shuffles the order in which the candidate answers are presented; we also study ensembles built from corrupted audio and from swapped option labels. A pseudo-ensemble gives several predictive distributions per question, so it supports the full family of ensemble-based uncertainty measures (entropy of the expected distribution, expected entropy, and their difference, the mutual information) rather than entropy alone. Evaluating TinyMU on MuChoMusic, we find that averaging over four option orderings raises accuracy from 55.7% to 59.2%, and that the resulting uncertainty measures rank the model's errors better than the single-pass entropy baseline, reducing the area under the error retention curve from 0.293 to 0.261. All of this costs a few extra forward passes and no retraining, which makes abstention practical for compact music audio-language models.

Figures & tables

Figure 1: Overview of the task. The model returns both an answer and an uncertainty score, and abstains when the uncertainty exceeds a threshold τ\tau .
ModelSizeKnowledgeReasoningAll
MusiLingo [ 6 ]7.1B33.628.231.5
MU-LLaMA [ 28 ]7.7B32.333.532.7
Mellow [ 9 ]167M30.832.030.3
Audio Flamingo 2 [ 17 ]4.4B––56.5
Audio Flamingo 3 [ 18 ]8.3B––47.4
Qwen2-Audio-Instruct [ 5 ]8.4B69.465.567.8
Table 1: Accuracy (%) on MuChoMusic. Published numbers above, our runs of TinyMU below. Dashes mark values that are not reported in the original papers.
Uncertainty measureAUC-ERC ( ↓\downarrow )
Baseline entropy (single pass)0.293
Negative confidence0.264
Entropy of expected0.264
Expected entropy0.261
Mutual information0.306
Oracle ranking0.098
Table 2: Area under the error retention curve (lower is better). Ensemble measures use option shuffling with M=4M=4 .
Figure 3: Error retention curves for each M=4M=4 pseudo-ensemble, against the oracle and random references. A useful measure falls as the most uncertain questions are removed.
 Option shufflingAudio corruptionLabel swapping
Accuracy ( ↑\uparrow )59.2257.2042.70
Negative confidence0.2640.2840.451
Entropy of expected0.2640.2860.478
Expected entropy0.2610.2870.530
Mutual information0.3060.3060.576
Table 3: Accuracy (%) and AUC-ERC for each M=4M=4 pseudo-ensemble. Lower AUC-ERC is better. The single-pass baseline is 55.69%55.69\% and 0.2930.293 .
Figure 4: Option-shuffling accuracy versus ensemble size. For each M<24M<24 , five ensembles are drawn from the 2424 permutations; at M=24M=24 there is a single complete ensemble.
More in this topic

Audio-Language Models

Audio-language models connect audio and natural language for understanding, retrieval, dialogue, speech generation, and music or sound analysis.

See all 303 papers

Explore similar work

CardsList
Sep 24, 2026

Broadening Uncertainty Estimation for Audio Question Answering Across Methods, Formats, and Inputs

Audio-language models can produce confident answers unsupported by the audio, motivating uncertainty estimates that identify unreliable responses. We compare probability-based, sampling-based, self-verification, evidential, and contrastive measures across four open-weight models and five audio QA benchmarks. In multiple-choice evaluation, first-token measures are strongest overall, with top-1 probability achieving a mean AUROC of .740, compared with .708 for ten-sample discrete semantic entropy, while requiring no additional model calls. Across four benchmarks, shifting from multiple-choice to open-ended evaluation lowers mean accuracy from 57.6% to 36.6%, yet uncertainty remains predictive of errors: semantic entropy, maximum token entropy, and semantic agreement achieve mean AUROCs of .697, .694, and .693, respectively. To test whether uncertainty reflects the evidence available to answer the question, we perform input ablations that remove either the audio or the question. Across top-1 confidence, entropy, and sampling-based measures, removing audio reduces error-detection AUROC by .101 on average, compared with .010 when removing the question. Together, these results establish efficient uncertainty baselines and show that uncertainty in audio-language models depends substantially more on available audio evidence than on question text.

Aaron Isidore Grace, Weiran Wang

David R. Cheriton School of Computer Science, University of Waterloo, Waterloo, ON, Canada · Department of Computer Science, University of Iowa, Iowa City, IA, USA

Audio-Language Model EvaluationAudio QAConfidence Estimation in Language ModelsAudio-Language Models
AskPDF
Jun 30, 2026

Beyond Binary Instrument QA: Probing Instrument Grounding in Music Audio-Language Models

Recent music audio-language models achieve high accuracy on instrument question-answering benchmarks, but it remains unclear whether this reflects robust audio grounding or benchmark-specific shortcuts. In this paper, we introduce an OpenMIC-derived diagnostic benchmark sequence for instrument grounding in music audio-language models, extending binary instrument-presence QA to genre-prior-reduced examples, confusable instrument discrimination, longer audio context, and temporal localization. Across these settings, high binary QA accuracy often fails to predict model behavior: models can exhibit option-position bias, confusable-instrument errors, and temporal response bias. These results suggest that instrument grounding should be evaluated with multi-axis diagnostic benchmarks rather than a single aggregate accuracy.

Yujun Lee, Joonhyeok Shin, Hyoeun Kim +1

Sungkyunkwan University

Audio-Language Model EvaluationAudio QAAudio-Language ModelsMusic Understanding
AskPDF
Oct 21, 2026

Derivation Depth Exposes Reasoning Failures in Tool-Augmented Music Analysis

Audio-language models remain unreliable at extracting basic musical properties from audio, motivating pipelines that place a music information retrieval (MIR) tool between the audio and the model. The tool reports measurements such as keys, chords, and structural similarity, and the model is expected to derive the answer. We test whether it can. Holding the musical information fixed, we vary derivation depth: the number of reasoning steps between the tool output and the answer. When the report states the conclusion, all five models score near 100%, and they follow it even when it is wrong. One step away, accuracy drops. On modulation, open models fall to 57-81%, although they answer that single step correctly when asked directly, and reach 0% at greater depths. Frontier models degrade less, but still degrade. Adding the audio does not recover the loss. The bottleneck in tool-augmented music analysis is not access to musical information but reasoning from it, so tool design must account for how much derivation it leaves to the model.

Rupali Rajesh, Sethuraman T, Savya Khosla +4

IIT Madras · UIUC · Google +2

Music UnderstandingAudio-Language Model EvaluationAudio ReasoningMusic Information Retrieval
AskPDF