A large audio-language model (LALM) can answer "is a cat present?" about a mixture correctly, and its confidence can drop when the cat is removed, yet this does not show that it can tell the cat's presence from its absence. We introduce TRACE. It deletes the queried source and each rival in turn and asks whether every view still containing that sound outscores the view with it removed. Across 8 LALMs and 3,456 controlled mixtures, 6.3% of correct answers fail this ordering: the target-absent view scores at or above some target-present view, a conflict no shared threshold repairs. A target-only score check misses 42.5% of them. The share rises from 4.9% to 8.4% as target SNR falls from +5 to -5 dB, even on units correct at every level, and survives loudness and re-pairing controls. Accuracy certifies the answer; TRACE audits whether it orders presence above absence. Code and data: https://github.com/yikise/TRACE.
Figures & tables
Figure 1: Overview of TRACE. (a) A mixture (door wood creaks + footsteps; Qwen3-Omni-30B, 0 dB, partial overlap). The target-only check passes, yet removing the rival lowers the score further, so no shared threshold fits all views. (b) All 18,911 correct answers by Δt and maxjΔj (symlog): Δt≤0 ungrounded, on or above the dashed line competitor-confused, the rest ordering-consistent. Star: case in (a).
Check
Flagged
Conflict
Separable
Target-only score
3.6
3.6
0.0
Target-removed answer
27.3
5.6
21.7
Answer flip
30.8
6.3
24.5
TRACE (ordering)
6.3
6.3
0.0
Table 1: Decomposition of counterfactual checks on the same 18,911 correct mixture answers (%). All columns share that denominator and Flagged = Conflict + Separable throughout. The target-only and ordering rows have Conflict = Flagged by construction: set identities, not measured detection rates.
Model
Acc.
Conf.
Ungr.
RFail
Audio Flamingo 3
60.4
1.3
2.2
3.5
MiMo-Audio-7B
46.3
4.5
1.4
5.9
Qwen2-Audio-7B
74.4
3.5
5.6
9.1
Qwen2.5-Omni-3B
62.9
1.9
2.3
4.2
Qwen2.5-Omni-7B
78.6
2.1
1.4
3.6
+ AHA (LoRA)
72.0
1.9
1.3
3.2
Table 2: Attribution of correct mixture answers (%). Acc. is mixture accuracy on present-event questions; Conf. and Ungr. split the ranking failures into the two sub-cases, and RFail ( H≤0 ) comes from the same counts, so it can differ from their rounded sum. Correct-answer subsets differ across models, so these rates are unmatched.
Panel
Q1
Q2
Q3
Q4
Q5
Gap
Main (frozen targets)
22.3
10.4
4.4
2.0
0.8
+21.5
re-paired backgrounds
31.2
20.3
12.0
7.8
5.3
+25.9
targets frozen
21.4
–
–
–
0.5
+20.9
targets replaced
17.3
–
–
–
1.6
+15.7
Table 3: Where the ranking failures live (%). The 49 ESC-50 categories with ≥ 30 correct answers are split into quintiles by their failure share (Q1 highest); its Q1/Q5 role sets are then re-scored on the independently re-paired panel and re-rendered in the target control, which covers only those two quintiles. Shares are over each panel’s own correct answers; the frozen row is that control’s own baseline, not the main panel restated. Gap is Q1 − Q5, and its frozen-to-replaced difference-in-differences is -5.3 points ([-9.3, -1.9]). The re-paired row pools 3 models against 8 elsewhere.
Contrast
A
B
Δ
95% CI
Acoustic competition (common-correct units)
Target SNR, +5→−5 dB (5,466)
4.9
8.4
+3.5
[2.7, 4.3]
same, RMS-matched views
5.2
8.0
+2.9
[2.1, 3.8]
Distractors, 1→2 (5,120)
4.0
8.3
+4.3
[2.8, 5.9]
same, one randomly kept rival
4.0
6.5
+2.5
[1.3, 3.7]
Training (Qwen2.5-Omni-7B → +AHA, common-correct)
Table 4: Paired contrasts of the ranking-failure share (%): Δ= B − A in points, with 95% source-tuple bootstrap intervals. Competition rows use units correct at every level; the one-kept-rival row restricts maxj to one randomly kept rival against that cell’s two one-distractor conditions; the training row compares the AHA adapter with its base model on conditions both answer correctly.
Audio-language model evaluation tests whether systems understand speech, music, and environmental sound reliably, fairly, and with evidence grounded in audio.
We introduce TRACE (Trustworthy Reasoning from Acoustic Cues and Evidence), a benchmark for diagnosing the reliability of large audio-language models (LALMs) by revealing when, and under what acoustic conditions these models behave untrustworthily. Existing benchmarks primarily measure whether a model produces the correct answer, but correctness alone cannot determine whether the answer is grounded in acoustic evidence or instead reflects language and contextual priors. TRACE evaluates model behavior along three complementary dimensions: (1) task performance, whether the answer is correct; (2) evidence alignment, whether the model is attending to the audio regions required to answer the question; and (3) robustness, whether answers and attributed evidence respond coherently to controlled changes such as event order, overlap, signal-to-noise ratio, or background content. TRACE targets temporal ordering, counting, and event existence, three tasks that remain challenging for current models. We synthesize contextually informed acoustic scenes that allow us to test whether models hallucinate events associated with the context but absent from the audio. This synthesis also allows us to obtain a scalable benchmark for which we know the answers by design. For each scene, we create controlled variants in a programmatic way to stress-test model robustness. We introduce attribution-based metrics that quantify model attention at scale on target events and non-target events. Evaluation across five state-of-the-art models shows that current LALMs often detect where relevant sound evidence occurs, but they do not reliably interpret it, integrate it into the answer, or respond to controlled changes in that evidence. Our benchmark also shows that models still struggle the most with ordering and counting questions.
Giovana Morais, Yujin Kim, Atilay Kucukoglu +1
Music and Audio Research Lab, NYU · Integrated Design and Media, NYU
Audio-language models (ALMs) are increasingly used as judges for speech-to-speech systems, but a judge that receives audio may not actually use paralinguistic evidence. We introduce counterfactual audits for paralinguistic response evaluation. Each audit item holds the transcript fixed while varying affect, prosody, or the timing of an affective shift, forcing a valid judge to track the audio cue rather than lexical content or response style. We evaluate ALM judges using a native one-context judgment protocol and a contrastive recoverability control, then further decompose each item into its constituent perception and response-mapping skills. This yields useful diagnostic states that identify different sources of judge failures. Across Gemini, GPT, and open audio models, we find that contrastive success often overstates native judge reliability, and that similar aggregate accuracies can hide different failure modes. These results suggest that ALM judges should not be evaluated by accuracy alone, instead requiring thorough behavioral audits before deployment.
Recent advances in large audio language models (LALMs) have primarily been assessed using a multiple-choice question answering (MCQA) framework. However, subtle changes, such as shifting the order of choices, result in substantially different results. Existing MCQA frameworks do not account for this variability and report a single accuracy number per benchmark or category. We dive into the MCQA evaluation framework and conduct a systematic study spanning three benchmarks (MMAU, MMAR and MMSU) and four models: Audio Flamingo 2, Audio Flamingo 3, Qwen2.5-Omni-7B-Instruct, and Kimi-Audio-7B-Instruct. Our findings indicate that models are sensitive not only to the ordering of choices, but also to the paraphrasing of the question and the choices. Finally, we propose a simpler evaluation protocol and metric that account for subtle variations and provide a more detailed evaluation report of LALMs within the MCQA framework.
Fernando López, Santosh Kesiraju, Jordi Luque
Scientific Research, Telef´onica Innovaci´on Digital, Spain · Brno University of Technology, Czech Republic