TRACE: Trustworthy Reasoning from Acoustic Cues and Evidence

Giovana Morais; Yujin Kim; Atilay Kucukoglu; Magdalena Fuentes

cs.SDOct 15, 2026

TRACE: Trustworthy Reasoning from Acoustic Cues and Evidence

Authors: Giovana Morais, Yujin Kim, Atilay Kucukoglu, Magdalena Fuentes

Organizations: Music and Audio Research Lab, NYU · Integrated Design and Media, NYU

Abstract

We introduce TRACE (Trustworthy Reasoning from Acoustic Cues and Evidence), a benchmark for diagnosing the reliability of large audio-language models (LALMs) by revealing when, and under what acoustic conditions these models behave untrustworthily. Existing benchmarks primarily measure whether a model produces the correct answer, but correctness alone cannot determine whether the answer is grounded in acoustic evidence or instead reflects language and contextual priors. TRACE evaluates model behavior along three complementary dimensions: (1) task performance, whether the answer is correct; (2) evidence alignment, whether the model is attending to the audio regions required to answer the question; and (3) robustness, whether answers and attributed evidence respond coherently to controlled changes such as event order, overlap, signal-to-noise ratio, or background content. TRACE targets temporal ordering, counting, and event existence, three tasks that remain challenging for current models. We synthesize contextually informed acoustic scenes that allow us to test whether models hallucinate events associated with the context but absent from the audio. This synthesis also allows us to obtain a scalable benchmark for which we know the answers by design. For each scene, we create controlled variants in a programmatic way to stress-test model robustness. We introduce attribution-based metrics that quantify model attention at scale on target events and non-target events. Evaluation across five state-of-the-art models shows that current LALMs often detect where relevant sound evidence occurs, but they do not reliably interpret it, integrate it into the answer, or respond to controlled changes in that evidence. Our benchmark also shows that models still struggle the most with ordering and counting questions.

Figures & tables

Figure 1: TRACE pipeline. Co-occurrence statistics from temporally annotated recordings guide the composition of multi-event soundscapes from isolated sources. Programmatic templates generate binary existence, ordering, and counting questions together with task-specific temporal evidence. Each reference scene is paired with answer-preserving nuisance transformations and evidence-changing counterfactual interventions. LALMs are evaluated for answer accuracy, attribution to target and distractor regions, and whether their answers and evidence change as expected across transformations.

	Scenes			Questions
Task	Ref.	Nuis.	CF	Yes	No
Existence	2000	743	257	1743	1257
Ordering	1000	766	234	635	1365
Counting	1000	793	207	677	1323
Total	4000	2302	698	3055	3945

Table 1: TRACE statistics. Ref. denotes reference scene; Nuis. denotes answer-preserving nuisance transformations; CF denotes evidence-changing counterfactual interventions. We report the number of scenes by type and distribution of question answers for each task.

	Accuracy (%)
Model	Overall	Order	Exist.	Count.
AF-Next	55.0	31.2	73.5	42.0
AF3	63.0	52.1	69.8	60.3
Qwen2.5	67.8	63.5	74.5	58.5
Kimi	69.3	70.9	69.3	67.9
Qwen3	78.4	73.3	73.6	92.9

Table 2: Performance on reference scenes without transformations. Best model results are in bold and second-best model results are underlined. The All row pools predictions and attribution records across models.

Figure 2: Positive- and negative-existence accuracy by model. Diamonds show the fraction of predictions answered “yes”; the dashed line marks the balanced rate of 50%. Most models favor negative responses.

Figure 3: Positive and negative attribution mass on target event. Hatched plot shows the negative mass value. Existence questions show the highest positive attribution mass across models.

Figure 4: Robustness analysis. How nuisance and counterfactual changes affect model accuracy, response and grounding metrics.

	All questions			Incorrect
Model	Cons.	Contra.	Ungr.	% grounded
AF3	0.452	0.265	0.284	66.6
AF-Next	0.345	0.386	0.269	71.2
Kimi	0.394	0.297	0.309	67.2
Qwen2.5	0.414	0.317	0.269	72.2
Qwen3	0.488	0.236	0.276	72.5

Table 3: Grounded cases split by whether the dominant attribution sign argues for the correct answer (Consistent) or against it (Contradictory); the three shares sum to 1. The final column is the fraction of incorrect answers that were still grounded on the target event. Sign is anchored to the model’s own answer, so a case is consistent (Cons.) when the dominant attribution sign supports the answer the model gave and that answer is correct, and contradictory when the dominant sign supports the model’s answer but the answer is wrong.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Distribution of labels in the dataset. The plot is showing top 30 labels from the reference tracks.

		Correct			Incorrect
Task	Ans.	pos	neg	net	pos	neg	net
existence	yes	0.445	0.025	+0.420	0.191	0.093	+0.098
order	yes	0.223	0.057	+0.166	0.105	0.131	$-$ 0.026
order	no	0.089	0.140	$-$ 0.051	0.177	0.066	+0.112
counting	yes	0.161	0.110	+0.051	0.087	0.213	$-$ 0.126
counting	no	0.139	0.120	+0.019	0.264	0.067	+0.197

Appendix

Table 4: Positive and negative attribution mass on the target event, pooled over models and split by the correct answer and by whether the model answered correctly. net

=

pos

-

neg. The expected pattern is net

>0

when the correct answer is yes and net

<0

when it is no . Negative-existence questions are omitted because they have no target region.

Variation	Dur.	AF3	AF-Next	Kimi	Qwen2.5	Qwen3
Removal	0.641	0.468	0.434	0.545	0.443	0.432
Overlap	0.554	0.227	0.203	0.270	0.197	0.202
Distance	0.477	0.311	0.296	0.348	0.290	0.268
Inv. order	0.477	0.318	0.306	0.343	0.294	0.276
Background	0.456	0.363	0.357	0.403	0.369	0.368
SNR	0.443	0.327	0.321	0.361	0.311	0.302

Appendix

Table 5: Attribution mass assigned to regions outside annotated events, compared with those regions’ share of clip duration (Dur.). The duration share is the fraction of the clip lying outside the union of target and distractor events, so it accounts for the shared portion

A_{\cap}

. Every model in every condition falls below the duration baseline: models under-weight background rather than relying on it.

Figure 6: Accuracy versus mean target attribution mass. Points show model means. Higher grounding does not determine higher performance across models.

Figure 7: Qualitative analysis example for Audio Flamingo 3. In the first row, we have the reference scene and what is the question the model receives, as well as the model prediction and answer. Each plot has the Shapley values for each audio token and the sound event roll with target and distractor events. Each different plot is showcasing how the attribution changes for a given variation.

Figure 8: Model comparison for the same reference question. Each heatmap represents one model and its attributions.

Figure 9: Model comparison for the same question as Figure 8 , but applying the overlap variation.

Audio-Language Model Evaluation

Audio-language model evaluation tests whether systems understand speech, music, and environmental sound reliably, fairly, and with evidence grounded in audio.

See all 206 papers

Beyond Accuracy: ARIA-Rubrics for Evaluating Audio Reasoning in Large Audio Language Models
Sep 9, 2026Yupei Li, Qiyang Sun, Mohamed Mady +4Audio-Language Model Evaluation Audio Reasoning
Imperial College London, UK · Technische Universität München München, Germany · Mohamed bin Zayed University of Artificial Intelligence Abu Dhabi, AE +2
Multi-Source Evidence Fusion for Audio Question Answering
Mar 18, 2026Aivo Olev, Tanel AlumäeAudio QA Audio Reasoning
Tallinn University of Technology, Estonia
All That Glitters Is Not Audio: Rethinking Text Priors and Audio Reliance in Audio-Language Evaluation
Apr 27, 2026Leonardo Haw-Yang Foo, Chih-Kai Yang, Chen-An Li +2Audio-Language Model Evaluation Modality Reliance in Multimodal Models
National Taiwan University · NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE)

Sep 9, 2026J/K move · Enter open · S save

Beyond Accuracy: ARIA-Rubrics for Evaluating Audio Reasoning in Large Audio Language Models

Yupei Li, Qiyang Sun, Mohamed Mady, Chenxi Wang+3

Imperial College London, UK · Technische Universität München München, Germany · Mohamed bin Zayed University of Artificial Intelligence Abu Dhabi, AE+2

Audio-Language Model Evaluation Audio Reasoning Audio-Language Models

Open paper AskPDF arXiv

Large Audio Language Models (LALMs) have shown strong performance on audio reasoning benchmarks, but accuracy alone cannot distinguish true reasoning from superficial pattern matching, often overestimating reasoning ability since high scores may result from guessing rather than genuine audio understanding. Evaluating the reasoning process itself is essential for improving LALMs' reasoning ability, yet remains challenging. Existing methods either rely on costly human annotation or opaque LLM-as-judge approaches, making them impractical, biased, and lacking transparency. Moreover, audio reasoning introduces unique challenges absent in text-based settings, perceptual hallucination and cross-modal alignment between audio understanding and textual inference, hence text-based evaluation frameworks cannot be directly applied. Therefore, we propose ARIA-Rubrics (Audio Reasoning Integrity Assessment), a lightweight, annotation-free gold reasoning chains, automatic and transparent framework comprising six complementary metrics that evaluate audio reasoning quality across perceptual grounding, reasoning coherence, and answer consistency. We use Chain-of-Thought prompting as an externalization mechanism to make the reasoning process observable. Experiments on 9 models across 2 benchmarks identify three reasoning modes of current LALMs with actionable directions for future development, with ARIA-Rubrics achieving high correlation with human judgments. The code is available at the Github Repository.

Abstract

Audio-Language Model Evaluation

Explore similar work

Beyond Accuracy: ARIA-Rubrics for Evaluating Audio Reasoning in Large Audio Language Models

Multi-Source Evidence Fusion for Audio Question Answering

All That Glitters Is Not Audio: Rethinking Text Priors and Audio Reliance in Audio-Language Evaluation