Listen-to-Reason: Listen with Experts, Retrieve over a Graph, Reason with LLMs

Pooneh Mousavi; Mirco Ravanelli; Cem Subakan

cs.SDOct 7, 2026

Listen-to-Reason: Listen with Experts, Retrieve over a Graph, Reason with LLMs

Authors: Pooneh Mousavi, Mirco Ravanelli, Cem Subakan

Organizations: Concordia University · Mila – Quebec AI Institute · Laval University

Abstract

Large audio-language models (LALMs) fuse an audio encoder into a large language model (LLM) through multi-stage training. This coupling means that a new domain or a stronger LLM requires retraining, and their answers cannot be traced to what the model heard: a chain-of-thought is a post-hoc account. We propose Listen-to-Reason (L2R), an interpretable-by-design pipeline that passes audio to the LLM through an explicit, human-readable tree: small heads on frozen expert encoders map each chunk of a clip to semantically meaningful nodes on the tree (for speech, music and environmental sound), and a frozen text-only LLM answers from these nodes and an ASR transcript without hearing the clip. Every answer can therefore be traced to the nodes and transcript it read, and the nodes are causal: replacing the deciding node with a distractor overturns 78% of correct answers on SAKURA. With a 7B reader, L2R outperforms all LALMs we compare against on SAKURA and trails them by 6-12 points on MMAU and MMAR, despite training about 1,400x fewer parameters on orders of magnitude less audio data. However, because any LLM can serve as the reader, we show that a stronger reader narrows this gap without retraining any audio component. A new domain is added with one small head: with five labelled clips per species, it outperforms QLoRA fine-tuning of an LALM on the same clips by 13-26 points.

Figures & tables

Figure 1: Overview of Listen-to-Reason . Listen: a four-level tree describes speech, music and sound. Retrieve: router heads on frozen encoders map each chunk to tree nodes, which join an ASR transcript in a readable context. Reason: a frozen text-only LLM answers from this context. Adapt: a new domain is a new encoder–head branch (dashed).

		B	SAKURA $\uparrow$	MMAU $\uparrow$	MMAR $\uparrow$	MMAU-Pro $\uparrow$
Input ablation (Qwen2.5-7B)	Blind (question and options only)	7.6	32.6	50.2	37.8	38.9
	Transcript only	7.6	51.8	60.1	47.8	44.6
	Tree only	7.6	74.7	57.5	40.3	39.3
L2R (ours) (tree + transcript) by reader	Qwen2.5-7B	7.6	80.1	68.3	51.6	45.7
	Qwen2.5-14B	14.7	83.3	69.4	55.6	48.5
	Qwen2.5-32B	32.8	85.1	71.4	55.7	49.8

Table 1: Accuracy (%) on four audio reasoning benchmarks , full sets. Our readers never hear the clip; they read the same cached tree and transcript. B = parameters in billions (for audio LLMs, total size from the released weights). On MMAU-Pro the options are shuffled per item for all systems (Appendix D ).

Figure 2: Accuracy of Listen-to-Reason as a function of the frozen reader. Every reader reads the same cached tree and transcript contexts. Lines connect the Qwen2.5 (solid) and dense Qwen3 (dashed) families; grey points are the other 15 readers (Llama, Vicuna, Mistral, Phi, OLMo, Falcon, SmolLM, gpt-oss and the mixture-of-experts Qwen3-30B-A3B). Horizontal lines are audio LLMs that hear the clip. Full table in Appendix F .

Panel of Figure 3 — Figure 3: Data efficiency on new domains. One small head (0.34M parameters for birds, 0.17M for marine mammals) on a frozen encoder (BirdNET for birds, CLAP for marine mammals), read by a frozen Qwen2.5-7B, against Omni-7B fine-tuned with QLoRA on the same clips and Omni-7B given the clips in context.

Figure 4: Relatedness of the served nodes to the audio (1–5). Bars: mean of three judges with 95% bootstrap intervals; markers: each judge; numbers: share of items where the L2R nodes score higher.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Region	Attribute	Classes	Leaves	Decided per	Example classes	Router encoder	Expert labeller	Human labels
all	regions present*	3	–	chunk	speech, music, environmental sound	CLAP	PANNs (0.30)	–
speech	gender	2	–	chunk	male voice, female voice	WavLM-SV	WavLM-SV (0.05)	–
	age group	4	–	chunk	child, young adult, elderly adult	Whisper	–	–
	language	7	50	clip	western european ( french ), south asian	Whisper †	Whisper-LID (0.50)	–
	number of speakers	3	–	clip	one, two, three or more	Whisper	–	–
	emotion	7	50	chunk	angry, happy, sad, neutral	emotion2vec	emotion2vec (0.60)	–

Appendix

Table 2: Attributes of the tree. * = multiple-choice attribute. Router encoder = frozen encoder whose embedding the attribute’s head reads at training and inference. Expert labeller = expert classifier that labels the attribute when the tree is built, with its confidence threshold; it does not run at inference. Human labels = datasets whose labels are added to the training targets. † When a transcript is available, language is predicted from its text. New-domain attributes are added after the tree is built, each trained on

k

labelled clips per species plus clips without that domain.

Attribute	Classes	Leaves	Router encoder	Example classes
animal	26	96	BEATs	bird, cat, cattle, dog, insect
vehicle	12	40	CLAP	aircraft, boat, car, emergency vehicle, train
machine	2	37	CLAP	engine, mechanisms
tool	7	12	CLAP	hammer, jackhammer, power tool, sanding
household	23	38	CLAP	blender, dishes, door, tap or sink, typing
human non-speech	24	58	BEATs	applause, breathing, cough, laughter

Appendix

Table 3: Identity attributes of environmental sound (all multiple-choice), following the AudioSet ontology. Each attribute is read by CLAP or BEATs, whichever scores higher on held-out AudioSet and FSD50K clips. The new-domain heads of § 4.3 read BirdNET (birds) and CLAP (marine mammals).

Step	Rule
Normalize	Remove filler words ( a , some , sound ) and merge names that are nearly identical and have the same number of words.
Admit	Keep a leaf only if it appears in at least 2 clips from at least 2 sources, adds detail beyond its class, and contains no number.
Merge	The text LLM proposes groups of synonyms. A group is kept only if its names do not conflict in number, negation or polarity ( high-pitched vs low-pitched ), its chunks sound closer to each other than to the other leaves of the class, and it does not join two different sources ( goat bleating vs sheep bleating ).
Prune	Fold a leaf that only adds a filler word into its sibling ( alarm sound $\to$ alarm ). Remove attributes whose content lies in the spoken words (topic, intent, scenario, speaker role) or that carried no usable information (recording channel, voice quality, accent, weather, number of sources).

Appendix

Table 4: Leaf post-processing.

Heads	Architecture	Loss	Learning rate	Batch	Training
speech, music, scene	1–3-layer MLP body (width 256), one head per attribute	masked CE / BCE (Eq. 2 )	$3\!\times\!10^{-4}$ – $10^{-3}$ , one-cycle	64	60–80 epochs
environmental-sound identity	LayerNorm + 1,024-unit layer, sigmoid per node	masked BCE, positive weights	$10^{-3}$	512	25 epochs
music properties	2-layer MLP (width 256)	class-weighted CE	$10^{-3}$	64	60 epochs
event timeline	linear layer on PANNs frames	BCE on strong labels	$10^{-4}$	256	6 epochs
new domain	1-layer MLP body, classes + none	CE	$10^{-3}$	64	40 epochs

Appendix

Table 5: Head settings. All heads use AdamW; checkpoints are selected on held-out training data.

Dataset	Domain	Tree pool	Router training (labels, size)
WavCaps AudioSet-SL	sound	805	sound hierarchy: AudioSet labels, 81,325 clips
Clotho	sound	645	–
FSD50K	sound	413	sound hierarchy (CLAP heads): leaf labels, 24,238 dev clips
ESC-50	sound	280	sound hierarchy (CLAP heads): class labels, 1,793 clips
WavCaps SoundBible	sound	127	–
AudioCaps	sound	107	–

Appendix

Table 6: Datasets used to generate the tree and to train the router. Tree pool = clips in the tree-generation pool (4,054 in total, labelled by the annotators). Router = additional labelled data used to train heads; “–” means the dataset reaches the router only through the pool’s labels (the speech heads, including emotion, are trained on the pool alone). AudioSet test (9,042 clips) and FSD50K evaluation (10,161 clips) are held out. No benchmark audio is used.

removed group	SAKURA	MMAU	MMAR
every leaf (classes kept)	$-8.0$	$+0.3$	$-1.3$
speech gender	$-7.7$	$-1.0$	$+0.3$
sound source (event, action and material)	$-7.2$	$-1.3$	$-0.3$
speech emotion	$-7.0$	$+1.3$	$-0.7$
speech language	$-0.8$	$+1.0$	$+0.7$
music identity (genre, instrument, mood, era, vocals)	$0.0$	$-2.3$	$+0.3$

Appendix

Table 7: Attribute groups. Change in accuracy (points) when one attribute group is removed from a perfect-router tree, on the development clips (SAKURA 600 items, MMAU 300, MMAR 298; one item is 0.2, 0.3 and 0.3 points). Negative values mean the group helps.

Reader	B	MMAU $\uparrow$	MMAR $\uparrow$	SAKURA $\uparrow$
Qwen2.5-0.5B	0.5	40.5	36.5	47.6
Qwen2.5-1.5B	1.5	61.5	47.3	69.3
SmolLM2-1.7B	1.7	43.2	39.1	44.9
Qwen2.5-3B	3	63.1	50.1	72.0
Phi-3.5-mini	3.8	63.0	47.4	73.8
Qwen3-4B	4	63.9	50.9	78.2

Appendix

Table 8: Choice of reader. Accuracy (%) of every frozen reader on the same cached tree + transcript contexts, full sets. B = parameters in billions (Qwen3-30B-A3B has 3B active). Prompt template locked per reader on 100 development items per benchmark. Llama-2 and Vicuna are scored with an answer prefix; gpt-oss by generation. Audio LLMs: B is total size from the released weights.

	method (accuracy $\uparrow$ )	$k{=}1$	3	5	10	20	50
Birds (48 sp.)	L2R (ours, 0.34M trainable)	48.4	72.7	79.6	85.3	88.4	91.1
	Omni-7B + QLoRA (40.4M trainable)	–	–	54.1	61.0	69.0	75.2
	Omni-7B, clips in context	43.3	46.1	47.3	–	–	–
Marine (31 sp.)	L2R (ours, 0.17M trainable)	41.3	59.6	65.2	76.5	83.0	–
	Omni-7B + QLoRA (40.4M trainable)	–	–	52.2	68.6	73.5	83.6
	Omni-7B, clips in context	44.1	57.1	60.5	–	–	–

Appendix

Table 9: Few-shot domain adaptation , 4-way species identification, accuracy (%) by labelled clips per species (

k

). Ours = one new head on a frozen encoder, frozen Qwen2.5-7B reader, tree only. Omni-7B zero-shot: birds 32.3, marine 25.8. Marine has a median of 33 labelled clips per species, so

k{=}50

is not reachable for our head.

question type	n	ours $\uparrow$	Omni $\uparrow$	blind $\uparrow$	net $\uparrow$	n	ours $\uparrow$	Omni $\uparrow$	blind $\uparrow$	net $\uparrow$
	MMAU					MMAR
counting	100	47.0	53.0	43.0	-6	122	33.6	45.1	27.0	-14
order / timing	113	60.2	62.8	39.8	-3	106	47.2	57.5	37.7	-11
scene / place / activity	115	77.4	93.0	46.1	-18	111	58.6	71.2	27.0	-14
identify source / instrument / genre	148	71.6	83.8	60.1	-18	30	43.3	46.7	40.0	-1
music theory	102	57.8	61.8	52.9	-4	22	50.0	31.8	40.9	+4

Appendix

Table 10: Where Listen-to-Reason lacks , by question type (assigned from the question wording), against Omni-7B and the blind reader, scored against gold. Net = items we get right minus items Omni gets right. Qwen2.5-7B reader.

sub-category	n	ours $\uparrow$	Omni $\uparrow$	blind $\uparrow$	net $\uparrow$
Acoustic Scene Reasoning	48	56.2	81.2	33.3	-12
Musical Texture Interpretation	34	58.8	82.4	44.1	-8
Event-Based Sound Reasoning	48	64.6	79.2	62.5	-7
Acoustic Source Inference	48	75.0	87.5	72.9	-6
Temporal Event Reasoning	48	58.3	70.8	25.0	-6
Melodic Structure Interpretation	33	48.5	66.7	42.4	-6

Appendix

Table 11: MMAU by sub-category , Qwen2.5-7B reader. Net = items right for us minus items right for Omni-7B.

sub-category	n	ours $\uparrow$	Omni $\uparrow$	blind $\uparrow$	net $\uparrow$
Environmental Perception and Reasoning	149	57.7	73.2	30.9	-23
Content Analysis	301	62.1	67.4	44.9	-16
Counting and Statistics	99	31.3	46.5	25.3	-15
Correlation Analysis	50	44.0	68.0	28.0	-12
Professional Knowledge and Reasoning	70	47.1	62.9	34.3	-11
Speaker Analysis	48	52.1	68.8	29.2	-8

Appendix

Table 12: MMAR by sub-category , Qwen2.5-7B reader. Net = items right for us minus items right for Omni-7B.

Panel of Figure 5 — Figure 5: Interpretable by design. Real MMAU items with the complete tree context given to the frozen Qwen2.5-7B (no transcript is given for these clips: no chunk heard speech or singing). Bold marks the node that decides the answer.

	total params $\downarrow$	trained params $\downarrow$	audio training data $\downarrow$	MACs / item $\downarrow$	s / item $\downarrow$
L2R (ours)	10.50B	7.7M	$\approx$ 0.2M clips	5.3T	0.57
Qwen2.5-Omni-7B	10.73B	10.73B	300B tokens	2.7T	0.12
Audio Flamingo 3	8.27B	8.27B	9.6M QA + 13.2M captions	3.3T	0.11
Qwen3-Omni-30B-A3B	35.3B (3B active)	35.3B	20M h + 0.77T tokens	0.76T	0.93

Appendix

Table 13: Cost of Listen-to-Reason (Qwen2.5-7B reader) and audio LLMs. MACs and seconds are per MMAU item on one A100 GPU; MACs count executed operations, so a mixture-of-experts model counts only its active parameters.

Explore similar work

Beyond Accuracy: ARIA-Rubrics for Evaluating Audio Reasoning in Large Audio Language Models
Sep 9, 2026Yupei Li, Qiyang Sun, Mohamed Mady +4Audio-Language Model Evaluation Audio Reasoning
Imperial College London, UK · Technische Universität München München, Germany · Mohamed bin Zayed University of Artificial Intelligence Abu Dhabi, AE +2
Audio-DeepThinker: Progressive Reasoning-Aware Reinforcement Learning for High-Quality Chain-of-Thought Emergence in Audio Language Models
Apr 20, 2026Xiang He, Chenxing Li, Jinting Wang +5RL for Language Model Reasoning Audio Reasoning
Tencent AI Lab, Beijing, China · The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China
EChO-Agent: Evidence Chain Orchestration Agent for Audio Reasoning
Jun 13, 2026Siyuan Zhang, Jian Zong, Junyu Wang +7Audio QA Tool-Augmented Language Model Agents
School of Artificial Intelligence, Tianjin University, Tianjin, China

Sep 9, 2026, cs.SDJ/K move · Enter open · S save

Beyond Accuracy: ARIA-Rubrics for Evaluating Audio Reasoning in Large Audio Language Models

Yupei Li, Qiyang Sun, Mohamed Mady, Chenxi Wang+3

Imperial College London, UK · Technische Universität München München, Germany · Mohamed bin Zayed University of Artificial Intelligence Abu Dhabi, AE+2

Audio-Language Model Evaluation Audio Reasoning Audio-Language Models

Open paper AskPDF arXiv

Large Audio Language Models (LALMs) have shown strong performance on audio reasoning benchmarks, but accuracy alone cannot distinguish true reasoning from superficial pattern matching, often overestimating reasoning ability since high scores may result from guessing rather than genuine audio understanding. Evaluating the reasoning process itself is essential for improving LALMs' reasoning ability, yet remains challenging. Existing methods either rely on costly human annotation or opaque LLM-as-judge approaches, making them impractical, biased, and lacking transparency. Moreover, audio reasoning introduces unique challenges absent in text-based settings, perceptual hallucination and cross-modal alignment between audio understanding and textual inference, hence text-based evaluation frameworks cannot be directly applied. Therefore, we propose ARIA-Rubrics (Audio Reasoning Integrity Assessment), a lightweight, annotation-free gold reasoning chains, automatic and transparent framework comprising six complementary metrics that evaluate audio reasoning quality across perceptual grounding, reasoning coherence, and answer consistency. We use Chain-of-Thought prompting as an externalization mechanism to make the reasoning process observable. Experiments on 9 models across 2 benchmarks identify three reasoning modes of current LALMs with actionable directions for future development, with ARIA-Rubrics achieving high correlation with human judgments. The code is available at the Github Repository.

Abstract

Explore similar work

Beyond Accuracy: ARIA-Rubrics for Evaluating Audio Reasoning in Large Audio Language Models

Audio-DeepThinker: Progressive Reasoning-Aware Reinforcement Learning for High-Quality Chain-of-Thought Emergence in Audio Language Models

EChO-Agent: Evidence Chain Orchestration Agent for Audio Reasoning