Long Listening Thoughts: Eliciting Open Auditory Reasoning with Deliberative Perception and Cognitive Refinement

Jaeyeon Kim; Chao-Han Huck Yang; Luoyi Zhang; Chan-Jan Hsu; Fernando Ruiloba Portilla; Jinchuan Tian; Shinji Watanabe; Carlos Busso

cs.SDOct 17, 2026

Long Listening Thoughts: Eliciting Open Auditory Reasoning with Deliberative Perception and Cognitive Refinement

Authors: Jaeyeon Kim, Chao-Han Huck Yang, Luoyi Zhang, Chan-Jan Hsu, Fernando Ruiloba Portilla, Jinchuan Tian, Shinji Watanabe, Carlos Busso

Organizations: Carnegie Mellon University · NVIDIA

Abstract

We introduce Long Listening Thoughts (LLT), a framework for constructing post-training datasets that improves audio reasoning by pairing deliberate perceptual grounding with explicit cognitive behaviors. On audio understanding and reasoning benchmarks, large audio-language models (LALMs) are limited primarily by perceptual errors, yet existing post-training datasets rely on audio descriptions of limited diversity. Through a four-stage pipeline, LLT (1) builds multi-aspect dense audio descriptions spanning semantic, acoustic, and temporal information, (2) generates more challenging questions, (3) synthesizes incorrect reasoning traces, and (4) extends short reasoning traces into long ones via a thought-continuation strategy that elicits cognitive behaviors such as verification and backtracking. Using LLT, we synthesize 1.6M reasoning traces and train Qwen2.5-Omni-7B with a two-stage pipeline of supervised fine-tuning (SFT) followed by direct preference optimization (DPO). This improves accuracy on MMAU Test-mini (71.5 to 78.2), MMAR (56.7 to 63.7), and MMSU (60.6 to 66.0).

Figures & tables

Figure 1: Overview of the proposed Long Listening Thoughts pipeline. Stage 1 extracts multi-aspect audio descriptions (semantic, acoustic, temporal) from specialized models. Stage 2 augments more challenging questions using an LLM. Stage 3 generates incorrect reasoning traces. Stage 4 extends traces via thought continuation to elicit cognitive behaviors.

Panel of Figure 2 — Figure 2: Accuracy of an LLM across different benchmarks using different types of audio descriptions. Semantic, Acoustic, and Temporal correspond to the respective components of the multi-aspect audio descriptions constructed in Stage 1 of LLT.

Method	MMAU Test-mini	MMAR	MMSU
SFT with 50k QA Pairs
AF-Think ( $r^{+}$ )	74.1	58.8	63.4
LLT ( $r^{+}$ , $r^{+}\!\rightarrow\!r^{++}$ )	75.7	61.1	64.0
LLT ( $r^{+}$ , $r^{+}\!\rightarrow\!r^{++}$ , $r^{-}\!\rightarrow\!r^{++}$ )	74.6	60.2	63.3
SFT with Full Reasoning Traces
AF-Think ( $r^{+}$ )	73.9	59.9	63.7

Table 1: Comparison across different training setups. Notation for reasoning traces follows Section 3 . Parentheses indicate the trace types used for SFT. The last block reports the full two-stage pipeline (SFT

\rightarrow

DPO). Best result in each block is bolded.

Figure 3: Cognitive analysis of model responses on MMAR. (a) Mean occurrence of different cognitive behaviors (b) Distribution of response length (number of words).

Method	MMAU Test-mini	MMAR	MMSU
Large Audio Language Models
Qwen2-Audio-Instruct ( 2024 )	59.6	30.0	53.3
Kimi-Audio ( 2025 )	68.2	57.6	59.3
Audio Flamingo 3 ( 2025 )	73.3	58.5	61.4
Audio Reasoning Models
SFT-based

Table 2: Performance comparison across different models. Omni-R1 (AVQA) indicates the variant trained on the naive AVQA dataset. For clarity, we highlight in bold the best-performing existing method and the best-performing model among our approaches.

\dagger

denotes methods built on the same Qwen2.5-Omni-7B backbone as ours;

\ddagger

denotes results reproduced by us.

Setting	MMAU Test-mini	MMAR	MMSU
Qwen2.5-Omni baseline	71.5	56.7	60.6
Stage 3: Source of Incorrect Traces
SFT on 50k QA Pairs
From base LALM	72.9	58.3	64.4
From $M_{\text{instruct}}$	74.6	60.2	63.3
Two-Stage Training

Table 3: Ablation studies on the LLT pipeline. Each block isolates one design choice: the source of incorrect traces (Stage 3), DPO pair composition, and generalization to a different source dataset. The best result in each block is in bold.

Method	MMAU Test-mini	MMAR	MMSU
Qwen2-Audio-Instruct	59.6	30.0	53.3
+ LLT SFT	67.8	53.8	59.0
Qwen3-Omni-Instruct	77.5	66.9	69.0
+ LLT SFT	79.6	67.9	71.6

Table 4: Generalization of LLT to additional LALM backbones.

Figure 4: Qualitative comparison of responses from Qwen2.5-Omni-7B trained with AF-Think and with the proposed LLT on the same question.

Appendix figures & tables31 assets

Supplementary material from the paper’s appendix.

Appendix

Pre $\rightarrow$ Post	# Responses	%
correct $\rightarrow$ correct	668	59.0
incorrect $\rightarrow$ incorrect	309	27.3
incorrect $\rightarrow$ correct (correction)	99	8.7
correct $\rightarrow$ incorrect (over-correction)	56	4.9
Net correction	+43	+3.8

Appendix

Table 5: Pre-trigger vs. post-trigger answer transitions on MMAR and MMAU responses from the two-stage model (LLT SFT

\rightarrow

DPO) that contain a Stage 4 cognitive trigger, over the 1,132 responses with a determinable pre-trigger answer. The net correction is statistically significant (McNemar test,

p{<}0.001

Setting	MMAU Test-mini	MMAR	MMSU
Thought expansion	75.6	60.5	63.8
Thought continuation	75.7	61.1	64.0

Appendix

Table 6: Ablation on long-trace construction (Stage 4). The best result is in bold.

Trace (role in DPO)	Base	SFT ckpt (DPO init)
$r^{+}$ (chosen)	2.173	0.715
$r^{+}\!\rightarrow\!r^{++}$ (chosen)	2.153	0.831
$r^{-}\!\rightarrow\!r^{++}$ , synthetic (chosen)	2.348	1.428
$r^{-}\!\rightarrow\!r^{++}$ , model-sampled (chosen)	1.774	1.282
$r^{-}$ , synthetic (rejected)	2.510	1.948
$r^{-}$ , model-sampled (rejected)	0.767	1.615

Appendix

Table 7: Per-token NLL of each trace type (lower = more likely under the model).

Preference pair	Synthetic	Model-sampled
$r^{+}\succ r^{-}$	1.233	0.900
$r^{+}\!\rightarrow\!r^{++}\succ r^{-}$	1.117	0.784
$r^{-}\!\rightarrow\!r^{++}\succ r^{-}$	0.520	0.333

Appendix

Table 8: Separation between chosen and rejected responses under the DPO reference policy.

Model	Overall	$\leq$ 30s ( $n{=}1720$ )	$>$ 30s ( $n{=}2862$ )
Qwen2.5-Omni-7B (reported)	52.2	–	–
Qwen2.5-Omni-7B (reproduced)	50.1	42.6	57.1
+ LLT (SFT $\rightarrow$ DPO)	55.2	48.7	59.8

Appendix

Table 9: Results on MMAU-Pro.

Model	Weighted Avg	Long-Form	Cough	Heart	Lung	Speech	Speech + Sound	Multi-Turn	Voice QA
Qwen2.5-Omni-7B	47.5	54.7	44.4	38.0	36.1	55.6	46.1	100.0	56.1
+ LLT (SFT $\rightarrow$ DPO)	51.3	45.3	51.1	45.8	44.3	55.3	52.8	88.9	47.8

Appendix

Table 10: Results on the publicly released subset of MedMosaic.

Model	Overall	Perception	Cognition
Qwen2.5-Omni-7B	35.9	38.9	35.5
+ LLT (SFT $\rightarrow$ DPO)	40.4	44.6	39.6

Appendix

Table 11: Results on WoW-Bench.

Model	MMAU Test-mini	MMAR
Qwen2.5-Omni-7B (reported)	71.5	56.7
Qwen2.5-Omni-7B (reproduced)	74.6	57.9
+ Gaussian noise	49.2	36.9
+ No audio	48.4	34.6
LLT (SFT $\rightarrow$ DPO)	78.2	63.7
+ Gaussian noise	48.9	31.4

Appendix

Table 12: Audio ablation on MMAU Test-mini and MMAR.

Figure 5: Full annotation guidelines shown to annotators at the start of the labeling interface.

Panel of Figure 6 — Figure 6: Labeling interface presented to annotators for (a) Type A and (b) Type B trace pairs. Each item shows the audio clip, the question with its correct answer, the initial trace (left) and the continuation (right), and the per-trace and pair-level judgments.

	Type A		Type B
	$r^{+}$	$r^{++}$	$r^{-}$	$r^{++}$
Per-trace quality
Hallucination	13	19	–	18
Textual artifact	1	1	1	2
Reasoning valid	96	99	–	95
Plausibly incorrect	–	–	77	–

Appendix

Table 13: Human inspection results (% of annotations on the valid-question subset,

N{=}172

). Question validity is 90% overall (Type A 88%, Type B 92%).

Item	% Agr.	Cohen’s $\kappa$	Krippendorff’s $\alpha$
Question valid	85	0.22	0.22
Hallucination	79	0.21	0.22
Textual artifact	97	$-0.01$	$-0.01$
Reasoning valid	94	$-0.03$	$-0.03$
Plausibly incorrect	66	0.07	0.09
Better reasoning	50	$-0.09$	$-0.08$

Appendix

Table 14: Inter-annotator agreement over the 96 doubly-annotated pairs.

Figure 7: Spectrogram of ESC-50/3-151081-B-20 (Baby Crying)

Panel of Figure 8 — Figure 8: Acoustic description generated by Qwen2-Audio-Instruct.

Panel of Figure 9 — Figure 9: Acoustic description generated by Gemini-2.5-Pro.

Panel of Figure 10 — Figure 10: Acoustic description generated by our Spectrogram + Qwen3-VL-32B approach.

Panel of Figure 11 — Figure 11: Stage 1 example: semantic description generated by Qwen3-Omni-Captioner.

Panel of Figure 12 — Figure 12: Stage 1 example: acoustic description generated by Qwen3-VL-32B from the spectrogram.

Panel of Figure 13 — Figure 13: Stage 1 example: temporal description aggregating timestamps from Whisper, pyannote, BEATs-based sound event detection, allin1 music structure analysis, and autochord chord recognition.

Panel of Figure 14 — Figure 14: Example question and initial incorrect CoT trace used to compare thought expansion and thought continuation. Correct answer: (D) Drums.

Panel of Figure 15 — Figure 15: Example of thought expansion: $M_{\text{reason}}$ repeatedly restates the audio description and follows a circuitous reasoning pattern.

Panel of Figure 16 — Figure 16: Example of thought continuation: $M_{\text{reason}}$ directly references auditory cues and produces a more focused, refined rationale.

Panel of Figure 17 — Figure 17: Stage 1 prompt used with Qwen3-VL-32B to generate the acoustic description from the spectrogram image.

Panel of Figure 18 — Figure 18: Stage 2 prompt used with $M_{\text{instruct}}$ to generate a more challenging question from the original QA pair and the dense audio description.

Panel of Figure 19 — Figure 19: Stage 2 prompt used with $M_{\text{instruct}}$ to verify the generated question, reasoning, and answer.

Panel of Figure 20 — Figure 20: Stage 3 prompt used with $M_{\text{instruct}}$ to synthesize an incorrect reasoning trace $r^{-}$ from the original $(q,r^{+},a^{+})$ and the dense audio description.

Panel of Figure 21 — Figure 21: Stage 4 prompt used with $M_{\text{reason}}$ to continue an initial reasoning trace $r\oplus c$ with explicit cognitive behaviors.

Panel of Figure 22 — Figure 22: Stage 4 prompt used with $M_{\text{instruct}}$ to verify and (where possible) minimally repair long reasoning traces generated via thought continuation.

Source data
Audio clips	338k
Original QA pairs (AF-Think)	338k
Hardened QA pairs (Stage 2)	316k
Total QA pairs	654k
Reasoning traces
Short correct $r^{+}$	654k

Appendix

Table 15: LLT dataset statistics. Counts are approximate.

\mathcal{J}_{\mathrm{DPO}}(\theta)=\mathbb{E}_{(q,o_{w},o_{l})\sim\mathcal{D}}\left[\log\sigma\!\left(\beta\log\frac{\pi_{\theta}(o_{w}\mid q)}{\pi_{\mathrm{ref}}(o_{w}\mid q)}-\beta\log\frac{\pi_{\theta}(o_{l}\mid q)}{\pi_{\mathrm{ref}}(o_{l}\mid q)}\right)\right].

(1)

Appendix

Table 38

\begin{split}\mathcal{J}_{\mathrm{GRPO}}(\theta)={}&\mathbb{E}_{q\sim P(Q),\,\{o_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\mathrm{old}}}(O\mid q)}\Bigg[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_{i}|}\sum_{t=1}^{|o_{i}|}\bigg(\min\Big[\frac{\pi_{\theta}(o_{i,t}\mid q,o_{i,<t})}{\pi_{\theta_{\mathrm{old}}}(o_{i,t}\mid q,o_{i,<t})}\hat{A}_{i,t},\\ &\mathrm{clip}\Big(\frac{\pi_{\theta}(o_{i,t}\mid q,o_{i,<t})}{\pi_{\theta_{\mathrm{old}}}(o_{i,t}\mid q,o_{i,<t})},1-\epsilon,1+\epsilon\Big)\hat{A}_{i,t}\Big]-\beta\mathbb{D}_{\mathrm{KL}}\!\left[\pi_{\theta}\,\|\,\pi_{\mathrm{ref}}\right]\bigg)\Bigg]\end{split}

(3)

Appendix

Table 39

Audio Reasoning

Audio reasoning studies how AI systems infer events, relationships, intent, and context from sound, beyond recognizing words or matching labels.

See all 60 papers

Audio-DeepThinker: Progressive Reasoning-Aware Reinforcement Learning for High-Quality Chain-of-Thought Emergence in Audio Language Models
Apr 20, 2026Xiang He, Chenxing Li, Jinting Wang +5RL for Language Model Reasoning Audio Reasoning
Tencent AI Lab, Beijing, China · The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China
Listen Then Reason: Perception-Grounded Test-Time Reinforcement Learning for Large Audio-Language Models
Sep 20, 2026Jiaheng Dong, Xiaofeng Yu, Jean Honorio +3Audio Reasoning Audio-Language Models
The University of Melbourne, Australia · University of Auckland, New Zealand · ARC OPTIMA, Australia +1
AudioDER: A Deduplication-Enhanced Reasoning Dataset for Post-Training Large Audio-Language Models
Jun 12, 2026Hui Geng, Yi Su, Han Yin +7Audio Reasoning Audio-Language Models
College of Computer Science and Technology, National University of Defense Technology Changsha, Hunan, China · Korea Advanced Institute of Science and Technology (KAIST) Korea · Shanghai Jiaotong University Shanghai, China

Apr 20, 2026J/K move · Enter open · S save

Audio-DeepThinker: Progressive Reasoning-Aware Reinforcement Learning for High-Quality Chain-of-Thought Emergence in Audio Language Models

Xiang He, Chenxing Li, Jinting Wang, Yan Rong+4

Tencent AI Lab, Beijing, China · The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China

RL for Language Model Reasoning Audio Reasoning Audio Understanding Audio-Language Models Multimodal CoT Reasoning

Open paper AskPDF arXiv

Large Audio-Language Models (LALMs) have made significant progress in audio understanding, yet they primarily operate as perception-and-answer systems without explicit reasoning processes. Existing methods for enhancing audio reasoning rely either on supervised chain-of-thought (CoT) fine-tuning, which is limited by training data quality, or on reinforcement learning (RL) with coarse rewards that do not directly evaluate reasoning quality. As a result, the generated reasoning chains often appear well-structured yet lack specific acoustic grounding. We propose Audio-DeepThinker, a framework built on two core ideas. First, we introduce a hybrid reasoning similarity reward that directly supervises the quality of generated reasoning chains by combining an LLM evaluator assessing logical path alignment, key step coverage, and analytical depth with an embedding similarity component enforcing semantic alignment with reference reasoning chains. Second, we propose a progressive two-stage curriculum that enables high-quality CoT reasoning to emerge through pure RL exploration, without any supervised reasoning fine-tuning, from an instruction-tuned model that possesses no prior chain-of-thought capability. Stage 1 trains on foundational audio QA with the hybrid reward to foster basic reasoning patterns, while Stage 2 shifts to acoustically challenging boundary cases with an LLM-only reward for greater reasoning diversity. Audio-DeepThinker achieves state-of-the-art results on MMAR (74.0%), MMAU-test-mini (78.5%), and MMSU (77.26%), winning 1st Place in the Interspeech 2026 Audio Reasoning Challenge (Single Model Track). Interpretability analyses further reveal that RL training primarily reshapes upper-layer MoE gating mechanisms and that reasoning tokens crystallize progressively in the upper transformer layers, offering mechanistic insights into how audio reasoning emerges through exploration.

Abstract

Audio Reasoning

Explore similar work

Audio-DeepThinker: Progressive Reasoning-Aware Reinforcement Learning for High-Quality Chain-of-Thought Emergence in Audio Language Models

Listen Then Reason: Perception-Grounded Test-Time Reinforcement Learning for Large Audio-Language Models

AudioDER: A Deduplication-Enhanced Reasoning Dataset for Post-Training Large Audio-Language Models