Listen First, Then Answer: Timestamp-Grounded Speech Reasoning

Jihoon Jeong; Pooneh Mousavi; Mirco Ravanelli; Cem Subakan

cs.SDMar 19, 2026

Listen First, Then Answer: Timestamp-Grounded Speech Reasoning

Authors: Jihoon Jeong, Pooneh Mousavi, Mirco Ravanelli, Cem Subakan

Organizations: Mila-Quebec AI Institute · Universit´e Laval · Concordia University

Abstract

Large audio-language models (LALMs) can generate reasoning chains for their predictions, but it remains unclear whether these reasoning chains remain grounded in the input audio. In this paper, we propose an RL-based strategy that grounds the reasoning outputs of LALMs with explicit timestamp annotations referring to relevant segments of the audio signal. Our analysis shows that timestamp grounding leads the model to attend more strongly to audio tokens during reasoning generation. Experiments on four speech-based benchmark datasets demonstrate that our approach improves performance compared to both zero-shot reasoning and fine-tuning without timestamp grounding. Additionally, grounding amplifies desirable reasoning behaviors, such as region exploration, audiology verification, and consistency, underscoring the importance of grounding mechanisms for faithful multimodal reasoning.

Figures & tables

Figure 1: Grounded timestamp reasoning enables interpretable and accurate answers. Our method decomposes the task into a sequence of reasoning steps explicitly grounded in corresponding audio regions. In contrast, vanilla reasoning and zero-shot tend to produce ungrounded and incorrect responses.

Figure 2: Overview of the proposed framework. Stage 1 learns temporal localization through supervised timestamp alignment. Stage 2 performs GRPO-based optimization to encourage answers justified by concise timestamp-grounded reasoning.

Figure 3: Semantic attention distribution of a baseline LALM across input categories. It aggregates token-level attention by semantic role, highlighting under-attention to audio features.

Model	STA	IoU $\uparrow$	F1 $\uparrow$	IoU $\geq$ 0.7 (%) $\uparrow$
Qwen-2.5-Omni [ 40 ]	✗	0.2324	0.3074	11.32
Qwen-2.5-Omni [ 40 ]	$\checkmark$	0.7189	0.7675	69.79
AudioFlamingo3 [ 17 ]	✗	0.2838	0.3592	11.45
AudioFlamingo3 [ 17 ]	$\checkmark$	0.5439	0.6012	47.95
Gemini-2.5-Flash [ 44 ]	–	0.7356	0.8141	71.08

Table 1: Effect of Sentence-level Timestamp Alignment.

Figure 4: Layer-wise audio attention during timestamp generation. STA model allocates higher attention to audio tokens across transformer layers compared to the vanilla model, indicating stronger reliance on acoustic evidence.

Methods	Size	MMAU-mini	MMAR	AIR-Bench			MELD
Methods	Size	Speech (%)	Speech (%)	SER (%)	SNV (%)	SIC (%)	(%)
Proprietary Models
Gemini 2.5 Flash [ 44 ]	–	75.08	72.11	56.4	68.5	88.6	61.5
GPT-4o Audio [ 48 ]	–	66.67	70.41	51.2	61.6	89.3	62.5
Open-source Models
SALMONN [ 3 ]	7B	26.43	24.35	29.9	34.3	42.3	37.2

Table 2: Performance comparison on MMAU-mini-Speech, MMAR-Speech, AIR-Bench, and MELD.

Model	Regions Explored	Audiology Verify $\uparrow$	Consistency $\uparrow$	Acc. $\uparrow$
Zero-Shot
Standard CoT	1.3	0.27	0.72	69.4
RL-tuned
w/o compaction	3.6	0.48	0.78	72.8
w/o grounding	0.5	0.22	0.69	74.2
Ours	1.8	0.56	0.83	74.5

Table 3: Average visual reasoning behaviors per example on MMAU-mini-Speech.

Model	Vanilla	Only STA	Reasoning SFT	Ours
Audio Attention (All)	0.0748	0.0936	0.0975	0.1030
Reasoning Attention	0.0790	0.0952	0.1032	0.1138
Answer Attention	0.0421	0.0690	0.0714	0.0680

Table 4: Average audio attention behaviors when generating reasoning on MMAU-mini-Speech.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Layer-wise attention aggregation across different query types. Unlike Fig. 3 , the values shown here correspond to summed attention scores within each semantic block rather than per-token normalized values. The overall attention profiles remain highly consistent across layers, indicating that LALMs follow a similar internal processing pattern for audio-conditioned reasoning tasks.

Figure 6: Block-level attention aggregation at Layer 14 during timestamp generation. Attention weights from each output token are aggregated across four semantic input blocks. Tokens corresponding to timestamp prediction (highlighted in a red box) allocate increased attention to the audio block, indicating that temporal reasoning relies on acoustic evidence.

Panel of Figure 7 — Figure 7: Layer-wise attention aggregation during timestamp generation across two LALM architectures. (a) Audio Flamingo 3 and (b) Qwen2.5-Omni. Despite architectural differences, both models exhibit a similar shift toward increased reliance on acoustic representations during timestamp generation, supporting the tendency discussed in the main text.

Table 12

Figure 8: Comparison of reasoning traces between vanilla GRPO and our method on the MMAU dataset - 1.

Figure 9: Comparison of reasoning traces between vanilla GRPO and our method on the MMAU dataset - 2.

Audio-Language Models

Audio-language models connect audio and natural language for understanding, retrieval, dialogue, speech generation, and music or sound analysis.

See all 303 papers

Audio-DeepThinker: Progressive Reasoning-Aware Reinforcement Learning for High-Quality Chain-of-Thought Emergence in Audio Language Models
Apr 20, 2026Xiang He, Chenxing Li, Jinting Wang +5RL for Language Model Reasoning Audio Reasoning
Tencent AI Lab, Beijing, China · The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China
ALARM: Audio-Language Alignment for Reasoning Models
Mar 10, 2026Petr Grinberg, Hassan ShahmohammadiAudio-Language Models Audio Reasoning
EPFL, Switzerland · Sony Europe Ltd., Germany
TEMPO: Temporally-grounded Multi-task Post-training for Large Audio-Language Models
Aug 30, 2026Apoorva Kulkarni, Kaousheik Jayakumar, Sreyan Ghosh +3Audio Understanding Audio-Language Models
University of Maryland, College Park, USA · CNH Industrial India

Apr 20, 2026J/K move · Enter open · S save

Audio-DeepThinker: Progressive Reasoning-Aware Reinforcement Learning for High-Quality Chain-of-Thought Emergence in Audio Language Models

Xiang He, Chenxing Li, Jinting Wang, Yan Rong+4

Tencent AI Lab, Beijing, China · The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China

RL for Language Model Reasoning Audio Reasoning Audio Understanding Audio-Language Models Multimodal CoT Reasoning

Open paper AskPDF arXiv

Large Audio-Language Models (LALMs) have made significant progress in audio understanding, yet they primarily operate as perception-and-answer systems without explicit reasoning processes. Existing methods for enhancing audio reasoning rely either on supervised chain-of-thought (CoT) fine-tuning, which is limited by training data quality, or on reinforcement learning (RL) with coarse rewards that do not directly evaluate reasoning quality. As a result, the generated reasoning chains often appear well-structured yet lack specific acoustic grounding. We propose Audio-DeepThinker, a framework built on two core ideas. First, we introduce a hybrid reasoning similarity reward that directly supervises the quality of generated reasoning chains by combining an LLM evaluator assessing logical path alignment, key step coverage, and analytical depth with an embedding similarity component enforcing semantic alignment with reference reasoning chains. Second, we propose a progressive two-stage curriculum that enables high-quality CoT reasoning to emerge through pure RL exploration, without any supervised reasoning fine-tuning, from an instruction-tuned model that possesses no prior chain-of-thought capability. Stage 1 trains on foundational audio QA with the hybrid reward to foster basic reasoning patterns, while Stage 2 shifts to acoustically challenging boundary cases with an LLM-only reward for greater reasoning diversity. Audio-DeepThinker achieves state-of-the-art results on MMAR (74.0%), MMAU-test-mini (78.5%), and MMSU (77.26%), winning 1st Place in the Interspeech 2026 Audio Reasoning Challenge (Single Model Track). Interpretability analyses further reveal that RL training primarily reshapes upper-layer MoE gating mechanisms and that reasoning tokens crystallize progressively in the upper transformer layers, offering mechanistic insights into how audio reasoning emerges through exploration.

Abstract

Audio-Language Models

Explore similar work

Audio-DeepThinker: Progressive Reasoning-Aware Reinforcement Learning for High-Quality Chain-of-Thought Emergence in Audio Language Models

ALARM: Audio-Language Alignment for Reasoning Models

TEMPO: Temporally-grounded Multi-task Post-training for Large Audio-Language Models