Spoken Question Answering (Spoken QA) presents a challenging cross-modal problem: effectively aligning acoustic queries with textual knowledge while avoiding the latency and error propagation inherent in cascaded ASR-based systems. In this paper, we introduce Attention-guided Evidence Grounding (AEG), a novel end-to-end framework that leverages the internal cross-modal attention of Speech Large Language Models (SpeechLLMs) to explicitly locate and ground key evidence in the model's latent space. To address the diffuse attention distribution in pre-trained models, we propose Learning to Focus on Evidence (LFE), a supervised fine-tuning paradigm that calibrates the model's attention mechanism to distinguish query-relevant segments from irrelevant context. Experiments on SQuAD, HotpotQA, and MuSiQue demonstrate that AEG reduces hallucinations and achieves strong efficiency gains, outperforming large-scale cascaded baselines (Whisper-Large-v3 + Reranker) while reducing inference latency by approximately 62%.
Figures & tables
Fig. 1: A comparison demonstrating the critical role of Learning to Focus on Evidence (LFE). For the audio query ”When was the Governor ended?”, our complete AEG framework successfully grounds the answer in the correct evidence (Doc 3), while both AEG without LFE and baseline methods fail to identify relevant evidence, resulting in incorrect responses.
Fig. 2: Overview of the proposed Attention-guided Evidence Grounding (AEG) method. AEG comprises two components: (1) Learning to Focus on Evidence —a supervised fine-tuning stage that calibrates the SpeechLLM’s attention toward key evidence, and (2) Grounding with Attention —an inference stage that leverages learned attention patterns to highlight and ground key evidence.
Fig. 3: Heatmaps of attention weight evolution across layers (y-axis, 1-32) and training steps (x-axis, 0-190) in Qwen2-Audio. (a) Weights allocated to key evidence. (b) Weights assigned to irrelevant evidence. (c) The difference (diff) between key and irrelevant weights. Green boxes (10-28) highlight the most effective layers.
Fig. 4: Impact of the grounding threshold τ on evidence selection performance ( F1 ), comparing the baseline (base) and LFE (sft) models.
Model
Method
HotpotQA
MuSiQue
SQuAD
GPT-4o Audio
baseline
79.07
51.51
88.49
AEG (w/o LFE)
78.03
53.70
86.94
AEG (with LFE)
79.16
53.99
88.94
Qwen3-Omni Flash
baseline
73.44
44.72
79.51
AEG (w/o LFE)
73.02
44.93
80.36
AEG (with LFE)
74.17
45.72
80.92
TABLE I: EM (%) comparison of methods on context-based Spoken QA tasks across three SpeechLLMs. Best results are bolded.
Dataset
Method
Prec.
Rec.
Hit-Rate
F1
SQuAD
AEG (w/o LFE)
34.21
59.68
84.93
43.49
AEG (with LFE)
88.07
73.32
91.16
80.02
HotpotQA
AEG (w/o LFE)
26.12
58.90
81.86
36.19
AEG (with LFE)
56.99
64.57
94.19
60.55
MuSiQue
AEG (w/o LFE)
13.20
96.37
96.77
23.22
AEG (with LFE)
29.21
58.83
91.06
39.04
TABLE II: Ablation study of the Learning to Focus on Evidence (LFE) module. All metrics are reported in percentages (%).
Configuration
Params
WER
F1
Hit-Rate
Latency
Cascade Systems (ASR + Reranker)
Whisper-Small + bge-reranker-v2-m3
0.8B
4.71
73.85
87.53
405
Whisper-Medium + bge-reranker-v2-m3
1.3B
3.84
74.17
87.96
503
Whisper-Large-v3 + bge-reranker-v2-m3
2.1B
3.18
74.56
88.36
599
Whisper-Small + Qwen3-Reranker-4B
4.2B
4.71
73.56
88.97
420
Whisper-Medium + Qwen3-Reranker-4B
4.8B
3.84
73.88
87.32
518
TABLE III: Performance of Cascade Systems vs. AEG. Metrics are in % (Latency in ms). Best results are bold .
System Prompt
You are a helpful assistant. Your main job is to understand and refer to the doc in the context to accurately answer questions from users.
Reference doc is:
doc_1: doc_1 content
< EVIDENCE >
doc_2: doc_2 content
< /EVIDENCE >
TABLE IV: Prompt Template for Spoken QA
Fig. 5: Impact of the grounding threshold τ on evidence selection performance, including Hit Rate, Precision, Recall, and F1 score, comparing the baseline (base) and LFE (sft) models.
(Partial) Context Passage
Question & Answers
doc_1 Sacro Cuore di Cristo Re is a Roman Catholic church (minor basilica) in Rome, designed between the 1920s and 1930s by Marcello Piacentini. < EVIDENCE > doc_2 The post of Governor of Vatican City (Governatore dello Stato della Città del Vaticano in Italian) was held by Marchese Camillo Serafini from the foundation of the state in 1929 until his death in 1952. No successor was appointed, and the post itself was not mentioned in the Fundamental Law of Vatican City State issued by Pope John Paul II on 26 November 2000, which entered into force on 22 February 2001. < /EVIDENCE > … …doc_19 Bacarra Church is a Roman Catholic church located in the municipality of Bacarra, Ilocos Norte, Philippines under the jurisdiction of the Roman Catholic Diocese of Laoag. The church was founded by the Augustinians, who dedicated it to St. Andrew. doc_20 Start and end dates vary with location and year… …
Question: When was the position of Governor ended in the city that houses the head of Catholicism, and the basilica of the saint holding a knife in the Last Supper? True Answer: 1952. Baseline : 22 February 2001. AEG(w/o LFE) : 22 Feb-ruary 2001. AEG(with LFE) : 1952
doc_1 The Bellas are left in disgrace after the incident with no hopes of winning the competition until Stacie calls the girls to inform she has given birth to a baby girl and named her Bella. This motivates the Bellas to just perform their hearts out without trying to win… … < EVIDENCE > doc_7 The Battle of New Orleans was a series of engagements fought between December 14, 1814 and January 18, 1815, constituting the last major battle of the War of 1812. American combatants, commanded by Major General Andrew Jackson, prevented a much larger British force, commanded by Admiral Alexander Cochrane and General Edward Pakenham, from seizing New Orleans and the vast territory the United States had acquired with the Louisiana Purchase. < /EVIDENCE > doc_8 The Battle of Alto de los Godos was a battle that took place on 25 … … doc_20 The Battle of Alexandria or Battle of Canope … …
Question: Who was the British general in the birthplace of the performer of Here Come the Girls? True Answer: Edward Pakenham. Baseline : William Howe. AEG(w/o LFE) : Based on the given information, the question cannot be answered. AEG(with LFE) : Edward Pakenham.
TABLE V: Examples of how AEG helps LLM, with blue text highlighting the key evidence selected by AEG.
Large audio language models (LALMs) can answer questions about speech, music, and environmental sounds, yet their internal reasoning is largely opaque and difficult to validate. We describe TalTech's solution to the Agent Track of the Interspeech 2026 Audio Reasoning Challenge, in which systems are evaluated on reasoning process quality, specifically the factual accuracy, logical soundness, and completeness of their reasoning chains. Our multi-source ensemble pipeline uses two LALMs that generate independent observations, while a separate text-only reasoning model cross-checks these against outputs from 25 acoustic tools organized into reliability tiers. By grounding every inference step in explicit, reliability-tagged evidence, the system produces dense, verifiable reasoning chains. Our system ranked first in the challenge, outperforming all competing systems by a wide margin in challenge's reasoning quality metric.
Large audio-language models (LALMs) can generate reasoning chains for their predictions, but it remains unclear whether these reasoning chains remain grounded in the input audio. In this paper, we propose an RL-based strategy that grounds the reasoning outputs of LALMs with explicit timestamp annotations referring to relevant segments of the audio signal. Our analysis shows that timestamp grounding leads the model to attend more strongly to audio tokens during reasoning generation. Experiments on four speech-based benchmark datasets demonstrate that our approach improves performance compared to both zero-shot reasoning and fine-tuning without timestamp grounding. Additionally, grounding amplifies desirable reasoning behaviors, such as region exploration, audiology verification, and consistency, underscoring the importance of grounding mechanisms for faithful multimodal reasoning.
Jihoon Jeong, Pooneh Mousavi, Mirco Ravanelli +1
Mila-Quebec AI Institute · Universit´e Laval · Concordia University
Multi-turn, multi-audio temporal question answering requires models to track target events across follow-up questions, recording switches, and historical references, recovering complete instances and their boundaries for temporal calculation and comparison. We propose TEMA, which connects event perception with evidence-based answering through Route, specifying the audio scope, and Span, describing all relevant intervals as conditional audio captions. We construct TEMA-Dialog with 40,704 dialogs and per-turn evidence and answer supervision, and TEMA-Bench for joint evaluation of evidence and final answers. Training combines temporal grounding initialization, full-dialog supervised fine-tuning, and completeness-first Span-only GRPO. Experiments on Qwen2.5-Omni and AF-Next show improved temporal question answering, particularly event localization and cross-audio comparison. Reinforcement learning applied solely to evidence further improves interval recovery and answer accuracy.
Kaidi Yang, Hualei Wang, Zhaohui Wang +3
Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China. · University of Chinese Academy of Sciences, Beijing, China.