cs.CLMar 17, 2026
SaveAttention-guided Evidence Grounding for Spoken Question Answering
Organizations: Ant Group Hangzhou, China
Abstract
Spoken Question Answering (Spoken QA) presents a challenging cross-modal problem: effectively aligning acoustic queries with textual knowledge while avoiding the latency and error propagation inherent in cascaded ASR-based systems. In this paper, we introduce Attention-guided Evidence Grounding (AEG), a novel end-to-end framework that leverages the internal cross-modal attention of Speech Large Language Models (SpeechLLMs) to explicitly locate and ground key evidence in the model's latent space. To address the diffuse attention distribution in pre-trained models, we propose Learning to Focus on Evidence (LFE), a supervised fine-tuning paradigm that calibrates the model's attention mechanism to distinguish query-relevant segments from irrelevant context. Experiments on SQuAD, HotpotQA, and MuSiQue demonstrate that AEG reduces hallucinations and achieves strong efficiency gains, outperforming large-scale cascaded baselines (Whisper-Large-v3 + Reranker) while reducing inference latency by approximately 62%.
Figures & tables
Fig. 1: A comparison demonstrating the critical role of Learning to Focus on Evidence (LFE). For the audio query ”When was the Governor ended?”, our complete AEG framework successfully grounds the answer in the correct evidence (Doc 3), while both AEG without LFE and baseline methods fail to identify relevant evidence, resulting in incorrect responses.
Fig. 2: Overview of the proposed Attention-guided Evidence Grounding (AEG) method. AEG comprises two components: (1) Learning to Focus on Evidence —a supervised fine-tuning stage that calibrates the SpeechLLM’s attention toward key evidence, and (2) Grounding with Attention —an inference stage that leverages learned attention patterns to highlight and ground key evidence.
Fig. 3: Heatmaps of attention weight evolution across layers (y-axis, 1-32) and training steps (x-axis, 0-190) in Qwen2-Audio. (a) Weights allocated to key evidence. (b) Weights assigned to irrelevant evidence. (c) The difference (diff) between key and irrelevant weights. Green boxes (10-28) highlight the most effective layers.
Fig. 4: Impact of the grounding threshold on evidence selection performance ( ), comparing the baseline (base) and LFE (sft) models.
| Model | Method | HotpotQA | MuSiQue | SQuAD |
|---|---|---|---|---|
| GPT-4o Audio | baseline | 79.07 | 51.51 | 88.49 |
| AEG (w/o LFE) | 78.03 | 53.70 | 86.94 | |
| AEG (with LFE) | 79.16 | 53.99 | 88.94 | |
| Qwen3-Omni Flash | baseline | 73.44 | 44.72 | 79.51 |
| AEG (w/o LFE) | 73.02 | 44.93 | 80.36 | |
| AEG (with LFE) | 74.17 | 45.72 | 80.92 |
TABLE I: EM (%) comparison of methods on context-based Spoken QA tasks across three SpeechLLMs. Best results are bolded.
| Dataset | Method | Prec. | Rec. | Hit-Rate | |
|---|---|---|---|---|---|
| SQuAD | AEG (w/o LFE) | 34.21 | 59.68 | 84.93 | 43.49 |
| AEG (with LFE) | 88.07 | 73.32 | 91.16 | 80.02 | |
| HotpotQA | AEG (w/o LFE) | 26.12 | 58.90 | 81.86 | 36.19 |
| AEG (with LFE) | 56.99 | 64.57 | 94.19 | 60.55 | |
| MuSiQue | AEG (w/o LFE) | 13.20 | 96.37 | 96.77 | 23.22 |
| AEG (with LFE) | 29.21 | 58.83 | 91.06 | 39.04 |
TABLE II: Ablation study of the Learning to Focus on Evidence (LFE) module. All metrics are reported in percentages (%).
| Configuration | Params | WER | Hit-Rate | Latency | |
| Cascade Systems (ASR + Reranker) | |||||
| Whisper-Small + bge-reranker-v2-m3 | 0.8B | 4.71 | 73.85 | 87.53 | 405 |
| Whisper-Medium + bge-reranker-v2-m3 | 1.3B | 3.84 | 74.17 | 87.96 | 503 |
| Whisper-Large-v3 + bge-reranker-v2-m3 | 2.1B | 3.18 | 74.56 | 88.36 | 599 |
| Whisper-Small + Qwen3-Reranker-4B | 4.2B | 4.71 | 73.56 | 88.97 | 420 |
| Whisper-Medium + Qwen3-Reranker-4B | 4.8B | 3.84 | 73.88 | 87.32 | 518 |
TABLE III: Performance of Cascade Systems vs. AEG. Metrics are in % (Latency in ms). Best results are bold .
| System Prompt |
|---|
| You are a helpful assistant. Your main job is to understand and refer to the doc in the context to accurately answer questions from users. |
| Reference doc is: |
| doc_1: doc_1 content |
| EVIDENCE |
| doc_2: doc_2 content |
| /EVIDENCE |
TABLE IV: Prompt Template for Spoken QA
Fig. 5: Impact of the grounding threshold on evidence selection performance, including Hit Rate, Precision, Recall, and score, comparing the baseline (base) and LFE (sft) models.
| (Partial) Context Passage | Question & Answers |
|---|---|
| doc_1 Sacro Cuore di Cristo Re is a Roman Catholic church (minor basilica) in Rome, designed between the 1920s and 1930s by Marcello Piacentini. EVIDENCE doc_2 The post of Governor of Vatican City (Governatore dello Stato della Città del Vaticano in Italian) was held by Marchese Camillo Serafini from the foundation of the state in 1929 until his death in 1952. No successor was appointed, and the post itself was not mentioned in the Fundamental Law of Vatican City State issued by Pope John Paul II on 26 November 2000, which entered into force on 22 February 2001. /EVIDENCE … …doc_19 Bacarra Church is a Roman Catholic church located in the municipality of Bacarra, Ilocos Norte, Philippines under the jurisdiction of the Roman Catholic Diocese of Laoag. The church was founded by the Augustinians, who dedicated it to St. Andrew. doc_20 Start and end dates vary with location and year… … | Question: When was the position of Governor ended in the city that houses the head of Catholicism, and the basilica of the saint holding a knife in the Last Supper? True Answer: 1952. Baseline : 22 February 2001. AEG(w/o LFE) : 22 Feb-ruary 2001. AEG(with LFE) : 1952 |
| doc_1 The Bellas are left in disgrace after the incident with no hopes of winning the competition until Stacie calls the girls to inform she has given birth to a baby girl and named her Bella. This motivates the Bellas to just perform their hearts out without trying to win… … EVIDENCE doc_7 The Battle of New Orleans was a series of engagements fought between December 14, 1814 and January 18, 1815, constituting the last major battle of the War of 1812. American combatants, commanded by Major General Andrew Jackson, prevented a much larger British force, commanded by Admiral Alexander Cochrane and General Edward Pakenham, from seizing New Orleans and the vast territory the United States had acquired with the Louisiana Purchase. /EVIDENCE doc_8 The Battle of Alto de los Godos was a battle that took place on 25 … … doc_20 The Battle of Alexandria or Battle of Canope … … | Question: Who was the British general in the birthplace of the performer of Here Come the Girls? True Answer: Edward Pakenham. Baseline : William Howe. AEG(w/o LFE) : Based on the given information, the question cannot be answered. AEG(with LFE) : Edward Pakenham. |
TABLE V: Examples of how AEG helps LLM, with blue text highlighting the key evidence selected by AEG.
More in this topic
Spoken QA
Spoken question answering (Spoken QA) builds systems that understand spoken questions and produce accurate, evidence-grounded answers.
See all 12 papers