Don't Just Listen, Try Planning: Graph-based Retrieval-Generation Agent for Long-form Audio Meeting Understanding

Quanwei Tang; Dong Zhang; Shoushan Li; Guodong Zhou

cs.SDAug 25, 2026

Don't Just Listen, Try Planning: Graph-based Retrieval-Generation Agent for Long-form Audio Meeting Understanding

Authors: Quanwei Tang, Dong Zhang, Shoushan Li, Guodong Zhou

Organizations: School of Computer Science & Technology, NLP Lab, Soochow University, China · Jiangsu Key Lab of Language Computing, Suzhou.

Abstract

While long-form audio meeting understanding (LAMU) is garnering growing attention, task-specific question answering (QA) datasets remain scarce. Existing speech QA paradigms and state-of-the-art Speech LLMs suffer from acoustic information loss and poor long-term context memory. To address these issues, we construct the LongAudioQA dataset and propose the GRGA model, which models heterogeneous audio features into a multi-dimensional graph and leverages agent planning for retrieval and answer generation.

Figures & tables

Figure 1: Existing Speech LLMs and our GRGA for long-form audio QA. By indexing conversations across Semantic , Speaker , and Timestamp dimensions, our model enables precise reasoning.

Category	Core Competency	Subtasks & Example Queries
Factual	Explicit Retrieval	$\bullet$ Entity Retrieval: “Who mentioned the project code ‘Alpha’?” $\bullet$ Attribute Association: “What is the budget proposed by the Manager?” $\bullet$ Keyword Locating: Matching specific text spans.
Inferential	Multi-hop Reasoning	$\bullet$ Causal Inference: Linking a problem raised early with the final decision. $\bullet$ Coreference Resolution: Identifying what “that plan” refers to. $\bullet$ Stance Analysis: Tracking how a speaker’s attitude evolves.
Temporal	Time Awareness	$\bullet$ Absolute Localization: “What topic was discussed at the 30-min mark?” $\bullet$ Relative Sequencing: “What was discussed after ‘market research’?” $\bullet$ Frequency Stats: Counting term occurrences in a window.
Summarization	Aggregation	$\bullet$ Topic Summarization: Synthesizing consensus on “backend architecture”. $\bullet$ Speaker Profiling: Summarizing a participant’s main contributions.
Acoustic-Aware	Multi-modal Alignment	$\bullet$ Emotion & Intensity: “Who seemed most agitated when discussing the budget?” $\bullet$ Cross-modal Localization & Causal: “What is the reason for the person’s sudden loud voice at the 15-minute mark?”

Table 1: Taxonomy of Meeting Questions in our LongAudioQA. Categories are distinguished by background colors: Factual , Inferential , Temporal , Summarization , and Acoustic-Aware . The Acoustic-Aware category uniquely requires grounding textual semantics with paralinguistic acoustic signals.

Dataset	Dur (h)	Avg. Dur (s)	Avg. Turns	# Q&A	# Dial.
AliMeeting	14.91	1,935	815	3,013	28
AMI	18.24	1,930	757	3,243	34
DailyTalk	21.59	610	186	10,200	128

Table 2: Statistical information among DailyTalk, AMI, and AliMeeting datasets. Dur : Duration, Avg : Average.

Figure 2: Question Distribution Across Corpora

Figure 3: The overall architecture of our graph-based retrieval-generation agent GRGA . The framework models long-form audio as a Multi-dimensional Graph (bottom-left) to capture semantic, temporal, and speaker dependencies. Our GRGA Planning Process consists of: Query Decomposition , Planning , Execution , Synthesis , and Reflection .

Method	AliMeeting					AMI Meeting					DailyTalk
Method	Fact.	Infer.	Temp.	Summ.	Acou.	Fact.	Infer.	Temp.	Summ.	Acou.	Fact.	Infer.	Temp.	Summ.	Acou.
Speech as Context
Qwen3-Omni	29.31	27.33	15.81	19.06	16.50	20.96	26.38	15.12	16.67	12.15	34.26	38.43	2.43	2.11	9.39
Audio Flamingo 3	35.70	35.23	13.19	18.09	32.89	16.04	27.94	12.62	10.24	17.06	65.13	61.31	3.83	2.44	40.09
MiMo-Audio	54.50	54.50	16.92	39.91	25.15	52.75	64.74	35.34	38.30	17.17	81.57	78.65	5.58	3.46	31.44
Transcription as Context

Table 3: Main results on accuracy (%). We compare our proposed method against End-to-End Speech LLM and standard RAG baselines across three datasets. Bold indicates the best performance. Fact. : Factual, Infer. : Inferential, Temp. : Temporal, Summ. : Summarization, Acou. : Acoustic reasoning.

Dataset	WER ( $\downarrow$ )	DER ( $\downarrow$ )
Dataset	WER ( $\downarrow$ )	Collar = 0 s	Collar = 0.25 s
DailyTalk	0.85	6.42	3.41
AliMeeting-far	20.38	16.80	13.10
AMI-sdm	18.84	17.60	14.90

Table 4: Error analysis on benchmark datasets. We report Word Error Rate (WER) and Diarization Error Rate (DER) under strict (

0\,\text{s}

) and standard (

0.25\,\text{s}

) collar (tolerance ) settings. All metrics are reported in percentage (%), and

\downarrow

denotes that lower values are better.

Figure 4: Human evaluation results. Our method shows significant improvements, particularly in Groundedness across complex meeting scenarios.

Figure 5: Citation Precision-Recall Trade-off . Results are reported in % with a

\pm 2

s tolerance.

Ablation Settings	Overall		AMI
Ablation Settings	Avg.	$\Delta$	Fact.	Infer.	Temp.	Summ.	Acou.
Ours (Full)	49.49	-	59.46	65.29	38.56	48.46	35.68
Validation of Query Decomposer
w/o Query Planner	46.39	-3.10	57.63	59.35	36.79	45.73	32.43
Validation of Query Planner
w/o Query Planner	44.68	-4.80	56.12	56.35	36.41	43.18	31.37

Table 5: Ablation study on the AMI dataset. We report the accuracy (%) drop when specific components are removed.

\Delta

: Performance degradation compared to the full framework.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Ablation Settings	Overall		AMI
Ablation Settings	Avg.	$\Delta$	Fact.	Infer.	Temp.	Summ.	Acou.
Ours (Full)	49.49	-	59.46	65.29	38.56	48.46	35.68
Validation of Query Decomposer
w/o Query Planner	46.39	-3.10	57.63	59.35	36.79	45.73	32.43
Validation of Query Planner
w/o Query Planner	44.68	-4.80	56.12	56.35	36.41	43.18	31.37

Appendix

Table 6: Ablation study on the AMI dataset. We report the accuracy (%) drop when specific components are removed.

\Delta

: Performance degradation compared to the full framework.

Method	DailyTalk		AMI		AliMeeting
Method	P	R	P	R	P	R
Qwen3-Omni	32.6	24.6	25.7	27.3	29.3	28.2
+ Text RAG	67.3	69.1	56.9	65.2	72.4	68.3
+ Audio RAG	61.6	63.6	45.2	42.7	57.3	49.6
+ Ours	84.2	77.2	76.2	69.7	79.1	74.1

Appendix

Table 7: Citation accuracy comparison. Results are reported in % with a

\pm 2

s tolerance. P: Precision, R: Recall.

Figure 6: Average reasoning steps across different question types. The agent exhibits adaptive computation : solving simple Factual queries requires minimal steps, while complex Inferential and Temporal queries trigger deeper reasoning chains. The stacked colors illustrate the distribution of tool usage, showing heavy reliance on Graph Traversal for multi-hop reasoning.

Figure 7: Noise Sensitivity Analysis. QA Accuracy comparison under varying upstream error rates. The rightmost panel highlights the widening performance gap ( $\Delta$ Accuracy), showing that GRGA’s advantage grows in noisier environments.

Dataset	Method	Human Ratings (1-5 Scale)			Avg.	IAA
Dataset	Method	Corr.	Grd.	Coh.
AMI	Qwen3-Omni	1.79	2.12	3.76	2.56	0.78
AMI	Ours	2.25	3.25	4.35	3.28	0.78
AllMeeting	Qwen3-Omni	1.93	2.24	4.32	2.83	0.75
AllMeeting	Ours	2.36	3.51	4.37	3.41	0.75
DailyTalk	Qwen3-Omni	3.43	3.72	4.12	3.76	0.82

Appendix

Table 8: Human evaluation results across different datasets. Models are assessed on three dimensions: Correctness (Corr.), Groundedness (Grd.), and Coherence (Coh.) using a 1-5 scale. We also report average scores (Avg.) and Inter-Annotator Agreement (IAA).

Stage	Module / Action	Execution Details & Internal State
User Query: “What is the team’s general attitude to the project deadline and workload?”
Phase 1	Intent Analysis	Entities: [‘team’, ‘project deadline’, ‘workload’] Concept: [‘general attitude’]
	Plan Execution	1. Hybrid Search: query=“team project deadline workload” ( $\rightarrow$ 10 hits) 2. Traverse Relations: Expand context ( $\rightarrow$ 64 nodes found).
	Draft Answer	“The team generally views the deadline as tight… members feeling urgency.”
		(Generic summary, lacks specific details)
	Reflection (Critic)	Assessment: ✗ Confidence: 2/5 Reasoning: “Supported is False. Need reverse confirmation for ‘tight’ and ‘work-intensive’ claims.” $\hookrightarrow$ Decision: Trigger Re-Plan .

Appendix

Table 9: Case Study on Self-Correction. Initially, the model retrieved broad context but generated a low-confidence (2/5), generic answer. The Reflection module detected the lack of specific evidence and triggered a re-planning step focusing on “focused effort.” This allowed the model to retrieve concrete complaints (e.g., “six days,” “hassle,” “superficially”), resulting in a highly grounded (5/5) final response.

Tool	Time Complexity	Description
Filter	$O(\|V\|+\|E\|)$	Metadata filter
Search	$O(\|V\|\cdot d_{\text{embed}}+k\log k)$	Vector retrieval + sorting
GraphTraversal	$O(\bar{d}\cdot\text{depth})$	BFS ( $\bar{d}$ : avg degree)
Temporal	$O(1)$	Indexed temporal query
AudioAccess	$O(1)$	Get audio clip

Appendix

Table 10: Time Complexity of Graph Operations

Category	Tool Signature	Description & Purpose
Retrieval	keyword_search(query)	BM25-based search for precise entity/term lookup.
	semantic_search(query)	Dense vector retrieval for abstract semantic concepts.
	hybrid_search(query, $\alpha=0.6$ )	Weighted combination of keyword and semantic scores to optimize recall.
Filtering	filte_time_range( $t_{start},t_{end}$ )	Retrieves utterances strictly within a specified time window.
Filtering	filte_speaker(nodes, spk_id)	Filters a set of candidate nodes by speaker identity.
Traversal	traverse_relations(nodes, depth=k)	Walks along graph edges (e.g., Next , Reply-To ) to trace dialogue threads for multi-hop reasoning.

Appendix

Table 11: The definitions of the atomic tools in our Action Space (

\mathcal{A}

). The Query Planner invokes these tools to interact with the meeting graph and retrieve evidence.

Audio QA

Audio question answering uses spoken, musical, and environmental sounds to answer questions grounded in acoustic evidence.

See all 45 papers

Event-Grounded Question Answering over Long Audio via Structured Retrieval
Feb 16, 2026Kartik Hegde, Arvind Krishna Sridhar, Naveen Vakada +2Multimodal RAG Audio QA
Qualcomm Technologies, Inc., India · Qualcomm Technologies, Inc., USA
EChO-Agent: Evidence Chain Orchestration Agent for Audio Reasoning
Jun 13, 2026Siyuan Zhang, Jian Zong, Junyu Wang +7Audio QA Tool-Augmented Language Model Agents
School of Artificial Intelligence, Tianjin University, Tianjin, China
Audio-Mind: An Auditable Agentic Framework for Audio Understanding
May 27, 2026Yucheng Wang, Jing Peng, Hanqi Li +6Audio QA Audio Reasoning
School of Intelligence Science and Technology, Nanjing University, China · X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University, China · School of Automation Science and Engineering, Xi’an Jiaotong University, China +1

Feb 16, 2026J/K move · Enter open · S save

Event-Grounded Question Answering over Long Audio via Structured Retrieval

Kartik Hegde, Arvind Krishna Sridhar, Naveen Vakada, Yinyi Guo+1

Qualcomm Technologies, Inc., India · Qualcomm Technologies, Inc., USA

Multimodal RAG Audio QA Audio Understanding Audio-Language Models Temporal Audio Grounding Temporal QA

Open paper AskPDF arXiv

Answering natural-language questions over multi-hour audio requires reliable event recognition, temporal grounding, and efficient retrieval. We present LA-RAG (Long Audio Retrieval-Augmented Generation), a structured framework that converts audio into timestamped event records, stores them in an event database, and answers questions using intent-aware retrieval and LLM-based generation. LA-RAG supports two deployment settings: offline grounding mode, in which long recordings are pre-indexed for low-latency querying, and inference-time grounding mode, which performs query-conditioned grounding over shorter, open-ended clips. We evaluate LA-RAG on controlled 24-hour Home-IoT and Industrial-IoT benchmarks and on CASTELLA-QA, derived from real-world audio recordings. The results demonstrate effective long-audio question answering with low query-time latency after grounding and indexing. They also reveal a substantial gap between event detection and temporal localization in current large audio-language models, while showing that explicit timestamped grounding consistently improves temporal reasoning. These findings establish structured grounding and retrieval as a practical complement to generative audio-language models for deployment-oriented long-audio understanding.

Abstract

Audio QA

Explore similar work

Event-Grounded Question Answering over Long Audio via Structured Retrieval

EChO-Agent: Evidence Chain Orchestration Agent for Audio Reasoning

Audio-Mind: An Auditable Agentic Framework for Audio Understanding