cs.MANov 6, 2026
SaveReSUM: Reasoning-Driven Multi-Agent Framework for Query-Focused Summarization
Organizations: Indian Institute of Technology Patna, India · Indian Institute of Technology Jodhpur, India
Abstract
LLMs excel at language tasks, but long-meeting QFS remains difficult due to complex speaker dynamics, sparse key information, and limited data. Meetings involve disfluencies, long transcripts and complex discourse. QFS have high risk of information omission, hallucination, even for advanced models such as GPT-4-mini and Gemini. Existing end-to-end methods compress retrieval, evidence extraction, and summary generation into a single pass, failing to handle the complex reasoning required for targeted meeting summaries. We propose ReSUM, a modular multi-LLM framework coordinated by a planner that orchestrates retrieval and summarization agents. By distilling teacher reasoning into query- and plan-conditioned retrievers and summarizers, ReSUM enhances long-context robustness, minimizes irrelevant exploration, and generates more faithful and comprehensive query-focused summaries. Experiments show consistent gains on query-focused meeting summarization benchmarks.
Figures & tables
Figure 1: Single Agent that fails to capture query-specific context, our Multi-Agent pipeline divides tasks between specialized agents. By utilizing reasoning-driven knowledge distillation, the planning and retrieval agents ensure precise content selection, while the critique loop provides feedback-driven refinement for a highly accurate, query-focused summary.
Figure 2: Architectural Overview of ReSUM , where the upper part illustrates the training stage with teacher–student distillation and reasoning supervision for the Retriever and Summarizer, and the lower part depicts the multi-agent pipeline coordinating Planner, Retriever, Summarizer, and Critique agents for iterative, query-focused meeting summarization.
| Instructions |
| You are a Planner agent for long meeting QFS. Given user query Q, meeting transcript M, and current state S, construct a structured, dependency-aware execution plan for agents. Given agents description: Retriever (select fine-grained, query-relevant utterance spans from planner-selected discourse regions, focusing on evidence-bearing content), Summarizer (generate a query-focused summary grounded strictly in retrieved spans without hallucination), and Critique (evaluate planning quality, retrieval relevance, and summary faithfulness). Generate a step-by-step plan for each agent, to generate a query-focused summary. |
Table 1: Planning Agent: Role definition and multi-agent coordination instructions.
| Prompt |
|---|
| Given (i) a meeting transcript with turn-level identifiers, (ii) a user query, and (iii) a relevant span defined by its start and end turn IDs along with the corresponding span text, generate a structured rationale explaining how the span addresses the query. The rationale should: (1) identify key evidence lines within the span, (2) explain their relevance to the query, (3) articulate any implicit reasoning linking the span to the query, and (4) rely exclusively on transcript content within the specified turn range. The output must present a coherent, step-by-step explanation grounded in the provided span. |
| Rationale |
| The Project Manager introduces the prototype by outlining the meeting agenda and emphasizing trend-driven design elements, such as a fashionable and fruit-inspired look. The User Interface then describes the prototype’s banana-shaped design, directly aligning with these trends. Together, these details explain how they introduced the prototype in response to the query. |
Table 2: Retriever Agent: System prompt and generated rationale for evidence extraction.
| Prompt |
|---|
| Given (i) a user query, (ii) a relevant transcript span, and (iii) a query-focused summary, generate a structured rationale explaining how the span supports the summary with respect to the query. The rationale should: (1) Identify key evidence lines from the span, (2) Explain how each summary point addresses the query, (3) Clarify implicit reasoning linking the span to the summary, and (4) Rely only on information within the provided span. Produce a clear, step-by-step explanation grounded strictly in the given content. |
| Rationale |
| The team introduced a fashion-inspired prototype, described as bright yellow and banana-shaped. They emphasized simplicity, ergonomics, and usability for both left- and right-handed users, along with advanced technology. The discussion about size and design choices supports the summary’s focus on creativity, functionality, and ease of use. |
Table 3: Summarizer Agent: Instructional prompt and evidence-grounded rationale.
| Instructions |
|---|
| You are a Critique agent in a multi-agent QFS system. Evaluate outputs from the Planner, Retriever, and Summarizer, and provide actionable feedback. 1. Planner (Planning Alignment) |
| (i) Are selected topics appropriate for the query and meeting context? (ii) Were necessary agents correctly invoked and assigned roles? (iii) Are instructions clear and query-aligned? 2. Retriever (Retrieval Quality) |
| (i) Are retrieved turn-ID ranges directly relevant to the query? (ii) Are retrieved spans evidence-bearing rather than loosely related? (iii) Is important query-relevant content missing? 3. Summarizer (Summary Quality) |
| (i) Is the summary grounded only in retrieved evidence? (ii) Does it fully and correctly answer the query? (iii) Is it faithful, coherent, and free of hallucinations? Conclude whether the overall output is acceptable or requires refinement. |
Table 4: Critique Agent: Evaluation and refinement instructions.
| Model | QMSum | SQuALITY | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| R1 | R2 | RL | BS | Mover | COMET | chrF | R1 | R2 | RL | BS | Mover | COMET | chrF | |
| Zero-shot | ||||||||||||||
| LLaMA3-3B | 0.27 | 0.06 | 0.17 | 0.15 | 0.53 | 0.66 | 34.01 | 0.32 | 0.06 | 0.17 | 0.08 | 0.53 | 0.67 | 35.52 |
| Gemma3-4B | 0.20 | 0.04 | 0.13 | 0.06 | 0.52 | 0.65 | 30.39 | 0.32 | 0.06 | 0.17 | 0.04 | 0.53 | 0.66 | 37.38 |
| DeepSeek-1.5B | 0.17 | 0.02 | 0.13 | 0.00 | 0.51 | 0.57 | 13.10 | 0.26 | 0.04 | 0.15 | 0.04 | 0.52 | 0.59 | 31.99 |
| Qwen3-4B | 0.27 | 0.07 | 0.17 | 0.16 | 0.54 | 0.68 | 35.22 | 0.32 | 0.07 | 0.18 | 0.11 | 0.53 | 0.67 | 32.35 |
Table 5: Evaluation results on QMSum and SQuALITY.
| Method | ROUGE-L |
|---|---|
| ReSUM | 0.38 |
| w/o Planner | 0.32 ( 0.06) |
| w/o Reasoning Retriever | 0.31 ( 0.07) |
| w/o Critique | 0.33 ( 0.05) |
Table 6: Ablation study of core agents in ReSUM on QMSum using ROUGE-L.
Figure 3: Evaluating QMSum retriever performance via Recall@k and Precision@k: A comparative study of sparse, dense, hybrid, Zero-shot LLM (ZS), FT models, reasoning-driven fine-tuned (R) models.
Figure 4: Performance of summarization agents on QMSum across different settings. Zero-shot models, fine-tuned models, and reasoning-driven fine-tuned models. Each point represents a model.
| Model | Rel | Fact | Fluency | Complete | Mean |
|---|---|---|---|---|---|
| 0-Shot | 1.1 | 1.0 | 3.5 | 1.0 | 1.65 |
| CoT | 1.4 | 1.2 | 3.2 | 1.2 | 1.75 |
| RAG | 3.5 | 3.4 | 2.8 | 3.2 | 3.23 |
| FT | 4.2 | 3.8 | 4.5 | 4.1 | 4.15 |
| Ours | 4.5 | 4.5 | 4.3 | 4.4 | 4.43 |
Table 7: Human evaluation scores on a 1-5 Likert scale. Rel - Relevance, Fact - Factuality, Complete - Completeness
| Iteration 1: Initial (Failure) | |
|---|---|
| Planner | Requested Efficacy [2, 15] and Implementation [51, 74] . Instructions focused on CPS readiness and data collection. |
| Retriever | Fetched [2–15]; reported failure for [51–74]. |
| Summarizer | (Truncated) The law regarding reasonable punishment is confused. Removing the defence clarifies the framework. |
| Critique | Reject: Planner requested spans [51-74] which do not exist. Retrieval and Summary are incomplete. |
| Iteration 2: Refined (Success) | |
Table 8: Case Study 1: Summarize the efficacy of law. Initial outputs contain errors or omissions, which are corrected in subsequent iterations through refined planning and retrieval.
| Category | ID | Failure Mode Description |
| Planning (PL) | PL-01 | Span over/under-selection |
| PL-02 | Ambiguous or vague instructions | |
| PL-03 | Neglected critique feedback | |
| Retrieval (RE) | RE-01 | Missed relevant dialogue turns |
| RE-02 | Inclusion of irrelevant content | |
| Summarization (SU) | SU-01 | Model hallucinations |
Table 9: Taxonomy of Failure Modes in the Multi-Agent Pipeline
| Iteration 1 (Failed) | |
|---|---|
| Planner | Requested “Efficacy” [2–15] and “Implementation” [51–74] . The latter span does not exist in context. |
| Retriever | Fetched [2–15]. Reported failure to locate content for span [51–74]. |
| Summarizer | Summary focuses on efficacy but fails to address implementation instructions. |
| Critique | Reject: Flags planner overreach ([51-74]) as the root cause of downstream incompleteness. |
| Error(s) | PL-01, RE-01, SU-02 |
Table 10: Agentic Trace: Iterative Error Identification and Correction. Question: “Summarise the efficacy of the law.”
| Iteration 1 (Failed) | |
|---|---|
| Planner | Selected “Spectral subtraction” [402–415]. Instructions focus on Wiener filter, signal subspace, and cascaded system concepts. |
| Retriever | Successfully retrieved Professor C’s queries and PhD A’s clarifications on filtering and KL transform. |
| Summarizer | Reports Wiener filtering and signal subspace relations, but omits the specific conceptualization of the cascaded system. |
| Critique | Reject: Validates retrieval quality but identifies the summary as incomplete (SU-02) due to the cascaded system omission. |
| Error(s) | SU-02 |
Table 11: Agentic Trace: Iterative Performance for Signal Processing Discussion. Question: “What did the professor think about the Wiener filter?”