LLMs excel at language tasks, but long-meeting QFS remains difficult due to complex speaker dynamics, sparse key information, and limited data. Meetings involve disfluencies, long transcripts and complex discourse. QFS have high risk of information omission, hallucination, even for advanced models such as GPT-4-mini and Gemini. Existing end-to-end methods compress retrieval, evidence extraction, and summary generation into a single pass, failing to handle the complex reasoning required for targeted meeting summaries. We propose ReSUM, a modular multi-LLM framework coordinated by a planner that orchestrates retrieval and summarization agents. By distilling teacher reasoning into query- and plan-conditioned retrievers and summarizers, ReSUM enhances long-context robustness, minimizes irrelevant exploration, and generates more faithful and comprehensive query-focused summaries. Experiments show consistent gains on query-focused meeting summarization benchmarks.
Figures & tables
Figure 1: Single Agent that fails to capture query-specific context, our Multi-Agent pipeline divides tasks between specialized agents. By utilizing reasoning-driven knowledge distillation, the planning and retrieval agents ensure precise content selection, while the critique loop provides feedback-driven refinement for a highly accurate, query-focused summary.
Figure 2: Architectural Overview of ReSUM , where the upper part illustrates the training stage with teacher–student distillation and reasoning supervision for the Retriever and Summarizer, and the lower part depicts the multi-agent pipeline coordinating Planner, Retriever, Summarizer, and Critique agents for iterative, query-focused meeting summarization.
Instructions
You are a Planner agent for long meeting QFS. Given user query Q, meeting transcript M, and current state S, construct a structured, dependency-aware execution plan for agents. Given agents description: Retriever (select fine-grained, query-relevant utterance spans from planner-selected discourse regions, focusing on evidence-bearing content), Summarizer (generate a query-focused summary grounded strictly in retrieved spans without hallucination), and Critique (evaluate planning quality, retrieval relevance, and summary faithfulness). Generate a step-by-step plan for each agent, to generate a query-focused summary.
Table 1: Planning Agent: Role definition and multi-agent coordination instructions.
Prompt
Given (i) a meeting transcript with turn-level identifiers, (ii) a user query, and (iii) a relevant span defined by its start and end turn IDs along with the corresponding span text, generate a structured rationale explaining how the span addresses the query. The rationale should: (1) identify key evidence lines within the span, (2) explain their relevance to the query, (3) articulate any implicit reasoning linking the span to the query, and (4) rely exclusively on transcript content within the specified turn range. The output must present a coherent, step-by-step explanation grounded in the provided span.
Rationale
The Project Manager introduces the prototype by outlining the meeting agenda and emphasizing trend-driven design elements, such as a fashionable and fruit-inspired look. The User Interface then describes the prototype’s banana-shaped design, directly aligning with these trends. Together, these details explain how they introduced the prototype in response to the query.
Table 2: Retriever Agent: System prompt and generated rationale for evidence extraction.
Prompt
Given (i) a user query, (ii) a relevant transcript span, and (iii) a query-focused summary, generate a structured rationale explaining how the span supports the summary with respect to the query. The rationale should: (1) Identify key evidence lines from the span, (2) Explain how each summary point addresses the query, (3) Clarify implicit reasoning linking the span to the summary, and (4) Rely only on information within the provided span. Produce a clear, step-by-step explanation grounded strictly in the given content.
Rationale
The team introduced a fashion-inspired prototype, described as bright yellow and banana-shaped. They emphasized simplicity, ergonomics, and usability for both left- and right-handed users, along with advanced technology. The discussion about size and design choices supports the summary’s focus on creativity, functionality, and ease of use.
Table 3: Summarizer Agent: Instructional prompt and evidence-grounded rationale.
Instructions
You are a Critique agent in a multi-agent QFS system. Evaluate outputs from the Planner, Retriever, and Summarizer, and provide actionable feedback. 1. Planner (Planning Alignment)
(i) Are selected topics appropriate for the query and meeting context? (ii) Were necessary agents correctly invoked and assigned roles? (iii) Are instructions clear and query-aligned? 2. Retriever (Retrieval Quality)
(i) Are retrieved turn-ID ranges directly relevant to the query? (ii) Are retrieved spans evidence-bearing rather than loosely related? (iii) Is important query-relevant content missing? 3. Summarizer (Summary Quality)
(i) Is the summary grounded only in retrieved evidence? (ii) Does it fully and correctly answer the query? (iii) Is it faithful, coherent, and free of hallucinations? Conclude whether the overall output is acceptable or requires refinement.
Table 4: Critique Agent: Evaluation and refinement instructions.
Model
QMSum
SQuALITY
R1
R2
RL
BS
Mover
COMET
chrF
R1
R2
RL
BS
Mover
COMET
chrF
Zero-shot
LLaMA3-3B
0.27
0.06
0.17
0.15
0.53
0.66
34.01
0.32
0.06
0.17
0.08
0.53
0.67
35.52
Gemma3-4B
0.20
0.04
0.13
0.06
0.52
0.65
30.39
0.32
0.06
0.17
0.04
0.53
0.66
37.38
DeepSeek-1.5B
0.17
0.02
0.13
0.00
0.51
0.57
13.10
0.26
0.04
0.15
0.04
0.52
0.59
31.99
Qwen3-4B
0.27
0.07
0.17
0.16
0.54
0.68
35.22
0.32
0.07
0.18
0.11
0.53
0.67
32.35
Table 5: Evaluation results on QMSum and SQuALITY.
Method
ROUGE-L
ReSUM
0.38
w/o Planner
0.32 ( ↓ 0.06)
w/o Reasoning Retriever
0.31 ( ↓ 0.07)
w/o Critique
0.33 ( ↓ 0.05)
Table 6: Ablation study of core agents in ReSUM on QMSum using ROUGE-L.
Figure 3: Evaluating QMSum retriever performance via Recall@k and Precision@k: A comparative study of sparse, dense, hybrid, Zero-shot LLM (ZS), FT models, reasoning-driven fine-tuned (R) models.
Figure 4: Performance of summarization agents on QMSum across different settings. Zero-shot models, fine-tuned models, and reasoning-driven fine-tuned models. Each point represents a model.
Model
Rel
Fact
Fluency
Complete
Mean
0-Shot
1.1
1.0
3.5
1.0
1.65
CoT
1.4
1.2
3.2
1.2
1.75
RAG
3.5
3.4
2.8
3.2
3.23
FT
4.2
3.8
4.5
4.1
4.15
Ours
4.5
4.5
4.3
4.4
4.43
Table 7: Human evaluation scores on a 1-5 Likert scale. Rel - Relevance, Fact - Factuality, Complete - Completeness
Iteration 1: Initial (Failure)
Planner
Requested Efficacy [2, 15] and Implementation [51, 74] . Instructions focused on CPS readiness and data collection.
Retriever
Fetched [2–15]; reported failure for [51–74].
Summarizer
(Truncated) The law regarding reasonable punishment is confused. Removing the defence clarifies the framework.
Critique
Reject: Planner requested spans [51-74] which do not exist. Retrieval and Summary are incomplete.
Iteration 2: Refined (Success)
Table 8: Case Study 1: Summarize the efficacy of law. Initial outputs contain errors or omissions, which are corrected in subsequent iterations through refined planning and retrieval.
Category
ID
Failure Mode Description
Planning (PL)
PL-01
Span over/under-selection
PL-02
Ambiguous or vague instructions
PL-03
Neglected critique feedback
Retrieval (RE)
RE-01
Missed relevant dialogue turns
RE-02
Inclusion of irrelevant content
Summarization (SU)
SU-01
Model hallucinations
Table 9: Taxonomy of Failure Modes in the Multi-Agent Pipeline
Iteration 1 (Failed)
Planner
Requested “Efficacy” [2–15] and “Implementation” [51–74] . The latter span does not exist in context.
Retriever
Fetched [2–15]. Reported failure to locate content for span [51–74].
Summarizer
Summary focuses on efficacy but fails to address implementation instructions.
Critique
Reject: Flags planner overreach ([51-74]) as the root cause of downstream incompleteness.
Error(s)
PL-01, RE-01, SU-02
Table 10: Agentic Trace: Iterative Error Identification and Correction. Question: “Summarise the efficacy of the law.”
Iteration 1 (Failed)
Planner
Selected “Spectral subtraction” [402–415]. Instructions focus on Wiener filter, signal subspace, and cascaded system concepts.
Retriever
Successfully retrieved Professor C’s queries and PhD A’s clarifications on filtering and KL transform.
Summarizer
Reports Wiener filtering and signal subspace relations, but omits the specific conceptualization of the cascaded system.
Critique
Reject: Validates retrieval quality but identifies the summary as incomplete (SU-02) due to the cascaded system omission.
Error(s)
SU-02
Table 11: Agentic Trace: Iterative Performance for Signal Processing Discussion. Question: “What did the professor think about the Wiener filter?”
QMSum provides no scorer, making query-focused meeting summarization results difficult to compare. We rescore or generate 15 systems under one implementation. Through a common inference port, a released 406M Fusion-in-Decoder specialist loses 6.30 ROUGE-1 when moved from capped long input to 2,000-word retrieved spans. Fine-tuning it on this span regime recovers the loss. On test it scores 36.33 ROUGE-1 versus 35.41 for our 1.2B system; the meeting-cluster 95% interval for the difference is [-0.27, +2.22], so QMSum does not statistically separate them. The smaller system uses about one-third as many total parameters and less than half the peak inference memory. Within the fixed 1.2B base, span-regime fine-tuning adds 5.29 [+4.02, +6.56], while replacing the first 4,500 transcript words with 2,000 retrieved words adds 1.55 on test and 0.29 on validation. Separately, under one concise prompt and reference-overlap scorer, a released 406M specialist exceeds five proprietary hosted models by at least 6.2 ROUGE-1, but output length and absent human or factuality evaluation limit this ordering. Conclusions are limited to QMSum and automatic metrics.
Meeting documents are challenging to summarize due to their length and complex conversational structure. Existing approaches typically adopt multi-stage pipelines that extract information prior to summarization; however, these approaches often suffer from cumulative error propagation without intermediate validation, a limitation further amplified by short and low-quality reference summaries. We propose segment-level summarization via Monte Carlo Tree Search (S3), a training-free framework that constructs a final summary by composing segment-level summary candidates. S3 partitions a long document into segments and generates multiple summary candidates per segment, forming nodes of a search tree. The best-scoring combination is selected via self-reward-guided tree search and refined into the final output. Despite using a 7B model, S3 achieves performance comparable to larger 72B models while producing length-appropriate summaries.
Sangwon Ryu, Heejin Do, Jun Seo +4
GSAI, POSTECH · ETH Zurich, ETH AI Center · Agentic AI Lab, KT +2
Large language models (LLMs) demonstrate strong performance in text summarization, yet their effectiveness drops significantly across languages with restricted training resources. This work addresses the challenge of query-focused summarization (QFS) in less-resourced languages, where labeled datasets and evaluation tools are limited. We present a novel QFS framework, QFS-Composer, that integrates query decomposition, question generation (QG), question answering (QA), and abstractive summarization to improve the factual alignment of a summary with user intent. We test our approach on the Slovenian language. To enable high-quality supervision and evaluation, we develop the Slovenian QA and QG models based on a Slovene LLM and adapt evaluation approaches for reference-free summary evaluation. Empirical evaluation shows that the QA-guided summarization pipeline yields improved consistency and relevance over baseline LLMs. Our work establishes an extensible methodology for advancing QFS in less-resourced languages.
Vuk Đuranović, Marko Robnik Šikonja
Faculty of Computer and Information Science, University of Ljubljana, Slovenia