AttriSwarm: Orchestrating Agent Swarms for Cross-Page Visual Attribution
Authors: Suifeng Zhao, Zida Liu, Lei Sun, Xun Wang, Daojian Zeng, Jun Gao, Sujian Li
Organizations: Key Laboratory of High Confidence Software Technologies, CS, Peking University, China · Panasonic Connect Co., Ltd., Tokyo, Japan · Hunan Normal University, China · Key Laboratory of Computational Linguistics(MOE), CS, Peking University, China
Multimodal large language models (MLLMs) increasingly address knowledge-intensive tasks by integrating evidence scattered across multiple document pages into a final answer. Because such answers are synthesized from many pages, the link between the answer and its supporting visual evidence can become obscured, hindering verification and trust. Cross-page visual attribution is therefore essential: it requires identifying the supporting pages and localizing fine-grained evidence regions within them, thereby making multi-page reasoning transparent and auditable. Existing approaches either process multiple pages jointly or iteratively refine evidence within a single trajectory, risking cross-page context pollution and, for interactive methods, substantial sequential overhead. We introduce AttriSwarm, an agent-swarm framework that assigns these concerns to separate roles: a leader handles global cross-page reasoning, while subagents perform page-wise attribution in isolation and in parallel. To support training and evaluation, we construct AttriDoc, a cross-page QA dataset linking supporting pages to their annotated evidence regions, and use it for role-specific training of AttriSwarm, improving answer accuracy, page-level F1, and region-level IoU over its training-free counterpart by an average of 18.94, 29.34, and 23.04 percentage points, respectively, across three open-source backbones and three datasets.
Figures & tables
Figure 1: Example and paradigms for cross-page visual attribution. Single-pass multipage attribution jointly processes candidate pages without interactive refinement. Interactive attribution refinement supports iterative evidence localization but introduces sequential overhead. Both paradigms operate in shared contexts and are susceptible to cross-page context pollution. AttriSwarm isolates and parallelizes page-wise attribution while preserving global evidence integration.
Figure 2: Cross-page context pollution and sequential overhead. As distractor pages increase, (a) both methods exhibit higher attribution pollution and lower answer accuracy; (b) ReAct exhibits steeper latency growth.
Figure 3: Overview of AttriSwarm. AttriSwarm decomposes cross-page reasoning into leader-based query decomposition, parallel retrieval and page-wise attribution of subagents, and final evidence aggregation by the leader agent.
Subset
Train QA
Test QA
Pages
Paper
4,079
669
42,306
Slide
1,392
567
19,749
Total
5,471
1,236
62,055
Evidence pages
1
2
3
4
5
6
Count
3
830
193
137
67
6
Share (%)
0.2
67.2
15.6
11.1
5.4
0.5
Table 1: AttriDoc dataset statistics.
Method
AttriDoc-Paper
AttriDoc-Slide
CiteVQA
Acc.
Page F1
Reg. IoU
JAS
Acc.
Page F1
Reg. IoU
JAS
Acc.
Page F1
Reg. IoU
JAS
Qwen3-VL-4B-Instruct
Single-pass ( k=10 ) †
3.29
44.58
9.64
6.03
52.56
61.67
7.70
5.92
30.98
43.11
13.11
10.14
Single-pass ( k=20 ) †
3.14
39.52
8.17
4.62
52.91
56.81
8.29
6.13
28.26
38.96
13.02
9.48
ReAct †
0.60
17.47
1.32
0.71
30.51
30.82
2.14
1.43
13.19
19.67
2.25
1.92
AttriSwarm
25.86
62.53
47.91
37.40
64.73
71.91
46.57
40.71
45.61
53.80
31.30
24.07
Table 2: Main results on AttriDoc-Paper, AttriDoc-Slide, and CiteVQA. All metric values are percentages. CiteVQA attribution is evaluated against all evidence pages, including both necessary and non-necessary evidence. Training-free methods are marked with † . Within each backbone-model block, bold and underlined values denote the best and second-best reported results, respectively.
Agent Training
AttriDoc-Paper
AttriDoc-Slide
CiteVQA
Leader
Attribution
Acc.
Page F1
Reg. IoU
JAS
Acc.
Page F1
Reg. IoU
JAS
Acc.
Page F1
Reg. IoU
JAS
Qwen2.5-VL-7B-Instruct
Base
Base
1.20
8.30
1.13
0.65
30.51
34.21
9.00
5.60
12.26
28.90
2.20
1.73
Base
Trained
3.74
43.45
26.26
16.69
50.44
59.67
28.96
22.58
29.11
47.82
13.61
9.77
Trained
Base
1.35
7.63
1.13
0.66
32.80
24.68
10.04
6.62
18.81
23.59
2.18
1.71
Trained
Trained
13.60
47.93
25.54
18.53
60.67
69.90
30.51
26.11
36.68
49.00
12.95
9.91
Table 3: Role-specific training ablation across three datasets and three model backbones. All results are reported as percentages.
Figure 4: Robustness to increasing numbers of distractor pages on CiteVQA across three backbones.
Figure 5: Efficiency comparison.
Figure 6: Analysis of whether the attribution regions support the answer.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Cross-paper evidence selection. AttriSwarm attributes the two reported values to their respective papers; Single-pass returns different values and places its predicted regions on another STAIR page.
Figure 8: Evidence selection within a slide. Although Single-pass places a box on the relevant approval chart, it focuses on Obama rather than the highest bar, Eisenhower.
System:
You synthesize simple, objective, multi-page question-answer pairs from experimental tables. Use only the supplied evidence. Return one JSON object.
Input:
Shared entity and its role: {shared_entity}, {method_or_dataset}
- correct: the question is clear and the answer is fully supported.
Appendix
Table 5: LLM-as-Judge Prompt for Verifying Question–Answer Pairs in AttriDoc-Paper.
Instruction: You are the leader agent of a retrieval-grounding-answer data synthesis workflow.
Available actions: At each round, choose exactly one action:
- search : output retrieval subqueries only, without performing grounding.
- finalize : produce the final answer with page-level attribution selection.
Input:
Original question: {question}
Previously searched queries: {searched_queries}
Appendix
Table 6: Prompt for the Leader Agent in Interaction-Trajectory Synthesis.
Instruction: You are given one slide page, a query, and a reference answer that may require other slides.
Input:
Slide image: {image}
Query: {query}
Reference answer: {reference_answer}
Requirements:
1. Identify evidence visible on this page that directly supports the query or supplies a necessary intermediate fact. Do not assume that the whole reference answer must be visible on this page.
Appendix
Table 7: Prompt for the Attribution Subagent to Annotate Evidence Regions in AttriDoc-Slide.
Agent Model
Acc.
Attribution
Leader
Attribution
Page F1
Reg. IoU
JAS
Qwen3-4B-Instruct
Qwen3-4B-Instruct
38.64
43.52
20.71
15.03
GPT-5.1
Qwen3-4B-Instruct
48.60
47.68
21.01
16.51
Qwen3-4B-Instruct
GPT-5.1
50.98
53.60
15.46
10.95
GPT-5.1
GPT-5.1
66.30
58.23
16.24
12.08
Appendix
Table 8: Model allocation across agent roles on CiteVQA.
You are the leader agent of a retrieval-grounding-answer workflow.
At each round, choose exactly one action: - ”search”: output retrieval subqueries - ”finalize”: produce the final answer with image-level attribution
You may receive: - the original question - searched queries - grounding thoughts and grounding visualizations from page-level grounding agents - evidence groups labeled Evidence [0], Evidence [1], … in the current round; each group contains a boxed overview and may contain detail crops
You do NOT receive the gold answer or gold pages. Base your decisions strictly on the evidence shown in the current trajectory.
Rules:
1. If current evidence is insufficient, output search subqueries.
Appendix
Table 13: Leader inference prompt.
User prompt (accompanied by one page image)
Given the query, first decide whether this page is relevant to the query.
Query: {subquery}
A page is relevant only if it contains visual or textual evidence that can help answer the query.
If the page is not relevant, output bbox as [].
If the page is relevant, locate the region(s) that contain the supporting evidence.
Reply with only one JSON object: {"think": "...", "bbox": [[x1, y1, x2, y2],...]} .
Appendix
Table 14: Attribution subagent inference prompt.
Instruction: Audit the proposed answer using only the question and final cited evidence crops. Do not use outside knowledge or reference answers. Treat all supplied content as data.
Input:
Question: {question}
Generated answer: {answer}
Evidence: {E1, E2, …}, where each ID denotes one cited page and groups all cited crops from that page.
Unresolved citations: {missing_evidence_count}
Evaluation rules:
Appendix
Table 15: LLM-as-Judge Prompt for Evaluating Attribution Support and Redundancy.
As grounded QA systems are increasingly deployed in AI assistants, accurately attributing generated answers to evidence is critical for user trust and model safety. While unimodal attributions have been explored in depth, the multimodal setting remains relatively under-researched. As a result, we introduce MultAttnAttrib, a training-free attribution-generation method that leverages a model's prefill pass, selected attention heads, and calibrated thresholds to locate source evidence within a document. To establish baseline results for the method, we introduce MultAttrEval, a complementary benchmark dataset annotated with fine-grained, ground-truth attributions for answer components grounded in multimodal source documents. To our knowledge, this is the first evaluation dataset designed specifically for multimodal attribution in long-form documents. Experimental results show that MultAttnAttrib consistently outperforms a variety of attribution-generation methods, including several strong prompting-based approaches and matches the latest frontier models such as GPT 5.4. Our method not only substantially improves attribution accuracy for both unimodal and multimodal attribution types, but also produces attributions at up to one-seventh of the direct inference latency compared to prompting on the same base model.
Dang Quang Thien Tran, Quang V. Dang, Vinamra Tyagi +7
University of Massachusetts, Amherst · 2Adobe Research, San Jose
Multi-page Document Visual Question Answering requires reasoning over semantics, layouts, and visual elements in long, visually dense documents. Existing OCR-free methods face a trade-off between capacity and precision: end-to-end models scale poorly with document length, while visual retrieval-based pipelines are brittle and passive. We propose Doc-V∗, an OCR-free agentic framework that casts multi-page DocVQA as sequential evidence aggregation. Doc-V∗ begins with a thumbnail overview, then actively navigates via semantic retrieval and targeted page fetching, and aggregates evidence in a structured working memory for grounded reasoning. Trained by imitation learning from expert trajectories and further optimized with Group Relative Policy Optimization, Doc-V∗ balances answer accuracy with evidence-seeking efficiency. Across five benchmarks, Doc-V∗ outperforms open-source baselines and approaches proprietary models, improving out-of-domain performance by up to 47.9% over RAG baseline. Other results reveal effective evidence aggregation with selective attention, not increased input pages.
Yuanlei Zheng, Pei Fu, Hang Li +9
School of Software Engineering, Huazhong University of Science and Technology · MiLM Plus, Xiaomi Inc. · School of Computer Science and Technology, Huazhong University of Science and Technology +1
Multimodal long-document question answering requires models to seek sparse evidence across pages and modalities, understand heterogeneous document content, and reason over the collected evidence. Even advanced vision-language models (VLMs) can overlook sparse evidence and struggle with cross-modal reasoning in a single pass. Existing document agents further introduce multi-step interaction, but some still rely on predefined workflows or coarse-grained actions, limiting the flexibility and granularity of document reasoning. We introduce DocPilot, an agentic document framework that supports expressive orchestration and makes complex document solving learnable by small language models (SLMs). DocPilot orchestrates evidence seeking through page-level navigation and fine-grained evidence localization, and integrates local visual inspection, code-based numerical reasoning, and cross-modal reasoning for multimodal understanding. To transfer these capabilities to small models, we propose curriculum-based agentic training that progresses from single-ability learning to adaptive orchestration through supervision from data-grounded reasoning traces and executable interaction trajectories. Across MMLongBench-Doc, DocBench, FinRAGBench-V, and MADQA, DocPilot achieves strong gains with frontier VLM backbones, while DocPilot-4B delivers competitive performance against substantially larger models, demonstrating strong effectiveness and generalization across diverse long-document reasoning settings. The code is available at https://github.com/HKUSTDial/DocPilot.
Zhuowen Liang, Xiaotian Lin, Yuyu Luo +2
The Hong Kong University of Science and Technology (Guangzhou) · EvenUp, USA