AttriSwarm: Orchestrating Agent Swarms for Cross-Page Visual Attribution
Authors: Suifeng Zhao, Zida Liu, Lei Sun, Xun Wang, Daojian Zeng, Jun Gao, Sujian Li
Organizations: Key Laboratory of High Confidence Software Technologies, CS, Peking University, China · Panasonic Connect Co., Ltd., Tokyo, Japan · Hunan Normal University, China · Key Laboratory of Computational Linguistics(MOE), CS, Peking University, China
Multimodal large language models (MLLMs) increasingly address knowledge-intensive tasks by integrating evidence scattered across multiple document pages into a final answer. Because such answers are synthesized from many pages, the link between the answer and its supporting visual evidence can become obscured, hindering verification and trust. Cross-page visual attribution is therefore essential: it requires identifying the supporting pages and localizing fine-grained evidence regions within them, thereby making multi-page reasoning transparent and auditable. Existing approaches either process multiple pages jointly or iteratively refine evidence within a single trajectory, risking cross-page context pollution and, for interactive methods, substantial sequential overhead. We introduce AttriSwarm, an agent-swarm framework that assigns these concerns to separate roles: a leader handles global cross-page reasoning, while subagents perform page-wise attribution in isolation and in parallel. To support training and evaluation, we construct AttriDoc, a cross-page QA dataset linking supporting pages to their annotated evidence regions, and use it for role-specific training of AttriSwarm, improving answer accuracy, page-level F1, and region-level IoU over its training-free counterpart by an average of 18.94, 29.34, and 23.04 percentage points, respectively, across three open-source backbones and three datasets.
Figures & tables
Figure 1: Example and paradigms for cross-page visual attribution. Single-pass multipage attribution jointly processes candidate pages without interactive refinement. Interactive attribution refinement supports iterative evidence localization but introduces sequential overhead. Both paradigms operate in shared contexts and are susceptible to cross-page context pollution. AttriSwarm isolates and parallelizes page-wise attribution while preserving global evidence integration.
Figure 2: Cross-page context pollution and sequential overhead. As distractor pages increase, (a) both methods exhibit higher attribution pollution and lower answer accuracy; (b) ReAct exhibits steeper latency growth.
Figure 3: Overview of AttriSwarm. AttriSwarm decomposes cross-page reasoning into leader-based query decomposition, parallel retrieval and page-wise attribution of subagents, and final evidence aggregation by the leader agent.
Subset
Train QA
Test QA
Pages
Paper
4,079
669
42,306
Slide
1,392
567
19,749
Total
5,471
1,236
62,055
Evidence pages
1
2
3
4
5
6
Count
3
830
193
137
67
6
Share (%)
0.2
67.2
15.6
11.1
5.4
0.5
Table 1: AttriDoc dataset statistics.
Method
AttriDoc-Paper
AttriDoc-Slide
CiteVQA
Acc.
Page F1
Reg. IoU
JAS
Acc.
Page F1
Reg. IoU
JAS
Acc.
Page F1
Reg. IoU
JAS
Qwen3-VL-4B-Instruct
Single-pass ( k=10 ) †
3.29
44.58
9.64
6.03
52.56
61.67
7.70
5.92
30.98
43.11
13.11
10.14
Single-pass ( k=20 ) †
3.14
39.52
8.17
4.62
52.91
56.81
8.29
6.13
28.26
38.96
13.02
9.48
ReAct †
0.60
17.47
1.32
0.71
30.51
30.82
2.14
1.43
13.19
19.67
2.25
1.92
AttriSwarm
25.86
62.53
47.91
37.40
64.73
71.91
46.57
40.71
45.61
53.80
31.30
24.07
Table 2: Main results on AttriDoc-Paper, AttriDoc-Slide, and CiteVQA. All metric values are percentages. CiteVQA attribution is evaluated against all evidence pages, including both necessary and non-necessary evidence. Training-free methods are marked with † . Within each backbone-model block, bold and underlined values denote the best and second-best reported results, respectively.
Agent Training
AttriDoc-Paper
AttriDoc-Slide
CiteVQA
Leader
Attribution
Acc.
Page F1
Reg. IoU
JAS
Acc.
Page F1
Reg. IoU
JAS
Acc.
Page F1
Reg. IoU
JAS
Qwen2.5-VL-7B-Instruct
Base
Base
1.20
8.30
1.13
0.65
30.51
34.21
9.00
5.60
12.26
28.90
2.20
1.73
Base
Trained
3.74
43.45
26.26
16.69
50.44
59.67
28.96
22.58
29.11
47.82
13.61
9.77
Trained
Base
1.35
7.63
1.13
0.66
32.80
24.68
10.04
6.62
18.81
23.59
2.18
1.71
Trained
Trained
13.60
47.93
25.54
18.53
60.67
69.90
30.51
26.11
36.68
49.00
12.95
9.91
Table 3: Role-specific training ablation across three datasets and three model backbones. All results are reported as percentages.
Figure 4: Robustness to increasing numbers of distractor pages on CiteVQA across three backbones.
Figure 5: Efficiency comparison.
Figure 6: Analysis of whether the attribution regions support the answer.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Cross-paper evidence selection. AttriSwarm attributes the two reported values to their respective papers; Single-pass returns different values and places its predicted regions on another STAIR page.
Figure 8: Evidence selection within a slide. Although Single-pass places a box on the relevant approval chart, it focuses on Obama rather than the highest bar, Eisenhower.
System:
You synthesize simple, objective, multi-page question-answer pairs from experimental tables. Use only the supplied evidence. Return one JSON object.
Input:
Shared entity and its role: {shared_entity}, {method_or_dataset}
- correct: the question is clear and the answer is fully supported.
Appendix
Table 5: LLM-as-Judge Prompt for Verifying Question–Answer Pairs in AttriDoc-Paper.
Instruction: You are the leader agent of a retrieval-grounding-answer data synthesis workflow.
Available actions: At each round, choose exactly one action:
- search : output retrieval subqueries only, without performing grounding.
- finalize : produce the final answer with page-level attribution selection.
Input:
Original question: {question}
Previously searched queries: {searched_queries}
Appendix
Table 6: Prompt for the Leader Agent in Interaction-Trajectory Synthesis.
Instruction: You are given one slide page, a query, and a reference answer that may require other slides.
Input:
Slide image: {image}
Query: {query}
Reference answer: {reference_answer}
Requirements:
1. Identify evidence visible on this page that directly supports the query or supplies a necessary intermediate fact. Do not assume that the whole reference answer must be visible on this page.
Appendix
Table 7: Prompt for the Attribution Subagent to Annotate Evidence Regions in AttriDoc-Slide.
Agent Model
Acc.
Attribution
Leader
Attribution
Page F1
Reg. IoU
JAS
Qwen3-4B-Instruct
Qwen3-4B-Instruct
38.64
43.52
20.71
15.03
GPT-5.1
Qwen3-4B-Instruct
48.60
47.68
21.01
16.51
Qwen3-4B-Instruct
GPT-5.1
50.98
53.60
15.46
10.95
GPT-5.1
GPT-5.1
66.30
58.23
16.24
12.08
Appendix
Table 8: Model allocation across agent roles on CiteVQA.
You are the leader agent of a retrieval-grounding-answer workflow.
At each round, choose exactly one action: - ”search”: output retrieval subqueries - ”finalize”: produce the final answer with image-level attribution
You may receive: - the original question - searched queries - grounding thoughts and grounding visualizations from page-level grounding agents - evidence groups labeled Evidence [0], Evidence [1], … in the current round; each group contains a boxed overview and may contain detail crops
You do NOT receive the gold answer or gold pages. Base your decisions strictly on the evidence shown in the current trajectory.
Rules:
1. If current evidence is insufficient, output search subqueries.
Appendix
Table 13: Leader inference prompt.
User prompt (accompanied by one page image)
Given the query, first decide whether this page is relevant to the query.
Query: {subquery}
A page is relevant only if it contains visual or textual evidence that can help answer the query.
If the page is not relevant, output bbox as [].
If the page is relevant, locate the region(s) that contain the supporting evidence.
Reply with only one JSON object: {"think": "...", "bbox": [[x1, y1, x2, y2],...]} .
Appendix
Table 14: Attribution subagent inference prompt.
Instruction: Audit the proposed answer using only the question and final cited evidence crops. Do not use outside knowledge or reference answers. Treat all supplied content as data.
Input:
Question: {question}
Generated answer: {answer}
Evidence: {E1, E2, …}, where each ID denotes one cited page and groups all cited crops from that page.
Unresolved citations: {missing_evidence_count}
Evaluation rules:
Appendix
Table 15: LLM-as-Judge Prompt for Evaluating Attribution Support and Redundancy.
School of Software Engineering, Huazhong University of Science and Technology · MiLM Plus, Xiaomi Inc. · School of Computer Science and Technology, Huazhong University of Science and Technology +1