Multimodal retrieval-augmented generation over visually rich documents requires evidence at a finer granularity than entire pages. Existing visual retrievers typically index or return full pages, whereas training region-level retrievers often requires costly query-to-region annotations or noisy model-generated labels. We introduce GARRET, a region-retrieval framework trained from (page, question, answer) triplets through differentiable generative feedback. A shared context-preserving retriever produces page and region multi-vector representations. During training, their retrieval scores define a soft routing distribution over candidate region crops. An auxiliary vision-language model attends to each crop separately, aggregates crop-specific contexts using the routing weights, and predicts the question followed by the answer under a next-token prediction objective. Gradients through the routing distribution assign query-conditioned credit to the indexed region representations, without requiring query-to-region annotations. At inference, the auxiliary model is removed. GARRET also reuses the contextualized region representations, together with page summary tokens, to construct a compact page representation. We introduce ViDoRe-V3-Region, a corpus-wide benchmark containing 200,682 fixed candidate regions across eight domains. GARRET achieves a macro nDCG@10 of 0.4126, outperforming the strongest visual baseline by 2.94 points. On page retrieval, it remains comparable to ColQwen2.5-v0.2 while indexing 47.8% fewer vectors per page.
Figures & tables
Figure 1: Overview of the GARRET framework and its performance. Left: During training, the retriever computes routing weights for candidate regions. An auxiliary VLM uses these weights via in-batch cross-attention to process isolated visual crops. By predicting the paired question and answer, the VLM backpropagates a next-token prediction (NTP) loss as generative feedback, teaching the retriever latent region relevance. Right: GARRET achieves state-of-the-art region retrieval across eight domains in ViDoRe V3 while requiring significantly fewer indexed vectors than existing baselines.
Figure 2: Architecture of GARRET’s context-preserving multi-granularity retriever. A specialized attention mask allows region tokens to absorb global page context without mutual interference. After projection, base page embeddings are discarded. The retained region and summary embeddings form a highly compact representation, enabling both fine-grained region retrieval and efficient page-level retrieval.
Method
Avg.
CS
Ene.
Fin-EN
Fin-FR
HR
Ind.
Pha.
Phy.
Vec./reg.
Text-based Document Retrievers
BM25
16.24
25.93
5.97
19.67
5.47
28.70
18.28
21.84
4.05
–
BGE-M3
25.26
33.32
29.63
26.17
12.57
35.51
24.45
22.17
18.28
1.0
Qwen3-Emb.-8B
35.22
49.97
38.17
35.09
19.20
38.67
31.72
33.20
35.77
1.0
Vision-based Document Retrievers
ColPali-v1.3
31.51
50.15
28.51
27.98
14.26
39.74
31.18
30.47
29.75
1031.0
Table 1: Region retrieval on ViDoRe-V3-Region (nDCG@10, ×100 ). GARRET achieves the best average performance with the fewest indexed vectors among dense visual retrievers. Bold and underlined values denote the best and second-best results.
Method
nDCG
Layout Recall
@5
@10
R@1
R@5
R@10
Text-based regions
BM25-OCR
32.73
34.65
20.82
36.00
42.17
BM25-VLM
41.68
44.21
28.72
50.78
58.70
BGE-M3
42.48
45.58
27.90
50.88
59.75
Qwen3-Emb.-8B
42.14
45.32
24.87
47.98
56.73
Table 2: Region retrieval on MMDocIR. All metrics are reported on a ×100 scale. GARRET achieves the best nDCG and layout recall at all evaluated cutoffs, demonstrating strong cross-benchmark generalization. Bold and underlined values denote the best and second-best results.
Method
V1
V2
V3
Vec./page
Text-based Document Retrievers
BM25
69.94
21.61
38.29
-
BGE-M3
68.27
49.75
44.63
1.0
Qwen3-Emb.-8B
75.82
55.28
53.93
1.0
Vision-based Document Retrievers
ColPali-v1.3
85.42
57.81
48.86
1031.0
Table 3: Page retrieval on ViDoRe V1–V3 using nDCG@10 ( ×100 ). GARRET remains competitive with the strongest visual retrievers while using substantially fewer indexed vectors per page. Bold and underlined values denote the best and second-best results.
Method / context
Units
Area/P5
Acc
Page-level retrieval
BM25
5 pages
100%
27.53
BGE-M3
5 pages
100%
33.85
Qwen3-Emb.-8B
5 pages
100%
36.71
ColPali-v1.3
5 pages
100%
33.51
ColQwen2.5-v0.2
5 pages
100%
37.00
Table 4: RAG results on ViDoRe V3. Page-level methods use the top-5 retrieved pages, while GARRET uses the top- k retrieved layout regions. Area/P5 denotes retrieved area relative to five full pages. Acc is reported as a percentage.
Variant
Region
Page
Vectors/page
GARRET
41.26
57.29
392.9
Generative-target controls
w/o generative feedback
36.33
57.04
392.9
Answer target only
41.02
56.92
392.9
Question target only
39.98
56.89
392.9
Model architecture variants
Table 5: Ablation studies on ViDoRe V3 using nDCG@10 ( ×100 ). Generative feedback primarily improves region retrieval, while joint question–answer generation and the full shared architecture yield the best performance. Bold and underlined values denote the best and second-best results.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Component
Setting
Retriever / auxiliary VLM initialization
Qwen2.5-VL-3B-Instruct
Document embedding dimension
128
Retriever LoRA
rank 64, α=128
Auxiliary VLM LoRA
rank 32, α=64
Page-summary tokens
100
Region routing score
sR+sD
Appendix
Table 6: Implementation settings for the full GARRET model. These hyperparameters are used throughout the experiments unless an ablation explicitly changes them.
Statistic
Value
Domains
8
Pages
19,252
Queries
2,419
Fixed layout candidates
200,682
Appendix
Table 7: Statistics of ViDoRe-V3-Region . MinerU extracts 200,682 query-independent layout candidates from 19,252 pages across eight domains.
Summary tokens
Vectors/page
Region
Page
0
292.9
40.06
55.82
50
342.9
40.95
56.59
100
392.9
41.26
57.29
150
442.9
41.30
57.20
Appendix
Table 9: Effect of the page-summary-token budget on ViDoRe V3 nDCG@10 ( ×100 ). We use 100 tokens in the full model, as indicated by the bold token budget. Bold and underlined metric values denote the best and second-best results, respectively.
Hard
Random
Total pages
Region
Page
0
1
2
38.51
57.08
0
2
3
39.42
57.32
1
0
2
40.39
57.15
2
0
3
40.01
57.21
1
1
3
41.26
57.29
Appendix
Table 10: Ablation of negative-page sampling for the local candidate pool on ViDoRe V3 nDCG@10 ( ×100 ). “Total pages” includes the paired page, which is present in every row. The full model uses one hard-negative and one random-negative page.
Method family
Candidate input
Candidate representation
Region score
BM25
Parsed candidate text
Sparse lexical index
Lexical matching score
BGE-M3 / Qwen3-Embedding
Parsed candidate text
One dense text vector
Dense similarity
ColPali / ColQwen
Exact candidate crop
Independently encoded visual vectors
Late interaction
RegionRAG
Query-dependent predicted regions
Dynamic region output mapped to benchmark boxes
Mapped region score
GARRET
Full page and candidate box
Contextual region token group
sR+sD
Appendix
Table 11: Adaptation of each retriever to fixed-candidate region ranking. Text retrievers encode parsed candidate text, ColPali and ColQwen encode isolated candidate crops, and RegionRAG maps its query-dependent outputs to benchmark boxes. GARRET directly scores persistent, page-contextualized candidates.
View
Variant
Avg.
CS
Ene.
Fin-EN
Fin-FR
HR
Ind.
Pha.
Phy.
Region
GARRET
41.26
55.46
42.47
34.05
22.37
46.07
41.93
49.72
38.04
Region
No generative feedback
36.33
50.96
34.92
34.43
19.38
39.78
34.63
44.73
31.80
Page
GARRET
57.29
76.46
60.44
59.10
39.26
61.06
52.71
63.04
46.24
Page
No generative feedback
57.04
76.27
59.66
62.21
37.58
61.02
49.48
64.57
45.51
Appendix
Table 12: Per-domain ablation of generative feedback on ViDoRe V3 using nDCG@10 ( ×100 ). Removing the auxiliary VLM and NTP objective reduces average region retrieval by 4.93 points but page retrieval by only 0.25 points, showing that generative feedback primarily improves fine-grained ranking. Values are point estimates from one trained model per variant.
Method
Avg.
ArXiv
DocV
InfoV
Shift
Syn-AI
Syn-En
Syn-Gov
Syn-HC
TabFQ
TatD
Text-based document retrievers
BM25
69.94
36.65
39.45
68.36
68.29
95.85
90.90
86.52
90.93
61.67
60.73
BGE-M3
68.27
32.83
29.50
67.95
73.10
90.56
82.71
84.81
91.16
89.69
40.38
Qwen3-Emb.-8B
75.82
46.05
36.58
81.81
77.73
95.44
90.02
90.84
93.93
93.92
51.89
Vision-based document retrievers
ColPali-v1.3
85.42
84.36
59.86
86.83
77.51
97.27
94.87
95.79
96.47
88.44
72.77
Appendix
Table 13: Per-subset page retrieval on ViDoRe V1 using nDCG@10 ( ×100 ). The Avg. column is the unweighted mean across the ten published subsets. Bold and underlined values denote the best and second-best results.
Method
Avg.
Biomedical
Economics
ESG-Human
ESG
Text-based document retrievers
BM25
21.61
15.79
16.28
35.58
18.79
BGE-M3
49.75
44.71
48.79
58.27
47.21
Qwen3-Emb.-8B
55.28
59.56
50.98
59.61
50.96
Vision-based document retrievers
ColPali-v1.3
57.81
59.91
49.42
63.04
58.85
Appendix
Table 14: Per-subset page retrieval on ViDoRe V2 using nDCG@10 ( ×100 ) under the published all-language protocol. The Avg. column is the unweighted mean across the four subsets. Bold and underlined values denote the best and second-best results.
Method
Avg.
CS
Ene.
Fin-EN
Fin-FR
HR
Ind.
Pha.
Phy.
Text-based document retrievers
BM25
38.29
63.89
15.64
50.75
11.87
49.35
45.76
55.27
13.75
BGE-M3
44.63
63.63
50.09
43.86
24.24
45.35
39.02
54.63
36.21
Qwen3-Emb.-8B
53.93
73.78
59.09
56.49
36.01
53.70
44.94
62.11
45.31
Vision-based document retrievers
ColPali-v1.3
48.86
73.14
46.09
45.02
22.61
54.57
47.89
59.38
42.16
Appendix
Table 15: Per-domain page retrieval on ViDoRe V3 using nDCG@10 ( ×100 ) and the official page judgments. The Avg. column is the unweighted mean across the eight domains. Bold and underlined values denote the best and second-best results.
Figure 3: Two examples from the training set. Each training record contains only a page–query–answer triplet and query-independent layout bounding boxes generated by MinerU; no query-to-region relevance annotations are used.
Figure 4: ViDoRe-V3-Region construction for an example query. (a) The original ViDoRe V3 annotation is a free-form evidence box covering three paragraphs. (b) MinerU, run without access to the query, divides the page into persistent layout regions. (c) The VLM independently scores each region that intersects the original evidence box. (d) Green boxes mark regions with scores of at least 4 as provisional positives. Two human annotators subsequently review every scored candidate and determine the final qrels.
Figure 5: Top-ranked fixed candidates from GARRET, ColQwen2.5, and RegionRAG. Top : for an example from the pharmaceuticals subset of ViDoRe V3, the first relevant candidate appears at rank 1 for GARRET, rank 4 for ColQwen2.5, and rank 5 for RegionRAG. Bottom : for an example from the computer-science subset, the first relevant candidate appears at rank 2 for GARRET, rank 5 for ColQwen2.5, and rank 6 for RegionRAG.
Figure 6: Query–page spatial similarity for an example from the pharmaceuticals subset of ViDoRe V3. Panel (a) shows the reference page, with the evidence outlined by the teal box; panels (b)–(d) show similarity maps for GARRET, ColQwen2.5, and RegionRAG, respectively. Each cross marks the peak similarity for that method. GARRET’s peak falls inside the reference evidence, whereas both baselines peak outside it.
Figure 7: Query–page spatial similarity for an example from the computer-science subset of ViDoRe V3. Panel (a) shows the reference page, with the evidence outlined by the teal box; panels (b)–(d) show similarity maps for GARRET, ColQwen2.5, and RegionRAG, respectively. Each cross marks the peak similarity for that method. GARRET’s peak falls inside the reference passage, whereas both baselines peak elsewhere.
Annual reports, diligence packs, and infographic dashboards bury numbers in page images: axes, cell grids, and footnotes that OCR pipelines flatten and that page-level visual retrievers still treat as interchangeable in-context examples. We keep a frozen Qwen2.5-VL-7B-Instruct generator and a ColPali / VisRAG-Ret page index, and insert three modules. A capability-aware visual router (CAVR) tags each retrieved page as text, table, chart, layout, or mixed and mixes specialist experts before generation. Weak-to-strong page selection (WSPS) distils a frozen 7B answerability teacher into a 3B selection head so ranking is no longer a single InfoNCE score. Visual evidence threading (VET) builds layout-anchored paths of length at most three and lets the generator read the thread rather than a flat top-k list. On gold-page DocVQA / ChartQA / InfographicVQA the 7B system reaches 96.3 / 90.1 / 85.4. Under the VisRAG top-3 protocol the mean generation accuracy is 62.74 versus 59.39 for the same backbone with concatenation. On MMLongBench-Doc retrieve-then-read, F1 moves from 19.2 to 22.6 and multi-page accuracy from 16.4 to 21.2. ViDoRe nDCG@5 after WSPS reranking is 83.6, with TAT-DQA financial reports at 70.4.
Multi-modal retrieval-augmented generation (MM-RAG) relies heavily on re-rankers to surface the most relevant evidence for image-question queries. However, standard re-rankers typically process the full query image as a global embedding, making them susceptible to visual distractors (e.g., background clutter) that skew similarity scores. We propose Region-R1, a query-side region cropping framework that formulates region selection as a decision-making problem during re-ranking, allowing the system to learn to retain the full image or focus only on a question-relevant region before scoring the retrieved candidates. Region-R1 learns a policy with a novel region-aware group relative policy optimization (r-GRPO) to dynamically crop a discriminative region. Across two challenging benchmarks, E-VQA and InfoSeek, Region-R1 delivers consistent gains, achieving state-of-the-art performances by increasing conditional Recall@1 by up to 20%. These results show the great promise of query-side adaptation as a simple but effective way to strengthen MM-RAG re-ranking.
Visual document retrieval is a critical component of multimodal retrieval-augmented generation, aiming to identify query-relevant pages from document collections where evidence is distributed across text, layout, charts, and visual structures. Recent efforts toward finer-grained supervision primarily rely on textual descriptions or localized visual regions as evidence proxies. However, such supervision signals may either overlook complex visual structures or provide incomplete and inaccurate representations of the underlying evidence. To address these limitations, we propose ConceptFormer, a latent concept representation learning framework for visual document retrieval. ConceptFormer models query-relevant evidence as continuous, query-conditioned latent concepts that explicitly bridge localized visual evidence and semantic relevance, without requiring either textual intermediate representations or direct reliance on raw visual annotations. During training, ConceptFormer employs a strong vision-language model to dynamically determine the number of latent concept tokens and uses these concepts as an intermediate representation to bridge the semantic gap between queries and documents, thereby guiding the learning of the embedding space. Experiments on diverse visual document retrieval benchmarks demonstrate that ConceptFormer achieves 16.7% and 22.1% relative improvements in average NDCG@10 over the strongest visual retrieval baseline and the strongest OCR-based text retrieval baseline, respectively. Further analysis reveals that latent concepts effectively connect localized visual evidence with semantic relevance, enabling the retriever to capture both fine-grained textual cues and complex document-level visual structures while preserving strong retrieval alignment. Codes and data are available at https://github.com/Neuir/ConceptFormer.
Chunyi Peng, Zhipeng Xu, Yukun Yan +9
Northeastern University Shenyang, China · Tsinghua University Beijing, China · Peking University Beijing, China