GARRET: Generative Feedback for Region Retrieval in Visual Documents

Weijian Jian; Siyang Chen; Yuguang Yang; Yajun Zhang; Chunyu Xie; Dawei Leng; Yuhui Yin

cs.CVOct 20, 2026

GARRET: Generative Feedback for Region Retrieval in Visual Documents

Authors: Weijian Jian, Siyang Chen, Yuguang Yang, Yajun Zhang, Chunyu Xie, Dawei Leng, Yuhui Yin

Organizations: AI Research · SAIS, UCAS · CASIA · SEIE, BUAA

Abstract

Multimodal retrieval-augmented generation over visually rich documents requires evidence at a finer granularity than entire pages. Existing visual retrievers typically index or return full pages, whereas training region-level retrievers often requires costly query-to-region annotations or noisy model-generated labels. We introduce GARRET, a region-retrieval framework trained from (page, question, answer) triplets through differentiable generative feedback. A shared context-preserving retriever produces page and region multi-vector representations. During training, their retrieval scores define a soft routing distribution over candidate region crops. An auxiliary vision-language model attends to each crop separately, aggregates crop-specific contexts using the routing weights, and predicts the question followed by the answer under a next-token prediction objective. Gradients through the routing distribution assign query-conditioned credit to the indexed region representations, without requiring query-to-region annotations. At inference, the auxiliary model is removed. GARRET also reuses the contextualized region representations, together with page summary tokens, to construct a compact page representation. We introduce ViDoRe-V3-Region, a corpus-wide benchmark containing 200,682 fixed candidate regions across eight domains. GARRET achieves a macro nDCG@10 of 0.4126, outperforming the strongest visual baseline by 2.94 points. On page retrieval, it remains comparable to ColQwen2.5-v0.2 while indexing 47.8% fewer vectors per page.

Figures & tables

Figure 1: Overview of the GARRET framework and its performance. Left: During training, the retriever computes routing weights for candidate regions. An auxiliary VLM uses these weights via in-batch cross-attention to process isolated visual crops. By predicting the paired question and answer, the VLM backpropagates a next-token prediction (NTP) loss as generative feedback, teaching the retriever latent region relevance. Right: GARRET achieves state-of-the-art region retrieval across eight domains in ViDoRe V3 while requiring significantly fewer indexed vectors than existing baselines.

Figure 2: Architecture of GARRET’s context-preserving multi-granularity retriever. A specialized attention mask allows region tokens to absorb global page context without mutual interference. After projection, base page embeddings are discarded. The retained region and summary embeddings form a highly compact representation, enabling both fine-grained region retrieval and efficient page-level retrieval.

Text-based Document Retrievers
Method	Avg.	CS	Ene.	Fin-EN	Fin-FR	HR	Ind.	Pha.	Phy.	Vec./reg.
BM25	16.24	25.93	5.97	19.67	5.47	28.70	18.28	21.84	4.05	–
BGE-M3	25.26	33.32	29.63	26.17	12.57	35.51	24.45	22.17	18.28	1.0
Qwen3-Emb.-8B	35.22	49.97	38.17	35.09	19.20	38.67	31.72	33.20	35.77	1.0
Vision-based Document Retrievers
ColPali-v1.3	31.51	50.15	28.51	27.98	14.26	39.74	31.18	30.47	29.75	1031.0

Table 1: Region retrieval on ViDoRe-V3-Region (nDCG@10,

\times 100

). GARRET achieves the best average performance with the fewest indexed vectors among dense visual retrievers. Bold and underlined values denote the best and second-best results.

Text-based regions
Method	nDCG		Layout Recall
	@5	@10	R@1	R@5	R@10
BM25-OCR	32.73	34.65	20.82	36.00	42.17
BM25-VLM	41.68	44.21	28.72	50.78	58.70
BGE-M3	42.48	45.58	27.90	50.88	59.75
Qwen3-Emb.-8B	42.14	45.32	24.87	47.98	56.73

Table 2: Region retrieval on MMDocIR. All metrics are reported on a

\times 100

scale. GARRET achieves the best nDCG and layout recall at all evaluated cutoffs, demonstrating strong cross-benchmark generalization. Bold and underlined values denote the best and second-best results.

Text-based Document Retrievers
Method	V1	V2	V3	Vec./page
BM25	69.94	21.61	38.29	-
BGE-M3	68.27	49.75	44.63	1.0
Qwen3-Emb.-8B	75.82	55.28	53.93	1.0
Vision-based Document Retrievers
ColPali-v1.3	85.42	57.81	48.86	1031.0

Table 3: Page retrieval on ViDoRe V1–V3 using nDCG@10 (

\times 100

). GARRET remains competitive with the strongest visual retrievers while using substantially fewer indexed vectors per page. Bold and underlined values denote the best and second-best results.

Method / context	Units	Area/P5	Acc
Page-level retrieval
BM25	5 pages	100%	27.53
BGE-M3	5 pages	100%	33.85
Qwen3-Emb.-8B	5 pages	100%	36.71
ColPali-v1.3	5 pages	100%	33.51
ColQwen2.5-v0.2	5 pages	100%	37.00

Table 4: RAG results on ViDoRe V3. Page-level methods use the top-5 retrieved pages, while GARRET uses the top-

k

retrieved layout regions. Area/P5 denotes retrieved area relative to five full pages. Acc is reported as a percentage.

Generative-target controls
Variant	Region	Page	Vectors/page
GARRET	41.26	57.29	392.9
w/o generative feedback	36.33	57.04	392.9
Answer target only	41.02	56.92	392.9
Question target only	39.98	56.89	392.9
Model architecture variants

Table 5: Ablation studies on ViDoRe V3 using nDCG@10 (

\times 100

). Generative feedback primarily improves region retrieval, while joint question–answer generation and the full shared architecture yield the best performance. Bold and underlined values denote the best and second-best results.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Setting
Retriever / auxiliary VLM initialization	Qwen2.5-VL-3B-Instruct
Document embedding dimension	128
Retriever LoRA	rank 64, $\alpha=128$
Auxiliary VLM LoRA	rank 32, $\alpha=64$
Page-summary tokens	100
Region routing score	$s_{R}+s_{D}$

Appendix

Table 6: Implementation settings for the full GARRET model. These hyperparameters are used throughout the experiments unless an ablation explicitly changes them.

Statistic	Value
Domains	8
Pages	19,252
Queries	2,419
Fixed layout candidates	200,682

Appendix

Table 7: Statistics of ViDoRe-V3-Region . MinerU extracts 200,682 query-independent layout candidates from 19,252 pages across eight domains.

Summary tokens	Vectors/page	Region	Page
0	292.9	40.06	55.82
50	342.9	40.95	56.59
100	392.9	41.26	57.29
150	442.9	41.30	57.20

Appendix

Table 9: Effect of the page-summary-token budget on ViDoRe V3 nDCG@10 (

\times 100

). We use 100 tokens in the full model, as indicated by the bold token budget. Bold and underlined metric values denote the best and second-best results, respectively.

Hard	Random	Total pages	Region	Page
0	1	2	38.51	57.08
0	2	3	39.42	57.32
1	0	2	40.39	57.15
2	0	3	40.01	57.21
1	1	3	41.26	57.29

Appendix

Table 10: Ablation of negative-page sampling for the local candidate pool on ViDoRe V3 nDCG@10 (

\times 100

). “Total pages” includes the paired page, which is present in every row. The full model uses one hard-negative and one random-negative page.

Method family	Candidate input	Candidate representation	Region score
BM25	Parsed candidate text	Sparse lexical index	Lexical matching score
BGE-M3 / Qwen3-Embedding	Parsed candidate text	One dense text vector	Dense similarity
ColPali / ColQwen	Exact candidate crop	Independently encoded visual vectors	Late interaction
RegionRAG	Query-dependent predicted regions	Dynamic region output mapped to benchmark boxes	Mapped region score
GARRET	Full page and candidate box	Contextual region token group	$s_{R}+s_{D}$

Appendix

Table 11: Adaptation of each retriever to fixed-candidate region ranking. Text retrievers encode parsed candidate text, ColPali and ColQwen encode isolated candidate crops, and RegionRAG maps its query-dependent outputs to benchmark boxes. GARRET directly scores persistent, page-contextualized candidates.

View	Variant	Avg.	CS	Ene.	Fin-EN	Fin-FR	HR	Ind.	Pha.	Phy.
Region	GARRET	41.26	55.46	42.47	34.05	22.37	46.07	41.93	49.72	38.04
Region	No generative feedback	36.33	50.96	34.92	34.43	19.38	39.78	34.63	44.73	31.80
Page	GARRET	57.29	76.46	60.44	59.10	39.26	61.06	52.71	63.04	46.24
Page	No generative feedback	57.04	76.27	59.66	62.21	37.58	61.02	49.48	64.57	45.51

Appendix

Table 12: Per-domain ablation of generative feedback on ViDoRe V3 using nDCG@10 (

\times 100

). Removing the auxiliary VLM and NTP objective reduces average region retrieval by 4.93 points but page retrieval by only 0.25 points, showing that generative feedback primarily improves fine-grained ranking. Values are point estimates from one trained model per variant.

Text-based document retrievers
Method	Avg.	ArXiv	DocV	InfoV	Shift	Syn-AI	Syn-En	Syn-Gov	Syn-HC	TabFQ	TatD
BM25	69.94	36.65	39.45	68.36	68.29	95.85	90.90	86.52	90.93	61.67	60.73
BGE-M3	68.27	32.83	29.50	67.95	73.10	90.56	82.71	84.81	91.16	89.69	40.38
Qwen3-Emb.-8B	75.82	46.05	36.58	81.81	77.73	95.44	90.02	90.84	93.93	93.92	51.89
Vision-based document retrievers
ColPali-v1.3	85.42	84.36	59.86	86.83	77.51	97.27	94.87	95.79	96.47	88.44	72.77

Appendix

Table 13: Per-subset page retrieval on ViDoRe V1 using nDCG@10 (

\times 100

). The Avg. column is the unweighted mean across the ten published subsets. Bold and underlined values denote the best and second-best results.

Text-based document retrievers
Method	Avg.	Biomedical	Economics	ESG-Human	ESG
BM25	21.61	15.79	16.28	35.58	18.79
BGE-M3	49.75	44.71	48.79	58.27	47.21
Qwen3-Emb.-8B	55.28	59.56	50.98	59.61	50.96
Vision-based document retrievers
ColPali-v1.3	57.81	59.91	49.42	63.04	58.85

Appendix

Table 14: Per-subset page retrieval on ViDoRe V2 using nDCG@10 (

\times 100

) under the published all-language protocol. The Avg. column is the unweighted mean across the four subsets. Bold and underlined values denote the best and second-best results.

Text-based document retrievers
Method	Avg.	CS	Ene.	Fin-EN	Fin-FR	HR	Ind.	Pha.	Phy.
BM25	38.29	63.89	15.64	50.75	11.87	49.35	45.76	55.27	13.75
BGE-M3	44.63	63.63	50.09	43.86	24.24	45.35	39.02	54.63	36.21
Qwen3-Emb.-8B	53.93	73.78	59.09	56.49	36.01	53.70	44.94	62.11	45.31
Vision-based document retrievers
ColPali-v1.3	48.86	73.14	46.09	45.02	22.61	54.57	47.89	59.38	42.16

Appendix

Table 15: Per-domain page retrieval on ViDoRe V3 using nDCG@10 (

\times 100

) and the official page judgments. The Avg. column is the unweighted mean across the eight domains. Bold and underlined values denote the best and second-best results.

Figure 3: Two examples from the training set. Each training record contains only a page–query–answer triplet and query-independent layout bounding boxes generated by MinerU; no query-to-region relevance annotations are used.

Figure 4: ViDoRe-V3-Region construction for an example query. (a) The original ViDoRe V3 annotation is a free-form evidence box covering three paragraphs. (b) MinerU, run without access to the query, divides the page into persistent layout regions. (c) The VLM independently scores each region that intersects the original evidence box. (d) Green boxes mark regions with scores of at least 4 as provisional positives. Two human annotators subsequently review every scored candidate and determine the final qrels.

Figure 5: Top-ranked fixed candidates from GARRET, ColQwen2.5, and RegionRAG. Top : for an example from the pharmaceuticals subset of ViDoRe V3, the first relevant candidate appears at rank 1 for GARRET, rank 4 for ColQwen2.5, and rank 5 for RegionRAG. Bottom : for an example from the computer-science subset, the first relevant candidate appears at rank 2 for GARRET, rank 5 for ColQwen2.5, and rank 6 for RegionRAG.

Figure 6: Query–page spatial similarity for an example from the pharmaceuticals subset of ViDoRe V3. Panel (a) shows the reference page, with the evidence outlined by the teal box; panels (b)–(d) show similarity maps for GARRET, ColQwen2.5, and RegionRAG, respectively. Each cross marks the peak similarity for that method. GARRET’s peak falls inside the reference evidence, whereas both baselines peak outside it.

Figure 7: Query–page spatial similarity for an example from the computer-science subset of ViDoRe V3. Panel (a) shows the reference page, with the evidence outlined by the teal box; panels (b)–(d) show similarity maps for GARRET, ColQwen2.5, and RegionRAG, respectively. Each cross marks the peak similarity for that method. GARRET’s peak falls inside the reference passage, whereas both baselines peak elsewhere.

Visual Document Retrieval

Visual document retrieval finds relevant pages or regions in image-based documents using visual and multimodal representations.

See all 66 papers

Capability-Routed Visual Retrieval and Evidence Threading for Long-Context Document Question Answering
Sep 7, 2026Amirul Rahman, Aisha Karim, Kenji Nakamura +1Document QA Long-Document VQA
University of Malaya
Region-R1: Reinforcing Query-Side Region Cropping for Multi-Modal Re-Ranking
Apr 7, 2026Chan-Wei Hu, Zhengzhong TuMultimodal Reranking Multimodal RAG
Texas A&M University
ConceptFormer: Learning Adaptive Latent Concepts for Query-Document Alignment in Visual Document Retrieval
Aug 16, 2026Chunyi Peng, Zhipeng Xu, Yukun Yan +9Visual Document Retrieval Representation Learning
Northeastern University Shenyang, China · Tsinghua University Beijing, China · Peking University Beijing, China

Sep 7, 2026J/K move · Enter open · S save

Capability-Routed Visual Retrieval and Evidence Threading for Long-Context Document Question Answering

Amirul Rahman, Aisha Karim, Kenji Nakamura, Yi-Fan Ng

University of Malaya

Document QA Long-Document VQA Multimodal RAG Visual Document Retrieval Document Image Understanding Long-Context VLMs+1

Open paper AskPDF arXiv

Annual reports, diligence packs, and infographic dashboards bury numbers in page images: axes, cell grids, and footnotes that OCR pipelines flatten and that page-level visual retrievers still treat as interchangeable in-context examples. We keep a frozen Qwen2.5-VL-7B-Instruct generator and a ColPali / VisRAG-Ret page index, and insert three modules. A capability-aware visual router (CAVR) tags each retrieved page as text, table, chart, layout, or mixed and mixes specialist experts before generation. Weak-to-strong page selection (WSPS) distils a frozen 7B answerability teacher into a 3B selection head so ranking is no longer a single InfoNCE score. Visual evidence threading (VET) builds layout-anchored paths of length at most three and lets the generator read the thread rather than a flat top-

k

list. On gold-page DocVQA / ChartQA / InfographicVQA the 7B system reaches 96.3 / 90.1 / 85.4. Under the VisRAG top-3 protocol the mean generation accuracy is 62.74 versus 59.39 for the same backbone with concatenation. On MMLongBench-Doc retrieve-then-read, F1 moves from 19.2 to 22.6 and multi-page accuracy from 16.4 to 21.2. ViDoRe nDCG@5 after WSPS reranking is 83.6, with TAT-DQA financial reports at 70.4.

Abstract

Visual Document Retrieval

Explore similar work

Capability-Routed Visual Retrieval and Evidence Threading for Long-Context Document Question Answering

Region-R1: Reinforcing Query-Side Region Cropping for Multi-Modal Re-Ranking

ConceptFormer: Learning Adaptive Latent Concepts for Query-Document Alignment in Visual Document Retrieval