Region-R1: Reinforcing Query-Side Region Cropping for Multi-Modal Re-Ranking

Chan-Wei Hu; Zhengzhong Tu

cs.CVApr 7, 2026

Region-R1: Reinforcing Query-Side Region Cropping for Multi-Modal Re-Ranking

Authors: Chan-Wei Hu, Zhengzhong Tu

Organizations: Texas A&M University

Abstract

Multi-modal retrieval-augmented generation (MM-RAG) relies heavily on re-rankers to surface the most relevant evidence for image-question queries. However, standard re-rankers typically process the full query image as a global embedding, making them susceptible to visual distractors (e.g., background clutter) that skew similarity scores. We propose Region-R1, a query-side region cropping framework that formulates region selection as a decision-making problem during re-ranking, allowing the system to learn to retain the full image or focus only on a question-relevant region before scoring the retrieved candidates. Region-R1 learns a policy with a novel region-aware group relative policy optimization (r-GRPO) to dynamically crop a discriminative region. Across two challenging benchmarks, E-VQA and InfoSeek, Region-R1 delivers consistent gains, achieving state-of-the-art performances by increasing conditional Recall@1 by up to 20%. These results show the great promise of query-side adaptation as a simple but effective way to strengthen MM-RAG re-ranking.

Figures & tables

Figure 1: Overview of query-side region cropping approach. Conventional re-rankers treat the query as fixed and only re-order candidates. Our method instead automatically adapt the query by selecting an informative region or keeping the original query before scoring candidates.

Method	E-VQA						InfoSeek
Method	MRR	NDCG	R@1	R@5	R@10	R@20	MRR	NDCG	R@1	R@5	R@10	R@20
EVA-CLIP	0.224	0.289	14.2	33.4	47.4	50.8	0.553	0.614	46.3	71.2	77.3	81.7
ReflectiVA	-	-	15.6	36.1	-	49.8	-	-	56.1	77.6	-	86.4
Wiki-LLaVA	-	-	3.3	-	9.9	13.2	-	-	36.9	-	66.1	77.9
mR 2 AG	-	-	-	-	-	-	-	-	38.0	-	65.0	71.0
Random	0.060	0.160	1.2	5.9	20.7	50.8	0.193	0.323	10.6	27.1	40.7	81.7

Table 1: Re-ranking results on E-VQA and InfoSeek with top-20 candidates. We report MRR, NDCG, and Recall@

\{1,5,10,20\}

. Best in bold, second-best underlined.

Methods	E-VQA			InfoSeek
Methods	CondR@1	CondR@5	CondR@10	CondR@1	CondR@5	CondR@10
EVA-CLIP Sun et al. (2023)	0.28	0.66	0.93	0.57	0.87	0.95
ReflectiVA Cocchi et al. (2025)	0.31	0.72	-	0.65	0.90	-
Wiki-LLaVA Caffagni et al. (2024)	0.25	-	0.75	0.47	-	0.85
mR 2 AG Zhang et al. (2024)	-	-	-	0.54	-	0.92
Random	0.02	0.12	0.41	0.13	0.33	0.49

Table 2: Conditional recall on E-VQA and InfoSeek. CondR@

K

denotes CondRecall@

K

(Eq. 10 ). Best in bold, second-best underlined.

Panel of Table 3 — Table 3: Qualitative examples. Left-to-right: original query, top-1 from EchoSight Yan and Xie (2024) and OMGM Yang et al. (2025) after re-ranking , our query with region cropping, and the ground-truth positive candidate. Our region cropping flips the rank-1 prediction to the correct positive by removing distractors and emphasizing question-relevant region.

Panel of Table 4 — Table 4: Quantitative results of our region cropping (RC) behavior across two datasets. (a,b) show cases where the original query already ranks the positive at #1 on InfoSeek and E-VQA, respectively. (c,d) show harder cases where the original rank is $>1$ . RC rate indicates how often does the model choose the crop a region.

Panel of Figure 2 — Figure 2: Ablation on the margin term in the reward design.

Reward Component	InfoSeek	E-VQA
$\Delta$ MRR only	0.611	0.408
+ $\Delta$ NDCG	0.613 ( $\uparrow$ )	0.425 ( $\uparrow$ )
+ $\Delta$ Rank	0.613 (-)	0.426 ( $\uparrow$ )
+ $\Delta$ Margin (Full)	0.706	0.473

Table 5: Ablation on reward design. Each row adds one component to the region-cropping reward (Eq. 7 ).

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

⬇ You are an intelligent region cropping assistant for re - ranking tasks . Given an image and the user ’ s question , your task is to decide whether finding a specific region would help remove the redundant information and rank more relevant information to a higher rank . # Instructions 1. Carefully analyze the image in the context of the user ’ s question . 2. Based on the user ’ s question , decide whether region selection would improve re - ranking accuracy among many candidates , and push the most relevant candidate to the rank 1. You can decide : - If you think a specific region is most relevant to answering the question , select it to focus on it . - If you think the full image is already optimal for the question , output " FULL ". 3. Output your decision with " FULL " or " REGION ", if your decision is " REGION ", you have to specify a region . # Tools < tools > {" type ":" function "," function ":{" name ":" image_zoom_in_tool "," description ":" Zoom in on a specific region of an image ."," parameters ":{" type ":" object "," properties ":{" bbox_2d ":{" type ":" array "," items ":{" type ":" number "}," minItems ":4," maxItems ":4," description ":" Bounding box as [ x1 , y1 , x2 , y2 ], where ( x1 , y1 ) is top - left and ( x2 , y2 ) is bottom - right ."}," label ":{" type ":" string "}}," required ":[" bbox_2d "]}}} </ tools > # How to call a tool Return a json object with function name and arguments within < tool_call ></ tool_call > XML tags : < tool_call > {" name ": < function - name >, " arguments ": < args - json - object >} </ tool_call > # Output format {" Decision ": " FULL " or " REGION ", " Tool ": < Call the tool with parameters if the decision is " REGION ">} Again , if you call the tool , you MUST follow the format exactly as specified . Otherwise , I will be unable to parse your response .

Appendix

Table 6: System Prompt

Multimodal Reranking

Multimodal reranking reorders retrieved candidates using joint visual, textual, and other modality-specific evidence.

See all 35 papers

PixSearch: Region-Grounded Retrieval for Knowledgeable Large Multimodal Models
Jan 27, 2026Jeonghwan Kim, Renjie Tao, Sanat Sharma +8Multimodal RAG Visual Question Answering
Meta Reality Labs · University of Illinois Urbana-Champaign
R3G: A Reasoning-Retrieval-Reranking Framework for Vision-Centric Answer Generation
Jan 25, 2026Zhuohong Chen, Zhengxian Wu, Zirui Liao +6Multimodal RAG Visual Question Answering
1The Shenzhen International Graduate School, Tsinghua University, China · School of Computer Science and Information Engineering, Hefei University of Technology, China · 2State Key Laboratory of Nuclear Power Safety Technology and Equipment, China
EviRank: Structured Relevance Evidence for Multimodal Image Re-ranking
Aug 21, 2026Enjun Du, Siyi Liu, Zirong Chen +8Multimodal Reranking Multimodal IR
The Hong Kong University of Science and Technology (Guangzhou) · Tencent Yuanbao · The University of Hong Kong

Jan 27, 2026J/K move · Enter open · S save

PixSearch: Region-Grounded Retrieval for Knowledgeable Large Multimodal Models

Jeonghwan Kim, Renjie Tao, Sanat Sharma, Jiaqi Wang+7

Meta Reality Labs · University of Illinois Urbana-Champaign

Multimodal RAG Visual Question Answering Large Vision-Language Models Vision-Language Grounding Image Retrieval

Open paper AskPDF arXiv

Multimodal Retrieval-Augmented Generation (MM-RAG) augments visual question answering (VQA) with external knowledge, but many systems rely on modular pipelines that separate visual grounding, query construction, and answer generation. This separation can propagate errors from intermediate detections, masks, or captions into retrieval while adding latency and limiting coordination with ongoing reasoning. These limitations are especially consequential for egocentric images, where questions often concern small, off-center, or long-tail entities that global retrieval can overlook. We propose PixSearch, an end-to-end segmenting Large Multimodal Model (LMM) that unifies region-level visual grounding with retrieval-augmented reasoning. During autoregressive generation, PixSearch learns when to retrieve, how to route queries across text, whole-image, and region-level retrieval, and how to crop the target entity as a visual query using a jointly trained mask decoder. A two-stage supervised fine-tuning regimen with search-interleaved supervision teaches retrieval triggering, query selection, and evidence-grounded answer generation while preserving segmentation ability. On egocentric and entity-centric VQA benchmarks, PixSearch achieves an 18.3% relative accuracy gain on CRAG-MM over whole-image retrieval, with particularly strong gains on egocentric images, while remaining competitive on broader VQA and text-only QA tasks. It also reduces inference latency by 36.4% compared with a cached GroundedSAM-LLaVA pipeline.

Abstract

Multimodal Reranking

Explore similar work

PixSearch: Region-Grounded Retrieval for Knowledgeable Large Multimodal Models

R3G: A Reasoning-Retrieval-Reranking Framework for Vision-Centric Answer Generation

EviRank: Structured Relevance Evidence for Multimodal Image Re-ranking