PixSearch: Region-Grounded Retrieval for Knowledgeable Large Multimodal Models
Authors: Jeonghwan Kim, Renjie Tao, Sanat Sharma, Jiaqi Wang, Kai Sun, Zhaojiang Lin, Seungwhan Moon, Lambert Mathias, Anuj Kumar, Heng Ji, Xin Luna Dong
Organizations: Meta Reality Labs · University of Illinois Urbana-Champaign
Multimodal Retrieval-Augmented Generation (MM-RAG) augments visual question answering (VQA) with external knowledge, but many systems rely on modular pipelines that separate visual grounding, query construction, and answer generation. This separation can propagate errors from intermediate detections, masks, or captions into retrieval while adding latency and limiting coordination with ongoing reasoning. These limitations are especially consequential for egocentric images, where questions often concern small, off-center, or long-tail entities that global retrieval can overlook. We propose PixSearch, an end-to-end segmenting Large Multimodal Model (LMM) that unifies region-level visual grounding with retrieval-augmented reasoning. During autoregressive generation, PixSearch learns when to retrieve, how to route queries across text, whole-image, and region-level retrieval, and how to crop the target entity as a visual query using a jointly trained mask decoder. A two-stage supervised fine-tuning regimen with search-interleaved supervision teaches retrieval triggering, query selection, and evidence-grounded answer generation while preserving segmentation ability. On egocentric and entity-centric VQA benchmarks, PixSearch achieves an 18.3% relative accuracy gain on CRAG-MM over whole-image retrieval, with particularly strong gains on egocentric images, while remaining competitive on broader VQA and text-only QA tasks. It also reduces inference latency by 36.4% compared with a cached GroundedSAM-LLaVA pipeline.
Figures & tables
Figure 1: Egocentric images from wearable devices often render small entities. Full-image or caption-only MM-RAG can retrieve noisy evidence, while PixSearch learns when to search, whether to query text, image, or region, and how to reason over retrieved evidence. On CRAG-MM Wang et al. (2025) , PixSearch improves overall accuracy by 18.3% and reduces hallucination by 19.4% relative to whole-image retrieval.
Figure 2: Overview of the proposed PixSearch framework. The model learns to decide when retrieval is needed, how to query (text, whole image, or segmented region), and grounds its answers in retrieved evidence while preserving mask-generation capabilities.
Figure 3: Data construction pipeline for Stage-2 training.
Dataset
Records
Unique IDs
Used
OVEN
7,000
5,277
✓
InfoSeek
7,000
6,045
✓
A-OKVQA
7,000
6,841
✓
OKVQA
7,000
6,078
✓
TextVQA
7,000
6,937
✓
Selected total
35,000
31,178
Table 1: Generated training-data statistics. “Used” indicates inclusion in the documented Stage-2 run. Unique counts refer to source IDs within each dataset, not a deduplicated training mixture.
Model
ADE20K
COCOStuff
MaskFormer
52.70
47.20
PLUM
55.08
49.97
PixSearch (1:9)
9.85 (-82.1%)
11.07 (-77.8%)
PixSearch (5:9)
47.19 (-14.3%)
42.55 (-14.9%)
PixSearch (7:9)
55.98 (+1.6%)
49.81 (-0.3%)
Table 3: Segmentation experiments. (a) Referring expression segmentation performance in gIoU. (b) Semantic segmentation performance in mIoU. PixSearch preserves competitive segmentation ability, with the best performance obtained at the 7:9 Stage-1/Stage-2 sampling ratio. Baselines: LISA ( Lai et al., 2024 ) , GLaMM ( Rasheed et al., 2024 ) , MaskFormer ( Cheng et al., 2021 ) , and PLUM ( Blume et al., 2025 ) .
Model (13B)
TextVQA EM / F1
InfoSeek EM / F1
OVEN EM / F1
LLaVA
22.84 / 26.15
13.94 / 14.98
2.20 / 3.12
PLUM
30.11 / 32.62
15.60 / 16.54
2.64 / 3.81
PixSearch No Search
33.0 / 34.9
18.6 / 19.7
4.0 / 5.6
PixSearch Question
31.2 / 33.3
21.0 / 22.6
5.5 / 7.0
PixSearch Whole Image
32.6 / 34.5
24.6 / 26.4
15.6 / 17.3
PixSearch Full
32.8 / 35.5
24.9 / 27.0
16.2 / 17.4
Table 4: Performance on additional VQA benchmarks. Scores are reported in %. For PixSearch variants, we report mean scores over three independent training seeds. Each cell shows EM / F1; all models have 13B parameters.
Figure 7
Method
Time (s)
p90 (s)
PixSearch-13B (Full)
2.8±0.9
3.3
PixSearch-13B (Whole Image)
2.45±0.72
2.96
GroundedSAM+LLaVA-13B †
4.4±1.3
4.1
GroundedSAM+LLaVA-13B ‡
5.0±3.4
5.9
Table 5: Inference latency. Wall-clock latency on CRAG-MM, reported as mean ± std over five runs. † : cached, GPU-only pipeline. ‡ : pipeline including GPT-4.1 entity extraction.
Training
Acc.
gIoU
Stage-2 SFT only
18.5
14.8
Single-stage joint SFT
32.6
67.4
Two-stage SFT
36.9
78.9
Search-R1-style GRPO
36.9
71.6
Table 6: Training comparison. CRAG-MM accuracy and RefCOCO gIoU (%). Expanded metrics are in Appendix F .
Table 9: Qualitative comparison between baseline and PixSearch -Interleaved outputs. Each example shows the input image (left), the corresponding question and model responses (middle), and the pixel-level grounding mask generated by the model (right). The payloads between the \informationtag \informationend are truncated to show entity names / titles only for brevity.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
PLUM
PixSearch
Backbone
Language model
LLaVA-13B
LLaVA-13B (PLUM init.)
Vision tower
CLIP ViT-L/14
CLIP ViT-L/14
Mask decoder
SAM ViT-H
SAM ViT-H
Training schedule
Input resolution
10242
10242
Appendix
Table 11: Hyperparameters for PLUM and PixSearch. Stage-2 settings follow the documented completed run.
Question Type
Example Question
OCR Read
“What does it say near the tail of the plane?”
OCR + Visual Reasoning
“Which team is winning the game?”
Multi-hop External Knowledge Reasoning
“When was the soft-drink company shown first created?”
Fine-grained Entity Identification
“What class of animal is this creature?”
Visual Reasoning – Attribute
“What is the color of the car in the background?”
Visual Reasoning – Counting
“How many cars are there in the image?”
Appendix
Table 12: Overview of the ten visual question types used in Stage-2.
Image File
Question
Retrieval
cragmm/4ec6f8ae.png
How many hybrid variations of this car were there in 2024?
Yes
cragmm/08629717.png
Is that drink good for my gut health?
Yes
cragmm/fb2fed47.png
How many arms does this statue typically have?
Yes
cragmm/1c613a06.png
Translate this.
No
cragmm/569a3617.png
Which station has more tracks, this one or Penn Station?
Yes
cragmm/a97e2470.png
Where was the designer who developed this car originally from?
Yes
Appendix
Table 13: Question Selection Examples. We show representative questions, whether retrieval was needed or not, and the associated image filenames. The Image File paths were truncated to have only the prefix of the image path for brevity.
Image File
Original Question
Sub-Questions
cragmm/4dcc84dc.png
When did the king of that country become king?
1) Who is the king of Spain? 2) When did the king of Spain become king?
cragmm/da33192e.png
What’s the capacity of this arena?
1) What is this place? 2) What’s the capacity of this arena?
cragmm/f73ab93c.png
Where was the first sign accompanying this erected?
1) Where was the first pedestrian crossing signal erected?
cragmm/878088c7.png
What’s the ideal temperature for this plant?
1) What is this plant? 2) What is the ideal temperature for this plant?
cragmm/b03b7dd6.png
How long can I use it without turning it off?
1) What is the model name of this generator? 2) How long can I use this generator?
cragmm/ec87776d.png
Translate this into English.
1) Translate this into English.
Appendix
Table 14: Question Decomposition Examples. Each example lists the original question, whether it was decomposed, and the resulting sub-questions.
Image File
Question + Model Answer
Ground-Truth Answer
cragmm/4dcc84dc.png
Q: When did the king of that country become king? A: The paper says “… independence from Spain.” Since Mexico has no king, this refers to Spain. The king of Spain <search> is King Felipe VI. Felipe VI came to the throne <search> on June 19, 2014.
June 19, 2014
cragmm/da33192e.png
Q: What’s the capacity of this arena? A: The place shown <search> is Madison Square Garden. It can house <search> between 18,006 and 20,789 people.
18,006–20,789 depending on event
cragmm/f73ab93c.png
Q: Where was the first sign accompanying this erected? A: The first pedestrian crossing signal was erected in <search> Bridge Street, Westminster, London, in December 1868.
Bridge Street, Westminster, London (1868)
cragmm/878088c7.png
Q: What’s the ideal temperature for this plant? A: The plant <search> appears to be Ravenea rivularis. Ravenea rivularis prefers <search> 65–85°F.
Prefers 65–85°F
cragmm/b03b7dd6.png
Q: How long can I use it without turning it off? A: The generator appears to be <search> a Firman Tri Fuel Generator. Its usage time is <search> a maximum of 12 hours with a 2–3 hour recommended break.
12 hours, with 2–3 hour breaks
cragmm/ec87776d.png
Q: Translate this into English. A: It translates to: “What mixer is used in a Paloma?”
“What mixer is used in a Paloma?”
Appendix
Table 15: Response Generation Examples. Each example includes the question, the model-generated answer (which includes <search> calls), and the ground-truth answer.
Image File
Question + <search> -Interleaved Response
Query Types
Queries
cragmm/4dcc84dc.png
Q: When did the king of that country become king? A: The king of Spain <search> is King Felipe VI. </search> He came to the throne <search> on June 19, 2014. </search>
TEXT, TEXT
1) Who is the king of Spain? 2) When did King Felipe VI become king?
cragmm/da33192e.png
Q: What’s the capacity of this arena? A: The place shown <search> is Madison Square Garden. </search> It can house <search> between 18,006 and 20,789 people. </search>
IMAGE, TEXT
1) image 2) What is the capacity of this arena?
cragmm/f73ab93c.png
Q: Where was the first sign accompanying this erected? A: The first pedestrian crossing signal was erected <search> in Bridge Street, Westminster, London. </search>
TEXT
Where was the first pedestrian crossing signal erected?
cragmm/878088c7.png
Q: What’s the ideal temperature for this plant? A: The plant <search> appears to be Ravenea rivularis. </search> It prefers <search> 65–85°F. </search>
REGION, TEXT
1) plant 2) Ideal temperature for Ravenea rivularis
cragmm/b03b7dd6.png
Q: How long can I use it without turning it off? A: The generator <search> appears to be a Firman Tri Fuel Generator. </search> Its usage time is <search> a maximum of 12 hours with breaks. </search>
REGION, TEXT
1) generator 2) Use time for Firman Tri Fuel Generator
cragmm/ec87776d.png
Q: Translate this into English. A: The sentence translates to: “What mixer is used in a Paloma?”
—
—
Appendix
Table 16: Query-Type Assignment Examples. Each example includes the original question, the model’s <search> -interleaved response, and the corresponding query types and queries.
Training
Acc.
R@1
R@10
gIoU
mIoU
Stage-2 SFT only
18.5
0.05
0.19
14.8
10.2
Joint SFT
32.6
0.18
0.35
67.4
46.8
Two-stage SFT
36.9
0.22
0.41
78.9
55.98
Search-R1-style GRPO
36.9
0.20
0.38
71.6
50.3
Appendix
Table 17: Expanded training comparison. Accuracy is on CRAG-MM, gIoU on RefCOCO, and mIoU on ADE20K; these three metrics are percentages. Retrieval recall is reported on a [0,1] scale.
Component
Full
Whole
Cached
+GPT
Model inference (s)
1.76
1.72
1.84
2.44
Mask/crop (s)
0.23
—
1.60
1.60
Query encoding (s)
0.12
0.11
0.13
0.13
Retrieval (s)
0.69
0.62
0.83
0.83
Total (s)
2.80
2.45
4.40
5.00
p90 (s)
3.30
2.96
4.10
5.90
Appendix
Table 18: Latency components on CRAG-MM. Full and Whole denote PixSearch with full and whole-image retrieval, respectively. Cached and +GPT denote the cached and GPT-4.1-assisted GroundedSAM–LLaVA pipelines.
Vision-centric retrieval for VQA requires retrieving images to supply missing visual cues and integrating them into the reasoning process. However, selecting the right images and integrating them effectively into the model's reasoning remains challenging. To address this challenge, we propose R3G, a modular Reasoning-Retrieval-Reranking framework. It first produces a brief reasoning plan that specifies the required visual cues, then adopts a two-stage strategy, with coarse retrieval followed by fine-grained reranking, to select evidence images. On MRAG-Bench, R3G improves accuracy across six MLLM backbones and nine sub-scenarios, achieving state-of-the-art overall performance. Ablations show that sufficiency-aware reranking and reasoning steps are complementary, helping the model both choose the right images and use them well. We release code and data at https://github.com/czh24/R3G.
Zhuohong Chen, Zhengxian Wu, Zirui Liao +6
1The Shenzhen International Graduate School, Tsinghua University, China · School of Computer Science and Information Engineering, Hefei University of Technology, China · 2State Key Laboratory of Nuclear Power Safety Technology and Equipment, China
Knowledge-based visual question answering (KB-VQA) requires vision-language models to understand images and use external knowledge, especially for rare entities and long-tail facts. Most existing retrieval-augmented generation (RAG) methods adopt a fixed pipeline that sequentially retrieves information, filters it, and then produces an answer. Such a design makes it difficult to adapt to diverse question types. Moreover, it separates retrieval from reasoning, making it hard for the model to decide when to search, how to refine queries, or when to stop. As a result, the retrieved evidence is often poorly aligned with the question. To address these limitations, we reformulate KB-VQA as a search-agent problem and model the solving process as a multi-step decision-making procedure. At each step, the agent selects one of four actions-Answer, Image Retrieval, Text Retrieval, and Caption-based on its current information state. We further design an automated pipeline to collect multi-step trajectories that record the agent's reasoning process, tool usage, and intermediate decisions. These trajectories are then used as supervision for fine-tuning. Experiments on InfoSeek and E-VQA demonstrate that our method achieves state-of-the-art performance, consistently outperforming prior baselines and confirming the effectiveness of our framework.
Zhuohong Chen, Zhenxian Wu, Yunyao Yu +6
Tsinghua University · Hefei University of Technology · University of Arizona
Multi-modal retrieval-augmented generation (MM-RAG) relies heavily on re-rankers to surface the most relevant evidence for image-question queries. However, standard re-rankers typically process the full query image as a global embedding, making them susceptible to visual distractors (e.g., background clutter) that skew similarity scores. We propose Region-R1, a query-side region cropping framework that formulates region selection as a decision-making problem during re-ranking, allowing the system to learn to retain the full image or focus only on a question-relevant region before scoring the retrieved candidates. Region-R1 learns a policy with a novel region-aware group relative policy optimization (r-GRPO) to dynamically crop a discriminative region. Across two challenging benchmarks, E-VQA and InfoSeek, Region-R1 delivers consistent gains, achieving state-of-the-art performances by increasing conditional Recall@1 by up to 20%. These results show the great promise of query-side adaptation as a simple but effective way to strengthen MM-RAG re-ranking.