PixSearch: Region-Grounded Retrieval for Knowledgeable Large Multimodal Models

Jeonghwan Kim; Renjie Tao; Sanat Sharma; Jiaqi Wang; Kai Sun; Zhaojiang Lin; Seungwhan Moon; Lambert Mathias

cs.CVJan 27, 2026

PixSearch: Region-Grounded Retrieval for Knowledgeable Large Multimodal Models

Authors: Jeonghwan Kim, Renjie Tao, Sanat Sharma, Jiaqi Wang, Kai Sun, Zhaojiang Lin, Seungwhan Moon, Lambert Mathias,

Organizations: Meta Reality Labs · University of Illinois Urbana-Champaign

Abstract

Multimodal Retrieval-Augmented Generation (MM-RAG) augments visual question answering (VQA) with external knowledge, but many systems rely on modular pipelines that separate visual grounding, query construction, and answer generation. This separation can propagate errors from intermediate detections, masks, or captions into retrieval while adding latency and limiting coordination with ongoing reasoning. These limitations are especially consequential for egocentric images, where questions often concern small, off-center, or long-tail entities that global retrieval can overlook. We propose PixSearch, an end-to-end segmenting Large Multimodal Model (LMM) that unifies region-level visual grounding with retrieval-augmented reasoning. During autoregressive generation, PixSearch learns when to retrieve, how to route queries across text, whole-image, and region-level retrieval, and how to crop the target entity as a visual query using a jointly trained mask decoder. A two-stage supervised fine-tuning regimen with search-interleaved supervision teaches retrieval triggering, query selection, and evidence-grounded answer generation while preserving segmentation ability. On egocentric and entity-centric VQA benchmarks, PixSearch achieves an 18.3% relative accuracy gain on CRAG-MM over whole-image retrieval, with particularly strong gains on egocentric images, while remaining competitive on broader VQA and text-only QA tasks. It also reduces inference latency by 36.4% compared with a cached GroundedSAM-LLaVA pipeline.

Figures & tables

Figure 1: Egocentric images from wearable devices often render small entities. Full-image or caption-only MM-RAG can retrieve noisy evidence, while PixSearch learns when to search, whether to query text, image, or region, and how to reason over retrieved evidence. On CRAG-MM Wang et al. (2025) , PixSearch improves overall accuracy by 18.3% and reduces hallucination by 19.4% relative to whole-image retrieval.

Figure 2: Overview of the proposed PixSearch framework. The model learns to decide when retrieval is needed, how to query (text, whole image, or segmented region), and grounds its answers in retrieved evidence while preserving mask-generation capabilities.

Figure 3: Data construction pipeline for Stage-2 training.

Dataset	Records	Unique IDs	Used
OVEN	7,000	5,277	✓
InfoSeek	7,000	6,045	✓
A-OKVQA	7,000	6,841	✓
OKVQA	7,000	6,078	✓
TextVQA	7,000	6,937	✓
Selected total	35,000	31,178

Table 1: Generated training-data statistics. “Used” indicates inclusion in the documented Stage-2 run. Unique counts refer to source IDs within each dataset, not a deduplicated training mixture.

Model	ADE20K	COCOStuff
MaskFormer	52.70	47.20
PLUM	55.08	49.97
PixSearch (1:9)	9.85 (-82.1%)	11.07 (-77.8%)
PixSearch (5:9)	47.19 (-14.3%)	42.55 (-14.9%)
PixSearch (7:9)	55.98 (+1.6%)	49.81 (-0.3%)

Table 3: Segmentation experiments. (a) Referring expression segmentation performance in gIoU. (b) Semantic segmentation performance in mIoU. PixSearch preserves competitive segmentation ability, with the best performance obtained at the 7:9 Stage-1/Stage-2 sampling ratio. Baselines: LISA ( Lai et al., 2024 ) , GLaMM ( Rasheed et al., 2024 ) , MaskFormer ( Cheng et al., 2021 ) , and PLUM ( Blume et al., 2025 ) .

Model (13B)	TextVQA EM / F1	InfoSeek EM / F1	OVEN EM / F1
LLaVA	22.84 / 26.15	13.94 / 14.98	2.20 / 3.12
PLUM	30.11 / 32.62	15.60 / 16.54	2.64 / 3.81
PixSearch ${}_{\texttt{No Search}}$	33.0 / 34.9	18.6 / 19.7	4.0 / 5.6
PixSearch ${}_{\texttt{Question}}$	31.2 / 33.3	21.0 / 22.6	5.5 / 7.0
PixSearch ${}_{\texttt{Whole Image}}$	32.6 / 34.5	24.6 / 26.4	15.6 / 17.3
PixSearch ${}_{\texttt{Full}}$	32.8 / 35.5	24.9 / 27.0	16.2 / 17.4

Table 4: Performance on additional VQA benchmarks. Scores are reported in %. For PixSearch variants, we report mean scores over three independent training seeds. Each cell shows EM / F1; all models have 13B parameters.

Method	Time (s)	p90 (s)
PixSearch-13B (Full)	$2.8\pm 0.9$	3.3
PixSearch-13B (Whole Image)	$\mathbf{2.45}\pm 0.72$	$\mathbf{2.96}$
GroundedSAM+LLaVA-13B †	$4.4\pm 1.3$	4.1
GroundedSAM+LLaVA-13B ‡	$5.0\pm 3.4$	5.9

Table 5: Inference latency. Wall-clock latency on CRAG-MM, reported as mean

\pm

std over five runs.

\dagger

: cached, GPU-only pipeline.

\ddagger

: pipeline including GPT-4.1 entity extraction.

Training	Acc.	gIoU
Stage-2 SFT only	18.5	14.8
Single-stage joint SFT	32.6	67.4
Two-stage SFT	36.9	78.9
Search-R1-style GRPO	36.9	71.6

Table 6: Training comparison. CRAG-MM accuracy and RefCOCO gIoU (%). Expanded metrics are in Appendix F .

Panel of Table 9 — Table 9: Qualitative comparison between baseline and PixSearch -Interleaved outputs. Each example shows the input image (left), the corresponding question and model responses (middle), and the pixel-level grounding mask generated by the model (right). The payloads between the \informationtag \informationend are truncated to show entity names / titles only for brevity.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	PLUM	PixSearch
Backbone
Language model	LLaVA-13B	LLaVA-13B (PLUM init.)
Vision tower	CLIP ViT-L/14	CLIP ViT-L/14
Mask decoder	SAM ViT-H	SAM ViT-H
Training schedule
Input resolution	$1024^{2}$	$1024^{2}$

Appendix

Table 11: Hyperparameters for PLUM and PixSearch. Stage-2 settings follow the documented completed run.

Question Type	Example Question
OCR Read	“What does it say near the tail of the plane?”
OCR + Visual Reasoning	“Which team is winning the game?”
Multi-hop External Knowledge Reasoning	“When was the soft-drink company shown first created?”
Fine-grained Entity Identification	“What class of animal is this creature?”
Visual Reasoning – Attribute	“What is the color of the car in the background?”
Visual Reasoning – Counting	“How many cars are there in the image?”

Appendix

Table 12: Overview of the ten visual question types used in Stage-2.

Image File	Question	Retrieval
cragmm/4ec6f8ae.png	How many hybrid variations of this car were there in 2024?	Yes
cragmm/08629717.png	Is that drink good for my gut health?	Yes
cragmm/fb2fed47.png	How many arms does this statue typically have?	Yes
cragmm/1c613a06.png	Translate this.	No
cragmm/569a3617.png	Which station has more tracks, this one or Penn Station?	Yes
cragmm/a97e2470.png	Where was the designer who developed this car originally from?	Yes

Appendix

Table 13: Question Selection Examples. We show representative questions, whether retrieval was needed or not, and the associated image filenames. The Image File paths were truncated to have only the prefix of the image path for brevity.

Image File	Original Question	Sub-Questions
cragmm/4dcc84dc.png	When did the king of that country become king?	1) Who is the king of Spain? 2) When did the king of Spain become king?
cragmm/da33192e.png	What’s the capacity of this arena?	1) What is this place? 2) What’s the capacity of this arena?
cragmm/f73ab93c.png	Where was the first sign accompanying this erected?	1) Where was the first pedestrian crossing signal erected?
cragmm/878088c7.png	What’s the ideal temperature for this plant?	1) What is this plant? 2) What is the ideal temperature for this plant?
cragmm/b03b7dd6.png	How long can I use it without turning it off?	1) What is the model name of this generator? 2) How long can I use this generator?
cragmm/ec87776d.png	Translate this into English.	1) Translate this into English.

Appendix

Table 14: Question Decomposition Examples. Each example lists the original question, whether it was decomposed, and the resulting sub-questions.

Image File	Question + Model Answer	Ground-Truth Answer
cragmm/4dcc84dc.png	Q: When did the king of that country become king? A: The paper says “… independence from Spain.” Since Mexico has no king, this refers to Spain. The king of Spain <search> is King Felipe VI. Felipe VI came to the throne <search> on June 19, 2014.	June 19, 2014
cragmm/da33192e.png	Q: What’s the capacity of this arena? A: The place shown <search> is Madison Square Garden. It can house <search> between 18,006 and 20,789 people.	18,006–20,789 depending on event
cragmm/f73ab93c.png	Q: Where was the first sign accompanying this erected? A: The first pedestrian crossing signal was erected in <search> Bridge Street, Westminster, London, in December 1868.	Bridge Street, Westminster, London (1868)
cragmm/878088c7.png	Q: What’s the ideal temperature for this plant? A: The plant <search> appears to be Ravenea rivularis. Ravenea rivularis prefers <search> 65–85°F.	Prefers 65–85°F
cragmm/b03b7dd6.png	Q: How long can I use it without turning it off? A: The generator appears to be <search> a Firman Tri Fuel Generator. Its usage time is <search> a maximum of 12 hours with a 2–3 hour recommended break.	12 hours, with 2–3 hour breaks
cragmm/ec87776d.png	Q: Translate this into English. A: It translates to: “What mixer is used in a Paloma?”	“What mixer is used in a Paloma?”

Appendix

Table 15: Response Generation Examples. Each example includes the question, the model-generated answer (which includes <search> calls), and the ground-truth answer.

Image File	Question + <search> -Interleaved Response	Query Types	Queries
cragmm/4dcc84dc.png	Q: When did the king of that country become king? A: The king of Spain <search> is King Felipe VI. </search> He came to the throne <search> on June 19, 2014. </search>	TEXT, TEXT	1) Who is the king of Spain? 2) When did King Felipe VI become king?
cragmm/da33192e.png	Q: What’s the capacity of this arena? A: The place shown <search> is Madison Square Garden. </search> It can house <search> between 18,006 and 20,789 people. </search>	IMAGE, TEXT	1) image 2) What is the capacity of this arena?
cragmm/f73ab93c.png	Q: Where was the first sign accompanying this erected? A: The first pedestrian crossing signal was erected <search> in Bridge Street, Westminster, London. </search>	TEXT	Where was the first pedestrian crossing signal erected?
cragmm/878088c7.png	Q: What’s the ideal temperature for this plant? A: The plant <search> appears to be Ravenea rivularis. </search> It prefers <search> 65–85°F. </search>	REGION, TEXT	1) plant 2) Ideal temperature for Ravenea rivularis
cragmm/b03b7dd6.png	Q: How long can I use it without turning it off? A: The generator <search> appears to be a Firman Tri Fuel Generator. </search> Its usage time is <search> a maximum of 12 hours with breaks. </search>	REGION, TEXT	1) generator 2) Use time for Firman Tri Fuel Generator
cragmm/ec87776d.png	Q: Translate this into English. A: The sentence translates to: “What mixer is used in a Paloma?”	—	—

Appendix

Table 16: Query-Type Assignment Examples. Each example includes the original question, the model’s <search> -interleaved response, and the corresponding query types and queries.

Training	Acc.	R@1	R@10	gIoU	mIoU
Stage-2 SFT only	18.5	0.05	0.19	14.8	10.2
Joint SFT	32.6	0.18	0.35	67.4	46.8
Two-stage SFT	36.9	0.22	0.41	78.9	55.98
Search-R1-style GRPO	36.9	0.20	0.38	71.6	50.3

Appendix

Table 17: Expanded training comparison. Accuracy is on CRAG-MM, gIoU on RefCOCO, and mIoU on ADE20K; these three metrics are percentages. Retrieval recall is reported on a

[0,1]

scale.

Component	Full	Whole	Cached	+GPT
Model inference (s)	1.76	1.72	1.84	2.44
Mask/crop (s)	0.23	—	1.60	1.60
Query encoding (s)	0.12	0.11	0.13	0.13
Retrieval (s)	0.69	0.62	0.83	0.83
Total (s)	2.80	2.45	4.40	5.00
p90 (s)	3.30	2.96	4.10	5.90

Appendix

Table 18: Latency components on CRAG-MM. Full and Whole denote PixSearch with full and whole-image retrieval, respectively. Cached and +GPT denote the cached and GPT-4.1-assisted GroundedSAM–LLaVA pipelines.

Multimodal RAG

Multimodal retrieval-augmented generation grounds AI answers in retrieved text, images, video, audio, and structured visual evidence.

See all 157 papers

R3G: A Reasoning-Retrieval-Reranking Framework for Vision-Centric Answer Generation
Jan 25, 2026Zhuohong Chen, Zhengxian Wu, Zirui Liao +6Multimodal RAG Visual Question Answering
1The Shenzhen International Graduate School, Tsinghua University, China · School of Computer Science and Information Engineering, Hefei University of Technology, China · 2State Key Laboratory of Nuclear Power Safety Technology and Equipment, China
Learning to Search: A Decision-Based Agent for Knowledge-Based Visual Question Answering
Apr 8, 2026Zhuohong Chen, Zhenxian Wu, Yunyao Yu +6Visual Question Answering Tool Use in VLMs
Tsinghua University · Hefei University of Technology · University of Arizona
Region-R1: Reinforcing Query-Side Region Cropping for Multi-Modal Re-Ranking
Apr 7, 2026Chan-Wei Hu, Zhengzhong TuMultimodal Reranking Multimodal RAG
Texas A&M University

Jan 25, 2026J/K move · Enter open · S save

R3G: A Reasoning-Retrieval-Reranking Framework for Vision-Centric Answer Generation

Zhuohong Chen, Zhengxian Wu, Zirui Liao, Shenao Jiang+5

1The Shenzhen International Graduate School, Tsinghua University, China · School of Computer Science and Information Engineering, Hefei University of Technology, China · 2State Key Laboratory of Nuclear Power Safety Technology and Equipment, China

Multimodal RAG Visual Question Answering Large Vision-Language Models VLM Reasoning Visually Grounded Reasoning Vision-Language Retrieval+1

Open paper AskPDF arXiv

Vision-centric retrieval for VQA requires retrieving images to supply missing visual cues and integrating them into the reasoning process. However, selecting the right images and integrating them effectively into the model's reasoning remains challenging. To address this challenge, we propose R3G, a modular Reasoning-Retrieval-Reranking framework. It first produces a brief reasoning plan that specifies the required visual cues, then adopts a two-stage strategy, with coarse retrieval followed by fine-grained reranking, to select evidence images. On MRAG-Bench, R3G improves accuracy across six MLLM backbones and nine sub-scenarios, achieving state-of-the-art overall performance. Ablations show that sufficiency-aware reranking and reasoning steps are complementary, helping the model both choose the right images and use them well. We release code and data at https://github.com/czh24/R3G.

Abstract

Multimodal RAG

Explore similar work

R3G: A Reasoning-Retrieval-Reranking Framework for Vision-Centric Answer Generation

Learning to Search: A Decision-Based Agent for Knowledge-Based Visual Question Answering

Region-R1: Reinforcing Query-Side Region Cropping for Multi-Modal Re-Ranking