SnapBench: Benchmarking Snap-and-Ask Multimodal Retrieval for Mobile Interactions

Zirong Chen; Fuda Ye; Kuan Zhang; Enjun Du; Junfu Pu; Xinlei Wang; Xinyu Zuo; Lisheng Duan

cs.CVAug 30, 2026

SnapBench: Benchmarking Snap-and-Ask Multimodal Retrieval for Mobile Interactions

Authors: Zirong Chen, Fuda Ye, Kuan Zhang, Enjun Du, Junfu Pu, Xinlei Wang, Xinyu Zuo, Lisheng Duan,

Organizations: The Hong Kong University of Science and Technology (Guangzhou) · Tencent Yuanbao · Tsinghua University · The University of Hong Kong ·

Abstract

Mobile AI acts as a visual oracle, empowering users to snap a picture of something and ask for information. Snap-and-ask retrieval is now one of the most common entry points for mobile AI, yet photos are often blurry, while text questions may be short or mistyped. Existing benchmarks only test on clean inputs or do not isolate paired robustness in snap-and-ask retrieval. Therefore, we introduce SnapBench, the first paired benchmark for robust snap-and-ask multimodal retrieval, spanning 1,145 queries, 9,085 gallery items under 53 controlled corruption conditions with human annotations. We evaluate 16 multimodal retrievers, covering dual-tower encoders and embedding-based VLMs. Results show that image corruptions substantially degrade retrieval, while text corruptions mainly affect text-only retrieval and have limited impact on joint retrieval. Clean image-only retrieval often outperforms joint retrieval, indicating the coarse-text drag and the lack of cross-modal fallback under noisy inputs. SnapBench provides a controlled testbed for evaluating robust retrieval in snap-and-ask scenarios. We further propose MOOR (Modality-anchored, Outlier-aware, Optimal Reweighting), a simple adaptive fusion approach, highlighting the need for reliability-aware modality calibration in snap-and-ask retrieval.

Figures & tables

Figure 1: Illustration of the mobile snap-and-ask retrieval scenario. Given a snapped image and a short text question, the system retrieves items from a gallery. The pilot analysis setting is detailed in Appendix A .

Dimension	ImageNet-C	TextAttack	SnapBench
Modality corrupted	image only	text only	image, text, or both
Paired protocol	✗	✗	✓
Joint multimodal corruption	✗	✗	✓

Table 1: SnapBench against existing robustness benchmarks. Paired : target entity, gallery, and labels are fixed across clean and corrupted conditions.

Figure 2: Overview of the SnapBench construction pipeline. This pipeline creates entity-centric snap-and-ask retrieval instances and simulates two sources of mobile interaction artifacts through controlled image and text question corruptions. Lv. , Pos. , and Neg. denote level, positive, and negative, respectively.

Model	Base.	C+C			D+D			L+W			M+S			E+W			R+S			Mean			Overall
Model	Base.	Exp.	Obs.	$\delta$	Exp.	Obs.	$\delta$	Exp.	Obs.	$\delta$	Exp.	Obs.	$\delta$	Exp.	Obs.	$\delta$	Exp.	Obs.	$\delta$	Exp.	Obs.	$\delta$	Overall
Dual-Encoder
SigLIP2 - SO400M	39.8	5.8	18.9	13.0	27.6	35.5	7.9	11.3	23.8	12.5	10.3	23.3	13.0	8.3	16.9	8.6	23.5	32.6	9.1	14.5	25.2	10.7	25.2
CLIP - ViT - L/14	49.5	-0.8	21.8	22.6	4.8	29.5	24.7	6.7	23.5	16.7	8.5	24.0	15.5	5.7	22.4	16.7	6.8	38.7	31.9	5.3	26.6	21.4	26.6
BLIP - ITM - L	57.3	0.2	6.1	5.9	5.6	13.0	7.3	7.2	12.9	5.7	6.0	11.8	5.8	6.8	13.4	6.6	9.2	24.8	15.6	5.8	13.7	7.8	13.7
SigLIP - SO400M	67.0	-3.3	2.2	5.5	9.5	13.4	3.9	8.0	11.5	3.4	5.7	11.7	6.0	7.5	9.0	1.6	-0.1	7.0	7.1	4.5	9.1	4.6	9.1

Table 3: Interaction between image-side and text-side artifacts. Exp.

=\Delta_{I}+\Delta_{T}

(naive additive prediction); Obs. = observed joint drop;

\bm{\delta}

=

Obs.

-

Exp. (

{>}0{=}

super-additive, bold ). Overall = mean Obs. across all six joint conditions. C+C = Compression

+

CharInsert; D+D = Downscale

+

CharDelete; L+W = LowLight

+

WordSwap; M+S = Mosaic

+

SentAdd; E+W = Overexposure

+

WordRepeat; R+S = Rotation

+

SentReplace.

Figure 3: Severity response of image-side corruptions. Thick lines show group means; faint lines are individual operators. Cliff (

n{=}7

): Mosaic, Low Light, Downscale, Crop, Watermark, Low Res, Defocus. Gradual (

n{=}8

): Perspective, UI Elements, Scribble, Motion Blur, JPEG, Rotation, Lens Distortion, Overexposure.

Figure 4: Average performance of MOOR in joint retrieval. Darker regions denote gains over fixed fusion.

Appendix figures & tables44 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Frequency
Clean (neither flagged)	29.4%
Snap-side only	23.5%
Ask-side only	12.4%
Both-side	34.7%
Any issue	$\mathbf{70.6\%}$

Appendix

Table 4: Four-way distribution of

5{,}000

real mobile snap-and-ask uploads.

Primitive	Frequency
Degrade	38.4%
Remove	24.6%
Add	17.3%
Transform	8.7%
Any image corruption	$\mathbf{58.2\%}$

Appendix

Table 5: Image-side primitive distribution among flagged uploads.

Dimension	Score	Definition
Visual similarity	0	Completely unrelated entity, no visual overlap.
	1	Different entity, partial visual overlap ( $<$ 50% shared elements).
	2	Same entity, different viewpoint or background.
	3	Same entity, matching viewpoint.
Image content relevance	0	Retrieved entity unrelated to the query need.
	1	Tangentially related only.

Appendix

Table 6: Per-dimension score definitions (

0

–

3

	Case	Vis	Cont	Cap	TN	Fitness
Image-only	I-0	—	0	—	—	0
	I-1	—	1	—	—	1
	I-2a	—	2	—	—	2
	I-2b	—	3	$\leq$ 1	—	2
	I-3	—	3	3	—	3
Image+text	IT-0	$\leq$ 1	$\leq$ 1	—	$\leq$ 1	0

Appendix

Table 7: Intent-conditioned fitness aggregation. Vis = visual similarity; Cont = image content relevance; Cap = caption/title accuracy; TN = text-vs-need relevance. “—” = not evaluated.

\leq

1 means score is 0 or 1;

\geq

2 means score is 2 or 3. Apply the first matching row within each block.

Level	Operator	Example
Char.	char_add	cbar
	char_delete	cr
	char_change	csr
	char_swap	acr
Word	word_repeat	What car car is this?
Word	word_swap	What is car this?

Appendix

Table 8: Text corruptions (

8

operators, all targeting the entity word; entity

{}={}

car ).

Primitive	Operator	$s_{1}$	$s_{2}$	$s_{3}$
Degrade	low_light	brightness $=0.70$ , $\sigma_{n}=3$	$0.40$ , $\sigma_{n}=10$	$0.15$ , $\sigma_{n}=22$
	overexposure	factor $=1.4$	$2.2$	$3.0$
	defocus_blur	$\sigma=1.0$	$3.5$	$10.0$
	motion_blur	kernel $=5$	$13$	$25$
	compression	quality $=50$	$18$	$5$
	low_resolution	scale $=0.50$	$0.25$	$0.08$

Appendix

Table 9: Image corruption grid. Four primitives,

15

operators,

3

severities each (

\,=45

image conditions). All severity parameters are deterministic and seeded for reproducibility.

Algorithm 1. MOOR (Modality-anchored, Outlier-aware, Optimal Reweighting)

Input:

\mathbf{q}^{I}\!\in\!\mathbb{R}^{d}

\mathbf{q}^{T}\!\in\!\mathbb{R}^{d}

(query image and text embeddings);

G^{I},G^{T}\!\in\!\mathbb{R}^{N\times d}

(gallery); whitening stats

\mu^{I/T},\sigma^{I/T}

Output: fused score vector

\bm{s}^{MOOR}\!\in\!\mathbb{R}^{N}

1. Gallery-side whitening. Whiten

G^{I},\mathbf{q}^{I}

with

(\mu^{I},\sigma^{I})

\!\to\!\tilde{G}^{I},\tilde{\mathbf{q}}^{I}

; whiten

G^{T},\mathbf{q}^{T}

with

(\mu^{T},\sigma^{T})

\!\to\!\tilde{G}^{T},\tilde{\mathbf{q}}^{T}

; cross-whiten

\hat{\mathbf{q}}^{I}

with

(\mu^{T},\sigma^{T})

and

\hat{\mathbf{q}}^{T}

with

(\mu^{I},\sigma^{I})

2. Four similarity paths.

\bm{s}^{II}\!=\!\tilde{\mathbf{q}}^{I}\tilde{G}^{I\top}

\bm{s}^{TT}\!=\!\tilde{\mathbf{q}}^{T}\tilde{G}^{T\top}

\bm{s}^{IT}\!=\!\hat{\mathbf{q}}^{I}\tilde{G}^{T\top}

\bm{s}^{TI}\!=\!\hat{\mathbf{q}}^{T}\tilde{G}^{I\top}

3. Bell-gated rank consistency. For

ab\!\in\!\{TT,IT,TI\}

r^{ab}\!=\!\operatorname{Pearson}(\bm{s}^{II},\bm{s}^{ab})

;

g^{ab}\!=\!\max(0,r^{ab})^{2}\!\cdot\!\max(0,1\!-\!r^{ab})^{2}

4. Variance weights.

w^{II}\!=\!\operatorname{Var}(\bm{s}^{II})

;

w^{ab}\!=\!g^{ab}\!\cdot\!\operatorname{Var}(\bm{s}^{ab})

for

ab\!\in\!\{TT,IT,TI\}

Appendix

Table 16

Figure 6: Coarse-text drag and MOOR recovery. Each bar is

\Delta

R@1 relative to image-only (I

\to

I). Top : standard joint fusion (IT

\to

IT) underperforms image-only even on clean inputs

		Baseline	Text Corruption			Image Corruption
Model	Method	Clean	Char	Word	Sent	Degrade			Transform			Remove			Add
						s1	s2	s3	s1	s2	s3	s1	s2	s3	s1	s2	s3
Dual-Encoder
CLIP-ViT-L/14	Joint	49.5	45.0	50.0	43.0	48.6	45.2	36.6	48.8	44.6	37.3	50.0	47.4	40.4	48.3	44.2	34.2
CLIP-ViT-L/14	MOOR	$+26.7$	$+31.6$	$+26.4$	$+33.2$	$+26.9$	$+27.3$	$+24.8$	$+27.3$	$+26.8$	$+23.0$	$+25.6$	$+27.6$	$+26.8$	$+27.0$	$+28.1$	$+23.3$
SigLIP-SO400M	Joint	67.0	68.5	67.1	68.2	66.3	63.5	52.0	66.9	62.0	52.8	66.3	64.7	56.1	64.3	61.8	48.6

Appendix

Table 10: MOOR robustness on SnapBench (Recall@1, %). Joint : for dual-encoders, fixed equal-weight fusion (joint-0.5); for VLM-based models, native IT

\to

IT multimodal performance (from Table 2 ). MOOR rows: signed change over Joint

\bm{+}

gain /

\bm{-}

loss . Text averages: Char (4) / Word (2) / Sent (2) conditions. Image: per-group averages at each severity level.

Primitive	Baseline	$s_{1}$	$s_{2}$	$s_{3}$
Degrade	0.065	0.065	0.067	0.070
Transform	0.065	0.066	0.067	0.069
Remove	0.065	0.066	0.068	0.071
Add	0.065	0.066	0.069	0.072

Appendix

Table 11: Average text-path weight share

\bar{w}_{\text{txt}}

of MOOR across

10

models at each corruption group and severity.

Figure 7: Gate-Function Ablation for MOOR. R@1 gain over the image-only baseline for each of six gate functions (Bell / Linear / Sigmoid / Inverse Bell / Pure-

r^{2}

/ No Gate) across 16 joint-mode models, averaged over all 53 corruption conditions. The Bell gate (ours, darkest blue) consistently achieves the highest gain.

Model	Mode	Clean	Char	Word	Sent	Degrade	Transform	Remove	Add
			Text Pert.			Image Pert. (avg)
CLIP-ViT-L/14	IT $\rightarrow$ IT	49.5	45.0	49.9	43.1	44.3	45.0	41.4	42.3
	I $\rightarrow$ I	73.5	73.5	73.5	73.5	65.0	66.5	59.1	61.6
	I $\rightarrow$ IT	70.3	70.3	70.3	70.3	64.0	65.2	58.8	60.9
	T $\rightarrow$ T	14.6	5.6	14.8	6.8	14.6	14.6	14.6	14.6
	T $\rightarrow$ IT	19.0	7.6	19.0	8.1	19.0	19.0	19.0	19.0

Appendix

Table 12: Retrieval-mode ablation on SnapBench (Recall@1, %). IT

\to

IT is the joint multimodal mode from Tab. 2 . Image columns show averages over severities s1/s2/s3. N/A cells (mode cannot be affected by that corruption type) are filled with the Clean baseline value. Dual-Encoder models.

Method	Mean R@1	$\bm{\Delta}$ vs. MOOR
Borda Count	43.98	$+25.83$
RRF	45.64	$+24.16$
CombMNZ	51.95	$+17.86$
GBM-Meta (learned)	54.89	$+14.91$
LR-Meta (learned)	55.97	$+13.83$
wh-Uniform	60.72	$\phantom{0}+9.09$

Appendix

Table 13: MOOR vs. fusion baselines, mean R@1 averaged over all 54 states (53 corrupted

+

1 clean) and 16 models.

Figure 8: Typos Are Harmless—Only Semantic Replacement Hurts Retrieval. (a) Token change rate across text operators, grouped by tokenizer family. (b) Despite high token change rate for character-level operators, the mean joint R@1 change is negligible or mildly positive. Only sentence-level semantic replacement ( sent_replace ) causes meaningful retrieval degradation.

Figure 9: Sneaky Effect: Word Operators Hurt Retrieval Without Disrupting Tokens. Scatter of token change rate (

x

-axis) vs. R@1 drop (

y

-axis); each point is one (model, operator, severity) triple. Char-level operators cluster at high token change rate with moderate R@1 drop; word-level operators cluster near zero token change rate yet cause substantial R@1 drop—the “sneaky” failure mode invisible to token-level diagnostics.

Figure 10: Image Attention Mass Across Input Conditions. Mean IAM per model under five conditions: image-only ( I ), clean image+text ( I+T ), and image paired with char-, word-, or sentence-level corrupted text. GME models maintain near-unity IAM regardless of text quality, while Qwen3-VL-Emb models collapse to near-zero IAM once any text is present. Models with larger IAM drop from I to I+T also exhibit higher mean R@1 degradation under text corruption (

r=0.628

Figure 11: Effect of removing the query text on clean IT

\to

IT retrieval. For each VLM embedding model, we replace the user question with an empty string while keeping the snapped image and the IT gallery fixed. Bars report

\Delta

R@1 between image-only query retrieval (I

\to

IT) and full query retrieval (IT

\to

IT). Positive values indicate that dropping the text improves retrieval, consistent with coarse-text drag : coarse user questions are over-weighted under fixed fusion and dilute a strong visual ranking. Ops-MM-7B is the only model for which the query text helps on clean inputs.

Model	Base.	C+C			D+D			L+W			M+S			E+W			R+S			Mean
	$s_{1}$	$s_{2}$	$s_{3}$	$s_{1}$	$s_{2}$	$s_{3}$	$s_{1}$	$s_{2}$	$s_{3}$	$s_{1}$	$s_{2}$	$s_{3}$	$s_{1}$	$s_{2}$	$s_{3}$	$s_{1}$	$s_{2}$	$s_{3}$	$s_{1}$	$s_{2}$	$s_{3}$
Dual-Encoder
CLIP - ViT - L/14	63.8	20.4	22.6	24.9	22.5	25.3	26.3	16.5	17.1	16.6	16.0	15.8	14.7	15.9	18.7	15.5	31.1	32.4	32.2	20.4	22.0	21.7
SigLIP - SO400M	68.1	5.1	5.5	6.0	4.3	2.0	5.5	2.4	2.6	5.2	5.0	4.5	8.4	0.6	1.7	2.4	6.9	8.3	6.1	4.1	4.1	5.6
SigLIP2 - SO400M	49.7	12.8	12.1	14.1	10.2	9.1	4.4	11.8	13.5	12.2	11.2	12.3	15.4	8.9	8.6	8.3	8.8	8.3	10.3	10.6	10.7	10.8
BLIP - ITM - L	64.4	5.4	6.9	5.5	4.6	6.1	11.3	5.8	5.4	5.8	5.0	5.3	7.1	7.5	6.7	5.5	11.1	15.6	20.2	6.6	7.7	9.3

Appendix

Table 14: Appendix: Per-severity super-additivity gap

\delta=\text{Obs.}-\text{Exp.}

(R@1 drop in pp) for six joint corruption conditions at severity levels

s_{1}

s_{2}

s_{3}

. Positive (bold) = super-additive; negative = sub-additive. Condition abbreviations follow Table 3 .

Model	I $\to$ I	IT $\to$ IT	Drag
CLIP ViT-L/14	73.45	49.52	$-23.93$
BLIP ITM-L	73.71	57.29	$-16.42$
Ops-MM-7B	80.52	69.61	$-10.91$
RzenEmbed	80.61	75.55	$-5.06$
Mean (16 models)	—	—	$-11.21$

Appendix

Table 15: Coarse-text drag (clean IT

\to

-

\to

I R@1) for representative models; full 16-model table follows the same pattern.

Model	Full-set drag	Tier-1 drag	$\bm{\Delta}$
SigLIP-SO400M	$-15.63$	$-1.45$	$+14.18$
BLIP ITM-L	$-16.42$	$-4.34$	$+12.08$
CLIP ViT-L/14	$-23.93$	$-10.15$	$+13.78$
Jina-Emb-v4	$-10.13$	$-2.90$	$+7.23$
GME-2B	$-3.58$	$0.00$	$+3.58$
Mean (16 models)	$-10.06$	$-8.78$	$+1.28$

Appendix

Table 16: Full-set vs. text-essential-subset (Tier-1) coarse-text drag.

Figure 12: Query-level stability under joint corruption (IT

\to

IT). Stacked bars show the fraction of queries in each of four classes per model (16 models, sorted by stably-correct rate): stably correct (green), drag from correct to wrong (red), persistently wrong (grey), and boost from wrong to correct (blue). Percentages are shown inside segments

\geq 5\%

; small boost segments are omitted for readability.

Figure 13: Asymmetry of Image-Side vs. Text-Side Corruption. (a) Extra R@1 drop from the second corruption. (b) Text-side extra drop minus image-side extra drop (percentage points).

Figure 14: Domain difficulty in IT

\to

IT retrieval. Mean Recall@1 under clean joint IT queries, aggregated across models and queries per domain (bars sorted hard

\to

easy). Color encodes difficulty (warmer = lower Recall@1).

Figure 15: Clean R@1 vs. corruption loss under two retrieval modes. (a) I

\to

I: clean R@1 vs. mean R@1 loss under image-side corruptions. (b) IT

\to

IT: clean R@1 vs. mean R@1 loss under joint corruptions. Each colored point is one model (legend above); dashed line shows the Pearson trend; statistics are in the upper-right box. Weak correlation in (b) shows that clean IT

\to

IT accuracy alone does not predict joint-corruption robustness.

Primitive	R@1 drop (pp)	GT rank displacement
Remove	$-26.8$	$7.84$ positions
Add	$-17.9$	$4.97$ positions
Degrade	$-14.5$	$3.47$ positions
Transform	$\phantom{0}{-7.9}$	$1.38$ positions

Appendix

Table 17: Retrieval impact at severe severity (

s_{3}

) by image primitive (model-average over 4 models).

Model	Mode	Clean	Char	Word	Sent	Degrade	Transform	Remove	Add
			Text Pert.			Image Pert. (avg)
GME-2B	IT $\rightarrow$ IT	61.8	60.6	61.5	58.9	58.1	60.0	60.2	61.4
	I $\rightarrow$ I	65.4	65.4	65.4	65.4	60.1	64.1	64.6	62.5
	I $\rightarrow$ IT	53.5	53.5	53.5	53.5	49.2	52.3	53.2	52.8
	T $\rightarrow$ T	22.5	15.1	22.2	9.0	22.5	22.5	22.5	22.5
	T $\rightarrow$ IT	24.1	15.7	23.8	8.8	24.1	24.1	24.1	24.1

Appendix

Table 18: Retrieval-mode ablation on SnapBench (Recall@1, %). IT

\to

IT is the joint multimodal mode from Tab. 2 . Image columns show averages over three severities. N/A cells (mode cannot be affected by that corruption type) are filled with the clean baseline value. VLM Embedding models.

Operator type	Target ID’able	% of $s_{3}$ drop
Cliff-type (mosaic, watermark, low light, downscale, cropping)	10%	89%
Gradual-type (compression, motion blur, rotation)	95%	6%

Appendix

Table 19: Target identifiability at severity 3, by operator type (

750

images,

15

operators).

Panel of Figure 16 — Figure 16: Degrade operators (lighting & blur) applied to Case B at $s_{1}\!\to\!s_{2}\!\to\!s_{3}$ .

Panel of Figure 17 — Figure 17: Degrade operators (resolution & compression) applied to Case B at $s_{1}\!\to\!s_{2}\!\to\!s_{3}$ . Signal-quality degradation progressively destroys discriminative visual cues.

Panel of Figure 18 — Figure 18: Transform operators applied to Case B at $s_{1}\!\to\!s_{2}\!\to\!s_{3}$ . Geometric distortions misalign spatial features but preserve entity content, keeping failures in the Semantic Drift regime even at $s_{3}$ .

Panel of Figure 19 — Figure 19: Remove operator applied to Case B at $s_{1}\!\to\!s_{2}\!\to\!s_{3}$ . Aggressive cropping removes the entity from the frame, triggering Quality Degradation at $s_{3}$ .

Panel of Figure 20 — Figure 20: Add operators applied to Case B at $s_{1}\!\to\!s_{2}\!\to\!s_{3}$ . Overlaid elements occlude local regions without globally destroying the scene; at $s_{3}$ , coverage is dense enough to suppress the foreground entity and trigger Background Bias.

Level	Operator	Perturbed query (corruption in bold)
Original: What kind of fish is this?
Char.	char_add	What kind of fiish is this?
	char_delete	What kind of fsh is this?
	char_change	What kind of gish is this?
	char_swap	What kind of ifsh is this?
Word	word_repeat	What kind of fish fish is this?

Appendix

Figure 21: Text perturbation cases per operator (Case B). All operators target the entity word fish ; corrupted spans are in bold. Character-level noise degrades the entity token directly; word-level noise preserves the entity string but perturbs syntax; sent_replace removes the category keyword and is the strongest Semantic Drift trigger.

Multimodal IR

Multimodal information retrieval finds relevant images, videos, documents, audio, or other media using queries that may combine different modalities.

See all 128 papers

Aug 31, 2026

CrossModalQA: A Cross-modal and Multi-hop Benchmark for Multimodal Retrieval-augmented Generation

Despite the strong capabilities of multimodal large language models (MLLMs), their parametric knowledge remains incomplete and difficult to update, motivating multimodal retrieval-augmented generation (RAG) to ground responses in external text and images. However, existing benchmarks face two major limitations: (i) they typically emphasize single-hop retrieval or reasoning over a small set of provided contexts rather than open-domain evidence discovery; and (ii) they provide fragmented coverage of cross-modal reasoning paths, leaving complex multi-hop and multi-image reasoning underexplored. In this paper, we introduce CrossModalQA, an open-domain benchmark for evaluating multimodal retrieval and reasoning over heterogeneous corpora. CrossModalQA contains 1,863 question-answer pairs constructed from 4,987 Wikipedia articles and 4,431 Wikimedia Commons images. It covers five complementary reasoning paths: vision-to-text, text-to-vision, vision-to-text-to-vision, multi-image intersection, and image-set reasoning. Every question requires retrieving and composing distributed textual and visual evidence, with an average reasoning depth of 3.50 hops. We construct the benchmark through multimodal knowledge graph-guided subgraph sampling and apply rule-based consistency checking and LLM verification to ensure multimodal dependence and traceable evidence. Extensive experiments demonstrate that existing multimodal RAG systems struggle to recover complete evidence chains and can underperform closed-book models when incomplete retrieval introduces distracting context. Further analysis reveals that complete cross-modal retrieval contributes more to answer accuracy than generator scaling, while multi-image retrieval and reasoning remain the primary bottlenecks limiting end-to-end performance.

Jiacheng Cai, Zijin Hong, Zheng Yuan +3

The Hong Kong Polytechnic University · Jilin University

Multimodal RAG Multimodal QA Cross-Modal Retrieval

Ask PDF

Mar 5, 2026

MultiHaystack: Benchmarking Multimodal Retrieval and Reasoning over 40K Images, Videos, and Documents

Multimodal large language models (MLLMs) achieve strong performance on benchmarks that evaluate text, image, or video understanding separately. However, these settings do not assess a critical real-world requirement, which involves retrieving relevant evidence from large, heterogeneous multimodal corpora prior to reasoning. Most existing benchmarks restrict retrieval to small, single-modality candidate sets, substantially simplifying the search space and overstating end-to-end reliability. To address this gap, we introduce MultiHaystack, the first benchmark designed to evaluate both retrieval and reasoning under large-scale, cross-modal conditions. MultiHaystack comprises over 46,000 multimodal retrieval candidates across documents, images, and videos, along with 747 open yet verifiable questions. Each question is grounded in a unique validated evidence item within the retrieval pool, requiring evidence localization across modalities and fine-grained reasoning. In our study, we find that models perform competitively when provided with the corresponding evidence, but their performance drops sharply when required to retrieve that evidence from the full corpus. Additionally, even the strongest retriever, E5-V, achieves only 40.8% Recall@1, while state-of-the-art MLLMs such as GPT-5 experience a significant drop in reasoning accuracy from 80.86% when provided with the corresponding evidence to 51.4% under top-5 retrieval. These results indicate that multimodal retrieval over heterogeneous pools remains a primary bottleneck for MLLMs, positioning MultiHaystack as a valuable testbed that highlights underlying limitations obscured by small-scale evaluations and promotes retrieval-centric advances in multimodal systems.

Dannong Xu, Zhongyu Yang, Jun Chen +6

INSAIT, Sofia University “St. Kliment Ohridski” · Lanzhou University · King Abdullah University of Science and Technology +3

Multimodal IR Multimodal Model Evaluation Cross-Modal Retrieval Multimodal Large Language Models Multimodal Dataset Multimodal Reasoning

Ask PDF

Jun 5, 2026

M $^3$ Exam: Benchmarking Multimodal Memory for Realistic User-Agent Interactions

Language agents are increasingly deployed over accumulating multimodal information, yet existing benchmarks assume a human-human form with sparse visuals and straightforward content, evaluating neither reasoning over authentic multimodal file interaction nor the interpretation of concealed user information. We therefore introduce M

^3

Exam, a query-centric multimodal conversational memory benchmark built on realistic user-agent interaction, with multi-dimensional evaluation spanning cross-modal grounding and implicit information inference. Benchmarking MLLMs and memory systems reveals persistent gaps in cross-modal grounding, cross session reasoning, and the efficiency cost of accumulating multimodal context. We further propose M

^3

Proctor, a multimodal memory method that detects query modality bias and consumes raw visual sources only on demand, improving accuracy by 13% while cutting index-construction time and retrieved tokens by over 70%.

Zhengjun Huang, Wenxuan Liu, Zhoujin Tian +6

The Hong Kong University of Science and Technology · Peng Cheng Laboratory · Beijing University of Chemical Technology +4

Multimodal Understanding Multimodal Grounding Efficient Multimodal Inference Multimodal Memory AI Agent Benchmarks Memory Agent Benchmarks+3

Ask PDF

Abstract

Multimodal IR

Explore similar work

CrossModalQA: A Cross-modal and Multi-hop Benchmark for Multimodal Retrieval-augmented Generation

MultiHaystack: Benchmarking Multimodal Retrieval and Reasoning over 40K Images, Videos, and Documents

M3^33Exam: Benchmarking Multimodal Memory for Realistic User-Agent Interactions

M $^3$ Exam: Benchmarking Multimodal Memory for Realistic User-Agent Interactions