EviRank: Structured Relevance Evidence for Multimodal Image Re-ranking

Enjun Du; Siyi Liu; Zirong Chen; Xinyu Zuo; Jinwen Luo; Ruiwen Tao; Lisheng Duan; Haijin Liang

cs.CVAug 21, 2026

EviRank: Structured Relevance Evidence for Multimodal Image Re-ranking

Authors: Enjun Du, Siyi Liu, Zirong Chen, Xinyu Zuo, Jinwen Luo, Ruiwen Tao, Lisheng Duan, Haijin Liang, +3 more

Organizations: The Hong Kong University of Science and Technology (Guangzhou) · Tencent Yuanbao · The University of Hong Kong

Abstract

Real-world image search queries are multimodal and compositional: ``find this shirt in pink'' specifies an entity to retain, an attribute to modify, and context to ignore. Yet existing re-rankers either compress such multifaceted relevance into an opaque embedding or rely on free-form chain-of-thought that easily omits or hallucinates fine-grained constraints. Drawing on rubric- and checklist-based evaluation from NLP, we recast multimodal image re-ranking as a semantic constraint satisfaction problem and propose EviRank, which parses any query - text-only, image-only, or composed - into a unified evidence package: typed criteria across six semantic slots (e.g., entities, attributes, relations), each labelled required, forbidden, or ignorable. Re-ranking then reduces to evidence-conditioned verification, combining deterministic rubric scoring and evidence-grounded listwise comparison in a single training-free procedure. The explicit evidence can further serve as structured supervision for optionally distilling a lightweight student. Across five benchmarks spanning text-to-image, image-to-image, and composed image retrieval, EviRank achieves state-of-the-art performance, and the distilled student preserves over 90% of the teacher's capability at substantially lower cost.

Figures & tables

Explicit mechanism	Image Scope	CoT RR	CoT MR	Ours
Fixed, reusable slot taxonomy	$\circ$	✗	✗	✓
Typed forbidden evidence	✗	✗	$\circ$	✓
Typed ignorable evidence	✗	✗	✗	✓
One frame for T $\to$ I/I $\to$ I/CIR	$\circ$	✗	✗	✓
Deterministic auditable scoring	✗	✗	✗	✓
Teacher-free distilled student	✗	✗	✗	✓

Table 1: Novelty delineation against the closest reasoning-based re-rankers. ✓: reported as an explicit, fixed and reusable representational mechanism;

\circ

: partially or implicitly present; ✗: not reported as such a mechanism—which does not deny that the underlying MLLM may implicitly possess the capability.

Figure 1: Overview of EviRank. A multimodal query (text-only, image-only, or composed) and top- $K$ candidates from any retriever are first parsed by an MLLM teacher into a structured Evidence Frame of typed required / forbidden / ignorable statements over six semantic slots. Candidates are then re-ranked by evidence-conditioned verification (deterministic rubric + MLLM listwise refinement). The same evidence yields a structured supervision bundle that is distilled into a lightweight student; at inference the student consumes only the raw query and candidate image.

VLM	Methods	R@1	R@5	R@10	MRR@5
	No ReRank	$84.0$	$95.3$	$96.9$	$89.7$
	ImageScope	$84.8$	$95.6$	$97.0$	$90.1$
	CoTRR	$85.6$	$96.2$	$97.3$	$90.3$
	CoTMR	$85.9$	$96.8$	$97.6$	$90.9$
	EviRank-mini	$84.9$	$96.5$	$97.5$	$91.0$
	EviRank	$86.7$	$96.9$	$98.0$	$91.1$

Table 2: Retrieval performance (%) with different backbones on Flickr30k.

Figure 2: EviRank’s Recall@10 on Flickr30k across the four coarse-retrieval backbones of Table 2 .

VLM	Methods	R@1	R@5	R@10
	No ReRank	$39.4$	$65.5$	$75.6$
	ImageScope	$51.2$	$73.3$	$80.7$
	CoTRR	$58.3$	$77.2$	$83.5$
	EviRank-mini	$59.9$	$77.4$	$82.4$
	EviRank	$61.0$	$81.3$	$84.9$
	EviRank-plus	$62.3$	$83.7$	$86.6$

Table 3: Retrieval performance on COCO.

	SoP				CUB-200
Methods	R@1	R@5	R@10	mAP	R@1	R@2	R@4	mAP
No ReRank	$80.8$	$86.8$	$92.1$	$65.1$	$68.9$	$79.4$	$87.3$	$49.8$
SSR ReRank	$81.2$	$87.4$	$91.9$	$66.7$	$69.4$	$79.0$	$86.1$	$54.2$
$\alpha$ QE	$81.1$	$85.8$	$90.7$	$68.1$	$70.9$	$78.8$	$84.7$	$56.9$
RRT	$81.9$	$86.7$	$92.4$	$67.2$	$68.7$	$85.0$	$95.6$	$55.6$
LoCoRE-tiny	$82.4$	$87.8$	$93.1$	$68.0$	$71.4$	$86.8$	$96.4$	$58.1$

Table 4: Retrieval performance (%) on SoP and CUB-200 (DINOv2). mAP refers to mAP@R.

VLM	Methods	Shirt		Dress		Toptee
		R@10	R@50	R@10	R@50	R@10	R@50
	SEARLE	$24.4$	$41.6$	$18.5$	$39.5$	$25.7$	$46.4$
	iSEARLE-OTI	$27.0$	$43.4$	$21.2$	$42.1$	$26.8$	$48.7$
	CIReVL	$28.3$	$47.8$	$25.2$	$46.3$	$31.2$	$53.8$
	ImageScope	$31.6$	$37.4$	$26.8$	$46.3$	$35.8$	$55.9$
	EviRank-mini	$31.7$	$38.4$	$27.9$	$47.4$	$37.4$	$56.5$

Table 5: Retrieval performance (%) on FashionIQ.

	T $\rightarrow$ I		I $\rightarrow$ I
Variants	Flickr30K	COCO	SoP	CUB-200
EviRank-pro	96.9	89.4	87.6	99.4
w/o B0	94.5 $\downarrow$ 2.4%	88.6 $\downarrow$ 0.8%	82.3 $\downarrow$ 5.9%	96.5 $\downarrow$ 2.9%
w/o Slots	93.5 $\downarrow$ 3.4%	87.9 $\downarrow$ 1.6%	85.4 $\downarrow$ 2.4%	96.3 $\downarrow$ 3.1%
w/o Evidence	93.2 $\downarrow$ 3.8%	84.2 $\downarrow$ 5.7%	84.1 $\downarrow$ 3.9%	95.9 $\downarrow$ 3.5%
w/o Required	94.1 $\downarrow$ 2.8%	86.6 $\downarrow$ 3.0%	84.8 $\downarrow$ 3.0%	96.0 $\downarrow$ 3.4%

Table 6: Ablation results on T

\rightarrow

I and I

\rightarrow

I tasks with CLIP-ViT-L/14.

	(T,I) $\rightarrow$ I
Variants	Shirt	Dress	Toptee
EviRank-pro	40.1	38.9	49.7
w/o B0	37.6 $\downarrow$ 6.2%	35.4 $\downarrow$ 8.8%	46.5 $\downarrow$ 6.3%
w/o Slots	37.6 $\downarrow$ 6.0%	35.6 $\downarrow$ 8.2%	44.4 $\downarrow$ 10.5%
w/o Evidence	36.9 $\downarrow$ 7.8%	34.7 $\downarrow$ 10.5%	42.9 $\downarrow$ 13.5%
w/o Required	37.1 $\downarrow$ 7.3%	35.7 $\downarrow$ 8.0%	44.6 $\downarrow$ 10.0%

Table 7: Ablation results on composed multimodal retrieval tasks with CLIP-ViT-L/14.

Setting	COCO	SoP	FashionIQ
Rubric-only	59.9 / 82.4	83.7 / 69.8	32.1 / 50.9
Listwise-only	62.5 / 85.6	87.9 / 75.7	37.3 / 55.6
Rubric+Listwise	64.6 / 88.4	91.5 / 79.8	42.9 / 60.4

Table 8: Rubric vs. listwise component analysis. M1: R@1 (T

\rightarrow

I/I

\rightarrow

I) / R@10 (CIR). M2: R@10 / mAP / R@50 respectively.

Figure 3: Pairwise SBERT similarity between the six evidence slots over 10k queries (off-diagonal mean 0.18).

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Student backbone	Qwen3-VL-2B-Thinking
Training epochs	10
Batch size	32
Learning rate	$5\times 10^{-5}$
Optimizer	AdamW
LR schedule	Cosine decay

Appendix

Table 11: Distillation hyperparameters for Stage C.

Method	Flickr30k	COCO
VLM2Vec	80.0	49.2
RetLLM	82.0	54.1
UniME-v2	85.5	60.9
ReMatch	85.6	62.8
EviRank-pro	95.6	69.5

Appendix

Table 13: Comparison with recent SOTA T

\rightarrow

I baselines (R@1, %).

Backbone	Methods	R@1	R@5	R@10	MRR@5
EVA-CLIP-18B	No ReRank	$84.08$	$95.33$	$96.97$	$89.75$
	ImageScope	$84.89$	$95.68$	$97.01$	$90.12$
	CoTRR	$85.69$	$96.21$	$97.34$	$90.37$
	CoTMR	$85.93$	$96.82$	$97.69$	$90.96$
	EviRank-mini	$84.89$	$96.47$	$97.52$	$90.97$
	EviRank	$86.65$	$96.93$	$98.01$	$91.11$

Appendix

Table 14: Full Flickr30k results across all backbones and baselines.

	SoP				CUB-200
Methods	R@1	R@5	R@10	mAP	R@1	R@2	R@4	mAP
No ReRank	$80.80$	$86.87$	$92.10$	$65.10$	$68.90$	$79.40$	$87.30$	$49.80$
SSR ReRank	$81.20$	$87.42$	$91.90$	$66.70$	$69.40$	$79.00$	$86.10$	$54.20$
AQE	$76.90$	$73.69$	$89.30$	$66.10$	$66.90$	$76.80$	$82.70$	$58.40$
DQE	$67.90$	$72.71$	$81.50$	$47.80$	$67.00$	$75.50$	$82.00$	$54.60$
$\alpha$ QE	$81.10$	$85.86$	$90.70$	$68.10$	$70.90$	$78.80$	$84.70$	$56.90$

Appendix

Table 15: Full SoP and CUB-200 results with all I

\rightarrow

I baselines.

VLM	Methods	Shirt		Dress		Toptee
		R@10	R@50	R@10	R@50	R@10	R@50
CLIP-ViT-B/32	PALAVRA	$21.49$	$37.05$	$17.25$	$35.94$	$20.55$	$38.76$
	SEARLE	$24.44$	$41.61$	$18.54$	$39.51$	$25.70$	$46.46$
	SEARLE-OTI	$25.37$	$41.32$	$17.85$	$39.91$	$24.12$	$45.79$
	iSEARLE	$25.81$	$43.52$	$20.92$	$42.19$	$26.47$	$48.70$
	iSEARLE-OTI	$27.09$	$43.42$	$21.27$	$42.19$	$26.82$	$48.75$

Appendix

Table 16: Full FashionIQ results across all baselines and backbones.

Slot	Required	Forbidden	Ignore
Entities	The image must contain an acoustic guitar with a hollow wooden body and a visible sound hole. The guitar should have a traditional construction with a neck and fretboard clearly visible.	The guitar must not be an electric guitar with solid body and pickups. The instrument should not be a ukulele, bass guitar, banjo, or any other stringed instrument.	The specific brand or manufacturer of the guitar is not relevant for matching. The exact number of strings or whether a pick is being used does not affect the match.
Attributes	The dress must have blue as its primary base color. The fabric must display a floral or flower-based pattern with visible botanical motifs.	The dress should not be a solid single color without any pattern. The pattern must not be geometric stripes, polka dots, or abstract non-floral designs.	The exact shade or tone of blue does not need to match precisely. The specific type of flowers in the pattern and the overall dress length are not critical factors.
Actions	The woman must be actively engaged in playing the guitar with fingers positioned on strings. The posture should indicate ongoing musical performance rather than static positioning.	The woman should not be merely holding the guitar as a prop without playing engagement. The scene must not show her posing statically for a photograph with the guitar as an accessory.	The specific playing technique such as strumming versus fingerpicking is not relevant. The particular song or music being performed does not affect the matching.
Relations	The woman’s hands must be positioned on the guitar in a playing configuration, with one hand on the neck and one near the body. The guitar must be held in an active playing position against her body.	The guitar should not be lying flat on a surface or hanging on a wall. The guitar must not be stored in a case or held passively without playing intent.	Whether the woman is sitting or standing while playing is not relevant. The specific hand positioning for left-handed versus right-handed playing does not affect the match.
Scene	The setting must be an outdoor café environment with visible tables, chairs, or typical café furniture. The lighting should indicate an outdoor daytime setting.	The location must not be an indoor concert stage or formal performance venue. The setting should not be a recording studio, private bedroom, or enclosed indoor space.	The number of other patrons or customers visible in the scene is not relevant. The specific café name, branding, or current weather conditions do not affect the match.
KeyDetails	The overall atmosphere must suggest a casual, informal musical performance in a relaxed social setting. The context should indicate spontaneous or low-key entertainment rather than organized performance.	The scene must not depict a formal concert with audience seating arrangements. The context should not suggest a music competition, audition, or professional street busking for monetary collection.	The specific time of day or season when the scene takes place is not relevant. The geographic location or cultural context of the café does not affect the matching.

Appendix

Table 17: Slot Decomposition: structured evidence slots with required, forbidden, and ignored constraints.

Figure 10: Semantic similarity matrix across evidence slots.

Figure 11: Candidate images for slot-type ablation.

Explore similar work

Oct 1, 2026cs.AI

AiSearch: Interactive Multi-Modal Search with VLMs

Modern retrieval systems must both be automated and interactive, allowing users to search and refine results in real time. We present AiSearch, a flexible multimodal retrieval framework that leverages the zero shot capabilities of Vision Language Models (VLMs) for natural language search over images and videos. AiSearch supports interactive search refinement through user feedback to tailor results to the user's intent, and allows visual benchmarking across multiple VLMs, enabling users to select the most suitable model for their task.

Ali Koksal, Mei Chee Leong, Vicky Sintunata +2

Institute of Advanced Intelligence and Computing (IAIC), Agency for Science, Technology and Research (A*STAR), Singapore

Multimodal IR Vision-Language Models Information Retrieval Vision-Language Retrieval

Ask PDF

Sep 21, 2026cs.IR

Graded-Relevance Composed Multimodal Retrieval for E-commerce Visual Search at Scale

Visual search on large e-commerce catalogs must serve both "similarity" queries that ask for items resembling an uploaded image and "modifier" queries that comprise an image and text describing a desired modification (e.g. a color change or style swap). The latter is the setting known as composed image retrieval (CIR). Existing CIR methods, however, treat relevance as binary and train on triplets with a single positive target - a poor fit for real catalogs where many candidates partially satisfy a user query and ranking across that partial-match spectrum drives the customer experience. We propose a methodology for training CIR retrievers on graded relevance, consisting of: (i) a VLM to curate training data, generating both queries (object detection + modifier synthesis) and 4-level relevance labels without manual annotation, (ii) an iterative relevance-feedback loop that expands the training set by mining hard negatives from the in-training retriever, and (iii) a hierarchy-aware angular objective to train the retriever directly on the graded labels rather than collapsing them to a binary split. We call this methodology GradCIR and instantiate it on a PaliGemma2 bi-encoder trained on 3.5M graded pairs curated from raw Walmart catalog data. A controlled graded-vs-binary ablation isolates the supervision granularity and shows lift of 4.9%-5.9% in NDCG@10. The same recipe applied to other multimodal encoders lifts early-fusion backbones by up to 8.5% NDCG@10. On the public FashionIQ benchmark, GradCIR (applied to PaliGemma2) reaches 0.6703 average recall when fine-tuned, slightly ahead of the strongest peer-reviewed supervised baseline we compare against, and matching or exceeding all published CLIP-L-class zero-shot CIR methods. The system is deployed in production at Walmart, where it's serving live visual-search user traffic.

Anubhav Gupta, Hrushikesh Mohapatra, Prijith Chandra +6

Walmart Global Tech Bangalore, India · Walmart Global Tech Sunnyvale, United States

Information Retrieval Visual Search Composed Image Retrieval

Ask PDF

May 18, 2026cs.IR

TIGER-FG: Text-Guided Implicit Fine-Grained Grounding for E-commerce Retrieval

E-commerce image search often takes a cropped image as the query, while each candidate is represented by full item images and structured text. This image-to-multimodal retrieval setting presents two asymmetries: a modality disparity -- a visual query must match image--text items, and a granularity disparity -- a cropped query must be compared with full images containing background context and possible distractors. Detection-based pipelines handle the granularity disparity through explicit localization but incur extra cost and error propagation, whereas CLIP-style encoders avoid detection, but are vulnerable to backgrounds or irrelevant items. To address these limitations, we propose TIGER-FG, a text-guided implicit fine-grained grounding framework for image-to-multimodal e-commerce retrieval. TIGER-FG uses item text as semantic guidance to produce target-focused item representations without object detection for retrieval. We further introduce dual distillation objectives that preserve target-region spatial consistency and query--item similarity structure, yielding more stable and discriminative multimodal representations. In addition, we construct ECom-RF-IMMR, a realistic benchmark suite with a 10M-pair training set and two evaluation benchmarks covering standard and cluttered item layouts. TIGER-FG improves Recall@1 over the strongest baseline by 6.1 and 34.4 percentage points on the two evaluation benchmarks, respectively, with only 85.7M query-side parameters and 256-dim embeddings. Results on public e-commerce benchmarks further demonstrate its generalization to noisy and one-to-many retrieval scenarios. Code and data will be released.

Xinyu Sun, Huangyu Dai, Lingtao Mao +5

Kuaishou Technology · Beijing, China

Multimodal IR Multimodal Grounding Fine-Grained Image Retrieval E-Commerce Search E-Commerce Image Retrieval

Ask PDF