EviRank: Structured Relevance Evidence for Multimodal Image Re-ranking

Enjun Du; Siyi Liu; Zirong Chen; Xinyu Zuo; Jinwen Luo; Ruiwen Tao; Lisheng Duan; Haijin Liang

cs.CVAug 21, 2026

EviRank: Structured Relevance Evidence for Multimodal Image Re-ranking

Authors: Enjun Du, Siyi Liu, Zirong Chen, Xinyu Zuo, Jinwen Luo, Ruiwen Tao, Lisheng Duan, Haijin Liang, +3 more

Organizations: The Hong Kong University of Science and Technology (Guangzhou) · Tencent Yuanbao · The University of Hong Kong

Abstract

Real-world image search queries are multimodal and compositional: ``find this shirt in pink'' specifies an entity to retain, an attribute to modify, and context to ignore. Yet existing re-rankers either compress such multifaceted relevance into an opaque embedding or rely on free-form chain-of-thought that easily omits or hallucinates fine-grained constraints. Drawing on rubric- and checklist-based evaluation from NLP, we recast multimodal image re-ranking as a semantic constraint satisfaction problem and propose EviRank, which parses any query - text-only, image-only, or composed - into a unified evidence package: typed criteria across six semantic slots (e.g., entities, attributes, relations), each labelled required, forbidden, or ignorable. Re-ranking then reduces to evidence-conditioned verification, combining deterministic rubric scoring and evidence-grounded listwise comparison in a single training-free procedure. The explicit evidence can further serve as structured supervision for optionally distilling a lightweight student. Across five benchmarks spanning text-to-image, image-to-image, and composed image retrieval, EviRank achieves state-of-the-art performance, and the distilled student preserves over 90% of the teacher's capability at substantially lower cost.

Figures & tables

Explicit mechanism	Image Scope	CoT RR	CoT MR	Ours
Fixed, reusable slot taxonomy	$\circ$	✗	✗	✓
Typed forbidden evidence	✗	✗	$\circ$	✓
Typed ignorable evidence	✗	✗	✗	✓
One frame for T $\to$ I/I $\to$ I/CIR	$\circ$	✗	✗	✓
Deterministic auditable scoring	✗	✗	✗	✓
Teacher-free distilled student	✗	✗	✗	✓

Table 1: Novelty delineation against the closest reasoning-based re-rankers. ✓: reported as an explicit, fixed and reusable representational mechanism;

\circ

: partially or implicitly present; ✗: not reported as such a mechanism—which does not deny that the underlying MLLM may implicitly possess the capability.

Figure 1: Overview of EviRank. A multimodal query (text-only, image-only, or composed) and top- $K$ candidates from any retriever are first parsed by an MLLM teacher into a structured Evidence Frame of typed required / forbidden / ignorable statements over six semantic slots. Candidates are then re-ranked by evidence-conditioned verification (deterministic rubric + MLLM listwise refinement). The same evidence yields a structured supervision bundle that is distilled into a lightweight student; at inference the student consumes only the raw query and candidate image.

VLM	Methods	R@1	R@5	R@10	MRR@5
	No ReRank	$84.0$	$95.3$	$96.9$	$89.7$
	ImageScope	$84.8$	$95.6$	$97.0$	$90.1$
	CoTRR	$85.6$	$96.2$	$97.3$	$90.3$
	CoTMR	$85.9$	$96.8$	$97.6$	$90.9$
	EviRank-mini	$84.9$	$96.5$	$97.5$	$91.0$
	EviRank	$86.7$	$96.9$	$98.0$	$91.1$

Table 2: Retrieval performance (%) with different backbones on Flickr30k.

Figure 2: EviRank’s Recall@10 on Flickr30k across the four coarse-retrieval backbones of Table 2 .

VLM	Methods	R@1	R@5	R@10
	No ReRank	$39.4$	$65.5$	$75.6$
	ImageScope	$51.2$	$73.3$	$80.7$
	CoTRR	$58.3$	$77.2$	$83.5$
	EviRank-mini	$59.9$	$77.4$	$82.4$
	EviRank	$61.0$	$81.3$	$84.9$
	EviRank-plus	$62.3$	$83.7$	$86.6$

Table 3: Retrieval performance on COCO.

	SoP				CUB-200
Methods	R@1	R@5	R@10	mAP	R@1	R@2	R@4	mAP
No ReRank	$80.8$	$86.8$	$92.1$	$65.1$	$68.9$	$79.4$	$87.3$	$49.8$
SSR ReRank	$81.2$	$87.4$	$91.9$	$66.7$	$69.4$	$79.0$	$86.1$	$54.2$
$\alpha$ QE	$81.1$	$85.8$	$90.7$	$68.1$	$70.9$	$78.8$	$84.7$	$56.9$
RRT	$81.9$	$86.7$	$92.4$	$67.2$	$68.7$	$85.0$	$95.6$	$55.6$
LoCoRE-tiny	$82.4$	$87.8$	$93.1$	$68.0$	$71.4$	$86.8$	$96.4$	$58.1$

Table 4: Retrieval performance (%) on SoP and CUB-200 (DINOv2). mAP refers to mAP@R.

VLM	Methods	Shirt		Dress		Toptee
		R@10	R@50	R@10	R@50	R@10	R@50
	SEARLE	$24.4$	$41.6$	$18.5$	$39.5$	$25.7$	$46.4$
	iSEARLE-OTI	$27.0$	$43.4$	$21.2$	$42.1$	$26.8$	$48.7$
	CIReVL	$28.3$	$47.8$	$25.2$	$46.3$	$31.2$	$53.8$
	ImageScope	$31.6$	$37.4$	$26.8$	$46.3$	$35.8$	$55.9$
	EviRank-mini	$31.7$	$38.4$	$27.9$	$47.4$	$37.4$	$56.5$

Table 5: Retrieval performance (%) on FashionIQ.

	T $\rightarrow$ I		I $\rightarrow$ I
Variants	Flickr30K	COCO	SoP	CUB-200
EviRank-pro	96.9	89.4	87.6	99.4
w/o B0	94.5 $\downarrow$ 2.4%	88.6 $\downarrow$ 0.8%	82.3 $\downarrow$ 5.9%	96.5 $\downarrow$ 2.9%
w/o Slots	93.5 $\downarrow$ 3.4%	87.9 $\downarrow$ 1.6%	85.4 $\downarrow$ 2.4%	96.3 $\downarrow$ 3.1%
w/o Evidence	93.2 $\downarrow$ 3.8%	84.2 $\downarrow$ 5.7%	84.1 $\downarrow$ 3.9%	95.9 $\downarrow$ 3.5%
w/o Required	94.1 $\downarrow$ 2.8%	86.6 $\downarrow$ 3.0%	84.8 $\downarrow$ 3.0%	96.0 $\downarrow$ 3.4%

Table 6: Ablation results on T

\rightarrow

I and I

\rightarrow

I tasks with CLIP-ViT-L/14.

	(T,I) $\rightarrow$ I
Variants	Shirt	Dress	Toptee
EviRank-pro	40.1	38.9	49.7
w/o B0	37.6 $\downarrow$ 6.2%	35.4 $\downarrow$ 8.8%	46.5 $\downarrow$ 6.3%
w/o Slots	37.6 $\downarrow$ 6.0%	35.6 $\downarrow$ 8.2%	44.4 $\downarrow$ 10.5%
w/o Evidence	36.9 $\downarrow$ 7.8%	34.7 $\downarrow$ 10.5%	42.9 $\downarrow$ 13.5%
w/o Required	37.1 $\downarrow$ 7.3%	35.7 $\downarrow$ 8.0%	44.6 $\downarrow$ 10.0%

Table 7: Ablation results on composed multimodal retrieval tasks with CLIP-ViT-L/14.

Setting	COCO	SoP	FashionIQ
Rubric-only	59.9 / 82.4	83.7 / 69.8	32.1 / 50.9
Listwise-only	62.5 / 85.6	87.9 / 75.7	37.3 / 55.6
Rubric+Listwise	64.6 / 88.4	91.5 / 79.8	42.9 / 60.4

Table 8: Rubric vs. listwise component analysis. M1: R@1 (T

\rightarrow

I/I

\rightarrow

I) / R@10 (CIR). M2: R@10 / mAP / R@50 respectively.

Figure 3: Pairwise SBERT similarity between the six evidence slots over 10k queries (off-diagonal mean 0.18).

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Student backbone	Qwen3-VL-2B-Thinking
Training epochs	10
Batch size	32
Learning rate	$5\times 10^{-5}$
Optimizer	AdamW
LR schedule	Cosine decay

Appendix

Table 11: Distillation hyperparameters for Stage C.

Method	Flickr30k	COCO
VLM2Vec	80.0	49.2
RetLLM	82.0	54.1
UniME-v2	85.5	60.9
ReMatch	85.6	62.8
EviRank-pro	95.6	69.5

Appendix

Table 13: Comparison with recent SOTA T

\rightarrow

I baselines (R@1, %).

Backbone	Methods	R@1	R@5	R@10	MRR@5
EVA-CLIP-18B	No ReRank	$84.08$	$95.33$	$96.97$	$89.75$
	ImageScope	$84.89$	$95.68$	$97.01$	$90.12$
	CoTRR	$85.69$	$96.21$	$97.34$	$90.37$
	CoTMR	$85.93$	$96.82$	$97.69$	$90.96$
	EviRank-mini	$84.89$	$96.47$	$97.52$	$90.97$
	EviRank	$86.65$	$96.93$	$98.01$	$91.11$

Appendix

Table 14: Full Flickr30k results across all backbones and baselines.

	SoP				CUB-200
Methods	R@1	R@5	R@10	mAP	R@1	R@2	R@4	mAP
No ReRank	$80.80$	$86.87$	$92.10$	$65.10$	$68.90$	$79.40$	$87.30$	$49.80$
SSR ReRank	$81.20$	$87.42$	$91.90$	$66.70$	$69.40$	$79.00$	$86.10$	$54.20$
AQE	$76.90$	$73.69$	$89.30$	$66.10$	$66.90$	$76.80$	$82.70$	$58.40$
DQE	$67.90$	$72.71$	$81.50$	$47.80$	$67.00$	$75.50$	$82.00$	$54.60$
$\alpha$ QE	$81.10$	$85.86$	$90.70$	$68.10$	$70.90$	$78.80$	$84.70$	$56.90$

Appendix

Table 15: Full SoP and CUB-200 results with all I

\rightarrow

I baselines.

VLM	Methods	Shirt		Dress		Toptee
		R@10	R@50	R@10	R@50	R@10	R@50
CLIP-ViT-B/32	PALAVRA	$21.49$	$37.05$	$17.25$	$35.94$	$20.55$	$38.76$
	SEARLE	$24.44$	$41.61$	$18.54$	$39.51$	$25.70$	$46.46$
	SEARLE-OTI	$25.37$	$41.32$	$17.85$	$39.91$	$24.12$	$45.79$
	iSEARLE	$25.81$	$43.52$	$20.92$	$42.19$	$26.47$	$48.70$
	iSEARLE-OTI	$27.09$	$43.42$	$21.27$	$42.19$	$26.82$	$48.75$

Appendix

Table 16: Full FashionIQ results across all baselines and backbones.

Slot	Required	Forbidden	Ignore
Entities	The image must contain an acoustic guitar with a hollow wooden body and a visible sound hole. The guitar should have a traditional construction with a neck and fretboard clearly visible.	The guitar must not be an electric guitar with solid body and pickups. The instrument should not be a ukulele, bass guitar, banjo, or any other stringed instrument.	The specific brand or manufacturer of the guitar is not relevant for matching. The exact number of strings or whether a pick is being used does not affect the match.
Attributes	The dress must have blue as its primary base color. The fabric must display a floral or flower-based pattern with visible botanical motifs.	The dress should not be a solid single color without any pattern. The pattern must not be geometric stripes, polka dots, or abstract non-floral designs.	The exact shade or tone of blue does not need to match precisely. The specific type of flowers in the pattern and the overall dress length are not critical factors.
Actions	The woman must be actively engaged in playing the guitar with fingers positioned on strings. The posture should indicate ongoing musical performance rather than static positioning.	The woman should not be merely holding the guitar as a prop without playing engagement. The scene must not show her posing statically for a photograph with the guitar as an accessory.	The specific playing technique such as strumming versus fingerpicking is not relevant. The particular song or music being performed does not affect the matching.
Relations	The woman’s hands must be positioned on the guitar in a playing configuration, with one hand on the neck and one near the body. The guitar must be held in an active playing position against her body.	The guitar should not be lying flat on a surface or hanging on a wall. The guitar must not be stored in a case or held passively without playing intent.	Whether the woman is sitting or standing while playing is not relevant. The specific hand positioning for left-handed versus right-handed playing does not affect the match.
Scene	The setting must be an outdoor café environment with visible tables, chairs, or typical café furniture. The lighting should indicate an outdoor daytime setting.	The location must not be an indoor concert stage or formal performance venue. The setting should not be a recording studio, private bedroom, or enclosed indoor space.	The number of other patrons or customers visible in the scene is not relevant. The specific café name, branding, or current weather conditions do not affect the match.
KeyDetails	The overall atmosphere must suggest a casual, informal musical performance in a relaxed social setting. The context should indicate spontaneous or low-key entertainment rather than organized performance.	The scene must not depict a formal concert with audience seating arrangements. The context should not suggest a music competition, audition, or professional street busking for monetary collection.	The specific time of day or season when the scene takes place is not relevant. The geographic location or cultural context of the café does not affect the matching.

Appendix

Table 17: Slot Decomposition: structured evidence slots with required, forbidden, and ignored constraints.

Figure 10: Semantic similarity matrix across evidence slots.

Figure 11: Candidate images for slot-type ablation.

Explore similar work

AiSearch: Interactive Multi-Modal Search with VLMs
Oct 1, 2026Ali Koksal, Mei Chee Leong, Vicky Sintunata +2Multimodal IR Vision-Language Models
Institute of Advanced Intelligence and Computing (IAIC), Agency for Science, Technology and Research (A*STAR), Singapore
Graded-Relevance Composed Multimodal Retrieval for E-commerce Visual Search at Scale
Sep 21, 2026Anubhav Gupta, Hrushikesh Mohapatra, Prijith Chandra +6Information Retrieval Visual Search
Walmart Global Tech Bangalore, India · Walmart Global Tech Sunnyvale, United States
TIGER-FG: Text-Guided Implicit Fine-Grained Grounding for E-commerce Retrieval
May 18, 2026Xinyu Sun, Huangyu Dai, Lingtao Mao +5Multimodal IR Multimodal Grounding
Kuaishou Technology · Beijing, China

Oct 1, 2026, cs.AIJ/K move · Enter open · S save

AiSearch: Interactive Multi-Modal Search with VLMs

Ali Koksal, Mei Chee Leong, Vicky Sintunata, Ching Ling Chin+1

Institute of Advanced Intelligence and Computing (IAIC), Agency for Science, Technology and Research (A*STAR), Singapore

Multimodal IR Vision-Language Models Information Retrieval Vision-Language Retrieval

Open paper AskPDF arXiv

Modern retrieval systems must both be automated and interactive, allowing users to search and refine results in real time. We present AiSearch, a flexible multimodal retrieval framework that leverages the zero shot capabilities of Vision Language Models (VLMs) for natural language search over images and videos. AiSearch supports interactive search refinement through user feedback to tailor results to the user's intent, and allows visual benchmarking across multiple VLMs, enabling users to select the most suitable model for their task.

Abstract

Explore similar work

AiSearch: Interactive Multi-Modal Search with VLMs

Graded-Relevance Composed Multimodal Retrieval for E-commerce Visual Search at Scale

TIGER-FG: Text-Guided Implicit Fine-Grained Grounding for E-commerce Retrieval