Whispers from the Neighbors: Rethinking Caption Consensus as Inference-Time Evidence for Vision-Language Retrieval

Khang Nhat Hoang Vo; Belal Shoer; Abhirama Subramanyam Penamakuri

cs.CVOct 24, 2026

Whispers from the Neighbors: Rethinking Caption Consensus as Inference-Time Evidence for Vision-Language Retrieval

Authors: Khang Nhat Hoang Vo, Belal Shoer, Abhirama Subramanyam Penamakuri

Organizations: Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)

Abstract

An image is rarely described only once. Retrieval benchmarks and many collections attach several captions to each picture, yet dual encoders still judge every image-caption pair in isolation. We ask what these neighboring captions contribute when a frozen encoder consults them at inference time. Fusing image similarity with similarity to an image's own captions improves text-to-image R@1 by 3.4 to 16.8 points across nine frozen encoders when the captions come from the same annotators as the queries. The gain falls to 1.9 to 7.2 points when an off-the-shelf captioner writes them, and to at most one point when they come from a different annotation source; a paraphrase-trained text encoder strengthens the text side further. A natural refinement, scoring the part of the image that the captions leave unexplained, appears to add much more under a leave-one-out evaluation. We show that this advantage arises because the correct candidate's supports are built differently from those of the negative candidates, with set size and query removal each contributing about half, and that under a fixed support index the residual reduces to a reweighting of what the captions already provide. On compositional benchmarks, the captions help by matching candidates to human descriptions of the image, without improving image-text grounding. The code is available at: https://github.com/nhatkhangcs/clip_consensus

Figures & tables

Figure 1: With image similarity alone, the query is closest to a distractor. Adding the consensus of the correct image’s own captions recovers the correct match.

Figure 2: Overview of the consensus-residual scorer. For a candidate image, the support captions are averaged into a consensus direction, the image embedding is split into a component along this direction and a residual, and the query is scored against the image, the consensus, and the residual. Section 4.2 shows that, under a fixed support index, the residual path contributes little beyond the consensus path.

Figure 3: Three ways to exclude the query (

\star

) from the supports. Leave-one-out removes it from the positive only, so the positive’s support set differs from the negatives’ in both size and content. Size-matched LOO equalizes the size by dropping a random caption from each negative. The fixed support index holds out the same caption slot for every image, so support sets do not depend on the query.

	COCO eval2500			Flickr30K test1k
Encoder	Base	B+C	Full	Base	B+C	Full
OpenCLIP-B/32	46.90	57.11	58.41	66.76	78.90	80.42
MetaCLIP-B/16	48.84	56.85	57.86	70.72	80.56	81.36
DFN-B/16	50.26	57.84	59.42	67.78	78.44	79.82
DataComp-B/32	44.06	56.03	57.61	61.10	77.92	79.06
EVA02-B/16	49.25	57.64	58.62	71.38	80.32	81.04

Table 1: Text-to-image R@1 under the fixed support index, averaged over five folds; bold marks the best scorer per encoder and dataset. Every gain of B+C over Base has a 95% image-clustered bootstrap CI that excludes zero. Weights are tuned on COCO tune2500 only. Fixed-index I2T is in Appendix B.1 .

	COCO eval2500			Flickr30K test1k
Encoder	LOO	SM-LOO	Fixed	LOO	SM-LOO	Fixed
OpenCLIP-B/32	+7.60	+4.00	+0.17	+3.20	+1.40	+0.04
MetaCLIP-B/16	+7.92	+3.04	$-$ 0.03	+3.40	+0.40	+0.08
DFN-B/16	+6.24	+2.92	+0.07	+2.70	+0.70	+0.14
DataComp-B/32	+8.12	+4.60	+0.69	+3.00	+0.90	+0.58
EVA02-B/16	+5.28	+2.60	+0.24	+2.60	+0.60	+0.04

Table 2: Image-to-text gain of Full over a separately tuned B+C (

\Delta

R@1) under standard LOO, size-matched LOO (SM-LOO), and the fixed support index; bold marks gains whose 95% CI lies entirely above zero. Each scorer is tuned under the protocol it is evaluated with. Intervals and T2I are in Appendix C.1 .

	COCO eval2500					Flickr30K test1k
Encoder	Base	QB-Norm	B+C	Base $\oplus$ BM25	Base $\oplus$ GTE	Base	QB-Norm	B+C	Base $\oplus$ BM25	Base $\oplus$ GTE
OpenCLIP-B/32	46.90	48.90	57.11	55.71	59.76	66.76	66.70	78.90	77.38	82.46
MetaCLIP-B/16	48.84	50.95	56.85	56.11	60.64	70.72	70.52	80.56	77.80	83.86
DFN-B/16	50.26	52.38	57.84	56.82	60.69	67.78	67.60	78.44	76.14	81.72
DataComp-B/32	44.06	46.17	56.03	54.27	59.19	61.10	60.76	77.92	75.38	82.22
EVA02-B/16	49.25	51.11	57.64	55.93	60.25	71.38	70.00	80.32	77.82	83.18

Table 3: Text-to-image R@1 for metadata-free normalization and for different text sides, fixed support index; bold marks the best method per row and dataset. QB-Norm uses a query bank from tune2500. B+C scores supports with the encoder’s own text tower; Base

\oplus

BM25 and Base

\oplus

GTE fuse the image score with BM25 or GTE-large similarity to the supports (z-normalized, weight tuned on tune2500). Text-only retrieval with GTE-large alone reaches 54.68 (COCO) and 78.60 (Flickr30K). I2T and confidence intervals are in Appendix D.1 .

	Human, T2I		Generated, T2I		Generated, I2T
Encoder	COCO	F30K	COCO	F30K	COCO	F30K
OpenCLIP-B/32	+10.21	+12.14	+6.04	+5.20	+6.92	+4.20
MetaCLIP-B/16	+8.01	+9.84	+4.33	+4.54	+3.72	+3.10
DFN-B/16	+7.58	+10.66	+3.16	+4.10	+5.16	+3.90
DataComp-B/32	+11.97	+16.82	+6.05	+7.20	+7.32	+5.60
EVA02-B/16	+8.39	+8.94	+3.73	+3.10	+4.64	+2.80

Table 4: Gain of B+C over Base (

\Delta

R@1) with human same-source supports (fixed index) and with four supports generated by Qwen2.5-VL-3B ( Bai et al., 2025 ) . With generated supports, all five human captions serve as queries, so I2T is standard 5-positive R@1. Every gain has a 95% CI that excludes zero (Appendix E.1 ).

Figure 4: Text-to-image R@1 gain of B+C over Base on COCO. (a) Number of supports per image, with weights tuned for four supports. (b) Fraction of supports replaced by captions of other images, with weights kept at their clean values (dashed) or retuned at each replacement level (solid).

		ARO-Order		ARO-VG
Scorer	SugarC.	COCO	F30K	Rel.	Attr.
LM fluency (blind)	44.4	97.4	98.4	59.6	79.0
Random-image captions	63.1	18.6	22.1	53.6	55.3
Own-image captions	85.3	56.9	62.1	57.9	67.9
Base	81.4	31.1	38.1	49.9	58.8
B+C	86.6	48.2	55.4	54.7	64.9

Table 5: Compositional accuracy (%), averaged over the six encoders, with supports that exclude the positive caption and are identical for all candidates of an item. The first three rows never see the image: GPT-2 large log-likelihood of the candidate, and similarity to the consensus of a random other item’s captions or of the image’s own captions. SugarCrepe is averaged over items. Per-encoder results are in Appendix G.1 .

Figure 5: Text-to-image queries where support captions correct the ranking, drawn at random from all such queries. Each panel shows the top three images for one query under Base (top row) and Full (bottom row); green marks the correct image, which moves from rank 2 or 3 to rank 1.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

	Fixed		LOO
Backbone	B+C $\beta$	Full $\beta,\gamma$	B+C $\beta$	Full $\beta,\gamma$
SigLIP ViT-B/16	0.2	0.3, 0.6	0.1	1.0, 4.2
SigLIP2 ViT-B/16	0.2	0.3, 0.4	0.1	0.8, 2.6
SigLIP2 So400m/14	0.15	0.4, 0.8	0.1	0.8, 2.6
CLIP ViT-L/14	0.4	1.0, 1.0	0.2	2.2, 4.2
OpenCLIP ViT-B/32	0.4	1.8, 1.8	0.3	2.6, 4.2

Appendix

Table 6: Fusion weights selected on COCO tune2500 (

\alpha=1

). The six main encoders are SigLIP and the lower block; SigLIP2 ViT-B/16, SigLIP2 So400m/14, and CLIP ViT-L/14 are the additional encoders of Appendix H.1 .

	COCO			Flickr30K
Encoder	Base	B+C	Full	Base	B+C	Full
OpenCLIP-B/32	48.40	54.03	54.20	68.74	78.64	78.68
MetaCLIP-B/16	48.90	54.27	54.24	71.44	78.86	78.94
DFN-B/16	50.87	54.85	54.92	69.54	76.44	76.58
DataComp-B/32	46.15	51.95	52.64	63.38	74.82	75.40
EVA02-B/16	50.76	54.29	54.53	72.24	77.92	77.96

Appendix

Table 7: Fixed-index image-to-text R@1, averaged over five folds. Every gain of B+C over Base has a 95% CI that excludes zero.

	T2I				I2T
Encoder	COCO	CxC	ECCV	ECCV mAP@R	COCO	CxC	ECCV
OpenCLIP-B/32	+10.21	+10.21	+7.78	+6.26	+5.63	+5.71	+4.74
MetaCLIP-B/16	+8.01	+7.96	+4.82	+4.80	+5.37	+5.59	+5.13
DFN-B/16	+7.58	+7.61	+4.82	+5.17	+3.98	+4.16	+3.66
DataComp-B/32	+11.97	+12.00	+11.51	+8.64	+5.80	+6.08	+5.00
EVA02-B/16	+8.39	+8.47	+6.22	+4.65	+3.53	+3.78	+4.08

Appendix

Table 8: Gain of B+C over Base on COCO eval2500 with the original, CxC, and ECCV Caption positives (

\Delta

R@1 unless noted). ECCV uses its 643 text and 3,060 image queries that fall in eval2500.

Encoder	Pos. 0	Pos. 1	Pos. 2	Pos. 3	Pos. 4
Words (mean)	18.1	14.0	11.8	9.9	7.9
OpenCLIP-B/32	75.8 (+10.30)	69.1 (+13.60)	67.2 (+12.90)	64.8 (+10.60)	56.9 (+13.30)
MetaCLIP-B/16	79.5 (+7.80)	73.6 (+10.80)	72.3 (+10.10)	67.7 (+9.70)	60.5 (+10.80)
DFN-B/16	76.8 (+9.00)	71.9 (+10.20)	70.3 (+10.40)	64.3 (+11.10)	55.6 (+12.60)
DataComp-B/32	69.6 (+14.50)	63.9 (+19.20)	61.6 (+19.20)	57.9 (+16.50)	52.5 (+14.70)
EVA02-B/16	81.4 (+6.30)	72.8 (+10.60)	72.3 (+9.80)	67.9 (+8.70)	62.5 (+9.30)

Appendix

Table 9: Flickr30K Base T2I@1 by caption position in the Karpathy file, with the gain of B+C in parentheses.

		I2T@1			T2I@1
Encoder	$K_{s}$	Base	B+C	Full	Base	B+C	Full
SigLIP-B/16	1	54.96	54.31	55.51	54.60	54.43	55.97
	2	54.96	56.40	56.98	54.60	57.92	58.44
	3	54.96	57.02	57.66	54.60	59.02	59.54
	4	54.96	57.53	57.94	54.60	59.50	60.02
SigLIP2-B/16	1	54.02	53.17	53.73	55.80	55.07	55.91

Appendix

Table 10: COCO eval2500 R@1 with

K_{s}

supports per candidate, fixed index, weights tuned with four supports (mean over five folds and three seeds). Flickr30K shows the same trend.

	COCO			Flickr30K
Encoder	LOO	SM-LOO	Fixed	LOO	SM-LOO	Fixed
OpenCLIP-B/32	+0.75	$-$ 0.78	+1.30	+0.44	$-$ 0.82	+1.52
MetaCLIP-B/16	+0.18	$-$ 1.93	+1.01	$-$ 0.50	$-$ 1.98	+0.80
DFN-B/16	+1.87	+0.32	+1.58	+1.22	+0.12	+1.38
DataComp-B/32	+0.70	$-$ 0.95	+1.58	$-$ 1.20	$-$ 2.36	+1.14
EVA02-B/16	+1.65	+0.13	+0.98	+0.68	$-$ 0.30	+0.72

Appendix

Table 11: Text-to-image gain of Full over B+C (

\Delta

R@1) under the three support constructions; bold marks gains whose 95% CI lies entirely above zero. The image-to-text counterpart is Table 2 .

SigLIP ViT-B/16
Data	Supports	B+C $-$ Base	Full $-$ Base
COCO	real (LOO)	$-$ 1.88 / +2.94	+4.56 / +3.16
	random, asym	$-$ 8.16 / $-$ 2.27	$-$ 28.12 / $-$ 11.22
	random, sym	$-$ 5.72 / $-$ 0.50	$-$ 22.40 / $-$ 6.13
Flickr	real (LOO)	+1.40 / +4.20	+4.00 / +3.74
	random, asym	$-$ 8.80 / $-$ 1.68	$-$ 30.90 / $-$ 10.32

Appendix

Table 12: LOO with random supports. Cells are I2T@1 / T2I@1 differences from Base.

	COCO	Flickr30K
$s_{i}=\mathbf{v}_{i}^{\top}\mathbf{c}_{i}$	0.129 $\pm$ 0.026	0.142 $\pm$ 0.027
$\alpha_{\text{eff}}$	2.816 $\pm$ 0.006	2.819 $\pm$ 0.007
$\beta_{\text{eff}}$	0.165 $\pm$ 0.049	0.141 $\pm$ 0.051
$\beta_{\text{eff}}/\alpha_{\text{eff}}$	0.059	0.050

Appendix

Table 13: Effective weights of the full score for an illustrative setting (

\beta=0.4

\gamma=1.8

, SigLIP ViT-B/16), mean

\pm

std over images.

Variant	SigLIP ViT-B/16	SigLIP2 ViT-B/16	SigLIP2 So400m/14	CLIP ViT-L/14
V1	+0.09/+0.10; $-$ 0.42/ $-$ 0.16	0/0; 0/0 ( $\gamma=0$ )	0/0; 0/0 ( $\gamma=0$ )	$-$ 0.83/+1.02; $-$ 0.54/+0.38
V2	+0.61/+0.46; $-$ 0.06/ $-$ 0.06	+0.43/+0.38; +0.10/+0.54	$-$ 0.10/+0.08; +0.42/+0.80	+0.42/+0.54; +0.76/+0.24
V3	+0.42/ $-$ 0.62; $-$ 0.20/ $-$ 0.98	+0.21/ $-$ 0.15; +0.02/ $-$ 0.38	$-$ 0.04/ $-$ 0.06; +0.08/ $-$ 0.18	+0.14/ $-$ 0.67; +0.42/ $-$ 0.46
V4	+0.16/+0.50; $-$ 0.10/ $-$ 0.08	+0.27/+0.01; $-$ 0.08/ $-$ 0.12	$-$ 0.21/+0.34; $-$ 0.22/+0.26	+0.38/ $-$ 0.11; +0.86/+0.14

Appendix

Table 14: Change over tuned B+C, fixed protocol. Each cell is COCO (I2T@1 / T2I@1); Flickr30K (I2T@1 / T2I@1). “(

\gamma=0

)” means tuning switched the variant off.

Quantity	Shared	Specific
Consensus similarity $\rho$	0.41	$-$ 0.40
Caption residual norm $\eta$	$-$ 0.40	0.40
Residual–query similarity	$-$ 0.14	0.14

Appendix

Table 15: Correlations with shared and caption-specific entity counts on Flickr30K Entities. For unit vectors,

\eta=\sqrt{1-\rho^{2}}

, so the first two rows carry the same information.

Scorer	COCO I2T	COCO T2I	Flickr I2T	Flickr T2I
E5-large-v2, consensus	45.73	52.91	74.62	78.16
GTE-large, consensus	48.53	54.68	77.16	78.60
MPNet, consensus	44.91	52.94	75.24	77.70
E5-large-v2, max	41.89	48.42	70.58	74.52
GTE-large, max	43.75	49.65	72.70	75.08
MPNet, max	40.91	47.84	70.68	73.50

Appendix

Table 16: Text-only R@1 over the four support captions (no image), fixed index.

	Base $\oplus$ GTE				Base $\oplus$ BM25
	COCO		Flickr		COCO		Flickr
Encoder	T2I	I2T	T2I	I2T	T2I	I2T	T2I	I2T
OpenCLIP-B/32	+2.65	+3.94	+3.56	+5.04	$-$ 1.40	+1.09	$-$ 1.52	$-$ 0.56
MetaCLIP-B/16	+3.79	+3.82	+3.30	+4.78	$-$ 0.74	+2.04	$-$ 2.76	$-$ 0.40
DFN-B/16	+2.85	+4.20	+3.28	+6.72	$-$ 1.02	+2.82	$-$ 2.30	+1.38
DataComp-B/32	+3.16	+5.29	+4.30	+7.64	$-$ 1.76	+2.45	$-$ 2.54	+0.96

Appendix

Table 17: Change in R@1 relative to tuned B+C when the image score is fused with a separate text retriever over the supports. All GTE gains have 95% CIs that exclude zero.

	COCO				Flickr30K
Encoder	QB-Norm	Inv. softmax †	B+C	B+C + QB	QB-Norm	Inv. softmax †	B+C	B+C + QB
OpenCLIP-B/32	+2.00	+7.47	+10.21	+10.96	$-$ 0.06	+8.86	+12.14	+10.92
MetaCLIP-B/16	+2.11	+8.15	+8.01	+8.94	$-$ 0.20	+9.40	+9.84	+8.64
DFN-B/16	+2.12	+8.20	+7.58	+8.65	$-$ 0.18	+9.44	+10.66	+9.60
DataComp-B/32	+2.11	+8.61	+11.97	+12.56	$-$ 0.34	+10.04	+16.82	+15.26
EVA02-B/16	+1.86	+8.41	+8.39	+9.12	$-$ 1.38	+8.80	+8.94	+7.38

Appendix

Table 18: Text-to-image gain over Base (

\Delta

R@1) for normalization without metadata and for support captions. † Transductive: normalizes over the evaluated queries. Dual softmax is within 0.3 points of inverted softmax.

	COCO eval2500					Flickr30K test1k
	T2I			I2T		T2I			I2T
Encoder	Base	B+C	Full $-$ B+C	Base	B+C	Base	B+C	Full $-$ B+C	Base	B+C
OpenCLIP-B/32	46.90	52.94	+0.16	63.84	70.76	66.76	71.96	+0.78	84.20	88.40
MetaCLIP-B/16	48.84	53.17	+0.29	66.56	70.28	70.72	75.26	+0.14	85.50	88.60
DFN-B/16	50.26	53.42	+0.00	66.32	71.48	67.78	71.88	+0.52	83.10	87.00
DataComp-B/32	44.06	50.11	+0.02	61.84	69.16	61.10	68.30	+0.16	79.20	84.80

Appendix

Table 19: Retrieval with four generated supports per image; all five human captions are queries, so I2T is standard 5-positive R@1. Every gain of B+C over Base has a 95% CI that excludes zero.

	B+C $-$ Base
Backbone	Same-source (fixed)	X1	X2
SigLIP ViT-B/16	+2.57/+4.90; +6.00/+7.12	$-$ 0.52/+3.36; +0.50/+5.20	+0.60/+0.56; +0.30/+0.76
SigLIP2 ViT-B/16	+1.15/+4.02; +3.52/+5.18	+0.72/+0.80; +1.40/+2.40	0/0; 0/0 ( $\beta=0$ )
SigLIP2 So400m/14	+2.30/+3.43; +3.74/+3.48	+1.52/+2.36; +3.40/+2.20	0/0; 0/0 ( $\beta=0$ )
CLIP ViT-L/14	+4.44/+11.33; +6.82/+13.22	$-$ 0.32/+6.32; +5.00/+7.50	+0.24/+0.79; +1.00/+0.90
	Full $-$ B+C

Appendix

Table 20: Cross-source supports. Each cell is COCO (I2T@1 / T2I@1); Flickr30K (I2T@1 / T2I@1). X1: Localized Narrative query, human-caption supports. X2: human-caption query, narrative support.

Encoder	Set	Mismatch	Base	B+C fixed	B+C retuned	Full fixed	Full retuned
SigLIP-B/16	COCO	20%	54.60 / 54.96	58.09 / 55.87	58.02 / 56.52	56.33 / 56.30	58.85 / 56.46
SigLIP-B/16	Flickr	20%	74.60 / 74.90	79.79 / 78.98	79.42 / 79.16	79.09 / 79.00	80.16 / 79.10
SigLIP-B/16	COCO	40%	54.60 / 54.96	55.81 / 53.74	55.92 / 55.65	55.30 / 54.01	57.30 / 55.22
SigLIP-B/16	Flickr	40%	74.60 / 74.90	77.34 / 76.65	76.45 / 76.54	77.49 / 76.86	78.16 / 77.31
SigLIP-B/16	COCO	60%	54.60 / 54.96	52.97 / 51.05	55.11 / 55.09	54.94 / 51.12	55.11 / 55.09
SigLIP-B/16	Flickr	60%	74.60 / 74.90	73.97 / 73.93	75.35 / 75.40	76.09 / 73.82	75.35 / 75.40

Appendix

Table 21: Mismatched supports, fixed index; each cell is T2I@1 / I2T@1, averaged over five seeds (standard deviations at most 0.46). “Fixed” keeps the weights tuned on clean supports; “retuned” tunes them on tune2500 at each mismatch level.

Set	Method	I2T@1	T2I@1	$\Delta$ I2T vs. none	$\Delta$ T2I vs. none	$\Delta$ I2T Full $-$ B+C	$\Delta$ T2I Full $-$ B+C
COCO	ITC	69.17 $\pm$ 1.02	70.57 $\pm$ 0.67	+0.00 [+0.00, +0.00]	+0.00 [+0.00, +0.00]	–	–
COCO	ITC + B+C ( $\beta$ =0.2)	69.54 $\pm$ 0.88	71.52 $\pm$ 0.65	+0.37 [ $-$ 0.09, +0.77]	+0.95 [+0.58, +1.34]	–	–
COCO	ITC + Full ( $\beta$ =0.4, $\gamma$ =0.2)	69.52 $\pm$ 0.89	72.02 $\pm$ 0.64	+0.35 [ $-$ 0.07, +0.75]	+1.45 [+1.07, +1.84]	$-$ 0.02 [ $-$ 0.22, +0.20]	+0.50 [+0.24, +0.75]
COCO	ITM rerank top-16	74.68 $\pm$ 0.78	73.74 $\pm$ 1.08	+0.00 [+0.00, +0.00]	+0.00 [+0.00, +0.00]	–	–
COCO	ITM rerank + B+C ( $\beta_{1}$ =0.2, $\beta_{2}$ =4)	75.26 $\pm$ 0.93	74.66 $\pm$ 0.89	+0.58 [+0.20, +0.98]	+0.92 [+0.56, +1.27]	–	–
COCO	ITM rerank + Full ( $\beta_{1}$ , $\gamma_{1}$ =0.4,0.2; $\beta_{2}$ , $\gamma_{2}$ =6,1)	75.12 $\pm$ 0.83	74.83 $\pm$ 0.94	+0.44 [+0.02, +0.92]	+1.09 [+0.72, +1.46]	$-$ 0.14 [ $-$ 0.34, +0.06]	+0.17 [ $-$ 0.01, +0.37]

Appendix

Table 22: BLIP-2 ViT-g fine-tuned on COCO, fixed index (mean

\pm

std over folds; 95% CIs).

	SugarCrepe			COCO-Order			F30K-Order			VG-Rel.			VG-Attr.
Encoder	Rand.	Own	B+C	Rand.	Own	B+C	Rand.	Own	B+C	Rand.	Own	B+C	Rand.	Own	B+C
OpenCLIP-B/32	62.4	86.5	87.2	20.6	63.7	55.6	27.0	68.3	62.3	54.2	57.3	54.9	61.5	71.3	67.9
MetaCLIP-B/16	64.9	85.1	85.7	18.8	53.0	43.6	15.8	60.3	53.1	51.0	57.5	52.8	51.3	66.6	63.4
DFN-B/16	61.1	84.6	86.4	13.6	51.1	33.4	18.1	54.5	38.9	51.8	57.2	54.5	52.3	62.9	62.3
DataComp-B/32	62.0	85.5	86.7	18.3	58.4	54.1	23.2	62.4	59.1	54.3	57.9	56.4	56.8	67.7	66.6
EVA02-B/16	61.6	85.7	86.7	17.5	62.6	59.2	21.5	68.1	67.8	55.9	59.3	58.3	56.1	66.5	66.7

Appendix

Table 23: Per-encoder compositional accuracy (%): candidates scored against a random other item’s captions (Rand.), against the image’s own captions (Own), and with B+C. GPT-2 fluency does not depend on the encoder: 44.4 (SugarCrepe), 97.4 and 98.4 (ARO-Order), 59.6 and 79.0 (ARO-VG). Base and Full are in Table 5 (averaged).

Supports	Scorer	SigLIP ViT-B/16	SigLIP2 ViT-B/16	SigLIP2 So400m/14	CLIP ViT-L/14
fixed	Text only	79.9	76.8	77.6	80.2
fixed	Base	79.6	78.6	80.3	73.0
fixed	B+C	83.3	80.5	81.5	80.0
fixed	Full	82.9	80.2	81.6	79.3
leaky	B+C	86.3	84.0	84.7	86.1
leaky	Full	85.0	83.2	84.5	85.2

Appendix

Table 24: SugarCrepe accuracy (%), averaged over the seven subsets. Add-attribute and add-object gains come almost entirely from text (e.g., CLIP ViT-L/14 add-att: 71.4 Base, 93.4 text only, 90.0 B+C).

Backbone	Benchmark		Text only	Base	B+C	Full
SigLIP ViT-B/16	ARO-Order	COCO	52.6 (40.5)	33.0 (25.7)	43.6 (33.6)	41.2 (31.8)
		Flickr	58.8 (44.3)	40.6 (31.5)	51.3 (39.2)	49.4 (37.5)
	ARO-VG	Rel.	58.0	46.6	51.1	50.0
		Attr.	72.4	56.1	62.5	60.7
SigLIP2 ViT-B/16	ARO-Order	COCO	53.4 (41.4)	25.6 (19.8)	39.9 (30.6)	38.2 (29.3)
		Flickr	57.6 (43.2)	30.4 (23.0)	45.8 (33.6)	44.2 (32.4)

Appendix

Table 25: ARO accuracy (%) with fixed supports. ARO-Order uses argmax accuracy with strict tie-breaking in parentheses. With the positive left among the supports, B+C on ARO-Order rises by a further 7–13 points.

		COCO eval2500		Flickr30K test1k
Backbone	Protocol	$\Delta$ I2T@1	$\Delta$ T2I@1	$\Delta$ I2T@1	$\Delta$ T2I@1
SigLIP ViT-B/16	LOO	+6.44 [+5.44, +7.40]	+0.22 [ $-$ 0.69, +1.09]	+2.60 [+1.70, +3.60]	$-$ 0.46 [ $-$ 1.50, +0.62]
	Fixed	+0.41 [+0.10, +0.72]	+0.52 [+0.05, +0.99]	$-$ 0.46 [ $-$ 0.92, $-$ 0.02]	+0.04 [ $-$ 0.48, +0.58]
SigLIP2 ViT-B/16	LOO	+5.24 [+4.40, +6.12]	$-$ 0.50 [ $-$ 1.34, +0.31]	+2.90 [+1.90, +4.00]	$-$ 1.02 [ $-$ 2.12, +0.00]
	Fixed	+0.09 [ $-$ 0.12, +0.32]	+0.07 [ $-$ 0.34, +0.47]	$-$ 0.14 [ $-$ 0.46, +0.18]	+0.08 [ $-$ 0.48, +0.62]
SigLIP2 So400m/14	LOO	+4.40 [+3.60, +5.24]	+0.51 [ $-$ 0.20, +1.29]	+2.20 [+1.40, +3.20]	+0.26 [ $-$ 0.66, +1.14]

Appendix

Table 26: Gain of Full over a separately tuned B+C scorer (R@1, 95% image-clustered bootstrap CI); bold marks gains whose interval lies entirely above zero. Under LOO, the positive image’s supports exclude the query while negatives keep all captions; under the fixed protocol, all candidates share one support index. Each scorer is tuned on COCO tune2500 under the protocol it is evaluated with.

		COCO eval2500		Flickr30K test1k
Backbone	Scorer	I2T@1	T2I@1	I2T@1	T2I@1
SigLIP ViT-B/16	Base	54.96 $\pm$ 0.88	54.60 $\pm$ 1.34	74.90 $\pm$ 5.72	74.60 $\pm$ 6.62
	B+C	57.53 $\pm$ 1.44	59.50 $\pm$ 1.73	80.90 $\pm$ 4.94	81.72 $\pm$ 6.10
	Full	57.94 $\pm$ 1.18	60.02 $\pm$ 1.72	80.44 $\pm$ 5.55	81.76 $\pm$ 6.70
SigLIP2 ViT-B/16	Base	54.02 $\pm$ 0.46	55.80 $\pm$ 0.48	73.14 $\pm$ 6.79	73.80 $\pm$ 7.17
	B+C	55.17 $\pm$ 0.48	59.82 $\pm$ 0.50	76.66 $\pm$ 4.80	78.98 $\pm$ 7.86

Appendix

Table 27: R@1 under the fixed support index, mean

\pm

standard deviation over the five folds.

Backbone	COCO $\Delta$ I2T@1	COCO $\Delta$ T2I@1	Flickr $\Delta$ I2T@1	Flickr $\Delta$ T2I@1
SigLIP ViT-B/16	+2.57 [+1.88, +3.28]	+4.90 [+4.30, +5.52]	+6.00 [+4.98, +7.00]	+7.12 [+6.12, +8.12]
SigLIP2 ViT-B/16	+1.15 [+0.50, +1.79]	+4.02 [+3.44, +4.64]	+3.52 [+2.62, +4.58]	+5.18 [+4.20, +6.12]
SigLIP2 So400m/14	+2.30 [+1.67, +2.90]	+3.43 [+2.95, +3.95]	+3.74 [+2.96, +4.64]	+3.48 [+2.76, +4.22]
CLIP ViT-L/14	+4.44 [+3.58, +5.29]	+11.33 [+10.37, +12.11]	+6.82 [+5.60, +8.12]	+13.22 [+11.72, +14.72]

Appendix

Table 28: Gain of tuned B+C over Base under the fixed support index, with 95% image-clustered bootstrap CIs.

	T2I@1				I2T@1
Method	SigLIP	SigLIP2-B	So400m	CLIP-L	SigLIP	SigLIP2-B	So400m	CLIP-L
Base image–text	$-$ 4.90	$-$ 4.02	$-$ 3.43	$-$ 11.33	$-$ 2.57	$-$ 1.15	$-$ 2.30	$-$ 4.44
Text only: query vs. consensus	$-$ 8.89	$-$ 10.97	$-$ 12.62	$-$ 4.50	$-$ 17.02	$-$ 17.68	$-$ 18.93	$-$ 11.68
Text only: max support sim.	$-$ 13.10	$-$ 16.41	$-$ 17.66	$-$ 10.43	$-$ 23.86	$-$ 23.86	$-$ 24.70	$-$ 15.78
Text only: softmax support sim.	$-$ 12.20	$-$ 16.05	$-$ 15.48	$-$ 9.01	$-$ 18.47	$-$ 19.54	$-$ 20.60	$-$ 12.56
Z-norm fusion	+0.41	$-$ 0.38	$-$ 0.06	$-$ 0.40	+0.44	+0.42	+0.02	+0.39

Appendix

Table 29: Change in R@1 over tuned linear B+C on COCO eval2500, fixed protocol. ∗ Tuning set the consensus weight to zero, reducing RRF to Base. Embedding fusion equals B+C in I2T by construction. Flickr30K shows the same pattern.

Figure 6: Text-to-image queries where support captions hurt the ranking, drawn at random from all such queries. The correct image (green) drops from rank 1 under Base to rank 2 under Full; in each case the new top-ranked image is also a plausible match for the generic query.

Vision-Language Retrieval

Vision-language retrieval finds images, video, or documents that match natural-language queries and supports search across modalities.

See all 97 papers

A Paragraph is Worth a Thousand Captions: Rethinking Text Supervision for Vision-Language Retrieval
Aug 5, 2026Mahyar Ghazanfari, Amin Tabrizian, Arsyi Aziz +2Image-Text Retrieval Vision-Language Retrieval
The George Washington University
MINER: Multi-crop INference-time Enhancement for Rare-Object Retrieval with Frozen Dual Encoders
Sep 22, 2026Abdulmalik Alquwayfili, Faisal AlMeshal, Jumanah Almajnouni +2Vision-Language Retrieval Image Retrieval
Saudi Data and Artificial Intelligence Authority (SDAIA)
Re $^3$ Cap: Retrieval-Guided Refinement for Image Captioning Enhancement via Reinforcement Learning
Aug 21, 2026Haonan Jia, Shichao Dong, Zenghui Sun +7Image Captioning RL for VLMs
Taobao & Tmall Group of Alibaba · The Hong Kong University of Science and Technology (Guangzhou) · Shanghai Artificial Intelligence Laboratory

Aug 5, 2026J/K move · Enter open · S save

A Paragraph is Worth a Thousand Captions: Rethinking Text Supervision for Vision-Language Retrieval

Mahyar Ghazanfari, Amin Tabrizian, Arsyi Aziz, Binshuai Wang+1

The George Washington University

Image-Text Retrieval Vision-Language Retrieval

Open paper AskPDF arXiv

Contrastive vision-language models such as CLIP and BLIP are typically trained on short image captions, limiting their ability to retrieve images from detailed textual descriptions. While methods such as Long-CLIP extend the token limit through positional embedding interpolation, we ask a simpler question: does training text granularity alone determine long-text retrieval performance? We present a systematic study of supervision ranging from single captions to multi-sentence paragraphs for contrastive image-text retrieval. Using a synthetic pipeline based on Qwen2-VL and Llama 3.2 Vision, we generate diverse captions, hard negatives, and quality-scored paragraphs for 500K CC3M images. To isolate the effect of text granularity, we fine-tune only the BLIP text encoder while keeping the vision encoder frozen across 10 training configurations. Our paragraph-supervised models match Long-CLIP-L on ShareGPT4V and outperform it by more than 14 points on DOCCI for image-to-text retrieval, without architectural changes. We further show that paragraph supervision enables effective use of long token sequences, whereas caption-only training degrades beyond 60 tokens. Increasing caption diversity improves short-caption retrieval with diminishing returns, while paragraph supervision consistently benefits long-description benchmarks and hard negatives prove detrimental in text-only fine-tuning. Evaluations on Flickr30k, COCO, ShareGPT4V, and DOCCI provide a comprehensive analysis of the trade-offs between text granularity, retrieval direction, and description length.

Abstract

Vision-Language Retrieval

Explore similar work

A Paragraph is Worth a Thousand Captions: Rethinking Text Supervision for Vision-Language Retrieval

MINER: Multi-crop INference-time Enhancement for Rare-Object Retrieval with Frozen Dual Encoders

Re3^33Cap: Retrieval-Guided Refinement for Image Captioning Enhancement via Reinforcement Learning

Re $^3$ Cap: Retrieval-Guided Refinement for Image Captioning Enhancement via Reinforcement Learning