Which Negatives Matter? Ask Your Text Encoder: Adaptive Similarity Margins for Dense-Caption Retrieval

Haoyue Liu; Ye Chen; Zhichao Wang; Xiaoying Tang

cs.AIAug 19, 2026

Which Negatives Matter? Ask Your Text Encoder: Adaptive Similarity Margins for Dense-Caption Retrieval

Authors: Haoyue Liu, Ye Chen, Zhichao Wang, Xiaoying Tang

Organizations: School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen 518172, China · XJTU-POLIMI Joint School, Xi’an Jiaotong University, Xi’an 710049, China · Shenzhen Future Network of Intelligence Institute (FNii-Shenzhen)

Abstract

Dense-caption retrieval has recently been improved by introducing segmentation, edge maps, LLM-filtered captions, and cross-modal modules into contrastive fine-tuning. However, these methods largely inherit the same InfoNCE objective, whose optimization can prematurely saturate under a strong pre-trained initialization: on dense captions, the loss falls below 10^-3 on 80% of batches within the first epoch, while its gradient becomes numerically zero in 47% of measurements. We find that this behavior is closely related to the large number of near-duplicate captions in dense-caption benchmarks, where a few highly similar negatives remain unresolved after the easy majority has already been separated. As a remedy, we introduce HN-CLIP, which uses the text encoder's own text-text geometry to construct per-negative adaptive similarity margins. Specifically, a detached caption-similarity matrix is added to the negative logits, assigning larger margins to more similar captions without mining, synthesizing, or resampling negatives. The resulting objective requires only one caption-similarity matrix and a masked logit addition during training, with no auxiliary data, additional parameters, offline preprocessing, or inference-time overhead. Extensive experiments on four dense-caption retrieval benchmarks show that HN-CLIP improves over the strongest competitors by +2.5 to +4.0 R@1 while training 2.4x faster than GOAL and 5.4x faster than StructXLIP. Moreover, the proposed objective improves all six tested fine-tuning frameworks on the in-domain benchmarks and reaches the strongest full-data baseline with only 20% of the training data.

Figures & tables

Figure 1: Dense-caption benchmarks are dominated by hard negatives. Distributions of (a) all pairwise caption-caption cosine similarities and (b) each caption’s hardest-negative similarity, measured with the pre-trained Long-CLIP-L text encoder on the test sets. Dotted lines mark the means. The pairs in (b) receive the strongest boosts.

Figure 2: Overview of HN-CLIP. (a) A batch of image–text pairs is encoded by the jointly fine-tuned dual encoder. (b) Caption–caption similarity provides detached, per-negative adaptive margins for the global contrastive loss, combined with the token-level term. (c) More similar negatives receive larger margins, while the positive is unchanged. HN-CLIP requires no mining, resampling, extra data, additional parameters, or inference-time overhead.

Figure 3: Illustration of HN-CLIP. A real DOCCI query and its hardest in-batch negative (cos $0.89$ ). The strongest baseline ranks the ground truth 52nd (its top-1 (pink) is the near-twin’s own image); HN-CLIP ranks it first.

Figure 4: Empirical gradient analysis (Long-CLIP-L, DOCCI, 10 epochs; losses and full-parameter gradients measured every 5 optimizer steps, 148 measurements). Standard InfoNCE declares $80\%$ of batches solved within the first epoch and its gradient is numerically zero in $47\%$ of measurements (median zero at five epochs; plotted clamped to $10^{-10}$ ). The boosted loss keeps at least $26\%$ of batches active in every epoch and, where the standard gradient is nonzero, exceeds it by $10^{2}$ – $10^{6}\times$ in per-epoch median, with interquartile bands disjoint at nine epochs, and the retrieval error it buys keeps falling. Its gradient-norm distribution is bimodal across batches; see Appendix C .

	DOCCI						DCI
Method	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10
Long-CLIP [ECCV’24]	78.78	95.24	98.02	66.75	91.92	96.31	67.83	83.19	87.69	64.13	84.84	89.74
FineLIP [CVPR’25]	77.51	96.02	98.41	69.90	93.43	97.45	72.69	87.14	90.65	65.48	86.84	91.00
GOAL [CVPR’25]	81.53	97.02	98.80	80.86	96.24	98.63	77.29	90.25	93.30	74.84	89.94	93.25
StructXLIP [CVPR’26]	84.73	97.69	99.00	82.61	97.08	98.71	75.84	89.94	93.65	74.49	90.05	93.40
HN-CLIP	88.25	98.45	99.43	86.24	98.12	99.22	80.69	92.40	95.10	78.84	91.90	94.65

Table 1: Cross-modal retrieval performance of CLIP-based fine-tuning methods on four dense-caption benchmarks. We report Recall@K (%) on both Text

\rightarrow

Image and Image

\rightarrow

Text settings. All fine-tuned methods start from the same Long-CLIP-L backbone with an identical training budget; Long-CLIP denotes the released checkpoint. Best results in bold ; second best underlined .

\Delta

denotes the margin over the best competitor per column, with gain in

\uparrow

green .

Figure 5: Convergence comparison. Average R@1 (T $\rightarrow$ I, I $\rightarrow$ T) per fine-tuning epoch. HN-CLIP’s first epoch matches or exceeds most baselines’ final accuracy on all four benchmarks.

	DOCCI						Long-DCI
Method	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10
Long-CLIP [ECCV’24]	86.45	98.00	99.31	84.10	97.84	99.04	70.24	89.62	94.00	68.44	89.35	94.67
+ our $\mathcal{L}_{\mathrm{HN}}$	88.31	98.75	99.45	86.39	98.08	99.20	78.43	92.52	95.27	76.02	91.78	94.76
$\Delta$	$\uparrow$ 1.86	$\uparrow$ 0.75	$\uparrow$ 0.14	$\uparrow$ 2.29	$\uparrow$ 0.24	$\uparrow$ 0.16	$\uparrow$ 8.19	$\uparrow$ 2.90	$\uparrow$ 1.27	$\uparrow$ 7.58	$\uparrow$ 2.43	$\uparrow$ 0.09
FineLIP [CVPR’25]	77.51	96.02	98.41	69.90	93.43	97.45	59.24	77.86	83.19	49.52	75.08	82.39
+ our $\mathcal{L}_{\mathrm{HN}}$	85.47	97.71	99.04	83.88	97.41	98.82	74.92	90.52	93.62	73.39	89.87	93.23

Table 2: Plug-and-play enhancement of our

\mathcal{L}_{\mathrm{HN}}

on CLIP-based fine-tuning. Results on DOCCI and Long-DCI for Text

\rightarrow

Image and Image

\rightarrow

Text retrieval. Upper: full-parameter fine-tuning; lower: parameter-efficient tuning. Our method consistently boosts diverse CLIP variants, and the gains grow with caption length. Best in bold , with gain in

\uparrow

green .

Figure 6: Sample efficiency (avg R@1, identical subsets). HN-CLIP at $20\%$ of the data already clears the strongest baseline trained on $100\%$ (dotted line, from Table 1 ); full numbers in Table A9 .

	DOCCI		DCI		Long-DCI		Urban-1K
$\gamma$	R@1	R@1	R@1	R@1	R@1	R@1	R@1	R@1	Avg
$\gamma{=}0$	86.45	84.29	79.24	77.54	70.96	68.08	92.00	92.90	81.43
$\gamma{=}0.25$	87.82	86.63	80.69	79.19	76.37	74.69	91.70	93.10	83.77
$\gamma{=}0.5$ (default)	88.25	86.24	80.69	78.84	78.90	76.24	90.20	90.70	83.76
$\gamma{=}0.75$	88.20	85.73	80.64	78.39	79.82	76.59	88.80	89.70	83.48
$\gamma{=}1.0$	87.98	85.02	80.04	78.29	80.05	76.48	88.20	88.30	83.05

Table 3: Ablations. (a) Sensitivity to the boost strength

\gamma

(Recall@1): every

\gamma\in[0.25,1]

beats

\gamma{=}0

in-domain, while under transfer (Urban-1K) milder boosts generalize better. (b) Loss components under an identical recipe; only the loss changes. (c)

\bar{G}

recomputed from the live encoder (default) vs. frozen at initialization; R@1/5/10 in Table A2 . Best per column in bold .

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	#train	#test	words	pairwise sim	hardest sim
DOCCI	9450	5100	123	0.84	0.93
DCI	5445	1999	133	0.85	0.92
Long-DCI	5445 (=DCI)	7444	134	0.85	0.93
Urban-1K	14579 (VG)	1000	107	0.88	0.94

Appendix

Table A1: Benchmark statistics. Mean caption length is measured on the evaluation split in words; similarities use the pre-trained Long-CLIP-L text encoder.

Figure A1: Near-duplicate caption pairs on DOCCI. Each row: a test image, its caption, the three nearest other captions’ images (cosine printed underneath), and the caption texts with shared content words highlighted .

Figure A2: Near-duplicate caption pairs on DCI.

Figure A3: Near-duplicate caption pairs on Long-DCI (full-length captions).

Figure A4: Near-duplicate caption pairs on Urban-1K.

Figure A5: One real training batch per benchmark. Text–text similarity matrices $\bar{G}$ (pre-trained encoder; diagonal masked, max off-diagonal entry annotated). Uniformly dark $=$ every negative is hard; this matrix, detached and scaled by $\gamma$ , is the entire mechanism of HN-CLIP.

Figure A6: Statistical analysis of the caption geometry across the four benchmarks (pre-trained Long-CLIP-L text encoder, full test splits). (A) Mean pairwise caption similarity. (B) Mean similarity of each caption’s hardest companion. (C) Composition of in-batch negatives by hardness bucket.

	DOCCI						DCI
$\bar{G}$ source	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10
Recomputed $\bar{G}$ (default)	88.25	98.45	99.43	86.24	98.12	99.22	80.69	92.40	95.10	78.84	91.90	94.65
Frozen $\bar{G}$	85.86	97.61	99.18	82.43	96.82	98.73	77.79	90.80	93.35	75.84	89.49	92.95
$\Delta$ (default $-$ frozen)	$\uparrow$ 2.39	$\uparrow$ 0.84	$\uparrow$ 0.25	$\uparrow$ 3.81	$\uparrow$ 1.30	$\uparrow$ 0.49	$\uparrow$ 2.90	$\uparrow$ 1.60	$\uparrow$ 1.75	$\uparrow$ 3.00	$\uparrow$ 2.41	$\uparrow$ 1.70
best epoch (default / frozen)	6 / 5						5 / 1
	Long-DCI						Urban-1K

Appendix

Table A2: Recomputed vs. frozen

\bar{G}

(Recall@K, Text

\rightarrow

Image and Image

\rightarrow

Text ). Building the boost from a frozen copy of the pre-trained text encoder makes the margins exactly stationary but removes their implicit annealing: accuracy peaks at epoch 1 on DCI and Urban-1K and then declines, and the default wins all 24 columns. Best per column in bold . The final row reports the selected best epoch as default/frozen.

Figure A8: Gradient dynamics across data scales. Rows: fraction of the training set; columns: dataset. Per cell: (a) raw loss traces (thin) with running medians (thick), (b) gradient norms and their ratio (dashed), (c) gradient cosine. Log axes in (a,b); values clamped at $10^{-10}$ (numerically zero values).

Method	Aux. training inputs	Offline prep.	Wall-clock	Throughput
Long-CLIP (plain FT)	none	none	$\approx$ 16 min	55.5 img/s
FineLIP	none	none	—	—
GOAL	SAM segments	required	$\approx$ 41 min	$\approx$ 22 img/s
StructXLIP	edges + LLM lexicon	required	$\approx$ 92 min	$\approx$ 10 img/s
HN-CLIP (ours)	none	none	17 min	53 img/s

Appendix

Table A3: Training-efficiency comparison on DCI fine-tuning (10 epochs, 5.4k images, single Ascend 910B, effective batch 128). Wall-clock for GOAL/StructXLIP is measured from same-device sequential runs minus evaluation overhead; FineLIP ran in parallel and is not attributable. Offline preprocessing time (segmentation, edge extraction, LLM filtering) is not included in the wall-clock.

	DOCCI						Long-DCI
Method	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10
Long-CLIP	86.45	98.00	99.31	84.10	97.84	99.04	70.24	89.62	94.00	68.44	89.35	94.67
+our $\mathcal{L}_{\mathrm{HN}}$	88.31	98.75	99.45	86.39	98.08	99.20	78.43	92.52	95.27	76.02	91.78	94.76
$\Delta$	$\uparrow$ 1.86	$\uparrow$ 0.75	$\approx$ 0	$\uparrow$ 2.29	$\uparrow$ 0.24	$\uparrow$ 0.16	$\uparrow$ 8.19	$\uparrow$ 2.90	$\uparrow$ 1.27	$\uparrow$ 7.58	$\uparrow$ 2.43	$\approx$ 0
FineLIP	77.51	96.02	98.41	69.90	93.43	97.45	59.24	77.86	83.19	49.52	75.08	82.39
+our $\mathcal{L}_{\mathrm{HN}}$	85.47	97.71	99.04	83.88	97.41	98.82	74.92	90.52	93.62	73.39	89.87	93.23

Appendix

Table A4: Plug-and-play enhancement of

\mathcal{L}_{\mathrm{HN}}

on CLIP-based fine-tuning. Results on DOCCI and Long-DCI for Text

\rightarrow

Image and Image

\rightarrow

Text retrieval; per framework we report the official baseline, the same recipe with

\mathcal{L}_{\mathrm{HN}}

replacing its global InfoNCE term, and the per-column difference

\Delta

(gain in

\uparrow

green , drop in

\downarrow

gray ; differences within

\pm 0.15

are shown as

\approx

0 ). Bold marks the

+\mathcal{L}_{\mathrm{HN}}

value where it is the better of the pair.

		DOCCI										DCI
$\mathcal{L}_{\mathrm{tok}}$	$\mathcal{L}_{\mathrm{HN}}$	R@1	R@5	R@10	R@25	R@50	R@1	R@5	R@10	R@25	R@50	R@1	R@5	R@10	R@25	R@50	R@1	R@5	R@10	R@25	R@50
✗	✗	86.45	98.00	99.31	99.86	99.98	84.10	97.84	99.04	99.78	99.98	79.39	91.60	94.65	97.25	97.95	78.34	92.85	95.25	97.50	98.50
✓	✗	86.45	97.98	99.43	99.86	99.98	84.29	97.90	98.94	99.78	99.94	79.24	91.55	94.25	97.00	98.00	77.54	92.95	95.35	97.55	98.60
✗	✓	88.31	98.75	99.45	99.84	99.98	86.39	98.08	99.20	99.82	99.96	81.04	92.20	95.00	97.40	98.10	78.54	91.95	94.60	97.50	98.20
✓	✓	88.25	98.45	99.43	99.86	99.98	86.24	98.12	99.22	99.84	99.94	80.69	92.40	95.10	97.35	98.15	78.84	91.90	94.65	97.40	98.30
		Long-DCI										Urban-1K

Appendix

Table A5: Loss ablation, full resolution (all four benchmarks, all ranks). Best per column in bold .

	DOCCI										DCI
$\gamma$	R@1	R@5	R@10	R@25	R@50	R@1	R@5	R@10	R@25	R@50	R@1	R@5	R@10	R@25	R@50	R@1	R@5	R@10	R@25	R@50
$\gamma{=}0$	86.45	97.98	99.43	99.86	99.98	84.29	97.90	98.94	99.78	99.94	79.24	91.55	94.25	97.00	98.00	77.54	92.95	95.35	97.55	98.60
$\gamma{=}0.25$	87.82	98.63	99.45	99.88	99.98	86.63	98.04	99.16	99.84	99.96	80.69	92.40	95.35	97.55	98.30	79.19	92.85	95.40	97.70	98.50
$\gamma{=}0.5$ (default)	88.25	98.45	99.43	99.86	99.98	86.24	98.12	99.22	99.84	99.94	80.69	92.40	95.10	97.35	98.15	78.84	91.90	94.65	97.40	98.30
$\gamma{=}0.75$	88.20	98.67	99.49	99.82	99.96	85.73	98.00	99.25	99.78	99.98	80.64	92.20	94.70	97.20	98.25	78.39	91.05	93.90	96.90	98.25
$\gamma{=}1.0$	87.98	98.47	99.51	99.82	99.96	85.02	97.69	99.02	99.76	99.96	80.04	92.00	94.40	97.10	98.30	78.29	90.80	93.70	96.70	98.00

Appendix

Table A6:

\gamma

sweep, full resolution (all four benchmarks, all ranks). Best per column in bold .

	DOCCI $\rightarrow$ Long-DCI
Method	R@1	R@5	R@10	R@25	R@50	R@1	R@5	R@10	R@25	R@50
Long-CLIP (zero-shot)	54.61	72.80	78.33	85.29	89.15	47.35	73.04	80.10	86.66	90.60
FineLIP	54.54	72.93	79.42	85.87	89.98	46.44	72.96	80.47	87.04	90.68
GOAL	55.74	74.83	80.63	86.78	90.54	56.60	75.48	81.25	87.45	91.30
StructXLIP	57.70	76.03	81.68	88.06	91.60	57.90	76.36	82.09	87.71	91.03
HN-CLIP	62.60	79.16	84.19	89.88	93.05	63.77	79.89	84.22	89.05	92.14

Appendix

Table A7: Cross-domain transfer DOCCI

\rightarrow

Long-DCI. Best per column in bold .

Method	Seed	R@1	R@5	R@10	R@1	R@5	R@10	$\max\|\Delta\|$
HN-CLIP (ours)	seed 42	78.90	93.85	96.34	76.24	92.86	95.90
	seed 43	78.79	92.32	95.12	76.18	91.99	95.15	1.53
GOAL	seed 42	74.11	92.73	95.75	73.29	92.18	95.78
	seed 43	74.34	92.89	95.77	72.03	92.37	95.69	1.26
StructXLIP	seed 42	75.31	93.15	95.92	72.56	92.65	95.75
	seed 43	75.58	93.12	96.20	72.39	93.03	95.80	0.38

Appendix

Table A8: Seed replication on Long-DCI.

\max|\Delta|

is the largest absolute difference across the six metrics.

Data	Method	R@1	R@5	R@10	R@25	R@50	R@1	R@5	R@10	R@25	R@50	$\Delta$
DOCCI
5%	GOAL	77.88	95.35	97.88	99.59	99.84	75.45	94.65	97.76	99.35	99.84
	HN-CLIP	82.73	96.94	98.69	99.69	99.84	80.08	96.22	98.35	99.57	99.98	$\uparrow$ 4.74
20%	GOAL	78.22	95.65	98.22	99.55	99.86	76.47	94.88	97.96	99.39	99.84
	HN-CLIP	85.43	97.86	99.20	99.80	99.96	83.75	97.25	98.80	99.76	99.98	$\uparrow$ 7.25
50%	GOAL	80.20	96.47	98.33	99.63	99.88	78.18	95.37	97.96	99.43	99.92

Appendix

Table A9: Sample efficiency, full resolution. Best per pair in bold ;

\Delta

is the average-R@1 margin of HN-CLIP over GOAL at that fraction.

Train on DCI $\rightarrow$ Test on DCI vs. DOCCI
Setting	R@1	R@5	R@10	R@1	R@5	R@10
Long-CLIP (DCI $\rightarrow$ DCI)	67.83	83.19	87.69	64.13	84.84	89.74
Long-CLIP (DCI $\rightarrow$ DOCCI)	78.78	95.24	98.02	66.75	91.92	96.31
FineLIP (DCI $\rightarrow$ DCI)	72.69	87.14	90.65	65.48	86.84	91.00
FineLIP (DCI $\rightarrow$ DOCCI)	80.69	96.47	98.53	65.63	91.51	96.31
GOAL (DCI $\rightarrow$ DCI)	77.29	90.25	93.30	74.84	89.94	93.25

Appendix

Table A10: Cross-domain generalization between DCI and DOCCI. Train on one dataset and test on another. Values are Recall@K (%), using Text

\rightarrow

Image and Image

\rightarrow

Text retrieval. In-domain best in italic bold , cross-domain best in bold .

DOCCI
Method	R@1	R@5	R@10	R@1	R@5	R@10
Long-CLIP [ECCV’24] (zero-shot)	78.78	95.24	98.02	66.75	91.92	96.31
FineLIP [CVPR’25]	77.39 $\pm$ 0.33	95.82 $\pm$ 0.08	98.36 $\pm$ 0.02	69.80 $\pm$ 0.08	93.14 $\pm$ 0.14	97.37 $\pm$ 0.09
GOAL [CVPR’25]	81.89 $\pm$ 0.36	96.97 $\pm$ 0.04	98.78 $\pm$ 0.05	80.59 $\pm$ 0.51	96.19 $\pm$ 0.10	98.59 $\pm$ 0.07
StructXLIP [CVPR’26]	84.73 $\pm$ 0.09	97.64 $\pm$ 0.05	99.04 $\pm$ 0.09	82.52 $\pm$ 0.17	96.97 $\pm$ 0.09	98.78 $\pm$ 0.08
HN-CLIP	88.36 $\pm$ 0.14	98.50 $\pm$ 0.04	99.48 $\pm$ 0.05	86.41 $\pm$ 0.19	98.09 $\pm$ 0.03	99.24 $\pm$ 0.02

Appendix

Table A11: Three-seed replication (seeds 42, 1337, 3407), mean

\pm

std over the three runs; the epoch is selected per benchmark by the same rule used throughout. Bold marks the best mean per column, underline the runner-up. Long-CLIP is evaluated zero-shot without fine-tuning and therefore carries no seed variance.

Figure A9: Per-direction convergence. Recall@1 per epoch; top: Text $\rightarrow$ Image, bottom: Image $\rightarrow$ Text.

Figure A10: Qualitative T $\rightarrow$ I retrieval on DOCCI (green = ground truth).

Cross-Modal Retrieval

Cross-modal retrieval matches queries and items across different data types, such as text, images, video, audio, and sensor signals.

See all 184 papers

What CLIP Knows but Cannot Say: Recovering Negation from Frozen Intermediate Features
Jul 25, 2026Chen-Yi Lu, Yueh-Shao Chen, Somali ChaterjiVision-Language Models VLM Adaptation
Purdue University, West Lafayette, IN 47907, USA
Whispers from the Neighbors: Rethinking Caption Consensus as Inference-Time Evidence for Vision-Language Retrieval
Oct 24, 2026Khang Nhat Hoang Vo, Belal Shoer, Abhirama Subramanyam PenamakuriVision-Language Retrieval Cross-Modal Retrieval
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)
CLIP Is Shortsighted: Paying Attention Beyond the First Sentence
Feb 25, 2026Marc-Antoine Lavoie, Anas Mahmoud, Aldo Zaimi +2Position Bias in VLMs Vision-Language Alignment
University of Toronto Robotics Institute · Mila - Quebec AI Institute

Jul 25, 2026J/K move · Enter open · S save

What CLIP Knows but Cannot Say: Recovering Negation from Frozen Intermediate Features

Chen-Yi Lu, Yueh-Shao Chen, Somali Chaterji

Purdue University, West Lafayette, IN 47907, USA

Vision-Language Models VLM Adaptation VLM Reasoning Vision-Language Alignment

Open paper AskPDF arXiv

Contrastive vision-language models such as CLIP map semantically opposite phrases (e.g., "a dog" vs. "not a dog") to nearly identical embeddings, rendering them insensitive to negation. We attribute this failure to a phenomenon we call Representational Collapse: by tracking compositional divergence and visual alignment across the CLIP text encoder, we show that middle layers build compositional syntax, but the final layers collapse this structure as visual alignment rises, producing a syntax-blind final representation. To recover the lost negation signal without altering pretrained weights, we propose PeakPatch, a lightweight post-hoc correction system that intercepts the encoder at its compositional peak. An Embedding Correction Network (ECN) uses cross-attention to extract a negation-specific signal from the peak layer, anchored to a stable baseline, and predicts a deviation vector that re-injects the lost syntax into the final-layer embedding space. A complementary Score Correction Network (SCN) predicts bounded scalar score offsets for discriminative tasks. Both modules are trained jointly end-to-end while all CLIP parameters remain frozen, adding only 5.2M parameters (3.5% of the backbone) and preserving the standard cosine similarity interface. On NegBench, PeakPatch achieves 74.3% on COCO MCQ (+35.1 over CLIP, +17.8 over the best encoder fine-tuning method) and 65.5% on VOC MCQ, while outperforming all fine-tuning baselines on fully out-of-distribution negation retrieval despite training only 3.5% of the parameters. The corrected embeddings also transfer to text-to-image generation (+18.4 negation score) and generalize across ViT-B/32, ViT-L/14, and SigLIP backbones. Project URL: https://stevencylu.github.io/PeakPatch/.

Abstract

Cross-Modal Retrieval

Explore similar work

What CLIP Knows but Cannot Say: Recovering Negation from Frozen Intermediate Features

Whispers from the Neighbors: Rethinking Caption Consensus as Inference-Time Evidence for Vision-Language Retrieval

CLIP Is Shortsighted: Paying Attention Beyond the First Sentence