Which Negatives Matter? Ask Your Text Encoder: Adaptive Similarity Margins for Dense-Caption Retrieval
Authors: Haoyue Liu, Ye Chen, Zhichao Wang, Xiaoying Tang
Organizations: School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen 518172, China · XJTU-POLIMI Joint School, Xi’an Jiaotong University, Xi’an 710049, China · Shenzhen Future Network of Intelligence Institute (FNii-Shenzhen)
Dense-caption retrieval has recently been improved by introducing segmentation, edge maps, LLM-filtered captions, and cross-modal modules into contrastive fine-tuning. However, these methods largely inherit the same InfoNCE objective, whose optimization can prematurely saturate under a strong pre-trained initialization: on dense captions, the loss falls below 10^-3 on 80% of batches within the first epoch, while its gradient becomes numerically zero in 47% of measurements. We find that this behavior is closely related to the large number of near-duplicate captions in dense-caption benchmarks, where a few highly similar negatives remain unresolved after the easy majority has already been separated. As a remedy, we introduce HN-CLIP, which uses the text encoder's own text-text geometry to construct per-negative adaptive similarity margins. Specifically, a detached caption-similarity matrix is added to the negative logits, assigning larger margins to more similar captions without mining, synthesizing, or resampling negatives. The resulting objective requires only one caption-similarity matrix and a masked logit addition during training, with no auxiliary data, additional parameters, offline preprocessing, or inference-time overhead. Extensive experiments on four dense-caption retrieval benchmarks show that HN-CLIP improves over the strongest competitors by +2.5 to +4.0 R@1 while training 2.4x faster than GOAL and 5.4x faster than StructXLIP. Moreover, the proposed objective improves all six tested fine-tuning frameworks on the in-domain benchmarks and reaches the strongest full-data baseline with only 20% of the training data.
Figures & tables
Figure 1: Dense-caption benchmarks are dominated by hard negatives. Distributions of (a) all pairwise caption-caption cosine similarities and (b) each caption’s hardest-negative similarity, measured with the pre-trained Long-CLIP-L text encoder on the test sets. Dotted lines mark the means. The pairs in (b) receive the strongest boosts.
Figure 2: Overview of HN-CLIP. (a) A batch of image–text pairs is encoded by the jointly fine-tuned dual encoder. (b) Caption–caption similarity provides detached, per-negative adaptive margins for the global contrastive loss, combined with the token-level term. (c) More similar negatives receive larger margins, while the positive is unchanged. HN-CLIP requires no mining, resampling, extra data, additional parameters, or inference-time overhead.
Figure 3: Illustration of HN-CLIP. A real DOCCI query and its hardest in-batch negative (cos 0.89 ). The strongest baseline ranks the ground truth 52nd (its top-1 (pink) is the near-twin’s own image); HN-CLIP ranks it first.
Figure 4: Empirical gradient analysis (Long-CLIP-L, DOCCI, 10 epochs; losses and full-parameter gradients measured every 5 optimizer steps, 148 measurements). Standard InfoNCE declares 80% of batches solved within the first epoch and its gradient is numerically zero in 47% of measurements (median zero at five epochs; plotted clamped to 10−10 ). The boosted loss keeps at least 26% of batches active in every epoch and, where the standard gradient is nonzero, exceeds it by 102 – 106× in per-epoch median, with interquartile bands disjoint at nine epochs, and the retrieval error it buys keeps falling. Its gradient-norm distribution is bimodal across batches; see Appendix C .
DOCCI
DCI
Method
R@1
R@5
R@10
R@1
R@5
R@10
R@1
R@5
R@10
R@1
R@5
R@10
Long-CLIP [ECCV’24]
78.78
95.24
98.02
66.75
91.92
96.31
67.83
83.19
87.69
64.13
84.84
89.74
FineLIP [CVPR’25]
77.51
96.02
98.41
69.90
93.43
97.45
72.69
87.14
90.65
65.48
86.84
91.00
GOAL [CVPR’25]
81.53
97.02
98.80
80.86
96.24
98.63
77.29
90.25
93.30
74.84
89.94
93.25
StructXLIP [CVPR’26]
84.73
97.69
99.00
82.61
97.08
98.71
75.84
89.94
93.65
74.49
90.05
93.40
HN-CLIP
88.25
98.45
99.43
86.24
98.12
99.22
80.69
92.40
95.10
78.84
91.90
94.65
Table 1: Cross-modal retrieval performance of CLIP-based fine-tuning methods on four dense-caption benchmarks. We report Recall@K (%) on both Text → Image and Image → Text settings. All fine-tuned methods start from the same Long-CLIP-L backbone with an identical training budget; Long-CLIP denotes the released checkpoint. Best results in bold ; second best underlined . Δ denotes the margin over the best competitor per column, with gain in ↑ green .
Figure 5: Convergence comparison. Average R@1 (T → I, I → T) per fine-tuning epoch. HN-CLIP’s first epoch matches or exceeds most baselines’ final accuracy on all four benchmarks.
DOCCI
Long-DCI
Method
R@1
R@5
R@10
R@1
R@5
R@10
R@1
R@5
R@10
R@1
R@5
R@10
Long-CLIP [ECCV’24]
86.45
98.00
99.31
84.10
97.84
99.04
70.24
89.62
94.00
68.44
89.35
94.67
+ our LHN
88.31
98.75
99.45
86.39
98.08
99.20
78.43
92.52
95.27
76.02
91.78
94.76
Δ
↑ 1.86
↑ 0.75
↑ 0.14
↑ 2.29
↑ 0.24
↑ 0.16
↑ 8.19
↑ 2.90
↑ 1.27
↑ 7.58
↑ 2.43
↑ 0.09
FineLIP [CVPR’25]
77.51
96.02
98.41
69.90
93.43
97.45
59.24
77.86
83.19
49.52
75.08
82.39
+ our LHN
85.47
97.71
99.04
83.88
97.41
98.82
74.92
90.52
93.62
73.39
89.87
93.23
Table 2: Plug-and-play enhancement of our LHN on CLIP-based fine-tuning. Results on DOCCI and Long-DCI for Text → Image and Image → Text retrieval. Upper: full-parameter fine-tuning; lower: parameter-efficient tuning. Our method consistently boosts diverse CLIP variants, and the gains grow with caption length. Best in bold , with gain in ↑ green .
Figure 6: Sample efficiency (avg R@1, identical subsets). HN-CLIP at 20% of the data already clears the strongest baseline trained on 100% (dotted line, from Table 1 ); full numbers in Table A9 .
DOCCI
DCI
Long-DCI
Urban-1K
γ
R@1
R@1
R@1
R@1
R@1
R@1
R@1
R@1
Avg
γ=0
86.45
84.29
79.24
77.54
70.96
68.08
92.00
92.90
81.43
γ=0.25
87.82
86.63
80.69
79.19
76.37
74.69
91.70
93.10
83.77
γ=0.5 (default)
88.25
86.24
80.69
78.84
78.90
76.24
90.20
90.70
83.76
γ=0.75
88.20
85.73
80.64
78.39
79.82
76.59
88.80
89.70
83.48
γ=1.0
87.98
85.02
80.04
78.29
80.05
76.48
88.20
88.30
83.05
Table 3: Ablations. (a) Sensitivity to the boost strength γ (Recall@1): every γ∈[0.25,1] beats γ=0 in-domain, while under transfer (Urban-1K) milder boosts generalize better. (b) Loss components under an identical recipe; only the loss changes. (c) Gˉ recomputed from the live encoder (default) vs. frozen at initialization; R@1/5/10 in Table A2 . Best per column in bold .
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
#train
#test
words
pairwise sim
hardest sim
DOCCI
9450
5100
123
0.84
0.93
DCI
5445
1999
133
0.85
0.92
Long-DCI
5445 (=DCI)
7444
134
0.85
0.93
Urban-1K
14579 (VG)
1000
107
0.88
0.94
Appendix
Table A1: Benchmark statistics. Mean caption length is measured on the evaluation split in words; similarities use the pre-trained Long-CLIP-L text encoder.
Figure A1: Near-duplicate caption pairs on DOCCI. Each row: a test image, its caption, the three nearest other captions’ images (cosine printed underneath), and the caption texts with shared content words highlighted .
Figure A2: Near-duplicate caption pairs on DCI.
Figure A3: Near-duplicate caption pairs on Long-DCI (full-length captions).
Figure A4: Near-duplicate caption pairs on Urban-1K.
Figure A5: One real training batch per benchmark. Text–text similarity matrices Gˉ (pre-trained encoder; diagonal masked, max off-diagonal entry annotated). Uniformly dark = every negative is hard; this matrix, detached and scaled by γ , is the entire mechanism of HN-CLIP.
Figure A6: Statistical analysis of the caption geometry across the four benchmarks (pre-trained Long-CLIP-L text encoder, full test splits). (A) Mean pairwise caption similarity. (B) Mean similarity of each caption’s hardest companion. (C) Composition of in-batch negatives by hardness bucket.
DOCCI
DCI
Gˉ source
R@1
R@5
R@10
R@1
R@5
R@10
R@1
R@5
R@10
R@1
R@5
R@10
Recomputed Gˉ (default)
88.25
98.45
99.43
86.24
98.12
99.22
80.69
92.40
95.10
78.84
91.90
94.65
Frozen Gˉ
85.86
97.61
99.18
82.43
96.82
98.73
77.79
90.80
93.35
75.84
89.49
92.95
Δ (default − frozen)
↑ 2.39
↑ 0.84
↑ 0.25
↑ 3.81
↑ 1.30
↑ 0.49
↑ 2.90
↑ 1.60
↑ 1.75
↑ 3.00
↑ 2.41
↑ 1.70
best epoch (default / frozen)
6 / 5
5 / 1
Long-DCI
Urban-1K
Appendix
Table A2: Recomputed vs. frozen Gˉ (Recall@K, Text → Image and Image → Text ). Building the boost from a frozen copy of the pre-trained text encoder makes the margins exactly stationary but removes their implicit annealing: accuracy peaks at epoch 1 on DCI and Urban-1K and then declines, and the default wins all 24 columns. Best per column in bold . The final row reports the selected best epoch as default/frozen.
Figure A8: Gradient dynamics across data scales. Rows: fraction of the training set; columns: dataset. Per cell: (a) raw loss traces (thin) with running medians (thick), (b) gradient norms and their ratio (dashed), (c) gradient cosine. Log axes in (a,b); values clamped at 10−10 (numerically zero values).
Method
Aux. training inputs
Offline prep.
Wall-clock
Throughput
Long-CLIP (plain FT)
none
none
≈ 16 min
55.5 img/s
FineLIP
none
none
—
—
GOAL
SAM segments
required
≈ 41 min
≈ 22 img/s
StructXLIP
edges + LLM lexicon
required
≈ 92 min
≈ 10 img/s
HN-CLIP (ours)
none
none
17 min
53 img/s
Appendix
Table A3: Training-efficiency comparison on DCI fine-tuning (10 epochs, 5.4k images, single Ascend 910B, effective batch 128). Wall-clock for GOAL/StructXLIP is measured from same-device sequential runs minus evaluation overhead; FineLIP ran in parallel and is not attributable. Offline preprocessing time (segmentation, edge extraction, LLM filtering) is not included in the wall-clock.
DOCCI
Long-DCI
Method
R@1
R@5
R@10
R@1
R@5
R@10
R@1
R@5
R@10
R@1
R@5
R@10
Long-CLIP
86.45
98.00
99.31
84.10
97.84
99.04
70.24
89.62
94.00
68.44
89.35
94.67
+our LHN
88.31
98.75
99.45
86.39
98.08
99.20
78.43
92.52
95.27
76.02
91.78
94.76
Δ
↑ 1.86
↑ 0.75
≈ 0
↑ 2.29
↑ 0.24
↑ 0.16
↑ 8.19
↑ 2.90
↑ 1.27
↑ 7.58
↑ 2.43
≈ 0
FineLIP
77.51
96.02
98.41
69.90
93.43
97.45
59.24
77.86
83.19
49.52
75.08
82.39
+our LHN
85.47
97.71
99.04
83.88
97.41
98.82
74.92
90.52
93.62
73.39
89.87
93.23
Appendix
Table A4: Plug-and-play enhancement of LHN on CLIP-based fine-tuning. Results on DOCCI and Long-DCI for Text → Image and Image → Text retrieval; per framework we report the official baseline, the same recipe with LHN replacing its global InfoNCE term, and the per-column difference Δ (gain in ↑ green , drop in ↓ gray ; differences within ±0.15 are shown as ≈ 0 ). Bold marks the +LHN value where it is the better of the pair.
DOCCI
DCI
Ltok
LHN
R@1
R@5
R@10
R@25
R@50
R@1
R@5
R@10
R@25
R@50
R@1
R@5
R@10
R@25
R@50
R@1
R@5
R@10
R@25
R@50
✗
✗
86.45
98.00
99.31
99.86
99.98
84.10
97.84
99.04
99.78
99.98
79.39
91.60
94.65
97.25
97.95
78.34
92.85
95.25
97.50
98.50
✓
✗
86.45
97.98
99.43
99.86
99.98
84.29
97.90
98.94
99.78
99.94
79.24
91.55
94.25
97.00
98.00
77.54
92.95
95.35
97.55
98.60
✗
✓
88.31
98.75
99.45
99.84
99.98
86.39
98.08
99.20
99.82
99.96
81.04
92.20
95.00
97.40
98.10
78.54
91.95
94.60
97.50
98.20
✓
✓
88.25
98.45
99.43
99.86
99.98
86.24
98.12
99.22
99.84
99.94
80.69
92.40
95.10
97.35
98.15
78.84
91.90
94.65
97.40
98.30
Long-DCI
Urban-1K
Appendix
Table A5: Loss ablation, full resolution (all four benchmarks, all ranks). Best per column in bold .
DOCCI
DCI
γ
R@1
R@5
R@10
R@25
R@50
R@1
R@5
R@10
R@25
R@50
R@1
R@5
R@10
R@25
R@50
R@1
R@5
R@10
R@25
R@50
γ=0
86.45
97.98
99.43
99.86
99.98
84.29
97.90
98.94
99.78
99.94
79.24
91.55
94.25
97.00
98.00
77.54
92.95
95.35
97.55
98.60
γ=0.25
87.82
98.63
99.45
99.88
99.98
86.63
98.04
99.16
99.84
99.96
80.69
92.40
95.35
97.55
98.30
79.19
92.85
95.40
97.70
98.50
γ=0.5 (default)
88.25
98.45
99.43
99.86
99.98
86.24
98.12
99.22
99.84
99.94
80.69
92.40
95.10
97.35
98.15
78.84
91.90
94.65
97.40
98.30
γ=0.75
88.20
98.67
99.49
99.82
99.96
85.73
98.00
99.25
99.78
99.98
80.64
92.20
94.70
97.20
98.25
78.39
91.05
93.90
96.90
98.25
γ=1.0
87.98
98.47
99.51
99.82
99.96
85.02
97.69
99.02
99.76
99.96
80.04
92.00
94.40
97.10
98.30
78.29
90.80
93.70
96.70
98.00
Appendix
Table A6: γ sweep, full resolution (all four benchmarks, all ranks). Best per column in bold .
DOCCI → Long-DCI
Method
R@1
R@5
R@10
R@25
R@50
R@1
R@5
R@10
R@25
R@50
Long-CLIP (zero-shot)
54.61
72.80
78.33
85.29
89.15
47.35
73.04
80.10
86.66
90.60
FineLIP
54.54
72.93
79.42
85.87
89.98
46.44
72.96
80.47
87.04
90.68
GOAL
55.74
74.83
80.63
86.78
90.54
56.60
75.48
81.25
87.45
91.30
StructXLIP
57.70
76.03
81.68
88.06
91.60
57.90
76.36
82.09
87.71
91.03
HN-CLIP
62.60
79.16
84.19
89.88
93.05
63.77
79.89
84.22
89.05
92.14
Appendix
Table A7: Cross-domain transfer DOCCI → Long-DCI. Best per column in bold .
Method
Seed
R@1
R@5
R@10
R@1
R@5
R@10
max∣Δ∣
HN-CLIP (ours)
seed 42
78.90
93.85
96.34
76.24
92.86
95.90
seed 43
78.79
92.32
95.12
76.18
91.99
95.15
1.53
GOAL
seed 42
74.11
92.73
95.75
73.29
92.18
95.78
seed 43
74.34
92.89
95.77
72.03
92.37
95.69
1.26
StructXLIP
seed 42
75.31
93.15
95.92
72.56
92.65
95.75
seed 43
75.58
93.12
96.20
72.39
93.03
95.80
0.38
Appendix
Table A8: Seed replication on Long-DCI. max∣Δ∣ is the largest absolute difference across the six metrics.
Data
Method
R@1
R@5
R@10
R@25
R@50
R@1
R@5
R@10
R@25
R@50
Δ
DOCCI
5%
GOAL
77.88
95.35
97.88
99.59
99.84
75.45
94.65
97.76
99.35
99.84
HN-CLIP
82.73
96.94
98.69
99.69
99.84
80.08
96.22
98.35
99.57
99.98
↑ 4.74
20%
GOAL
78.22
95.65
98.22
99.55
99.86
76.47
94.88
97.96
99.39
99.84
HN-CLIP
85.43
97.86
99.20
99.80
99.96
83.75
97.25
98.80
99.76
99.98
↑ 7.25
50%
GOAL
80.20
96.47
98.33
99.63
99.88
78.18
95.37
97.96
99.43
99.92
Appendix
Table A9: Sample efficiency, full resolution. Best per pair in bold ; Δ is the average-R@1 margin of HN-CLIP over GOAL at that fraction.
Setting
R@1
R@5
R@10
R@1
R@5
R@10
Train on DCI → Test on DCI vs. DOCCI
Long-CLIP (DCI → DCI)
67.83
83.19
87.69
64.13
84.84
89.74
Long-CLIP (DCI → DOCCI)
78.78
95.24
98.02
66.75
91.92
96.31
FineLIP (DCI → DCI)
72.69
87.14
90.65
65.48
86.84
91.00
FineLIP (DCI → DOCCI)
80.69
96.47
98.53
65.63
91.51
96.31
GOAL (DCI → DCI)
77.29
90.25
93.30
74.84
89.94
93.25
Appendix
Table A10: Cross-domain generalization between DCI and DOCCI. Train on one dataset and test on another. Values are Recall@K (%), using Text → Image and Image → Text retrieval. In-domain best in italic bold , cross-domain best in bold .
DOCCI
Method
R@1
R@5
R@10
R@1
R@5
R@10
Long-CLIP [ECCV’24] (zero-shot)
78.78
95.24
98.02
66.75
91.92
96.31
FineLIP [CVPR’25]
77.39 ± 0.33
95.82 ± 0.08
98.36 ± 0.02
69.80 ± 0.08
93.14 ± 0.14
97.37 ± 0.09
GOAL [CVPR’25]
81.89 ± 0.36
96.97 ± 0.04
98.78 ± 0.05
80.59 ± 0.51
96.19 ± 0.10
98.59 ± 0.07
StructXLIP [CVPR’26]
84.73 ± 0.09
97.64 ± 0.05
99.04 ± 0.09
82.52 ± 0.17
96.97 ± 0.09
98.78 ± 0.08
HN-CLIP
88.36 ± 0.14
98.50 ± 0.04
99.48 ± 0.05
86.41 ± 0.19
98.09 ± 0.03
99.24 ± 0.02
Appendix
Table A11: Three-seed replication (seeds 42, 1337, 3407), mean ± std over the three runs; the epoch is selected per benchmark by the same rule used throughout. Bold marks the best mean per column, underline the runner-up. Long-CLIP is evaluated zero-shot without fine-tuning and therefore carries no seed variance.
Figure A9: Per-direction convergence. Recall@1 per epoch; top: Text → Image, bottom: Image → Text.
Figure A10: Qualitative T → I retrieval on DOCCI (green = ground truth).
Contrastive vision-language models such as CLIP map semantically opposite phrases (e.g., "a dog" vs. "not a dog") to nearly identical embeddings, rendering them insensitive to negation. We attribute this failure to a phenomenon we call Representational Collapse: by tracking compositional divergence and visual alignment across the CLIP text encoder, we show that middle layers build compositional syntax, but the final layers collapse this structure as visual alignment rises, producing a syntax-blind final representation. To recover the lost negation signal without altering pretrained weights, we propose PeakPatch, a lightweight post-hoc correction system that intercepts the encoder at its compositional peak. An Embedding Correction Network (ECN) uses cross-attention to extract a negation-specific signal from the peak layer, anchored to a stable baseline, and predicts a deviation vector that re-injects the lost syntax into the final-layer embedding space. A complementary Score Correction Network (SCN) predicts bounded scalar score offsets for discriminative tasks. Both modules are trained jointly end-to-end while all CLIP parameters remain frozen, adding only 5.2M parameters (3.5% of the backbone) and preserving the standard cosine similarity interface. On NegBench, PeakPatch achieves 74.3% on COCO MCQ (+35.1 over CLIP, +17.8 over the best encoder fine-tuning method) and 65.5% on VOC MCQ, while outperforming all fine-tuning baselines on fully out-of-distribution negation retrieval despite training only 3.5% of the parameters. The corrected embeddings also transfer to text-to-image generation (+18.4 negation score) and generalize across ViT-B/32, ViT-L/14, and SigLIP backbones. Project URL: https://stevencylu.github.io/PeakPatch/.
An image is rarely described only once. Retrieval benchmarks and many collections attach several captions to each picture, yet dual encoders still judge every image-caption pair in isolation. We ask what these neighboring captions contribute when a frozen encoder consults them at inference time. Fusing image similarity with similarity to an image's own captions improves text-to-image R@1 by 3.4 to 16.8 points across nine frozen encoders when the captions come from the same annotators as the queries. The gain falls to 1.9 to 7.2 points when an off-the-shelf captioner writes them, and to at most one point when they come from a different annotation source; a paraphrase-trained text encoder strengthens the text side further. A natural refinement, scoring the part of the image that the captions leave unexplained, appears to add much more under a leave-one-out evaluation. We show that this advantage arises because the correct candidate's supports are built differently from those of the negative candidates, with set size and query removal each contributing about half, and that under a fixed support index the residual reduces to a reweighting of what the captions already provide. On compositional benchmarks, the captions help by matching candidates to human descriptions of the image, without improving image-text grounding. The code is available at: https://github.com/nhatkhangcs/clip_consensus
CLIP models learn transferable multi-modal features via image-text contrastive learning on internet-scale data. They are widely used in zero-shot classification, multi-modal retrieval, text-to-image diffusion, and as image encoders in large vision-language models. However, CLIP's pretraining is dominated by images paired with short captions, biasing the model toward encoding simple descriptions of salient objects and leading to coarse alignment on complex scenes and dense descriptions. While recent work mitigates this by fine-tuning on small-scale long-caption datasets, we identify an important common bias: both human- and LLM-generated long captions typically begin with a one-sentence summary followed by a detailed description. We show that this acts as a shortcut during training, concentrating attention on the opening sentence and early tokens and weakening alignment over the rest of the caption. To resolve this, we introduce DeBias-CLIP, which removes the summary sentence during training and applies sentence sub-sampling and text token padding to distribute supervision across all token positions. DeBias-CLIP achieves state-of-the-art long-text retrieval, improves short-text retrieval, and is less sensitive to sentence order permutations. It is a drop-in replacement for Long-CLIP with no additional trainable parameters.
Marc-Antoine Lavoie, Anas Mahmoud, Aldo Zaimi +2
University of Toronto Robotics Institute · Mila - Quebec AI Institute