Mitigating Saliency Collapse: Robust Saliency-Aware Long-Text Image-Text Alignment

Qiuyu Kong; Zanxi Ruan; Marco Cristani; Yiming Wang

cs.AIOct 22, 2026

Mitigating Saliency Collapse: Robust Saliency-Aware Long-Text Image-Text Alignment

Authors: Qiuyu Kong, Zanxi Ruan, Marco Cristani, Yiming Wang

Organizations: Sapienza University of Rome · University of Verona · Reykjavik University · Fondazione Bruno Kessler

Abstract

Long-text image-text alignment is essential for fine-grained scene understanding, yet existing methods mainly focus on extending context length or improving local alignment. We identify an overlooked failure mode, saliency collapse, where models over-rely on text that features visual prominence while underutilizing non-salient context. Consequently, retrieval performance degrades sharply when salient content is reduced, despite that the non-salient context remains informative. To address this, we propose RoSA, a saliency-aware finetuning framework that decomposes image-text pairs into complementary salient and contextual views, enabling joint learning of prominent and contextual semantics. We propose a modality-independent decomposition strategy to combine object-level visual saliency estimation with LLM-based text decomposition to bridge vision and language. We employ auxiliary objectives to explicitly align these views, ensuring robust representation with both salient and contextual features. RoSA greatly improves long-text retrieval and demonstrates strong robustness to saliency collapse across benchmarks. Moreover, our auxiliary objectives are plug-and-play, bringing consistent gains to existing tuning methods. Code and checkpoints are available at: https://github.com/intelligolabs/RoSA.

Figures & tables

Figure 1: Saliency collapse in long-text alignment. a) Different users may describe the same image with long queries that cover different subsets of salient objects; b1) Existing long-text alignment methods suffer from saliency collapse : they degrade sharply when salient text is progressively reduced, at a faster pace than when random content of the matched-length is removed. b2) Under both mild salient text reduction ( $\alpha=75\%$ ) and complete salient text removal ( $\alpha=0\%$ ), our method significantly improves over prior methods in terms of R@1. The relative gain of ours over the best competitor StructXLIP ( Ruan et al., 2026 ) is highlighted in a darker tone.

Method	DOCCI					INSECT
Method	Full	$\alpha{=}100$	$\alpha{=}50$	$\alpha{=}25$	$\alpha{=}10$	Full	$\alpha{=}100$	$\alpha{=}50$	$\alpha{=}25$	$\alpha{=}10$
Long-CLIP [ECCV’24]	64.49	62.02 96%	48.67 75%	35.49 55%	28.24 44%	8.20	7.41 90%	5.35 65%	4.26 52%	3.81 46%
GOAL [CVPR’25]	79.47	66.40 84%	53.11 67%	39.46 50%	31.10 39%	8.81	7.73 88%	4.76 54%	3.60 41%	2.72 31%
StructXLIP [CVPR’26]	83.04	66.98 81%	53.20 64%	38.75 47%	30.40 37%	9.93	8.00 81%	5.93 60%	3.90 39%	3.28 33%

Table 1: Salient-text-only analysis. For each caption we retain only the first-

\alpha\%

salient text and remove all contextual text. Cells report text-to-image R@1 (%); Full denotes retrieval with the full caption paired with the image. Superscripts indicate the performance ratio relative to Full .

Figure 2: Length-controlled analysis under progressive salient reduction. We compare retrieval performance under salient reduction and random-span reduction on DOCCI and INSECT. For each retained salient-text ratio $\alpha$ , the random-span control removes the same number of tokens as salient deletion, isolating the effect of shortening from the effect of removing salient content. The annotations report the corresponding percentage of the original caption removed. Across all panels, salient deletion yields much larger degradation than matched random deletion, indicating that the drop is caused by the absence of salient text rather than by reduced query length alone.

Figure 3: Overview of RoSA . RoSA consists of two stages: Stage 1: Saliency-guided view decomposition of vision and text, where we independently decompose the text and image. On the text side, an LLM is used to split the original long caption into salient text and contextual text. On the visual side, an off-the-shelf saliency method is applied to derive salient/contextual visual regions. Stage 2: Saliency-aware alignment. During fine-tuning of the image and text encoders, RoSA augments the original image-text contrastive loss $\mathcal{L}{ori}$ with three saliency-aware objectives: the saliency-focused contrastive loss $\mathcal{L}{sal}$ , which explicitly aligns salient images with salient texts; the context-preserving contrastive loss $\mathcal{L}{ctx}$ , which aligns contextual images with contextual texts; and the visual anchoring loss $\mathcal{L}{anc}$ , which enforces representation consistency between the original image and the salient image, thereby preserving stable salient semantics.

Method	SKETCHY						INSECT						DOCCI						DCI
Method	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10
Long-CLIP [ECCV’24]	54.32	80.14	88.43	52.76	80.31	88.08	8.20	23.83	34.97	9.41	24.78	37.31	64.49	87.67	93.43	63.08	87.45	93.14	59.23	80.89	87.04	60.13	81.44	87.54
FineLIP [CVPR’25]	40.59	71.16	81.78	40.33	72.11	82.38	8.46	23.32	33.59	6.86	23.75	34.46	67.80	90.22	94.84	66.39	89.12	94.47	66.13	85.34	89.79	64.58	84.59	89.54
SmartCLIP [CVPR’25]	50.73	81.09	94.56	51.30	80.83	94.04	4.84	16.84	34.63	4.66	15.46	34.02	74.92	94.08	97.31	74.91	94.04	97.29	69.88	86.64	94.05	70.94	87.04	92.77
GOAL [CVPR’25]	63.21	87.13	93.44	62.44	87.82	92.31	8.81	24.35	35.84	8.55	25.91	36.18	79.47	96.65	98.69	79.43	96.14	97.25	72.64	89.89	93.70	72.84	90.50	93.20
StructXLIP [CVPR’26]	69.86	90.85	95.42	68.22	90.67	95.68	9.93	26.60	38.34	9.50	26.60	39.64	83.04	97.06	98.96	81.59	96.94	98.78	75.90	90.00	95.15	74.39	89.90	94.30

Table 2: Comparison against recent CLIP-based finetuning methods on long-text cross-modal retrieval . We report Recall@K (%) on both T

\rightarrow

I and I

\rightarrow

T settings and highlight best and second best results;

\Delta

denotes the margin of ours over the best competitor.

Method	INSECT						DOCCI
Method	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10
Long-CLIP	8.20	23.83	34.97	9.41	24.78	37.31	64.49	87.67	93.43	63.08	87.45	93.14
+our $\mathcal{L}_{\mathrm{RoSA}}$	9.41 (+1.21)	25.73 (+1.90)	37.82 (+2.85)	10.36 (+0.95)	28.58 (+3.80)	40.07 (+2.76)	67.73 (+3.24)	90.63 (+2.96)	94.98 (+1.55)	67.29 (+4.21)	89.37 (+1.92)	94.39 (+1.25)
FineLIP	8.46	23.32	33.59	6.86	23.75	34.46	67.80	90.22	94.84	66.39	89.12	94.47
+our $\mathcal{L}_{\mathrm{RoSA}}$	8.89 (+0.43)	24.18 (+0.86)	34.97 (+1.38)	8.12 (+1.26)	24.96 (+1.21)	35.58 (+1.12)	74.06 (+6.26)	93.86 (+3.64)	97.02 (+2.18)	72.71 (+6.32)	92.90 (+3.78)	96.90 (+2.43)
SmartCLIP	4.84	16.84	34.63	4.66	15.46	34.02	74.92	94.08	97.31	74.91	94.04	97.29

Table 3: Plug-and-play enhancement with our saliency-aware alignment objectives. Results on T

\rightarrow

I and I

\rightarrow

T retrieval for INSECT and DOCCI, with best highlighted and the margin in bracket.

Figure 4: Comparison of cross-modal Grad-CAM visualizations between StructXlip and RoSA . From left to right, the columns show examples from Sketchy, INSECT, DCI, and DOCCI.

Method	DOCCI									DCI									INSECT
	Salient-only			Salient 50% +Context			Context-only			Salient-only			Salient 50% +Context			Context-only			Salient-only			Salient 50% +Context			Context-only
	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10
Long-CLIP [ECCV’24]	61.54	84.71	90.82	58.87	83.35	89.52	23.96	43.90	52.34	51.64	71.73	80.15	54.01	77.00	84.07	24.64	45.12	53.43	6.55	20.41	31.04	4.30	17.11	24.88	3.11	11.60	17.40
FineLIP [CVPR’25]	52.62	78.30	85.48	52.30	78.50	86.43	20.59	39.03	48.13	53.03	74.67	82.23	60.36	81.27	86.96	25.91	44.49	53.26	8.17	22.45	30.61	7.52	19.08	30.36	3.11	10.74	16.76
SmartCLIP [CVPR’25]	61.88	85.48	91.69	54.17	81.36	88.84	24.92	45.01	53.30	57.24	77.38	84.25	65.03	84.19	88.82	30.01	49.45	57.24	2.36	7.20	10.74	2.18	7.41	11.71	1.40	4.83	7.09
GOAL [CVPR’25]	66.84	88.32	93.21	68.60	90.85	95.00	30.28	51.63	60.09	60.88	80.44	85.92	67.55	86.56	91.50	32.08	52.28	59.67	7.09	21.59	32.22	7.09	22.45	32.72	3.76	11.92	17.51

Table 4: Robustness under reduced salient text. We report T

\rightarrow

I Recall@K (%) on DOCCI, DCI, and INSECT under reduced salient text: Salient-only , Salient 50% +Context , and Context-only . We highlight best and second best results;

\Delta

is our margin over the best competitor.

$\mathcal{L}_{\mathrm{sal}}$	$\mathcal{L}_{\mathrm{ctx}}$	$\mathcal{L}_{\mathrm{anc}}$	INSECT															DOCCI
			Full-caption retrieval						Reduced-query T $\rightarrow$ I retrieval									Full-caption retrieval						Reduced-query T $\rightarrow$ I retrieval
			T $\rightarrow$ I			I $\rightarrow$ T			Salient-only			Salient 50% +Ctx			Context-only			T $\rightarrow$ I			I $\rightarrow$ T			Salient-only			Salient 50% +Ctx			Context-only
			R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10
✗	✗	✗	8.81	24.53	35.41	7.43	24.35	36.53	7.95	22.02	32.12	7.63	22.84	33.58	3.22	9.34	15.90	81.18	96.67	98.51	80.00	96.31	98.67	64.70	86.71	91.95	73.30	92.91	96.27	29.04	50.67	58.99
✓	✗	✗	9.67	25.39	38.17	9.41	25.47	36.36	8.38	23.52	34.16	9.13	25.28	35.59	4.62	10.63	16.65	82.27	97.20	98.82	80.39	96.51	98.67	68.20	89.42	93.75	74.52	93.68	96.78	32.79	54.38	62.63
✗	✓	✗	9.41	26.34	37.05	8.38	27.20	37.65	8.49	22.56	32.12	9.77	25.67	35.30	5.26	14.18	21.59	82.88	97.18	98.76	81.22	96.90	98.59	63.44	85.95	91.81	77.22	95.01	97.60	43.62	67.22	75.37

Table 5: Ablation on auxiliary saliency-aware alignment objectives. We report full-caption retrieval on T

\rightarrow

I and I

\rightarrow

T , together in T

\rightarrow

I robustness settings with reduced salient text including Salient-only , Salient 50% +Context , and Context-only . Rows ablate different combinations of

\mathcal{L}_{\mathrm{sal}}

\mathcal{L}_{\mathrm{ctx}}

, and

\mathcal{L}_{\mathrm{anc}}

. We report Recall@K (%) with best highlighted.

Salient	Context	INSECT				DOCCI
Salient	Context	R@1	R@10	R@1	R@10	R@1	R@10	R@1	R@10
✓	✗	10.36	40.67	9.84	40.41	84.35	99.18	82.18	98.94
✗	✓	9.59	39.12	8.55	38.60	83.14	98.90	80.24	98.75
✓	✓	9.33	37.82	9.07	38.43	84.18	99.04	81.73	98.86

Table 6: Ablation on visual anchoring on T

\rightarrow

I and I

\rightarrow

T retrieval, with correct pairing fixed; highlighted best and second best .

Text-side variants
Method	INSECT				DOCCI
Method	R@1	R@10	R@1	R@10	R@1	R@10	R@1	R@10
Original text	10.36	38.08	8.64	39.38	83.02	98.94	80.35	98.67
Random text	9.41	37.39	8.55	37.48	82.67	98.88	80.43	98.57
Visual-side variants
Random crop	9.84	37.22	8.89	37.31	83.06	98.96	80.24	98.69

Table 7: Ablation on decomposition methods on T

\rightarrow

I and I

\rightarrow

T retrieval, with highlighted best and second best .

Methods	INSECT						DOCCI
Methods	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10
Baseline	8.81	24.53	35.41	7.43	24.35	36.53	81.18	96.67	98.51	80.00	96.31	98.67
Qwen2.5vl	10.62	27.72	37.48	9.07	27.89	38.17	83.29	97.53	98.92	81.65	97.10	98.73
Unipixel	10.36	26.17	37.65	8.55	27.55	38.77	81.59	97.16	98.82	79.08	96.47	98.35
Ours	10.36	28.32	40.67	9.84	29.36	40.41	84.35	97.51	99.18	82.18	97.14	98.94

Table 8: Ablation on saliency-guided decomposition strategies on T

\rightarrow

I and I

\rightarrow

T retrieval, with highlighted best and second best .

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Illustration of decomposed image–text pairs across four datasets. For each sample from Sketchy, INSECT, DOCCI, and DCI, we show the original image, the salient image, and the contextual image, together with the caption decomposition. Salient text is highlighted in yellow, while contextual text is highlighted in blue.

Figure 6: Statistical analysis of the four datasets. We analyze the textual properties of DOCCI, INSECT, Sketchy, and DCI. (a) Caption length distribution, where the number in parentheses denotes the number of image-text pairs in each dataset. (b) Average caption length measured by the number of words. (c) Average number of clauses per caption. (d) Average number of semantic entities per caption.

Figure 7: Prompt and input-output examples. The salient and contextual texts are generated by an LLM during an offline preprocessing stage and kept fixed across all experiments to ensure fair comparisons.

Metric	Salient	Contextual
Visual verifiability	84.00	84.67
Visual discriminability	94.00	90.67
Annotator agreement	83.11	78.89

Appendix

Table 9: Human validation of salient and contextual text. Results report visual verifiability, visual discriminability, and annotator agreement (%).

Text decomposer	INSECT				DOCCI
Text decomposer	R@1	R@10	R@1	R@10	R@1	R@10	R@1	R@10
Llama-3.1	10.28	38.08	9.59	38.60	83.33	98.84	81.90	98.73
Qwen2.5 (default)	10.36	40.67	9.84	40.41	84.35	99.18	82.18	98.94
$\Delta$	$\mathbf{\uparrow 0.08}$	$\mathbf{\uparrow 2.59}$	$\mathbf{\uparrow 0.25}$	$\mathbf{\uparrow 1.81}$	$\mathbf{\uparrow 1.02}$	$\mathbf{\uparrow 0.34}$	$\mathbf{\uparrow 0.28}$	$\mathbf{\uparrow 0.21}$

Appendix

Table 10: Robustness to the textual decomposer. R@1 and R@10 (%) on T

\rightarrow

I and I

\rightarrow

T retrieval. RoSA is retrained using partitions generated by each LLM, with all other settings fixed. Best results are bold ; second best underlined .

\Delta

denotes the margin of Qwen2.5 over Llama-3.1.

Pairing	INSECT				DOCCI
Pairing	R@1	R@10	R@1	R@10	R@1	R@10	R@1	R@10
Swapped	9.76	38.43	9.67	39.46	82.04	98.96	79.37	98.69
Correctly matched	10.36	40.67	9.84	40.41	84.35	99.18	82.18	98.94
$\Delta$	$\mathbf{\uparrow 0.60}$	$\mathbf{\uparrow 2.24}$	$\mathbf{\uparrow 0.17}$	$\mathbf{\uparrow 0.95}$	$\mathbf{\uparrow 2.31}$	$\mathbf{\uparrow 0.22}$	$\mathbf{\uparrow 2.81}$	$\mathbf{\uparrow 0.25}$

Appendix

Table 11: Ablation on salient–contextual swapping control. Results on T

\rightarrow

I and I

\rightarrow

T retrieval, with salient-only anchoring fixed. Best results are bold .

Filtering ratio	INSECT						DOCCI
Filtering ratio	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10
20%	10.02	27.63	37.39	9.24	26.08	38.51	80.25	96.71	98.69	78.37	96.37	98.57
10%	10.02	26.34	37.74	9.50	26.42	36.96	79.63	96.65	98.47	78.35	96.22	98.37
5%	9.84	29.02	39.72	8.98	27.89	39.12	80.25	96.76	98.82	79.16	95.86	98.53
0% (Ours)	10.36	28.32	40.67	9.84	29.36	40.41	84.35	97.51	99.18	82.18	97.14	98.94

Appendix

Table 12: Robustness to cross-modal mismatch. Results on Text

\rightarrow

Image and Image

\rightarrow

Text retrieval for INSECT, DOCCI. We discard different proportions of generated pairs according to their similarity scores. For example, 5% filtering ratio means removing the lowest-scored 5% pairs.

Dataset	Model	Entity	TF–IDF	Position	E5
DOCCI	GOAL	+13.95	+17.25	+16.33	+17.32
	LongCLIP-FT	+14.15	+17.54	+16.68	+17.58
	StructXLIP	+13.14	+17.26	+15.98	+17.39
INSECT	GOAL	+2.47	+2.02	+3.09	+2.57
	LongCLIP-FT	+2.41	+2.51	+3.47	+2.59
	StructXLIP	+3.15	+2.83	+3.58	+3.32

Appendix

Table 13: Single-factor controlled deletion. Each entry reports contextual-deletion R@1 - salient-deletion R@1, averaged over five salient-text deletion ratios. Columns represent separate matching metric. Positive values indicate greater degradation under salient deletion.

Model	0%	25%	50%	75%	100%
LongCLIP-FT	$18.26\pm 0.83$	$15.48\pm 0.20$	$13.09\pm 0.55$	$10.66\pm 0.12$	$7.77\pm 0.36$
GOAL	$20.06\pm 0.87$	$17.85\pm 0.55$	$15.58\pm 0.49$	$13.13\pm 0.13$	$9.67\pm 0.46$
StructXLIP	$20.14\pm 0.86$	$17.62\pm 0.21$	$15.47\pm 0.18$	$12.88\pm 0.34$	$9.42\pm 0.01$
RoSA	$\mathbf{21.02\pm 0.31}$	$\mathbf{19.65\pm 0.10}$	$\mathbf{17.39\pm 0.16}$	$\mathbf{15.98\pm 0.27}$	$\mathbf{13.53\pm 0.48}$

Appendix

Table 14: Entity-controlled reconstruction with

K=4

. Columns indicate the percentage of salient entities replaced by matched contextual entities from the same caption. Entries report text-to-image R@1 (%) over three seeds. The best mean in each column is highlighted in bold .

Query construction	Method	INSECT						DOCCI
		Salient-only		Salient 50% +Context		Context-only		Salient-only		Salient 50% +Context		Context-only
		R@1	R@10	R@1	R@10	R@1	R@10	R@1	R@10	R@1	R@10	R@1	R@10
Rewritten prompt	StructXLIP [CVPR’26]	7.55	33.19	9.68	36.17	3.62	16.81	65.15	92.53	78.30	97.47	34.98	65.85
	RoSA (Ours)	7.98	35.11	9.47	39.04	3.72	22.66	65.81	93.13	79.83	98.28	47.16	78.34
	$\Delta$	$\mathbf{\uparrow 0.43}$	$\mathbf{\uparrow 1.92}$	$\mathbf{\downarrow 0.21}$	$\mathbf{\uparrow 2.87}$	$\mathbf{\uparrow 0.10}$	$\mathbf{\uparrow 5.85}$	$\mathbf{\uparrow 0.66}$	$\mathbf{\uparrow 0.60}$	$\mathbf{\uparrow 1.53}$	$\mathbf{\uparrow 0.81}$	$\mathbf{\uparrow 12.18}$	$\mathbf{\uparrow 12.49}$
Llama-3.1	StructXLIP [CVPR’26]	7.19	34.69	9.17	35.83	3.12	18.02	66.92	93.23	76.04	96.82	35.29	64.47

Appendix

Table 15: Robustness to alternative query construction. We report T

\rightarrow

I Recall@K (%) on INSECT and DOCCI using test queries reconstructed with a rewritten prompt, Llama-3.1, or human annotations under Salient-only , Salient 50% +Context , and Context-only . Model checkpoints and full-image retrieval galleries remain fixed. For each query construction, we highlight best and second best results;

\Delta

denotes the margin of RoSA over StructXLIP (%). Human queries are evaluated only on DOCCI; “–” indicates an unevaluated setting.

Methods	T $\rightarrow$ I			I $\rightarrow$ T
Methods	R@1	R@5	R@10	R@1	R@5	R@10
Long-CLIP [ECCV’24]	3.20	11.99	19.30	3.49	11.15	18.08
FineLIP [CVPR’25]	3.72	13.05	20.51	3.69	12.31	19.99
SmartCLIP [CVPR’25]	2.13	8.65	14.39	2.08	7.93	13.45
GOAL [CVPR’25]	3.52	13.72	21.72	3.54	13.23	21.45
StructXLIP [CVPR’26]	3.81	12.48	20.71	3.27	12.43	20.36

Appendix

Table 16: Cross-modal retrieval on PathGen. We report R@K under T

\rightarrow

I and I

\rightarrow

T retrieval. Best and second-best results are marked in bold and underline .

Figure 8: Qualitative examples from the PathGen benchmark after semantic condensation. Top: representative pathology image-caption pairs and corresponding retrieval results. Bottom: attention map comparisons between StructXLIP and our RoSA . Our method RoSA exhibits better alignment with clinically relevant pathological regions described in the captions, including spindle-shaped tumor cells and yeast-like fungal structures associated with pulmonary blastomycosis.

Methods	Text $\rightarrow$ Image			Image $\rightarrow$ Text
Methods	R@1	R@5	R@10	R@1	R@5	R@10
Long-CLIP	3.20	11.99	19.30	3.49	11.15	18.08
+our $\mathcal{L}_{\mathrm{RoSA}}$	3.67 (+0.47)	12.63 (+0.64)	20.44 (+1.14)	3.49(+0.00)	11.99 (+0.84)	19.74 (+1.66)
FineLIP	3.72	13.05	20.51	3.69	12.31	19.99
+our $\mathcal{L}_{\mathrm{RoSA}}$	4.01 (+0.29)	13.15 (+0.10)	20.41 (-0.10)	4.14 (+0.45)	13.10 (+0.79)	20.19 (+0.20)
GOAL	3.52	13.72	21.72	3.54	13.23	21.45

Appendix

Table 17: Plug-and-play enhancement with our saliency-guided module on PathGen. We report Recall@K (%) under both Text

\rightarrow

Image and Image

\rightarrow

Text retrieval settings. The values in parentheses denote the absolute change over the corresponding baseline.

Table 26

Figure 9: Qualitative results of RoSA and the competitive method StructXLIP across four datasets. We visualize the attention maps corresponding to the blue bold text in each caption. Compared with StructXLIP, RoSA shows more focused activations on the corresponding visual objects described by the text.

Vision-Language Alignment

Vision-language alignment connects visual and textual representations so models can match, interpret, and ground language in images and video.

See all 361 papers

A Paragraph is Worth a Thousand Captions: Rethinking Text Supervision for Vision-Language Retrieval
Aug 5, 2026Mahyar Ghazanfari, Amin Tabrizian, Arsyi Aziz +2Image-Text Retrieval Vision-Language Retrieval
The George Washington University
HyFL-CLIP: Hyperbolic Fine-Tuning of CLIP for Robust Long-Context Understanding
Jul 1, 2026Ji Ha Jang, Hayeon Kim, Chulwon Lee +2VLM Robustness Vision-Language Models
1Dept. of Electrical and Computer Engineering, 2IPAI, 3INMC & AIIS, Seoul National University, Republic of Korea
Human-Grounded Calibration for Long-Text Image-Text Congruence in Vision-Language Models
Sep 14, 2026Alessandro Gambetti, Qiwei HanText-Image Alignment Vision-Language Retrieval
NOVA School of Science and Technology · Universidade NOVA de Lisboa · Caparica, Portugal +2

Aug 5, 2026J/K move · Enter open · S save

A Paragraph is Worth a Thousand Captions: Rethinking Text Supervision for Vision-Language Retrieval

Mahyar Ghazanfari, Amin Tabrizian, Arsyi Aziz, Binshuai Wang+1

The George Washington University

Image-Text Retrieval Vision-Language Retrieval

Open paper AskPDF arXiv

Contrastive vision-language models such as CLIP and BLIP are typically trained on short image captions, limiting their ability to retrieve images from detailed textual descriptions. While methods such as Long-CLIP extend the token limit through positional embedding interpolation, we ask a simpler question: does training text granularity alone determine long-text retrieval performance? We present a systematic study of supervision ranging from single captions to multi-sentence paragraphs for contrastive image-text retrieval. Using a synthetic pipeline based on Qwen2-VL and Llama 3.2 Vision, we generate diverse captions, hard negatives, and quality-scored paragraphs for 500K CC3M images. To isolate the effect of text granularity, we fine-tune only the BLIP text encoder while keeping the vision encoder frozen across 10 training configurations. Our paragraph-supervised models match Long-CLIP-L on ShareGPT4V and outperform it by more than 14 points on DOCCI for image-to-text retrieval, without architectural changes. We further show that paragraph supervision enables effective use of long token sequences, whereas caption-only training degrades beyond 60 tokens. Increasing caption diversity improves short-caption retrieval with diminishing returns, while paragraph supervision consistently benefits long-description benchmarks and hard negatives prove detrimental in text-only fine-tuning. Evaluations on Flickr30k, COCO, ShareGPT4V, and DOCCI provide a comprehensive analysis of the trade-offs between text granularity, retrieval direction, and description length.

Abstract

Vision-Language Alignment

Explore similar work

A Paragraph is Worth a Thousand Captions: Rethinking Text Supervision for Vision-Language Retrieval

HyFL-CLIP: Hyperbolic Fine-Tuning of CLIP for Robust Long-Context Understanding

Human-Grounded Calibration for Long-Text Image-Text Congruence in Vision-Language Models