GoldiCLIP: The Goldilocks Approach for Balancing Explicit Supervision for Language-Image Pretraining

Deen Dayal Mohan; Hossein Souri; Vitali Petsiuk; Juhong Min; Gopal Sharma; Luowei Zhou; Suren Kumar

cs.CVMar 25, 2026

GoldiCLIP: The Goldilocks Approach for Balancing Explicit Supervision for Language-Image Pretraining

Authors: Deen Dayal Mohan, Hossein Souri, Vitali Petsiuk, Juhong Min, Gopal Sharma, Luowei Zhou, Suren Kumar

Organizations: AI Center – Mountain View, Samsung Electronics

Abstract

Until recently, the success of large-scale vision-language models (VLMs) has primarily relied on billion-sample datasets, posing a significant barrier to progress. Latest works have begun to close this gap by improving supervision quality, but each addresses only a subset of the weaknesses in contrastive pretraining. We present GoldiCLIP, a framework built on a Goldilocks principle of finding the right balance of supervision signals. Our multifaceted training framework synergistically combines three key innovations: (1) a text-conditioned self-distillation method to align both text-agnostic and text-conditioned features; (2) an encoder integrated decoder with Visual Question Answering (VQA) objective that enables the encoder to generalize beyond the caption-like queries; and (3) an uncertainty-based weighting mechanism that automatically balances all heterogeneous losses. Trained on just 30 million images, 300x less data than leading methods, GoldiCLIP achieves state-of-the-art among data-efficient approaches, improving over the best comparable baseline by 2.2 points on MSCOCO retrieval, 2.0 on fine-grained retrieval, and 5.9 on question-based retrieval, while remaining competitive with billion-scale models. Project page: https://petsi.uk/goldiclip.

Figures & tables

Figure 1: An overview of GoldiCLIP approach. Input images, captions, and auxiliary textual data (such as VQA) are processed by the student and teacher models, which consist of a text encoder and a vision encoder each. The teacher model is constructed as an EMA of a student, and is used to produce robust and stable global image representations which serve as a target for the local representations of the student model following the self-distillation approach (Sec 3.2 ). In our framework, the text embedding is contrastively aligned not only with the standard image embedding but also with the text-conditioned image embedding (Sec 3.1 ). Multimodal decoder uses image patch tokens along with the textual tokens from the text encoder to perform generative tasks (such as VQA) (Sec 3.3 ). Finally, all objectives are automatically weighted using a task-balancing approach (Sec 3.4 ).

		(a) Standard Benchmarks								(b) Fine-Grained				(c) Long Text
Method	Data size	MSCOCO				Flickr30k				DOCCI-FG		IIW-FG		DCI		Urban 1k
		T2I		I2T		T2I		I2T		T2I	I2T	T2I	I2T	T2I	I2T	T2I	I2T
		R1	R5	R1	R5	R1	R5	R1	R5	R1	R1	R1	R1	R1	R1	R1	R1
DreamLIP [ 41 ]	3M	29.8	55.4	40.8	68.4	53.6	78.4	69.2	91.5	10.3	23.3	22.7	59.2	-	-	-	-
FLAIR [ 34 ]	3M	37.7	64.5	51.6	77.2	65.7	86.8	78.7	95.2	15.2	35.7	30.5	70.6	50.3	47.3	69.5	63.5
COSMOS [ 12 ]	3M	40.1	66.8	53.1	78.3	68.6	88.7	84.1	96.5	14.2	32.9	28.9	70.9	52.5	50.2	73.2	67.1

Table 1: Zero-shot image-text retrieval on (a) standard benchmarks (validation splits of MSCOCO [ 15 ] , Flickr30k [ 24 ] ), (b) fine-grained retrieval setting introduced by [ 34 ] (sentence-level on DOCCI [ 21 ] , IIW [ 8 ] ), and (c) long-text retrieval setting (DCI [ 30 ] , Urban 1k [ 40 ] ).

Figure 2: Example from RetVQA [ 23 ] , where the task is to retrieve relevant images for a question that can be used as context for generative models. For each query, there are two relevant images and a set of distractors. Unlike FLAIR [ 34 ] , our model correctly retrieves images containing the door knob and the train that are relevant to the query.

Panel of Figure 3 — Figure 3: Attention maps indicate good vision-text alignment.

Method	Data	T2I R2
Method	size	Count	Color	Attribute	Shape	Relation	Average
FLAIR [ 34 ]	30M	57.6	39.1	44.4	49.0	47.7	47.6
COSMOS [ 41 ]	30M	60.7	42.4	46.1	51.6	48.9	49.9
GoldiCLIP	30M	66.8	48.5	51.5	56.4	55.9	55.8
OpenCLIP [ 6 ]	2B	62.1	45.0	47.7	51.3	48.5	50.9
PE ${}_{\textrm{core}}$ [ 2 ]	5B	65.1	67.4	48.9	52.4	57.5	52.9

Table 2: R@2 for zero-shot retrieval for VQA task on [ 23 ] dataset build on top of VG [ 13 ] . The best and second-best results bolded and underlined respectively.

Method	Data size	VOC20	Cityscapes	Context59	ADE20K	COCO-Stuff	Average
SigLIP [ 38 ]	30M	14.5	5.5	5.8	2.2	3.8	6.4
DreamLIP [ 41 ]	30M	1.8	0.9	0.4	0.1	0.1	0.7
COSMOS [ 12 ]	30M	53.6	13.9	15.7	8.5	10.7	20.0
FLAIR [ 34 ]	30M	73.0	13.6	18.6	10.4	13.3	25.8
FLAIR ∗ [ 34 ]	30M	70.3	26.2	26.4	14.7	18.8	31.3
GoldiCLIP	30M	70.0	28.9	29.5	16.4	19.4	32.8

Table 3: Mean intersection over union (mIoU) for zero-shot semantic segmentation. The best and second-best results are bolded and underlined . ∗ No attention sink.

Method	Data size	Food-101	CIFAR-10	CIFAR-100	SUN397	Cars	Aircraft	DTD	Pets	Flowers	ImageNet	Average
CLIP [ 25 ]	30M	61.3	92.2	66.9	62.2	19.3	5.7	30.9	49.3	43.4	50.0	48.1
SigLIP [ 38 ]	30M	64.2	91.0	67.6	64.0	22.0	5.7	33.5	53.3	43.6	51.0	49.6
DreamLIP [ 41 ]	30M	75.4	92.3	70.7	63.7	22.7	7.9	33.9	64.1	51.1	58.1	54.0
FLAIR [ 34 ]	30M	72.5	93.1	69.6	66.9	31.1	7.2	37.3	55.6	48.4	56.6	53.8
COSMOS [ 12 ]	30M	73.9	93.4	72.3	67.8	31.2	5.9	37.3	62.9	54.6	57.6	55.7
GoldiCLIP	30M	74.0	92.9	70.7	67.2	34.5	6.6	33.9	62.7	51.4	57.4	55.1

Table 4: Top-1 accuracy for zero-shot classification. The best and second-best results per section are bolded and underlined .

Method	COCO		DOCCI		RVQA	VOC20	ImageNet
Method	T2I	I2T	T2I	I2T	T2I	mIOU	Top-1
CA	37.7	51.6	15.1	35.7	51.0	59.0	36.2
$+$ DC	42.1	56.4	16.7	40.3	51.4	50.6	37.5
$+$ VD	42.1	56.9	16.7	39.0	51.7	53.9	37.6
$+$ UN	42.9	58.2	17.6	42.3	52.0	55.1	38.8
$+$ SD	45.1	58.6	18.7	43.0	51.7	62.9	40.1

Table 5: Ablation study on different components of our method on the CC3M-recap dataset. CA : Cross Attention Pooling, DC : Multi-Task Decoder without VQA objective, VD : Multi-Task Decoder with VQA, UN : Uncertainty-based Task Weighting, SD : Text-Conditioned Self-Distillation.

Method	COCO		Flickr30k
Method	T2I	I2T	T2I	I2T
SILC	48.6	35.4	79.0	62.1
COSMOS	53.1	40.1	84.1	68.6
FLAIR	51.6	37.7	78.7	65.5
FLAIR + TC-SD	56.5	42.0	87.5	70.3

Table 6: Comparison of Prior Self-distillation methods to our text conditioned self-distillation applied to FLAIR. All methods trained on CC3M-recap dataset. Recall @1 reported for both I2T and T2I. Best results bolded.

Config	CC3M-recap	Open-30M
Batch size	$2,560$	$6,400$
Optimizer	AdamW
Learning rate	$5\times 10^{-4}$
Weight decay	$0.5$	$0.2$
Adam $\beta$	$\beta_{1}=0.9,\beta_{2}=0.98$
Adam $\epsilon$	$1\times 10^{-8}$	$1\times 10^{-6}$

Table 7: Pre-training hyper-parameters for our method.

Figure 4: Values of task coefficients changing over training epochs.

		(a) Standard Benchmarks								(b) Fine-Grained				(c) Long Text
Method	Data size	MSCOCO				Flickr30k				DOCCI-FG		IIW-FG		DCI		Urban 1k
		T2I		I2T		T2I		I2T		T2I	I2T	T2I	I2T	T2I	I2T	T2I	I2T
		R1	R5	R1	R5	R1	R5	R1	R5	R1	R1	R1	R1	R1	R1	R1	R1
DreamLIP [ 41 ]	3M	29.8	55.4	40.8	68.4	53.6	78.4	69.2	91.5	10.3	23.3	22.7	59.2	-	-	-	-
FLAIR [ 34 ]	3M	37.7	64.5	51.6	77.2	65.7	86.8	78.7	95.2	15.2	35.7	30.5	70.6	50.3	47.3	69.5	63.5
COSMOS [ 12 ]	3M	40.1	66.8	53.1	78.3	68.6	88.7	84.1	96.5	14.2	32.9	28.9	70.9	52.5	50.2	73.2	67.1

Table 8: Our model remains competetive on various retrieval benchmarks even when evaluated in text-agnostic mode, denoted as GoldiCLIP ta .

Method	Data	T2I R2
Method	size	Count	Color	Attribute	Shape	Relation	Average
FLAIR [ 34 ]	30M	57.6	39.1	44.4	49.0	47.7	47.6
COSMOS [ 41 ]	30M	60.7	42.4	46.1	51.6	48.9	49.9
GoldiCLIP	30M	66.8	48.5	51.5	56.4	55.9	55.8
GoldiCLIP ta	30M	65.5	46.9	48.8	55.0	54.5	54.1
OpenCLIP [ 6 ]	2B	62.1	45.0	47.7	51.3	48.5	50.9

Table 9: Our model remains competetive on the retrieval for VQA task even when evaluated in text-agnostic mode, denoted as GoldiCLIP ta .

Vision-Language Pretraining

Vision-language pretraining learns visual representations by relating images, video, or other visual inputs to language.

See all 80 papers

CLIMP: Contrastive Language-Image Mamba Pretraining
Jan 11, 2026Nimrod Shabtay, Itamar Zimerman, Eli Schwartz +1Vision-Language Pretraining VLM Robustness
Tel-Aviv University · IBM Research
A Paragraph is Worth a Thousand Captions: Rethinking Text Supervision for Vision-Language Retrieval
Aug 5, 2026Mahyar Ghazanfari, Amin Tabrizian, Arsyi Aziz +2Image-Text Retrieval Vision-Language Retrieval
The George Washington University
Hyper3-CLIP: Hierarchy-Conditioned Hyperbolic Vision-Language Training
Aug 29, 2026Matin Mahmood, Antonio Rueda-Toicen, Mohamed ElBassat +3Vision-Language Models Vision-Language Alignment
hyper3labs, Berlin, Germany · Hasso Plattner Institute, University of Potsdam, Potsdam, Germany · Faculty of Computers and Data Science, Alexandria University, Alexandria, Egypt +1

Jan 11, 2026J/K move · Enter open · S save

CLIMP: Contrastive Language-Image Mamba Pretraining

Nimrod Shabtay, Itamar Zimerman, Eli Schwartz, Raja Giryes

Tel-Aviv University · IBM Research

Vision-Language Pretraining VLM Robustness Mamba Vision-Language Retrieval Multimodal Foundation Models Cross-Modal Learning+2

Open paper AskPDF arXiv

Contrastive Language-Image Pre-training (CLIP) relies on Vision Transformers whose attention mechanism is susceptible to spurious correlations, and scales quadratically with resolution. To address these limitations, We present CLIMP, the first fully Mamba-based contrastive vision-language model that replaces both the vision and text encoders with Mamba. The new architecture encodes sequential structure in both vision and language, with VMamba capturing visual spatial inductive biases, reducing reliance on spurious correlations and producing an embedding space favorable for cross-modal retrieval and out-of-distribution robustness-surpassing OpenAI's CLIP-ViT-B by 7.5% on ImageNet-O. CLIMP naturally supports variable input resolutions without positional encoding interpolation or specialized training, achieving up to 6.6% higher retrieval accuracy at 16x training resolution while using 5x less memory and 1.8x fewer FLOPs. The autoregressive text encoder further overcomes CLIP's fixed context limitation, enabling dense captioning retrieval. Our findings suggest that Mamba exhibits advantageous properties for vision-language learning, making it a compelling alternative to Transformer-based CLIP.The code and models are publicly available at https://github.com/NimrodShabtay/CLIMP}

Abstract

Vision-Language Pretraining

Explore similar work

CLIMP: Contrastive Language-Image Mamba Pretraining

A Paragraph is Worth a Thousand Captions: Rethinking Text Supervision for Vision-Language Retrieval

Hyper3-CLIP: Hierarchy-Conditioned Hyperbolic Vision-Language Training