Tune-Up Open-Weight CLIP: Optimization Framework for Self-Supervised Fine-tuning of CLIP

Anant Mehta; Xingyu Chen; Xiyuan Wei; Tianbao Yang

cs.CVJan 14, 2026

Tune-Up Open-Weight CLIP: Optimization Framework for Self-Supervised Fine-tuning of CLIP

Authors: Anant Mehta, Xingyu Chen, Xiyuan Wei, Tianbao Yang

Organizations: Department of Computer Science & Engineering, Texas A&M University, USA

Abstract

CLIP has become a cornerstone of multimodal representation learning, yet improving its performance typically requires a prohibitively costly process of training from scratch on billions of samples. We ask a different question: Can we improve the performance of open-weight CLIP models across various downstream tasks using only existing self-supervised datasets? Unlike supervised fine-tuning, which adapts a pretrained model to a single downstream task, our setting seeks to improve general performance across various tasks. However, as both our experiments and prior studies reveal, simply applying standard training protocols starting from an open-weight CLIP model often fails, leading to performance degradation. In this paper, we introduce TuneCLIP, a self-supervised fine-tuning framework that overcomes the performance degradation. TuneCLIP has two key components: (1) a warm-up stage of recovering optimization statistics to reduce cold-start bias, inspired by theoretical analysis, and (2) a fine-tuning stage of optimizing a new contrastive loss to mitigate the penalization on false negative pairs. Our extensive experiments show that TuneCLIP consistently improves performance across model architectures and scales. Notably, it elevates leading open-weight models like SigLIP (ViT-B/16), achieving gains of up to +2.5% on ImageNet and related out-of-distribution benchmarks, and +1.2% on the highly competitive DataComp benchmark, setting a new strong baseline for efficient post-pretraining adaptation. The code is available at https://github.com/Optimization-AI/TuneCLIP.

Figures & tables

Figure 1: Early-stage performance drop on ImageNet when fine-tuning OpenAI ViT-B/16 model on DFN-12M dataset using OpenCLIP and FastCLIP pipelines.

Panel of Figure 2 — Figure 2: In supervised fine-tuning (red), OSR+GCL outperforms OSR+HGCL (TuneCLIP) because true negative labels justify separating negatives. In contrast, under self-supervised fine-tuning (blue), the absence of such labels makes OSR+HGCL more suitable, leading to improved retrieval performance on Flickr when fine-tuned with SSFT (see Appendix E for OpenAI CLIP & details).

Base Model	Method	Wall C.T. (hrs)	IN & Variants	Retrieval	DataComp
OpenAI ViT-B/16	Baseline	N/A	57.67	57.46	56.26
	FastCLIP	4.21	54.57 ( $\downarrow$ )	51.88 ( $\downarrow$ )	53.53 ( $\downarrow$ )
	OpenCLIP	5.46	54.99 ( $\downarrow$ )	57.81	55.11 ( $\downarrow$ )
	CLIP-Ref.	1.01	55.96( $\downarrow$ )	58.82	56.18( $\downarrow$ )
	TuneCLIP	8.62	59.36 (+1.69)	64.12 (+6.66)	58.62 (+2.36)
SigLIP ViT-B/16	Baseline	N/A	63.12	69.32	62.32

Table 1: Summary of mean zero-shot performance across ImageNet & distributional variants, retrieval benchmarks, and the DataComp benchmark, together with wall-clock training time per GPU.

( $m_{t}$ , $v_{t}$ ) $u_{t}$	IN & Variants	Retrieval	DataComp	Mean
( $\times$ , $\times$ ) $\times$	54.91	58.64	54.49	56.01
(✓, ✓) $\times$	59.48 (+4.57)	63.70	58.56	60.58
(✓, ✓) ✓	59.36	64.12 (+5.48)	58.62 (+4.13)	60.70 (+4.69)

Table 2: Impact of OSR statistics transfers on HGCL fine-tuning (OpenAI ViT-B/16 CLIP):

(m_{t},v_{t})

yield a large gain, with

u_{t}

providing additional improvement for the best overall performance.

Panel of Figure 3 — Figure 3: (a) Comparison of different false negative mitigation strategies in the second stage. DCL extends debiased contrastive learning Chuang et al. (2020) to CLIP, TNR removes top negatives during gradient estimation (based on Huynh et al. (2022) ). All use OSR in first stage, initialized from OpenAI B/16. (b) Effect of data scaling on TuneCLIP, demonstrating stable monotonic improvements as data scale increases.

Method	Data	IN & Variants	Retrieval	DataComp
OpenAI CLIP	–	57.67	57.46	56.26
TuneCLIP	CC12M	57.68 (+0.01)	65.83 (+8.37)	56.47 (+0.21)
TuneCLIP	DFN12M	59.36 (+1.69)	64.12 (+6.66)	58.62 (+2.36)

Table 3: TuneCLIP performance across datasets of varying noise levels.

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Image Size	Layers	Width	Patch Size
CLIP ViT-B/32	224	12	768	32
CLIP ViT-B/16	224	12	768	16
SigLIP ViT-B/16	224	12	768	16

Appendix

Table 4: Vision tower configurations of CLIP models.

Model	Context Length	Vocab Size	Width	Heads	Layers
CLIP ViT-B/32	77	49408	512	8	12
CLIP ViT-B/16	77	49408	512	8	12
SigLIP ViT-B/16	64	32000	768	12	12

Appendix

Table 5: Text tower configurations of CLIP models.

Model	Vision Encoder	Text Encoder	Parameters (M)	Developer
CLIP ViT-B/32	ViT	Transformer	151.28	OpenAI [ 1 ]
CLIP ViT-B/16	ViT	Transformer	149.62	OpenAI [ 1 ]
CLIP ViT-B/32	ViT	Transformer	151.28	LAION [ 41 ]
SigLIP ViT-B/16	ViT	Transformer	203.16	Google [ 42 ]

Appendix

Table 6: Model specifications of different CLIP variants.

Hyperparameter	Value
Image size	224x224
Learning rate (lr)	1e-5
Optimizer	AdamW
Beta1, Beta2	0.9, 0.98
Weight decay (wd)	0.02
Scheduler	Cosine

Appendix

Table 7: Key hyperparameters used for training the models.

Learning Rate	ImageNet-1k (%)	MS COCO (Avg R@1) (%)
1e-4	69.66	48.98
1e-5	70.57	50.11
5e-6	70.23	49.30

Appendix

Table 8: Learning rate sweep on TuneCLIP

Figure 4: Sensitivity of margin ( $m$ ).

Method	Loss	Optimization Strategy
FastCLIP [ 9 ]	GCL	SogCLR + AdamW
OpenCLIP [ 2 ]	MBCL	AdamW
CLIP-Refine [ 23 ]	RaFA+HyCD [ 23 ]	AdamW
TuneCLIP (ours)	HGCL (ours)	OSR (ours) + SogCLR + AdamW

Appendix

Table 9: Training configurations for compared methods.

Panel of Figure 5 — Figure 5: Similar to Figure 2 , supervised fine-tuning (red) shows stronger performance with OSR+GCL than with OSR+HGCL (TuneCLIP), since true negative labels justify separating negatives. By contrast, in self-supervised fine-tuning (blue), the absence of such labels makes OSR+HGCL more effective.

Panel of Figure 6 — Figure 6: Retrieval trends on the Flickr1k test set under supervised fine-tuning. In the supervised setting, explicit labels guide the separation of positives from negatives, making OSR+GCL outperform OSR+HGCL. By contrast, as shown in Figures 2(b) and 5(b) , the absence of supervision and margin regularization in SSFT reverses this trend, with OSR+HGCL (TuneCLIP) achieving superior retrieval performance.

Panel of Figure 8 — Figure 8: Zero-shot classification scores (ImageNet-1k) over 5 epochs of fine-tuning for all CLIPs.

Figure 9: ImageNet-1k accuracy after one epoch of fine-tuning on DFN-12M (a subset of DFN2B). Despite matching the pretraining data distribution, naïve fine-tuning methods (FastCLIP, OpenCLIP) exhibit clear performance degradation compared to the DFN2B-pretrained ViT-B/16 baseline.

Figure 10: Extended fine-tuning analysis of TuneCLIP and baselines on ImageNet-1k.

Base Model	Method	DataComp Average
OpenAI ViT-B/32	Baseline	52.45
	FastCLIP	49.78
	OpenCLIP	51.02
	CLIP-Refine	51.58
	TuneCLIP	54.34 (+1.89)
OpenAI ViT-B/16	Baseline	56.26

Appendix

Table 10: Performance of different CLIP models on DataComp Average. DataComp [ 40 ] is a highly comprehensive benchmark that spans a diverse collection of datasets, tasks, and distributional variants. Even small improvements on DataComp are particularly meaningful, as they indicate stable gains across heterogeneous and challenging settings rather than isolated benefits on individual datasets.

Base Model	Method	CIFAR-10	CIFAR-100	STL-10
OpenAI ViT-B/32	Baseline	89.83	64.23	97.13
	FastCLIP	90.54	69.51	92.68
	OpenCLIP	91.75	71.18	96.36
	CLIP-Refine	89.94	68.15	96.35
	TuneCLIP	93.63 (+3.80)	73.87 (+9.64)	97.20 (+0.07)
OpenAI ViT-B/16	Baseline	90.77	66.95	98.25

Appendix

Table 11: Performance of different CLIP models on small-scale classification benchmarks.

Base Model	Method	1K	Sketch	V2
OpenAI ViT-B/32	Baseline	63.32	42.29	55.92
	FastCLIP	59.59	41.26	52.25
	OpenCLIP	61.11	41.86	53.84
	CLIP-Refine	62.16	41.50	55.12
	TuneCLIP	65.29 (+1.97)	45.84 (+3.55)	57.45 (+1.53)
OpenAI ViT-B/16	Baseline	68.34	48.24	61.88

Appendix

Table 12: Performance of different CLIP models on ImageNet-1k, ImageNet-Sketch, ImageNet-V2.

Base Model	Method	A	O	R	ObjectNet
OpenAI ViT-B/32	Baseline	31.55	47.75	69.33	44.31
	FastCLIP	24.84	47.20	65.21	44.32
	OpenCLIP	27.10	48.85	67.13	46.02
	CLIP-Refine	28.74	48.90	68.46	48.15
	TuneCLIP	30.72 (-0.83)	51.11 (+3.36)	70.67 (+1.34)	46.45 (+2.14)
OpenAI ViT-B/16	Baseline	49.95	42.30	77.70	55.31

Appendix

Table 13: Performance of CLIP models on ImageNet out-of-distribution variants [ 15 ] (A, O, R, ObjectNet). ObjectNet [ 56 ] evaluates robustness under real-world viewpoint and background shifts.

Base Model	Method	VTAB Mean	Fairness Mean
OpenAI ViT-B/32	Baseline	51.81	68.02
	FastCLIP	48.51	69.52
	OpenCLIP	49.23	70.05
	CLIP-Refine	49.26	70.25
	TuneCLIP	53.50 (+1.69)	70.73 (+2.71)
OpenAI ViT-B/16	Baseline	53.80	72.45

Appendix

Table 14: Performance of different CLIP models on VTAB and Fairness. The Visual Task Adaptation Benchmark (VTAB) [ 57 ] evaluates performance across 12 diverse tasks. We also report performance on two fairness-oriented datasets, Dollar Street and GeoDE [ 58 ] , which measure robustness to geographic and socioeconomic diversity.

Base Model	Method	MSCOCO		Flickr
		IR@1	TR@1	IR@1	TR@1
OpenAI ViT-B/32	Baseline	30.44	50.12	58.78	78.90
	FastCLIP	28.75	43.23	51.49	68.19
	OpenCLIP	33.33	49.79	60.92	76.10
	CLIP-Refine	34.09	52.79	61.82	79.00
	TuneCLIP	36.74 (+6.30)	56.16 (+6.04)	64.71 (+5.93)	83.30 (+4.40)

Appendix

Table 15: Retrieval performance (Recall@1) on MSCOCO and Flickr datasets.

Model	Local Batch Size	Global Batch Size
OpenAI CLIP ViT-B/32	512	4096
OpenAI CLIP ViT-B/16	256	2048
LAION CLIP ViT-B/32	512	4096
SigLIP ViT-B/16	256	2048

Appendix

Table 16: Batch size specifications of different CLIP variants under our distributed training setup.

Base Model	Method	Wall-Clock Time (hrs)	GPU-hours	DataComp
OpenAI ViT-B/32	OpenCLIP	2.66	21.28	51.02
	FastCLIP	2.22	17.76	49.78
	CLIP-Refine	0.58	4.64	51.58
	TuneCLIP	5.05	40.40	54.34
OpenAI ViT-B/16	OpenCLIP	5.46	43.68	55.11
	FastCLIP	4.21	33.68	53.53

Appendix

Table 17: Unified comparison of training cost and performance across CLIP backbones.

Base Model	Method	Wall-Clock (hrs)	GPU-hours
OpenAI ViT-B/32	OpenCLIP	2.66	21.28
	FastCLIP	2.22	17.76
	OSR only	2.50	20.00
	HGCL only	2.55	20.40
OpenAI ViT-B/16	OpenCLIP	5.46	43.68
	FastCLIP	4.21	33.68

Appendix

Table 18: Isolated computational cost of the OSR and HGCL stages in TuneCLIP.

Cold-Start Mitigation	Stage II	OSR	IN & Variants	Retrieval	DataComp
Base Model	N/A	$\times$	57.67	57.46	56.26
Momentum SGD	Momentum SGD	$\times$	54.65	59.32	54.82
Larger Batch Gradients	AdamW	$\times$	58.27	62.92	58.05
OSR	Momentum SGD	$\checkmark$	57.99	62.08	57.82
OSR	AdamW	$\checkmark$	59.36	64.12	58.62

Appendix

Table 19: Comparison of cold-start mitigation strategies for TuneCLIP (on OpenAI CLIP ViT-B/16).

Category	Baseline	TuneCLIP
IN & Variants	71.80	73.23
Retrieval	74.78	73.78
DataComp	69.61	69.23

Appendix

Table 20: Performance of TuneCLIP on SOTA ViT-H/14-quickgelu across evaluation suites.

Method	ImageNet-1k	MS COCO
OpenAI CLIP ResNet-50	59.82	38.69
TuneCLIP	60.95	45.19

Appendix

Table 21: Performance of TuneCLIP on OpenAI CLIP with a ResNet-50 backbone.

Panel of Figure 11 — Figure 11: OSR+GCL shows higher variance among false negatives (std. dev. $0.030$ ). While, OSR+HGCL yields a substantially more compact distribution (std. dev. $0.0061$ ), reducing false negative bias and at the same time ensuring a clear separation between true positives ( $\mu=0.28$ ) and true negatives ( $\mu=0.23$ ).

Anchor Image	False Negative Captions ( $s_{1}$ , $s_{2}$ ) in Top-5 Retrieved texts
Caption: FABULOUS PEARL NECKLACE DIAMOND CLUSTER 14K GOLD CLASP \| eBay	• Women’s Chain Necklaces s 1 =0.2053, s 2 =0.2467 • necklace image New Arrivals s 1 =0.1720, s 2 =0.2429
Caption: Boys’ Soccer Cleats 0026 Shoes. Nike.com	• running shoes s 1 =0.1561, s 2 =0.2346
Caption: dress	• muslim wedding dresses 3d flower burgundy muslim wedding dresses 2018 arabic custom plus s 1 =0.1713, s 2 =0.2238 • Evening Gowns For Mother Of The Bride In Singapore - Prom Dresses s 1 =0.1913, s 2 =0.2333
Caption: Chicken noodle soup	• top down view of bowl filled with white bean tomatillo soup with items surrounding. s 1 =0.2061, s 2 =0.2047 • Butternut soup with sriracha. Made it. s 1 =0.2046, s 2 =0.2354
Caption: pallets chair	• B32 Office Chair by Armet 3 s 1 =0.0747, s 2 =0.2189 • Burlap Ruffle Chair #burlap #ruffle #furniture s 1 =0.3096, s 2 =0.2583
Caption: 2017 Audi S6 4.0T quattro Premium Plus	• gebraucht Audi S8 plus V8 4.0TFSI tiptr. UPE 154.100,- HeadUp/SD s 1 =0.2358, s 2 =0.2272 • Audi Q3 und Audi RS Q3: Facelift und mehr Leistung. s 1 =0.1056, s 2 =0.2174

Appendix

Table 22: Qualitative examples of false negatives (text captions) and their similarity scores with the anchor image. Here,

s_{1}

denotes the similarity under OSR+GCL and

s_{2}

under OSR+HGCL . These examples illustrate how OSR+HGCL mitigates excessive suppression of false negatives, allowing them to retain higher similarity scores compared to OSR+GCL. Conversely, false negatives that already exhibit reasonable similarity with the anchor tend to remain stable or slightly reduced, compensating for cases where suppression was more severe. Overall, this yields a calibrated similarity structure in which false negatives are assigned scores that better reflect semantic relatedness.

VLM Adaptation

Vision-language model adaptation tailors pretrained image-text and multimodal models to new tasks and domains while retaining transferable capabilities.

See all 551 papers

May 28, 2026

TRACER: Persistent Regularization for Robust Multimodal Finetuning

Mainstream strategies for finetuning pretrained multimodal models often degrade out-of-distribution (OOD) robustness, a phenomenon known as catastrophic forgetting. In this paper, we develop a theoretical framework for multimodal contrastive finetuning, yielding closed-form solutions and a geometric decomposition for each strategy. This framework shows that self-distillation is more effective than other regularization approaches to retain the knowledge of the pretrained model. Our analysis reveals a largely overlooked limitation: standard Exponential Moving Average (EMA) teachers, widely used in robust finetuning, suffer from collapse. To solve this, we prove that a Weighted Moving Average (WMA) teacher maintains a persistent regularizing force over finite horizons and yields bias-free convergence in the task subspace while preserving orthogonal knowledge. These insights motivate TRACER (Trajectory-Robust Anchoring for Contrastive Encoder Regularization), which combines contrastive learning with WMA-guided multi-perspective distillation. Extensive experiments on CLIP finetuning demonstrate consistent OOD accuracy and calibration gains across three backbone architectures, and comprehensive ablations confirm that TRACER is both principled and robust to hyperparameter choices. Code is available at https://github.com/HesamAsad/TRACER.

Hesam Asadollahzadeh, Feng Liu, Christopher Leckie +1

School of Computing and Information Systems (CIS), Faculty of Engineering and IT (FEIT), University of Melbourne, Australia.

VLM Robustness Neural Network Robustness OOD Generalization Self-Distillation Multimodal Foundation Models Multimodal Contrastive Learning+2

Ask PDF

Nov 3, 2026

Bridging CLIP with DINO: Cross-Modal Information Maximization for Online Test-Time Adaptation

Vision-language models such as CLIP enable zero-shot classification from arbitrary class names, but under test-time corruptions, their visual encoder projects distorted features onto incorrect textual anchors with high confidence: a phenomenon we term modality bias. Standard entropy minimization reinforces these overconfident errors, which creates a confirmation-bias loop and collapses the predicted label distribution. We propose an online test-time adaptation (TTA) method that breaks this loop at three levels. At the representation level, a frozen self-supervised backbone (e.g., DINO v2) is combined with CLIP through a cross-covariance logit ensemble. CLIP text prototypes are projected into the feature space of this text-free backbone, so the self-supervised branch classifies native image features and is less affected by the text bias of CLIP. At the optimization level, a gated test-time information-maximization objective pairs per-sample sharpening with batch-level diversity, while an adaptive gate excludes unreliable samples from the gradient. At the prediction level, an online logit adjustment corrects the residual class-distribution skew. CDIM outperforms the compared CLIP-based online TTA baselines on ImageNet-C, CIFAR10-C, and CIFAR100-C, and remains competitive in continual and mixed-domain TTA. Extensive analyses and ablation studies verify the contribution of each of its components. Our code is available at https://github.com/B1ueC1oud/CDIM

Taehoon Kim, Wonjun Hwang, Hyunsouk Cho

Korea University · Ajou University

Test-Time Adaptation of VLMs Test-Time Adaptation VLM Robustness Modality Gap in VLMs

Ask PDF

May 15, 2026

Sparse Autoencoders enable Robust and Interpretable Fine-tuning of CLIP models

Large-scale pre-trained vision-language models like CLIP demonstrate remarkable zero-shot performance across diverse tasks. However, fine-tuning these models to improve downstream performance often degrades robustness against distribution shifts. Recent approaches have attempted to mitigate this trade-off, but often rely on computationally expensive text-guidance. We propose a novel method for robust fine-tuning, SAE-FT, which operates only on the model's visual representations. SAE-FT regularizes changes to these representations by penalizing the addition and removal of semantically meaningful features identified by a Sparse Autoencoder trained on the pre-trained model. This constraint prevents catastrophic forgetting and makes the fine-tuning process interpretable, enabling direct analysis of semantic changes. SAE-FT is both mechanistically transparent and computationally efficient, matching or exceeding state-of-the-art performance on ImageNet and its associated distribution shift benchmarks. Code is publicly available at: https://github.com/Fabian-Mor/sae-ft.

Fabian Morelli, Arnas Uselis, Ankit Sonthalia +1

University of Tübingen ∗ · University of Tübingen

Distribution Shift Robustness VLM Robustness VLM Adaptation VLM Interpretability Sparse Autoencoders

Ask PDF

Abstract

VLM Adaptation

Explore similar work

TRACER: Persistent Regularization for Robust Multimodal Finetuning

Bridging CLIP with DINO: Cross-Modal Information Maximization for Online Test-Time Adaptation

Sparse Autoencoders enable Robust and Interpretable Fine-tuning of CLIP models