Bigger Text Encoders Can Hurt CLIP Zero-Shot Performance

Samir Char; Carles Domingo-Enrich; Randall Balestriero

cs.CVSep 4, 2026

Bigger Text Encoders Can Hurt CLIP Zero-Shot Performance

Authors: Samir Char, Carles Domingo-Enrich, Randall Balestriero

Organizations: Microsoft · Brown University

Abstract

Contrastive Language-Image Pretraining (CLIP) is a building block of many machine learning applications. Scaling laws have guided resource allocation for large-scale training, yet prior work treats total CLIP model size as a single variable, without exploring how the capacity split between encoders impacts downstream performance. Here, we train multiple CLIP models with different vision and text encoder sizes, revealing that for most vision encoders, there is an optimal text encoder size beyond which zero-shot performance degrades---even as total parameter count increases. Exploiting this behavior yields efficient configurations that match the zero-shot performance of the standard ViT-B/16 architecture with up to 55% fewer parameters. We further show that this degradation stems from overfitting induced by the oversized text encoder, and that using modality-specific weight decay coefficients not only recovers but improves performance across all degraded configurations. A geometric analysis reveals a trade-off in which scaling the text encoder improves embedding uniformity but worsens cross-modal alignment; we further show that these metrics are predictive of zero-shot performance. We hope these findings motivate CLIP architectures and training methods that counteract this degradation, a prerequisite for scaling CLIP reliably and efficiently.

Figures & tables

Figure 1: Naively scaling CLIP can degrade downstream performance. (a) Popular CLIP models by vision (x-axis) and text encoder (y-axis) size ( $n{=}45$ ; points jittered). Most models cluster just below the diagonal, leaving most vision-text capacity splits unexplored. (b) A grid of 30 architectures spanning five vision and six text encoder sizes, each pre-trained on CC3M and CC12M. (c) Every configuration is evaluated across 30 zero-shot and 31 linear-probe tasks, and its embedding geometry is quantified via alignment, uniformity, modality gap, and RankMe. (d) Counterintuitively, under standard training (solid) zero-shot performance rises, peaks and then degrades as the text encoder grows with the vision encoder fixed; modality-specific weight decay (dashed) removes the degradation by applying separate coefficients $\lambda_{v}$ and $\lambda_{t}$ to the encoders.

Name	Layers	Width	Heads	Params
Atto	12	256	4	10M
Tiny	10	640	10	50M
Base	12	768	12	86M
Giant	20	896	14	194M
Colossal	22	1152	18	352M

Table 1: Vision and text encoder architectures in the model grid: five vision encoders spanning 10M–352M parameters crossed with six text encoders spanning 3M–145M, for 30 configurations. All models share a vocabulary size of 49,408, context length of 77 tokens, patch size of 16, input resolution of

224\times 224

, and a shared embedding dimension of 512.

Figure 2: Beyond an optimal size, larger text encoders hurt zero-shot performance. Average zero-shot performance (y-axis; higher is better), averaged over 30 zero-shot tasks, as a function of text encoder size (x-axis), colored by vision encoder, for models pre-trained on (a) CC12M and (b) CC3M.

Figure 3: Exploiting the vision–text asymmetry yields configurations matching ViT-B/16 zero-shot performance with 30–55% fewer parameters. Average zero-shot performance (y-axis; higher is better) versus total model parameters (x-axis) on CC12M for all 30 architectures. Color encodes the vision encoder and marker shape encodes the text encoder; the red dotted line traces the Pareto frontier and the gray dashed lines mark the Base–Base (ViT-B/16) baseline.

Figure 4: The vision–text parameter split matters beyond the total parameter count. (a) Each node represents an encoder positioned on the y-axis according to its size. Left-to-right arrows (blue) point from each text encoder to the best vision encoder paired with it in terms of average zero-shot performance; right-to-left arrows (magenta) point from each vision encoder to the best text encoder. (b) Predicted average zero-shot performance (higher is better) from the fitted saturating-ceiling regression with a capacity-mismatch penalty (Eq. 1 ), as a function of the vision (x-axis) and text (y-axis) encoder size. Contour lines trace the regression’s iso-performance levels and the 30 architectures are overlaid as points colored by their observed average zero-shot performance.

Figure 5: CLIP demands modality-specific encoder capacity. (a) Final-epoch CLIP training loss on CC12M (y-axis) versus text-encoder size (x-axis), colored by vision encoder. (b) Average zero-shot performance (y-axis; higher is better) for the Base–Giant configuration as weight decay is increased on the text encoder only (

\lambda_{t}

), the vision encoder only (

\lambda_{v}

), or both jointly, all emanating from the Base–Giant baseline with joint weight decay of 0.5. The dashed line marks the best configuration using the Base vision encoder (Base–Tiny). (c) Average zero-shot performance (y-axis; higher is better) for each vision encoder paired with the oversized Giant text encoder (x-axis), comparing the degraded configuration (

\lambda_{t}{=}0.5

\lambda_{v}{=}0.5

; hatched bars) with the fixed configuration (

\lambda_{t}{=}1

\lambda_{v}{=}0.5

; solid bars). (d) Final-epoch CLIP training loss on CC12M (y-axis) for the same configurations as in (c), comparing the degraded configuration (hatched) with the fixed configuration (solid).

Figure 6: Scaling the text encoder improves embedding uniformity but worsens image–text alignment. (a) Alignment, (b) image uniformity, and (c) text uniformity (lower is better for all three; y-axes) of MSCOCO embeddings (

n{=}5,000

images-caption pairs) from the CC12M pre-trained models, plotted as a function of text encoder size (x-axis) and colored by vision encoder.

Figure 7: Modality-specific weight decay improves image-text alignment. Embedding alignment (x-axis) versus image uniformity (y-axis) on MSCOCO (lower is better for both), with arrows pointing from the degraded (crosses) to the fixed (circles) configuration using modality-specific weight decay. Colors denote the vision encoder, all paired with the Giant text encoder.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Table A1 : Pre-training hyperparameters for (a) CC3M and (b) CC12M datasets. All runs use the AdamW optimizer ( Loshchilov and Hutter, 2019 ) .

Task type	Dataset	Task	Test size	Classes	Main metric	ZS	LP
Classification	Caltech-101 ( Fei-Fei et al., 2007 )	Object recognition	6,085	102	mean per class	✓	✓
	CIFAR-10 ( Krizhevsky et al., 2009 )	Visual recognition	10,000	10	accuracy	✓	✓
	CIFAR-100 ( Krizhevsky et al., 2009 )	Visual recognition	10,000	100	accuracy	✓	✓
	CLEVR Counts ( Johnson et al., 2017 ; Zhai et al., 2019 )	Counting	15,000	8	accuracy		✓
	CLEVR Distance ( Johnson et al., 2017 ; Zhai et al., 2019 )	Distance prediction	15,000	6	accuracy		✓
	Country211 ( Radford et al., 2021 ; Thomee et al., 2016 )	Geolocation	21,100	211	accuracy	✓	✓

Appendix

Table A2 : Evaluation tasks in the benchmark suite. Columns ZS and LP indicate whether each dataset is used for zero-shot and linear probe evaluation, respectively; a checkmark (✓) marks inclusion.

Group	Datasets
ImageNet	ImageNet-1k
ImageNet Shifts	ImageNet Sketch, ImageNet v2, ImageNet-A, ImageNet-O, ImageNet-R
Fine-grained	Stanford Cars, Country211, FGVC Aircraft, Food-101, DTD,
	Flowers-102, Oxford-IIIT Pets, RESISC45, Diabetic Retinopathy, PatchCamelyon
Retrieval	Flickr30k, MSCOCO

Appendix

Table A3 : Dataset groups used for per-group analysis. Groups are not exhaustive of all evaluation datasets.

Seed	Base–Tiny (%)	Base–Giant (%)	$\Delta$ (pp; Tiny $-$ Giant)
$123$	$38.70$	$37.68$	$+1.02$
$321$	$38.23$	$37.15$	$+1.08$
$0$	$38.25$	$37.93$	$+0.32$
Mean	$38.39$	$37.59$	$+0.81$
SD	$0.27$	$0.40$	$0.42$

Appendix

Table A4 : Smallest zero-shot degradation is robust to the training seed. Average zero-shot performance for the Base–Tiny (best) and Base–Giant (largest text encoder; degraded) configurations, each trained at three seeds (123, 321, 0).

\Delta

is the per-seed paired difference; a one-sided paired

t

-test on

\Delta

(H 0 :

\Delta\leq 0

) gives

t(2){=}3.29

p{=}0.04

, Cohen’s

d_{z}{=}1.901

, and a one-sided

95\%

CI of

[+0.09,\ \infty)

Figure A1 : The zero-shot peak-and-degradation pattern holds across groups and is most pronounced for fine-grained tasks. Average zero-shot performance (y-axis; higher is better) versus text encoder size (x-axis), colored by vision encoder, for CC12M models split by groups defined in Table A3 : (a) ImageNet, (b) ImageNet distribution shifts, (c) fine-grained, and (d) retrieval.

Figure A2 : Linear probe scaling shows no degradation across groups. Average linear probe performance (y-axis; higher is better) versus text encoder size (x-axis), colored by vision encoder, for CC12M models split by groups defined in Table A3 : (a) ImageNet and (b) fine-grained.

Figure A3 : Using the Colossal vision encoder leads to strong overfitting on CC3M but generalizes on CC12M. Training loss (top row; a, b) and ImageNet zero-shot validation top-1 accuracy (bottom row; c, d; higher is better) versus epoch (x-axis) for the Colossal vision encoder pre-trained on (a, c) CC12M and (b, d) CC3M, colored by text encoder.

Figure A4 : Using the Base vision encoder leads to milder overfitting on CC3M. Training loss (top row; a, b) and ImageNet zero-shot validation top-1 accuracy (bottom row; c, d; higher is better) versus epoch (x-axis) for the Base vision encoder pre-trained on (a, c) CC12M and (b, d) CC3M, colored by text encoder.

Figure A5 : Unlike zero-shot, scaling the text encoder does not degrade linear probe performance. Average linear probe performance (y-axis; higher is better), averaged over 31 downstream tasks, as a function of text encoder size (x-axis) on (a) CC12M and (b) CC3M, colored by vision encoder.

Model	Functional form	# params	$R^{2}$	Adj. $R^{2}$	LOOCV $R^{2}$
linear-log: total params	$\beta_{0}+\beta_{1}\log_{10}(N_{v}{+}N_{t})$	2	0.336	0.312	0.240
linear-log: vision + text	$\beta_{0}+\beta_{1}\log_{10}N_{v}+\beta_{2}\log_{10}N_{t}$	3	0.639	0.613	0.523
linear-log: vision + text + interaction	$\beta_{0}+\beta_{1}\log_{10}N_{v}+\beta_{2}\log_{10}N_{t}+\beta_{3}(\log_{10}N_{v})(\log_{10}N_{t})$	4	0.724	0.693	0.543
saturating ceiling	$(\beta_{0}+\beta_{1}\log_{10}N_{v})(1-e^{-kN_{t}})$	3	0.880	0.871	0.826
saturating ceiling with mismatch penalty	$(\beta_{0}+\beta_{1}\log_{10}N_{v})(1-e^{-kN_{t}})\exp\!\left(-c\left(\log_{10}\tfrac{N_{v}}{N_{t}}\right)^{2}\right)$	4	0.960	0.956	0.937

Appendix

Table A5 : A saturating-ceiling model with a capacity-mismatch penalty best predicts zero-shot performance from encoder size. Fit statistics for five models of average zero-shot performance across the 30 CC12M architectures (final-epoch checkpoint), where

N_{v}

and

N_{t}

are the vision and text encoder parameter counts (millions): number of free parameters, in-sample

R^{2}

, adjusted

R^{2}

, and leave-one-out cross-validated

R^{2}

(LOOCV; higher is better). The mismatch-penalty model (last row) is defined in Eq. 1 . Best metrics are bolded.

Figure A6 : Large text encoders overfit held-out image caption datasets. Final-epoch CLIP contrastive loss on the CC12M training set (x-axis) versus a held-out set (y-axis; lower is better): (a) MSCOCO and (b) Flickr30k. For each vision encoder, its best text-encoder pairing (circles) is compared against the same vision encoder paired with the oversized Giant text encoder (crosses).

Architecture	$\lambda_{t}$	$\lambda_{v}$	Avg Performance (%)
Base–Giant	0.5	0.5	37.0
	0.75	0.5	38.4
	1.0	0.5	39.3
	0.5	0.75	36.8
	0.5	1.0	36.3
	0.75	0.75	38.3

Appendix

Table A6 : Modality-specific weight-decay ablations. Average zero-shot performance (higher is better) for every weight-decay ablation run of the Base–Giant and Giant–Giant configurations.

\lambda_{t}

and

\lambda_{v}

are the per-encoder weight decays (baseline

\lambda_{t}{=}\lambda_{v}{=}0.5

). Within each configuration, the two best-performing weight-decay settings are bolded.

Figure A7 : Weight-decay vision-text asymmetry replicates on the Giant–Giant configuration. Average zero-shot performance (y-axis; higher is better) as weight decay is increased on the text encoder only (

\lambda_{t}

), the vision encoder only (

\lambda_{v}

), or both jointly, all emanating from the Giant–Giant baseline with joint weight decay of 0.5. The dashed line marks the best configuration using the Giant vision encoder (Giant–Tiny).

Figure A8 : Learning collapses when weight decay is too strong. CLIP training loss (y-axis) versus training step (x-axis) for the Base–Giant configuration on CC12M with weight decay raised to

\lambda{=}1.5

on (a) the text encoder only and (b) both encoders jointly.

Architecture	$\lambda_{t}{=}0.5$	$\lambda_{t}{=}1.0$	$\Delta$ (pp)	Improved tasks ( $n/N$ )	$p$ -value
	Avg Performance (%)
Atto–Giant	30.3	33.0	+2.7	25/30	<0.001
Tiny–Giant	35.2	37.9	+2.7	27/30	<0.001
Base–Giant	37.0	39.3	+2.3	24/30	0.001
Giant–Giant	39.8	41.5	+1.7	21/29	0.024

Appendix

Table A7 : Modality-specific weight decay improves every degraded configuration. Average zero-shot performance (higher is better) for each vision encoder paired with the oversized Giant text encoder, at the baseline text weight decay (

\lambda_{t}{=}0.5

) and raised to

\lambda_{t}{=}1.0

(vision

\lambda_{v}{=}0.5

throughout). “Improved tasks” counts datasets (

n

N

) where

\lambda_{t}{=}1.0

beats the baseline;

p

is a two-sided sign test. Numbers behind Figure 5 (c).

Figure A9 : Modality-specific weight decay reduces held-out overfitting on MSCOCO and Flickr30k. Final-epoch CLIP contrastive loss on the CC12M training set (x-axis) versus the held-out set (y-axis; lower is better) on (a) MSCOCO and (b) Flickr30k, with an arrow from each degraded configuration (

\lambda_{t}{=}0.5

\lambda_{v}{=}0.5

; crosses) to its counterpart with raised text weight decay (

\lambda_{t}{=}1.0

\lambda_{v}{=}0.5

; circles). Each configuration pairs the labelled vision encoder with the Giant text encoder.

Figure A10 : Complementary to Figure 6 , scaling the vision encoder improves alignment when paired with larger text encoders but worsens it with the smallest ones. (a) Alignment, (b) image uniformity, and (c) text uniformity (lower is better for all three; y-axes) of MSCOCO embeddings from the CC12M pre-trained models, as a function of vision encoder size (x-axis) and colored by text encoder.

		MSCOCO			Flickr30k
Predictors	# params	$R^{2}$	Adj. $R^{2}$	LOOCV	$R^{2}$	Adj. $R^{2}$	LOOCV
Image uniformity	2	0.490	0.476	0.445	0.524	0.511	0.482
Alignment	2	0.150	0.127	0.022	0.098	0.072	-0.047
Text uniformity	2	0.347	0.328	0.275	0.435	0.420	0.381
Image uniformity + alignment	3	0.847	0.838	0.810	0.901	0.895	0.877
All three	4	0.848	0.835	0.794	0.901	0.893	0.867

Appendix

Table A8 : Image uniformity and alignment together predict zero-shot performance across models. In-sample

R^{2}

, adjusted

R^{2}

, and leave-one-out cross-validated

R^{2}

(LOOCV; higher is better) for linear models regressing average zero-shot performance on standardized geometry predictors (alignment, image/text uniformity), fit across the CC12M models. Geometry is measured on two probe sets (

n{=}38

models each; MSCOCO, Flickr30k). The best value in each column is in bold.

Figure A11 : The best models have similar image uniformity and alignment, while degraded models have worse alignment without a compensating gain in image uniformity. Predicted average zero-shot performance (color; brighter is higher) across the image uniformity (x-axis; $n{=}200$ grid points) and alignment (y-axis; $n{=}200$ grid points) plane, from a linear model fit on the MSCOCO geometry metrics using CC12M models to extract the embeddings. Markers show models by vision encoder, with the best text pairing per vision encoder outlined in red. The inset magnifies the boxed region.

Figure A12 : A geometry-only linear model generalizes to unseen vision encoders, capturing the peak-and-degradation pattern. For each vision encoder, the model regressing average zero-shot performance on image uniformity and alignment is fit on the four other vision encoders and used to predict the held-out one (leave-one-vision-out). Each panel shows the observed (solid line, filled circles, coloured by vision encoder) and predicted (dashed line, open grey squares) average zero-shot performance (y-axis; higher is better) versus text encoder size (x-axis). Each column is a vision encoder while rows give the held-out set used to measure geometry: (a) Flickr30k and (b) MSCOCO.

Figure A13 : Increasing text weight decay improves alignment and image uniformity. Alignment (x-axis) versus uniformity (y-axis; lower is better for both) for each vision encoder paired with the Giant text encoder, going from the degraded configuration (

\lambda_{t}{=}0.5

\lambda_{v}{=}0.5

; crosses) to the configuration with modality-specific weight decay (

\lambda_{t}{=}1

\lambda_{v}{=}0.5

; circles). Rows: (a,b) MSCOCO and (c,d) Flickr30k held-out sets. Columns: (a,c) image uniformity and (b,d) text uniformity (y-axis).

Figure A14 : Scaling the text encoder raises the effective rank (RankMe) of both image and text embeddings, whereas the vision encoder size has little effect. (a) Image and (b) text RankMe for the 30 architectures on CC12M training data, crossing five vision encoders (y-axis) with six text encoders (x-axis).

Figure A15 : As the text encoder grows, the modality gap follows a U-shaped curve that is more pronounced for larger vision encoders. Modality gap (y-axis) of MSCOCO embeddings as a function of text encoder size (x-axis) for CC12M models, colored by vision encoder.

Vision-Language Models

Vision-language models connect visual inputs with text for retrieval, question answering, generation, and grounded multimodal reasoning.

See all 736 papers

Rethinking Contrastive Loss in CLIP Post-training: A Complementary Framework with Frozen Text Encoder
Oct 8, 2026Zidan Wang, Yaqian Li, Xiaokai Zhang +3VLM Post-Training VLM Adaptation
Huazhong University of Science and Technology · Li Auto Inc.
CLIP Is Shortsighted: Paying Attention Beyond the First Sentence
Feb 25, 2026Marc-Antoine Lavoie, Anas Mahmoud, Aldo Zaimi +2Position Bias in VLMs Vision-Language Alignment
University of Toronto Robotics Institute · Mila - Quebec AI Institute
CLIMP: Contrastive Language-Image Mamba Pretraining
Jan 11, 2026Nimrod Shabtay, Itamar Zimerman, Eli Schwartz +1Vision-Language Pretraining VLM Robustness
Tel-Aviv University · IBM Research

Oct 8, 2026J/K move · Enter open · S save

Rethinking Contrastive Loss in CLIP Post-training: A Complementary Framework with Frozen Text Encoder

Zidan Wang, Yaqian Li, Xiaokai Zhang, Kaiwen Long+2

Huazhong University of Science and Technology · Li Auto Inc.

VLM Post-Training VLM Adaptation Vision-Language Models Multimodal Contrastive Learning

Open paper AskPDF arXiv

CLIP serves as a foundational vision-language model and the de facto vision encoder for downstream VLMs such as LLaVA. Post-training offers a lightweight route to refine CLIP, but recent work argues that the standard contrastive loss is unsuitable for post-training due to catastrophic forgetting under small batches, motivating designs that abandon the contrastive objective in favor of distillation. We revisit this premise and find that, for the InfoNCE objective, the reported forgetting is driven primarily not by insufficient negatives but by an inappropriate magnitude of the contrastive temperature

τ

: with

τ

set sufficiently small, contrastive post-training improves rather than degrades the pretrained CLIP, which we explain through the temperature dependence of the InfoNCE gradient. Building on this finding, we propose ComCLIP, a lightweight single-epoch post-training recipe that freezes CLIP's text encoder---so the refined vision encoder is a drop-in replacement with unchanged architecture and inference cost---and trains the vision encoder with a properly-tempered contrastive loss, an MSE anchoring loss against the original CLIP, and a relational distillation loss from DINOv2. Over multiple seeds, ComCLIP matches the self-distillation baseline CLIP-Refine on zero-shot classification while significantly improving the transferability of visual features, measured by linear probing (

48.99

vs.\

42.28

on ViT-B/16), and on ViT-L/14 it also improves MMVP over CLIP-Refine (

24.20

vs.\

19.01

); CLIP-Refine remains stronger on image-text retrieval. Used as a drop-in vision encoder for LLaVA-1.5-7B without re-aligning the projector or LLM, ComCLIP yields no net change across

8

VLM benchmarks, i.e., the refinement does not break downstream compatibility. Code and models are available at https://github.com/showstarpro/ComCLIP.git.

Abstract

Vision-Language Models

Explore similar work

Rethinking Contrastive Loss in CLIP Post-training: A Complementary Framework with Frozen Text Encoder

CLIP Is Shortsighted: Paying Attention Beyond the First Sentence

CLIMP: Contrastive Language-Image Mamba Pretraining