TIPSv2: Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignment

Bingyi Cao; Koert Chen; Kevis-Kokitsi Maninis; Kaifeng Chen; Arjun Karpur; Ye Xia; Sahil Dua; Tanmaya Dabral

cs.CVApr 13, 2026

TIPSv2: Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignment

Authors: Bingyi Cao, Koert Chen, Kevis-Kokitsi Maninis, Kaifeng Chen, Arjun Karpur, Ye Xia, Sahil Dua, Tanmaya Dabral,

Organizations: Google DeepMind · xAI · Epsilon Health · Seoul National University

Abstract

Recent progress in vision-language pretraining has enabled significant improvements to many downstream computer vision applications, such as classification, retrieval, segmentation and depth prediction. However, a fundamental capability that these models still struggle with is aligning dense patch representations with text embeddings of corresponding concepts. In this work, we investigate this critical issue and propose novel techniques to enhance this capability in foundational vision-language models. First, we reveal that a patch-level distillation procedure significantly boosts dense patch-text alignment -- surprisingly, the patch-text alignment of the distilled student model strongly surpasses that of the teacher model. This observation inspires us to consider modifications to pretraining recipes, leading us to propose iBOT++, an upgrade to the commonly-used iBOT masked image objective, where unmasked tokens also contribute directly to the loss. This dramatically enhances patch-text alignment of pretrained models. Additionally, to improve vision-language pretraining efficiency and effectiveness, we modify the exponential moving average setup in the learning recipe, and introduce a caption sampling strategy to benefit from synthetic captions at different granularities. Combining these components, we develop TIPSv2, a new family of image-text encoder models suitable for a wide range of downstream applications. Through comprehensive experiments on 9 tasks and 20 datasets, we demonstrate strong performance, generally on par with or better than recent vision encoder models. Code and models are released via our project page at https://gdm-tipsv2.github.io/ .

Figures & tables

Figure 1: TIPSv2’s improvement to the masked image modeling pretraining strategy. As part of our complete TIPSv2 method, we introduce iBOT++ (bottom), a simple modification to the well-known iBOT [ 68 ] self-supervised objective (top-left), where visible tokens also contribute directly to the loss. This enhancement dramatically improves patch-text alignment, as demonstrated by zero-shot image segmentation results (top-right).

Model	Zero-shot Segmentation (mIoU) $\uparrow$
Model	PC59	PC60	VOC21	ADE150
TIPS ViT-L	33.5	30.4	30.5	20.8
TIPS ViT-g	11.4	10.8	19.7	2.6

Table 1: Zero-shot segmentation with a large teacher model and its distilled student. Surprisingly, the TIPS ViT-L model, which is a student distilled from the TIPS ViT-g teacher, significantly surpasses it.

Figure 3: TIPSv2 pretraining overview . TIPSv2 introduces $3$ improvements (highlighted in green borders) to the combined contrastive and self-supervised approach to pretrain vision encoders. iBOT++ is an enhanced masked image modeling loss. Head-only EMA enables memory-efficient self-supervised losses. Multi-granularity captions provide a range of possible textual descriptions for images, increasing the robustness of the model.

Model	Zero-shot Segmentation (mIoU) $\uparrow$
Model	PC59	PC60	VOC21	ADE150
TIPS [ 33 ] (with iBOT)	14.2	13.4	29.1	3.5
TIPS with iBOT++	28.6	26.2	37.2	17.6

Table 3: Zero-shot segmentation in pretraining. Comparing TIPS ViT-g with iBOT or iBOT++ methods, showing significant improvements with our novel iBOT++.

Figure 4: Image captions at different granularities. Besides the alt-text and PaliGemma captions used in previous work, we propose additionally leveraging the more comprehensive Gemini captions, to make our vision encoder more robust.

Model	Seg. $\uparrow$	Depth $\downarrow$	Normals $\downarrow$	ImageNet $\uparrow$	I $\rightarrow$ T Ret. $\uparrow$	T $\rightarrow$ I Ret. $\uparrow$	Zero-shot Seg. $\uparrow$
Model	PASCAL	NYUv2	NYUv2	KNN	Flickr	Flickr	ADE150
TIPS ViT-g (reproduced)	82.8	0.375	23.1	83.2	92.0	81.0	3.5
+ iBOT++	82.5	0.369	22.7	84.4	93.9	81.7	17.6
+ Multi-granularity Captions	83.7	0.354	22.7	84.3	95.0	85.4	18.1
+ Head-only EMA	83.8	0.353	22.8	84.1	95.4	85.4	19.1

Table 4: Ablation studies for TIPSv2’s pretraining technique . Ablations are cumulative, running on a fixed schedule of

100

k steps at resolution

224

, for fair comparisons. We highlight the best and second-best number of each column.

Method	Zero-shot Segmentation $\uparrow$
Method	PC59	PC60	VOC21	ADE150
SigLIP2 SO/14	-	19.6	26.8	15.6
SILC B/16	31.6	-	-	19.3
DINOv2 (dino.txt) L/14	30.9	-	-	20.6
TIPS L/14	33.5	30.4	30.5	20.8
TIPSv2 L/14 (ours)	37.1	33.9	44.4	24.7

Table 5: Dense image-text evaluations , where TIPSv2 outperforms others in all cases, even though SILC and DINOv2 use the more expensive TCL protocol [ 6 ] . We highlight the best and second-best number of each column.

Model	I $\rightarrow$ T Retrieval $\uparrow$			T $\rightarrow$ I Retrieval $\uparrow$			0-Shot $\uparrow$
Model	COCO	Flickr	DOCCI	COCO	Flickr	DOCCI	ImageNet
CLIP L/14	56.3	85.2	44.4	36.5	65.2	40.4	75.5
OpenCLIP G/14	67.3	92.9	-	51.4	79.5	-	80.1
SigLIP2 g/16	72.8	95.4	-	56.1	86.0	-	85.0
SILC G/16	73.2	-	-	54.7	-	-	83.7
DINOv2 (dino.txt) L/14	-	-	-	45.4	77.1	-	81.4

Table 6: Global image-text evaluations , where TIPSv2 achieves best or second-best in

5

out of

7

cases. We highlight the best and second-best number of each column.

Method	Segmentation $\uparrow$		Depth $\downarrow$		Normals $\downarrow$		Fine-grained $\uparrow$	ImageNet Class. $\uparrow$
Method	PASCAL	ADE20k	NYUv2	NAVI	NYUv2	NAVI	retrieval (UnED)	KNN	lin
CLIP L/14	74.5	39.0	0.553	0.073	24.3	25.5	57.4	79.8	84.3
SigLIP2 SO/14	78.1	45.4	0.466	0.064	23.0	25.0	-	-	-
DINOv2 g/14	83.1	49.5	0.372	0.054	20.7	24.0	62.7	83.6	87.3
PE-core G/14	-	41.5	-	-	-	-	-	86.8	89.5
PE-spatial G/14	-	49.3	-	-	-	-	-	-	-

Table 7: Image-only evaluations , where TIPSv2 achieves the best or second-best performance in

7

out of

9

evaluations. We highlight the best and second-best number of each column.

Model	Segmentation $\uparrow$	Depth $\downarrow$	0-Shot $\uparrow$	I $\rightarrow$ T Ret. $\uparrow$	T $\rightarrow$ I Ret. $\uparrow$	Zero-shot Seg. $\uparrow$
Model	ADE20k	NYUv2	ImageNet	COCO	COCO	ADE150
DINOv3 L/16	54.9	0.352	82.3	63.7	45.6	24.7
TIPSv2 L/14 (ours)	51.4	0.339	79.7	73.5	57.4	25.1

Table 8: Comparison between DINOv3 and TIPSv2 , on largest comparable model size ViT-L. TIPSv2 achieves superior performance on 4 out of 6 metrics. We highlight the best performing model for each task.

Panel of Figure 5 — Figure 5: PCA maps. Comparing the first $3$ PCA components, using ViT-g models. Images are forwarded at 1372 resolution for patch size 14 models (TIPS, TIPSv2) and at 1568 resolution for patch size 16 models (SigLIP2). TIPSv2 produces smoother feature maps compared to other vision-language pretraining methods, with well-delineated objects.

Panel of Figure 6 — Figure 6: Zero-shot segmentation visualization. Comparing results for TIPSv2, TIPS and SigLIP 2, where classes are predicted directly by finding the closest text embedding to each image patch token, without any post-processing. TIPSv2 enhances patch-text alignment significantly compared to other models, showcasing strong off-the-shelf capabilities.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Seg.	Depth $\downarrow$	ImageNet	IT Ret.	TI Ret.	0-shot Seg.
ViT-L	ADE20k	NYUv2	KNN	COCO	COCO	PC60
CLIP	35.7	0.571	73.5	52.2	33.2	4.3
CLIP+iBOT	41.3	0.458	77.2	55.7	37.5	8.0
CLIP+iBOT++	42.8	0.434	78.7	58.1	40.6	22.9

Appendix

Table 9: Applying iBOT++ to CLIP , on a ViT-L backbone. iBOT++ significantly enhances CLIP performance across several tasks, beyond what can be obtained with iBOT.

Model	Seg.	Depth $\downarrow$	ImageNet	IT Ret.	TI Ret.	0-shot Seg.
ViT-L	ADE20k	NYUv2	KNN	COCO	COCO	PC60
CLIP	35.7	0.571	73.5	52.2	33.2	4.3
CLIP+iBOT	39.8	0.467	76.6	55.4	37.0	4.3
CLIP+iBOT++	42.5	0.437	78.2	56.6	39.3	22.1

Appendix

Table 10: Applying iBOT++ to CLIP with head-only EMA , on a ViT-L backbone. iBOT++ with head-only EMA significantly enhances CLIP performance.

Model	Seg.	Depth $\downarrow$	ImageNet	IT Ret.	TI Ret.	0-shot Seg.
ViT-g	ADE20k	NYUv2	KNN	COCO	COCO	PC60
CLIP 2 CLS	40.1	0.475	79.1	72.9	56.1	18.2
CLIP 2 CLS + iBOT	41.1	0.442	81.6	73.2	58.3	16.4
CLIP 2 CLS + iBOT++	47.2	0.366	82.7	73.6	58.9	28.2

Appendix

Table 11: Applying iBOT++ to CLIP with a dual CLS setup , on a ViT-g backbone. iBOT++ significantly enhances performance across several tasks, beyond what can be obtained with iBOT.

Masking	Segmentation $\uparrow$	Depth $\downarrow$	I $\rightarrow$ T Ret. $\uparrow$	Zero-shot Seg. $\uparrow$
Ratio	PASCAL	NYUv2	Flickr	ADE150
0.0	78.8	0.513	90.0	1.0
0.5	82.5	0.438	92.6	2.4
0.75	82.6	0.418	93.8	13.6

Appendix

Table 12: Masking ablation for iBOT++ pretraining. We conduct an ablation study on TIPS [ 33 ] ViT-L models to determine the optimal masking ratio for iBOT++ pretraining. The results show that

75\%

masking ratio is critical for achieving strong performance across all evaluations, particularly enhancing patch-text alignment. The model achieving the best overall performance is highlighted.

Text Strategy	Seg.	Depth $\downarrow$	ImageNet	IT Ret.	0-shot Seg.
Text Strategy	ADE20k	NYUv2	KNN	COCO	ADE150
1 CLS (web, PaliGemma)	46.3	0.375	80.4	72.3	16.4
1 CLS (web, PaliGemma, Gemini)	46.7	0.366	81.2	74.4	17.7
2 CLS (web / PaliGemma)	48.3	0.370	84.3	70.0	17.1
2 CLS (web / PaliGemma, Gemini)	49.1	0.354	84.3	76.2	18.1

Appendix

Table 13: Ablations for multi-granularity captions , on ViT-g. Texts for 2 CLS are separated by / and for each CLS, texts are uniformly sampled from sources.

Panel of Figure 7 — Figure 7: PCA maps at ViT-L size. Comparing the first $3$ PCA components from the ViT-L models of DINOv2 (with registers), DINOv3, and TIPSv2. Images are forwarded at $1372$ resolution for patch size $14$ models (DINOv2 and TIPSv2) and at $1568$ resolution for patch size $16$ models (DINOv3). DINOv3 features appear smoother, but TIPSv2 features show more semantically focused features, e.g., TIPSv2 maps show all windows clustered together in row 1, and the eyes and leash are distinct on the dog in row 4.

Panel of Figure 8 — Figure 8: PCA maps at ViT-g or ViT-7B size. Comparing the first $3$ PCA components from teacher models of DINOv2 (with registers, ViT-g), DINOv3 (ViT-7B), and TIPSv2 (ViT-g). Images are forwarded at $1372$ resolution for patch size $14$ models (DINOv2 and TIPSv2) and at $1568$ resolution for patch size $16$ models (DINOv3). As for the ViT-L PCA maps, DINOv3 features appear smoother, but TIPSv2 features capture more semantically focused details, e.g., notice the ceiling details in row 2 or the different colors contrasting people and backpacks in row 3.

Panel of Figure 9 — Figure 9: Zero-shot segmentation visualization. Comparing results with the iBOT loss (used in TIPS) vs the iBOT++ loss (used in TIPSv2), where classes are predicted directly by finding the closest text embedding to each image patch token, without any post-processing. Compared to baseline iBOT, iBOT++ achieves significantly improved patch-text alignment, as part of the TIPSv2 recipe.

Model	Zero-shot Segmentation (mIoU) $\uparrow$
Model	PC60	VOC21	ADE150
SigLIP2 B/16	22.6	25.8	16.4
SigLIP2 SO/14	19.6	26.8	15.6
SigLIP2 g/16	17.2	25.7	13.9

Appendix

Table 14: Zero-shot segmentation for SigLIP2 models. Similarly to TIPS, the smaller models tend to outperform larger ones in the same model family.

Model	Segmentation $\uparrow$	Depth $\downarrow$	Normals $\downarrow$	KNN $\uparrow$	I $\rightarrow$ T Ret. $\uparrow$	T $\rightarrow$ I Ret. $\uparrow$	Zero-shot Seg. $\uparrow$
Model	PASCAL	NYUv2	NYUv2	ImageNet	Flickr	Flickr	ADE150
TIPSv2 B/14	84.0	0.374	23.2	79.8	92.6	80.0	17.4
TIPSv2 L/14	85.1	0.339	21.9	82.5	95.4	83.3	24.7
TIPSv2 SO/14	85.2	0.339	21.7	82.8	94.8	84.0	23.3
TIPSv2 g/14	85.1	0.334	21.7	83.7	95.1	85.9	17.8

Appendix

Table 15: Evaluations for all TIPSv2 model variants. We report four distinct TIPSv2 model sizes (ViT-B, ViT-L, ViT-g, and SO-400m). Only the ViT-g model is pretrained; the other sizes (ViT-B, ViT-L, and SO-400m) are distilled from the ViT-g teacher. Best performance for each task is highlighted.

Model	Image # Params	Text # Params	Total # Params
TIPSv2-B/14	86.3M	109.6M	195.9M
TIPSv2-L/14	304.0M	183.9M	487.9M
TIPSv2-SO/14	413.3M	448.3M	861.7M
TIPSv2-g/14	1.1B	389.1M	1.5B

Appendix

Table 16: Number of parameters for all TIPSv2 model variants. We release

4

different model variants. For B, L and g model sizes, we use a fixed number of 12 layers in the text encoder; for the SO size, we use the same number of layers in both the image and text encoders.

Figure 10: Comparison of TIPSv2 against recent vision encoders. The chart illustrates the percentage of shared evaluation benchmarks where TIPSv2 secures the best result (green) compared head-to-head to other individual leading models (orange). TIPSv2 demonstrates a winning record on the majority of shared tasks. The integer displayed on the chart represents the count of metrics on which each respective model achieves the best result (i.e., the number of wins). For each comparison between individual vision encoders, we use the largest comparable model size (ViT-g against ViT-g, or ViT-g against ViT-G, or ViT-L against ViT-L).

Vision-Language Pretraining

Vision-language pretraining learns visual representations by relating images, video, or other visual inputs to language.

See all 79 papers

TEVI: Text-Conditioned Editing of Visual Representations via Sparse Autoencoders for Improved Vision-Language Alignment
Jun 5, 2026Sweta Mahajan, Sukrut Rao, Jiahao Xie +2Vision-Language Retrieval Vision-Language Alignment
Max Planck Institute for Informatics, Saarland Informatics Campus, Saarbrücken, Germany · Department of Language Science and Technology, Saarland University, Saarbrücken, Germany
Text as Partial Constraint: Core-Residual Alignment for Robust Vision-Language Learning
Jul 3, 2026Chengzhen Yu, Canran Xiao, Siyuan Ma +1VLM Robustness Contrastive Learning
1Nanyang Technological University (NTU) · 2Shenzhen Campus of Sun Yat-sen University
TextTeacher: What Can Language Teach About Images?
May 21, 2026Tobias Christian Nauen, Stanislav Frolov, Brian Bernhard Moser +3Cross-Modal Knowledge Distillation Visual Representation Learning
RPTU University Kaiserslautern-Landau · German Research Center for Artificial Intelligence (DFKI)

Jun 5, 2026J/K move · Enter open · S save

TEVI: Text-Conditioned Editing of Visual Representations via Sparse Autoencoders for Improved Vision-Language Alignment

Sweta Mahajan, Sukrut Rao, Jiahao Xie, Alexander Koller+1

Max Planck Institute for Informatics, Saarland Informatics Campus, Saarbrücken, Germany · Department of Language Science and Technology, Saarland University, Saarbrücken, Germany

Vision-Language Retrieval Vision-Language Alignment

Open paper AskPDF arXiv

Vision-language models such as CLIP are highly useful for diverse tasks due to their shared image-text embedding space. Despite this, the image and text embeddings are often poorly aligned, affecting downstream performance. Recent work has hypothesized that this can be attributed to an information imbalance: images contain more information than their captions describe. In this work, we propose TEVI, a framework that uses captions as a signal for what to retain from image embeddings. Specifically, we use sparse autoencoders to disentangle image embeddings and train a masking module to selectively reconstruct the embedding based on a given caption. In a controlled setup with synthetic captions, we show that TEVI is effective at preserving caption-described attributes while discarding others. We find that this extends to CLIP models trained on natural images, where TEVI learns to mask meaningfully and allows retrieval based on conditioning. Finally, we use TEVI to achieve improved retrieval performance across coarse-grained and fine-grained benchmarks. Code available at https://github.com/neuroexplicit-saar/TEVI.

Abstract

Vision-Language Pretraining

Explore similar work

TEVI: Text-Conditioned Editing of Visual Representations via Sparse Autoencoders for Improved Vision-Language Alignment

Text as Partial Constraint: Core-Residual Alignment for Robust Vision-Language Learning

TextTeacher: What Can Language Teach About Images?