Recent progress in vision-language pretraining has enabled significant improvements to many downstream computer vision applications, such as classification, retrieval, segmentation and depth prediction. However, a fundamental capability that these models still struggle with is aligning dense patch representations with text embeddings of corresponding concepts. In this work, we investigate this critical issue and propose novel techniques to enhance this capability in foundational vision-language models. First, we reveal that a patch-level distillation procedure significantly boosts dense patch-text alignment -- surprisingly, the patch-text alignment of the distilled student model strongly surpasses that of the teacher model. This observation inspires us to consider modifications to pretraining recipes, leading us to propose iBOT++, an upgrade to the commonly-used iBOT masked image objective, where unmasked tokens also contribute directly to the loss. This dramatically enhances patch-text alignment of pretrained models. Additionally, to improve vision-language pretraining efficiency and effectiveness, we modify the exponential moving average setup in the learning recipe, and introduce a caption sampling strategy to benefit from synthetic captions at different granularities. Combining these components, we develop TIPSv2, a new family of image-text encoder models suitable for a wide range of downstream applications. Through comprehensive experiments on 9 tasks and 20 datasets, we demonstrate strong performance, generally on par with or better than recent vision encoder models. Code and models are released via our project page at https://gdm-tipsv2.github.io/ .
Figures & tables
Figure 1: TIPSv2’s improvement to the masked image modeling pretraining strategy. As part of our complete TIPSv2 method, we introduce iBOT++ (bottom), a simple modification to the well-known iBOT [ 68 ] self-supervised objective (top-left), where visible tokens also contribute directly to the loss. This enhancement dramatically improves patch-text alignment, as demonstrated by zero-shot image segmentation results (top-right).
Model
Zero-shot Segmentation (mIoU) ↑
PC59
PC60
VOC21
ADE150
TIPS ViT-L
33.5
30.4
30.5
20.8
TIPS ViT-g
11.4
10.8
19.7
2.6
Table 1: Zero-shot segmentation with a large teacher model and its distilled student. Surprisingly, the TIPS ViT-L model, which is a student distilled from the TIPS ViT-g teacher, significantly surpasses it.
Figure 3
Figure 3: TIPSv2 pretraining overview . TIPSv2 introduces 3 improvements (highlighted in green borders) to the combined contrastive and self-supervised approach to pretrain vision encoders. iBOT++ is an enhanced masked image modeling loss. Head-only EMA enables memory-efficient self-supervised losses. Multi-granularity captions provide a range of possible textual descriptions for images, increasing the robustness of the model.
Model
Zero-shot Segmentation (mIoU) ↑
PC59
PC60
VOC21
ADE150
TIPS [ 33 ] (with iBOT)
14.2
13.4
29.1
3.5
TIPS with iBOT++
28.6
26.2
37.2
17.6
Table 3: Zero-shot segmentation in pretraining. Comparing TIPS ViT-g with iBOT or iBOT++ methods, showing significant improvements with our novel iBOT++.
Figure 4: Image captions at different granularities. Besides the alt-text and PaliGemma captions used in previous work, we propose additionally leveraging the more comprehensive Gemini captions, to make our vision encoder more robust.
Model
Seg. ↑
Depth ↓
Normals ↓
ImageNet ↑
I → T Ret. ↑
T → I Ret. ↑
Zero-shot Seg. ↑
PASCAL
NYUv2
NYUv2
KNN
Flickr
Flickr
ADE150
TIPS ViT-g (reproduced)
82.8
0.375
23.1
83.2
92.0
81.0
3.5
+ iBOT++
82.5
0.369
22.7
84.4
93.9
81.7
17.6
+ Multi-granularity Captions
83.7
0.354
22.7
84.3
95.0
85.4
18.1
+ Head-only EMA
83.8
0.353
22.8
84.1
95.4
85.4
19.1
Table 4: Ablation studies for TIPSv2’s pretraining technique . Ablations are cumulative, running on a fixed schedule of 100 k steps at resolution 224 , for fair comparisons. We highlight the best and second-best number of each column.
Method
Zero-shot Segmentation ↑
PC59
PC60
VOC21
ADE150
SigLIP2 SO/14
-
19.6
26.8
15.6
SILC B/16
31.6
-
-
19.3
DINOv2 (dino.txt) L/14
30.9
-
-
20.6
TIPS L/14
33.5
30.4
30.5
20.8
TIPSv2 L/14 (ours)
37.1
33.9
44.4
24.7
Table 5: Dense image-text evaluations , where TIPSv2 outperforms others in all cases, even though SILC and DINOv2 use the more expensive TCL protocol [ 6 ] . We highlight the best and second-best number of each column.
Model
I → T Retrieval ↑
T → I Retrieval ↑
0-Shot ↑
COCO
Flickr
DOCCI
COCO
Flickr
DOCCI
ImageNet
CLIP L/14
56.3
85.2
44.4
36.5
65.2
40.4
75.5
OpenCLIP G/14
67.3
92.9
-
51.4
79.5
-
80.1
SigLIP2 g/16
72.8
95.4
-
56.1
86.0
-
85.0
SILC G/16
73.2
-
-
54.7
-
-
83.7
DINOv2 (dino.txt) L/14
-
-
-
45.4
77.1
-
81.4
Table 6: Global image-text evaluations , where TIPSv2 achieves best or second-best in 5 out of 7 cases. We highlight the best and second-best number of each column.
Method
Segmentation ↑
Depth ↓
Normals ↓
Fine-grained ↑
ImageNet Class. ↑
PASCAL
ADE20k
NYUv2
NAVI
NYUv2
NAVI
retrieval (UnED)
KNN
lin
CLIP L/14
74.5
39.0
0.553
0.073
24.3
25.5
57.4
79.8
84.3
SigLIP2 SO/14
78.1
45.4
0.466
0.064
23.0
25.0
-
-
-
DINOv2 g/14
83.1
49.5
0.372
0.054
20.7
24.0
62.7
83.6
87.3
PE-core G/14
-
41.5
-
-
-
-
-
86.8
89.5
PE-spatial G/14
-
49.3
-
-
-
-
-
-
-
Table 7: Image-only evaluations , where TIPSv2 achieves the best or second-best performance in 7 out of 9 evaluations. We highlight the best and second-best number of each column.
Model
Segmentation ↑
Depth ↓
0-Shot ↑
I → T Ret. ↑
T → I Ret. ↑
Zero-shot Seg. ↑
ADE20k
NYUv2
ImageNet
COCO
COCO
ADE150
DINOv3 L/16
54.9
0.352
82.3
63.7
45.6
24.7
TIPSv2 L/14 (ours)
51.4
0.339
79.7
73.5
57.4
25.1
Table 8: Comparison between DINOv3 and TIPSv2 , on largest comparable model size ViT-L. TIPSv2 achieves superior performance on 4 out of 6 metrics. We highlight the best performing model for each task.
Figure 5: PCA maps. Comparing the first 3 PCA components, using ViT-g models. Images are forwarded at 1372 resolution for patch size 14 models (TIPS, TIPSv2) and at 1568 resolution for patch size 16 models (SigLIP2). TIPSv2 produces smoother feature maps compared to other vision-language pretraining methods, with well-delineated objects.
Figure 6: Zero-shot segmentation visualization. Comparing results for TIPSv2, TIPS and SigLIP 2, where classes are predicted directly by finding the closest text embedding to each image patch token, without any post-processing. TIPSv2 enhances patch-text alignment significantly compared to other models, showcasing strong off-the-shelf capabilities.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Seg.
Depth ↓
ImageNet
IT Ret.
TI Ret.
0-shot Seg.
ViT-L
ADE20k
NYUv2
KNN
COCO
COCO
PC60
CLIP
35.7
0.571
73.5
52.2
33.2
4.3
CLIP+iBOT
41.3
0.458
77.2
55.7
37.5
8.0
CLIP+iBOT++
42.8
0.434
78.7
58.1
40.6
22.9
Appendix
Table 9: Applying iBOT++ to CLIP , on a ViT-L backbone. iBOT++ significantly enhances CLIP performance across several tasks, beyond what can be obtained with iBOT.
Model
Seg.
Depth ↓
ImageNet
IT Ret.
TI Ret.
0-shot Seg.
ViT-L
ADE20k
NYUv2
KNN
COCO
COCO
PC60
CLIP
35.7
0.571
73.5
52.2
33.2
4.3
CLIP+iBOT
39.8
0.467
76.6
55.4
37.0
4.3
CLIP+iBOT++
42.5
0.437
78.2
56.6
39.3
22.1
Appendix
Table 10: Applying iBOT++ to CLIP with head-only EMA , on a ViT-L backbone. iBOT++ with head-only EMA significantly enhances CLIP performance.
Model
Seg.
Depth ↓
ImageNet
IT Ret.
TI Ret.
0-shot Seg.
ViT-g
ADE20k
NYUv2
KNN
COCO
COCO
PC60
CLIP 2 CLS
40.1
0.475
79.1
72.9
56.1
18.2
CLIP 2 CLS + iBOT
41.1
0.442
81.6
73.2
58.3
16.4
CLIP 2 CLS + iBOT++
47.2
0.366
82.7
73.6
58.9
28.2
Appendix
Table 11: Applying iBOT++ to CLIP with a dual CLS setup , on a ViT-g backbone. iBOT++ significantly enhances performance across several tasks, beyond what can be obtained with iBOT.
Masking
Segmentation ↑
Depth ↓
I → T Ret. ↑
Zero-shot Seg. ↑
Ratio
PASCAL
NYUv2
Flickr
ADE150
0.0
78.8
0.513
90.0
1.0
0.5
82.5
0.438
92.6
2.4
0.75
82.6
0.418
93.8
13.6
Appendix
Table 12: Masking ablation for iBOT++ pretraining. We conduct an ablation study on TIPS [ 33 ] ViT-L models to determine the optimal masking ratio for iBOT++ pretraining. The results show that 75% masking ratio is critical for achieving strong performance across all evaluations, particularly enhancing patch-text alignment. The model achieving the best overall performance is highlighted.
Text Strategy
Seg.
Depth ↓
ImageNet
IT Ret.
0-shot Seg.
ADE20k
NYUv2
KNN
COCO
ADE150
1 CLS (web, PaliGemma)
46.3
0.375
80.4
72.3
16.4
1 CLS (web, PaliGemma, Gemini)
46.7
0.366
81.2
74.4
17.7
2 CLS (web / PaliGemma)
48.3
0.370
84.3
70.0
17.1
2 CLS (web / PaliGemma, Gemini)
49.1
0.354
84.3
76.2
18.1
Appendix
Table 13: Ablations for multi-granularity captions , on ViT-g. Texts for 2 CLS are separated by / and for each CLS, texts are uniformly sampled from sources.
Figure 7: PCA maps at ViT-L size. Comparing the first 3 PCA components from the ViT-L models of DINOv2 (with registers), DINOv3, and TIPSv2. Images are forwarded at 1372 resolution for patch size 14 models (DINOv2 and TIPSv2) and at 1568 resolution for patch size 16 models (DINOv3). DINOv3 features appear smoother, but TIPSv2 features show more semantically focused features, e.g., TIPSv2 maps show all windows clustered together in row 1, and the eyes and leash are distinct on the dog in row 4.
Figure 8: PCA maps at ViT-g or ViT-7B size. Comparing the first 3 PCA components from teacher models of DINOv2 (with registers, ViT-g), DINOv3 (ViT-7B), and TIPSv2 (ViT-g). Images are forwarded at 1372 resolution for patch size 14 models (DINOv2 and TIPSv2) and at 1568 resolution for patch size 16 models (DINOv3). As for the ViT-L PCA maps, DINOv3 features appear smoother, but TIPSv2 features capture more semantically focused details, e.g., notice the ceiling details in row 2 or the different colors contrasting people and backpacks in row 3.
Figure 9: Zero-shot segmentation visualization. Comparing results with the iBOT loss (used in TIPS) vs the iBOT++ loss (used in TIPSv2), where classes are predicted directly by finding the closest text embedding to each image patch token, without any post-processing. Compared to baseline iBOT, iBOT++ achieves significantly improved patch-text alignment, as part of the TIPSv2 recipe.
Model
Zero-shot Segmentation (mIoU) ↑
PC60
VOC21
ADE150
SigLIP2 B/16
22.6
25.8
16.4
SigLIP2 SO/14
19.6
26.8
15.6
SigLIP2 g/16
17.2
25.7
13.9
Appendix
Table 14: Zero-shot segmentation for SigLIP2 models. Similarly to TIPS, the smaller models tend to outperform larger ones in the same model family.
Model
Segmentation ↑
Depth ↓
Normals ↓
KNN ↑
I → T Ret. ↑
T → I Ret. ↑
Zero-shot Seg. ↑
PASCAL
NYUv2
NYUv2
ImageNet
Flickr
Flickr
ADE150
TIPSv2 B/14
84.0
0.374
23.2
79.8
92.6
80.0
17.4
TIPSv2 L/14
85.1
0.339
21.9
82.5
95.4
83.3
24.7
TIPSv2 SO/14
85.2
0.339
21.7
82.8
94.8
84.0
23.3
TIPSv2 g/14
85.1
0.334
21.7
83.7
95.1
85.9
17.8
Appendix
Table 15: Evaluations for all TIPSv2 model variants. We report four distinct TIPSv2 model sizes (ViT-B, ViT-L, ViT-g, and SO-400m). Only the ViT-g model is pretrained; the other sizes (ViT-B, ViT-L, and SO-400m) are distilled from the ViT-g teacher. Best performance for each task is highlighted.
Model
Image # Params
Text # Params
Total # Params
TIPSv2-B/14
86.3M
109.6M
195.9M
TIPSv2-L/14
304.0M
183.9M
487.9M
TIPSv2-SO/14
413.3M
448.3M
861.7M
TIPSv2-g/14
1.1B
389.1M
1.5B
Appendix
Table 16: Number of parameters for all TIPSv2 model variants. We release 4 different model variants. For B, L and g model sizes, we use a fixed number of 12 layers in the text encoder; for the SO size, we use the same number of layers in both the image and text encoders.
Figure 10: Comparison of TIPSv2 against recent vision encoders. The chart illustrates the percentage of shared evaluation benchmarks where TIPSv2 secures the best result (green) compared head-to-head to other individual leading models (orange). TIPSv2 demonstrates a winning record on the majority of shared tasks. The integer displayed on the chart represents the count of metrics on which each respective model achieves the best result (i.e., the number of wins). For each comparison between individual vision encoders, we use the largest comparable model size (ViT-g against ViT-g, or ViT-g against ViT-G, or ViT-L against ViT-L).
Vision-language models such as CLIP are highly useful for diverse tasks due to their shared image-text embedding space. Despite this, the image and text embeddings are often poorly aligned, affecting downstream performance. Recent work has hypothesized that this can be attributed to an information imbalance: images contain more information than their captions describe. In this work, we propose TEVI, a framework that uses captions as a signal for what to retain from image embeddings. Specifically, we use sparse autoencoders to disentangle image embeddings and train a masking module to selectively reconstruct the embedding based on a given caption. In a controlled setup with synthetic captions, we show that TEVI is effective at preserving caption-described attributes while discarding others. We find that this extends to CLIP models trained on natural images, where TEVI learns to mask meaningfully and allows retrieval based on conditioning. Finally, we use TEVI to achieve improved retrieval performance across coarse-grained and fine-grained benchmarks. Code available at https://github.com/neuroexplicit-saar/TEVI.
Sweta Mahajan, Sukrut Rao, Jiahao Xie +2
Max Planck Institute for Informatics, Saarland Informatics Campus, Saarbrücken, Germany · Department of Language Science and Technology, Saarland University, Saarbrücken, Germany
Vision-language alignment powers open-vocabulary recognition, retrieval, and LVLM grounding, yet natural captions are often underspecified, making similarity brittle and overly confident under paraphrase and omitted details. We aim to learn representations whose matching is stable across caption views and whose confidence reflects how strongly text constrains an image. We propose Text as Partial Constraint (TPC), a core-residual alignment framework that treats multi-view captions as incomplete supervision. It distills a consensus semantic core as the alignment target, learns a single-view core predictor for standard inference with one query, and explicitly discourages vision-language similarity from depending on the orthogonal unsaid residual. An uncertainty-aware contrastive objective further softens alignment when caption views disagree, reducing overconfident updates under weak language constraints. Across zero-shot recognition and adversarial robustness, TPC achieves 81.42/64.05 Top-1 clean/robust accuracy on ImageNet and 76.19/52.03 on an Avg-14 transfer suite, while improving LVLM transfer with 85.16 POPE F1 and 59.57 OKVQA accuracy under an LLaVA-1.5-7B stack. These results suggest that modeling text as a partial constraint is a practical and principled route to more reliable vision-language representations under underspecified language supervision.
Chengzhen Yu, Canran Xiao, Siyuan Ma +1
1Nanyang Technological University (NTU) · 2Shenzhen Campus of Sun Yat-sen University
The platonic representation hypothesis suggests that sufficiently large models converge to a shared representation geometry, even across modalities. Motivated by this, we ask: Can the semantic knowledge of a language model efficiently improve a vision model? As an answer, we introduce TextTeacher, a simple auxiliary objective that injects text embeddings as additional information into image classification training. TextTeacher uses readily available image captions, a pre-trained and frozen text encoder, and a lightweight projection to produce semantic anchors that efficiently guide representations during training while leaving the inference-time model unchanged. On ImageNet with standard ViT backbones, TextTeacher improves accuracy by up to +2.7 percentage points (p.p.) and yields consistent transfer gains (on average +1.0 p.p.) under the same recipe and compute. It outperforms vision knowledge distillation, yielding more accuracy at a constant compute budget or similar accuracy, but 33% faster. Our analysis indicates that TextTeacher acts as a feature-space preconditioner, shaping deeper layers in the first stages of training, and aiding generalization by supplying complementary semantic cues. TextTeacher adds negligible overhead, requires no costly multimodal training of the target model and preserves the simplicity and latency of pure vision models. Project page with code and captions: https://nauen-it.de/publications/text-teacher
Tobias Christian Nauen, Stanislav Frolov, Brian Bernhard Moser +3
RPTU University Kaiserslautern-Landau · German Research Center for Artificial Intelligence (DFKI)