Vision foundation models such as DINOv2, SigLIP2, and MASt3R develop complementary capabilities from different pretraining objectives, yet their knowledge remains distributed across separate, specialized models. Multi-teacher knowledge distillation offers a path toward consolidating these capabilities into a single agglomerative backbone, but existing approaches assume a fixed set of teachers, and incorporating a new teacher requires repeating expensive joint distillation over the entire teacher set. We introduce GRAFT, a continual multi-teacher distillation framework that enables a unified backbone to progressively acquire capabilities from an open-ended sequence of foundation models. When a new teacher arrives, GRAFT treats the previously distilled model as a teacher for preserving learned capabilities, while the current student jointly learns from both the previous model and the incoming teacher. Furthermore, to reconcile the incompatible representation geometries of heterogeneous teachers, we introduce Teacher Specific Readout Tokens, which grant each teacher an independent read-out of the shared encoder, together with Geometry Agnostic Relational Loss that aligns a vision-language teacher by matching image-text similarity structures rather than raw feature values. We provide GRAFT model, which is a single, continually extensible backbone that unifies five domains, including image understanding, 2D dense prediction, 3D human pose estimation, 3D vision, and vision-language, delivering strong performance across all of them while acquiring each new capability at the cost of a single distillation rather than a full re-distillation.
Figures & tables
Figure 1: Compared to existing agglomerative models, GRAFT hold a comprehensive capability on various tasks with strong performance on each task.
Figure 2: Per-patch feature geometry across depth. Each cell visualizes the top-3 PCA components of patch tokens (mapped to RGB, fit per cell) at a shallow, deep, and final block for four ViT-L teachers and GRAFT (ViT-B) on three ADE20K images. The teachers diverge from the shallowest blocks: DINOv2 resolves object structure, MASt3R shows geometric texture, SigLIP2 carries only positional patterns, and Multi-HMR progressively organizes around people. GRAFT unifies these in a single trunk—its patch tokens retain sharp object/part structure from its dense teachers (DINOv2, MASt3R, Multi-HMR), while SigLIP2’s vision–language capability is grafted relationally onto a dedicated summary token (§ 3.3 ) and thus does not appear in this patch view.
Figure 3: Overview of GRAFT at round k . The active student Sk is warm-started from Sk−1 . A learnable Teacher Specific Readout Token (TSRT) is assigned to each task and jointly encoded with the shared patch tokens, producing task specific summary representations that are mapped by task-specific projectors. Training follows two concurrent paths. In the retention path, outputs for previously acquired tasks 1,…,k−1 are constrained by Lanchor to remain consistent with those of the frozen anchor model Sk−1 , which preserves previously acquired task functions. In the acquisition path, the new task readout is optimized through Ldistill to acquire knowledge from the incoming teacher Tk . Only the active student Sk is updated, while both Sk−1 and Tk remain frozen.
Image Understanding
2D Dense Prediction
3D Human Perception
3D Vision
Vision-Language
Model
Arch.
IN-1k cls ↑
Depth ↓
mIoU ↑
F1 ↑
PA-PVE ↓
AUC ↑
COCO (T → I/I → T) ↑
Flickr (T → I/I → T) ↑
Teachers (ceilings)
DINOv2
ViT-L
–
0.384
47.70
–
–
–
–
–
Multi-HMR
ViT-L
–
–
–
95.0
36.9
–
–
–
MASt3R
ViT-L
–
–
–
–
–
91.2
–
–
SigLIP2
ViT-L
83.8
–
–
–
–
–
55.7 / 72.1
83.2 / 95.1
Table 1: Main results across five task families. All models use a ViT-B backbone; teacher rows (ViT-L) are shown as per-task ceilings. Metric directions are marked in the header, best values are in bold and second-best underlined, and “-” indicates the model lacks that capability. ‡ Segmentation for RADIOv2.5 is distilled from SAM, and for SigLino and EUPE from DINOv3. § DUNE result obtained by fine-tuning on non-public data.
Acquisition
Retention
Acquisition
Retention
Variant
IN-1k cls ↑
mIoU ↑
Depth RMSE ↓
Variant
IN-1k cls ↑
mIoU ↑
Depth RMSE ↓
Distillation strategy
Vision–language projector
Direct distillation
82.6
35.4
0.584
Frozen encoder + Transformer
75.5
45.5
0.425
LoRA finetuning
76.2
45.7
0.425
Transformer, 1 layer
80.4
46.0
0.448
LPFT
78.0
40.5
0.442
Transformer, 3 layers
75.9
45.4
0.451
EWC
77.1
40.2
0.508
MLP + residual (ours)
81.2
46.0
0.450
Table 2: Ablation studies for vision language distillation. We study the distillation strategy, vision language projector architecture, and choice of vision language teacher. All experiments start from a student distilled from DINOv2 and further distill an incoming vision language teacher.
Figure 4: Qualitative comparison on depth estimation, semantic segmentation, multi-human reconstruction, and 3D reconstruction between GRAFT and corresponding teachers.
Table 3: Teacher sets and training data differ across methods , so the head-to-head in Table 1 is not like-for-like. Teacher scale is per model and counts are the number of distillation teachers. Entries are stated by each source where available and marked unavail when a source does not disclose them. SigLino reports image tokens rather than images, and EUPE additionally trains a from-scratch 1.9B proxy before distillation. GRAFT’s teachers are DUNE’s three, inherited through the warm start, plus the grafted vision–language teacher; its data subtotal counts DataComp-12M and ImageNet-19k and excludes Mapillary and GLDv2, whose sizes are undisclosed.
VCRE <45 px
VCRE <90 px
Median reproj.
Relative pose error
Method
Encoder
AUC ↑
Prec. ↑
AUC ↑
Prec. ↑
error (px) ↓
Trans. (m) ↓
Rot. ( ∘ ) ↓
Prec. ↑
AUC ↑
LoFTR ( Sun et al., 2021 )
CNN
39.7
18.2
61.8
33.5
166.8
2.31
39.4
26.9
9.8
DUSt3R ( Wang et al., 2024 )
ViT-L
45.9
28.7
69.8
50.4
115.8
0.99
7.1
39.4
21.4
MicKey ( Barroso-Laguna et al., 2024 )
ViT-L
57.2
31.2
74.8
49.3
129.5
1.59
26.0
28.3
12.0
MASt3R ( Leroy et al., 2024 )
ViT-L
81.7
63.0
93.3
79.3
48.8
0.37
2.2
74.0
54.7
GRAFT ( S4 , ours)
ViT-B
60.9
48.0
82.2
69.1
73.1
0.64
7.1
27.9
38.8
Appendix
Table 4: Map-free visual relocalization , official leaderboard (test split). The GRAFT encoder is substituted for the MASt3R encoder and the MASt3R decoder is fine-tuned, with the backbone frozen. AUC and precision (Prec.) are percentages, and metric directions are marked in the header. Baseline rows are quoted from Sarıyıldız et al. (2025) , Table 4.
Prompt distillation compresses large vision-language models (VLMs) such as CLIP into lightweight student models by matching teacher predictions on unlabeled domain images. PromptKD (CVPR 2024) established this paradigm with a single PromptSRC-finetuned ViT-L/14 teacher and a ViT-B/16 student. We propose TheProfessor, a multi-teacher extension that distills from a fixed two-teacher ensemble: a domain-finetuned PromptSRC ViT-L/14 teacher and a zero-shot EVA-CLIP-L/14 teacher whose logits are pre-computed per dataset. We evaluate single-teacher PromptKD, equal-probability ensembling, and confidence-weighted ensembling on four base-to-novel datasets: Caltech-101, DTD, UCF101, and EuroSAT. In a 12-run single-seed sweep, confidence-weighted ensembling improves average HM from 87.52 to 89.28 (+1.77 points), while equal averaging improves average HM to 88.88 (+1.37 points). Gains are dataset dependent: they are negligible on Caltech-101 (+0.16 HM for confidence weighting), modest on UCF101 (+0.62), and largest on domain-shifted EuroSAT (+5.78). These results update our earlier Caltech-only analysis and show that multi-teacher prompt distillation is most useful when the second teacher contributes complementary supervision under domain shift.
Ahmad Algadhi, Ahmed Alzuhair, Omar Alkhulaif +1
Department of Information and Computer Science King Fahd University of Petroleum and Minerals
The platonic representation hypothesis suggests that sufficiently large models converge to a shared representation geometry, even across modalities. Motivated by this, we ask: Can the semantic knowledge of a language model efficiently improve a vision model? As an answer, we introduce TextTeacher, a simple auxiliary objective that injects text embeddings as additional information into image classification training. TextTeacher uses readily available image captions, a pre-trained and frozen text encoder, and a lightweight projection to produce semantic anchors that efficiently guide representations during training while leaving the inference-time model unchanged. On ImageNet with standard ViT backbones, TextTeacher improves accuracy by up to +2.7 percentage points (p.p.) and yields consistent transfer gains (on average +1.0 p.p.) under the same recipe and compute. It outperforms vision knowledge distillation, yielding more accuracy at a constant compute budget or similar accuracy, but 33% faster. Our analysis indicates that TextTeacher acts as a feature-space preconditioner, shaping deeper layers in the first stages of training, and aiding generalization by supplying complementary semantic cues. TextTeacher adds negligible overhead, requires no costly multimodal training of the target model and preserves the simplicity and latency of pure vision models. Project page with code and captions: https://nauen-it.de/publications/text-teacher
Tobias Christian Nauen, Stanislav Frolov, Brian Bernhard Moser +3
RPTU University Kaiserslautern-Landau · German Research Center for Artificial Intelligence (DFKI)
Vision Foundation Models (VFMs) with Vision Transformer (ViT) backbones, such as DINOv2, have become essential for downstream tasks like object recognition and semantic segmentation. The immense computational requirements of backbones often necessitate distillation into smaller architectures for edge deployment. Feature-based knowledge distillation (KD) often suffers from the teacher-student gap; the student struggles to imitate teacher's complex feature map due to its limited capacity. To mitigate this bottleneck, we propose Dyna-DINO: Efficient ViT Distillation Via Adaptive Representation Anchoring, a training curriculum for ViT feature-based knowledge distillation. By utilizing the teacher's intermediate feature maps as a sequence of progressively more difficult targets, our curriculum allows the student to build a foundational representation before tackling higher-level abstractions. Our results demonstrate that this paradigm significantly accelerates convergence through adaptive difficulty selection across various student model sizes and dataset scales. With our curriculum, the Dyna-DINO distilled ViT-S achieves 90.1% accuracy on ImageNet-100, a +12.24% improvement compared with baseline. On ImageNet-1K, Dyna-DINO achieves +3.9% and +6.09% improvement for the instance retrieval task on the Oxford and Paris datasets, +1.93% improvements on semantic segmentation task, as well as meaningful performance gain on classification task. Furthermore, the curriculum enables 25.1% savings in training FLOPs and 21% savings in training time on ImageNet-100 by implementing early-stopping for teacher inference during the initial stages of training. Code is available at https://github.com/KevinZ0217/Dyna-DINO