Vision foundation models such as DINOv2, SigLIP2, and MASt3R develop complementary capabilities from different pretraining objectives, yet their knowledge remains distributed across separate, specialized models. Multi-teacher knowledge distillation offers a path toward consolidating these capabilities into a single agglomerative backbone, but existing approaches assume a fixed set of teachers, and incorporating a new teacher requires repeating expensive joint distillation over the entire teacher set. We introduce GRAFT, a continual multi-teacher distillation framework that enables a unified backbone to progressively acquire capabilities from an open-ended sequence of foundation models. When a new teacher arrives, GRAFT treats the previously distilled model as a teacher for preserving learned capabilities, while the current student jointly learns from both the previous model and the incoming teacher. Furthermore, to reconcile the incompatible representation geometries of heterogeneous teachers, we introduce Teacher Specific Readout Tokens, which grant each teacher an independent read-out of the shared encoder, together with Geometry Agnostic Relational Loss that aligns a vision-language teacher by matching image-text similarity structures rather than raw feature values. We provide GRAFT model, which is a single, continually extensible backbone that unifies five domains, including image understanding, 2D dense prediction, 3D human pose estimation, 3D vision, and vision-language, delivering strong performance across all of them while acquiring each new capability at the cost of a single distillation rather than a full re-distillation.
Figures & tables
Figure 1: Compared to existing agglomerative models, GRAFT hold a comprehensive capability on various tasks with strong performance on each task.
Figure 2: Per-patch feature geometry across depth. Each cell visualizes the top-3 PCA components of patch tokens (mapped to RGB, fit per cell) at a shallow, deep, and final block for four ViT-L teachers and GRAFT (ViT-B) on three ADE20K images. The teachers diverge from the shallowest blocks: DINOv2 resolves object structure, MASt3R shows geometric texture, SigLIP2 carries only positional patterns, and Multi-HMR progressively organizes around people. GRAFT unifies these in a single trunk—its patch tokens retain sharp object/part structure from its dense teachers (DINOv2, MASt3R, Multi-HMR), while SigLIP2’s vision–language capability is grafted relationally onto a dedicated summary token (§ 3.3 ) and thus does not appear in this patch view.
Figure 3: Overview of GRAFT at round k . The active student Sk is warm-started from Sk−1 . A learnable Teacher Specific Readout Token (TSRT) is assigned to each task and jointly encoded with the shared patch tokens, producing task specific summary representations that are mapped by task-specific projectors. Training follows two concurrent paths. In the retention path, outputs for previously acquired tasks 1,…,k−1 are constrained by Lanchor to remain consistent with those of the frozen anchor model Sk−1 , which preserves previously acquired task functions. In the acquisition path, the new task readout is optimized through Ldistill to acquire knowledge from the incoming teacher Tk . Only the active student Sk is updated, while both Sk−1 and Tk remain frozen.
Image Understanding
2D Dense Prediction
3D Human Perception
3D Vision
Vision-Language
Model
Arch.
IN-1k cls ↑
Depth ↓
mIoU ↑
F1 ↑
PA-PVE ↓
AUC ↑
COCO (T → I/I → T) ↑
Flickr (T → I/I → T) ↑
Teachers (ceilings)
DINOv2
ViT-L
–
0.384
47.70
–
–
–
–
–
Multi-HMR
ViT-L
–
–
–
95.0
36.9
–
–
–
MASt3R
ViT-L
–
–
–
–
–
91.2
–
–
SigLIP2
ViT-L
83.8
–
–
–
–
–
55.7 / 72.1
83.2 / 95.1
Table 1: Main results across five task families. All models use a ViT-B backbone; teacher rows (ViT-L) are shown as per-task ceilings. Metric directions are marked in the header, best values are in bold and second-best underlined, and “-” indicates the model lacks that capability. ‡ Segmentation for RADIOv2.5 is distilled from SAM, and for SigLino and EUPE from DINOv3. § DUNE result obtained by fine-tuning on non-public data.
Acquisition
Retention
Acquisition
Retention
Variant
IN-1k cls ↑
mIoU ↑
Depth RMSE ↓
Variant
IN-1k cls ↑
mIoU ↑
Depth RMSE ↓
Distillation strategy
Vision–language projector
Direct distillation
82.6
35.4
0.584
Frozen encoder + Transformer
75.5
45.5
0.425
LoRA finetuning
76.2
45.7
0.425
Transformer, 1 layer
80.4
46.0
0.448
LPFT
78.0
40.5
0.442
Transformer, 3 layers
75.9
45.4
0.451
EWC
77.1
40.2
0.508
MLP + residual (ours)
81.2
46.0
0.450
Table 2: Ablation studies for vision language distillation. We study the distillation strategy, vision language projector architecture, and choice of vision language teacher. All experiments start from a student distilled from DINOv2 and further distill an incoming vision language teacher.
Figure 4: Qualitative comparison on depth estimation, semantic segmentation, multi-human reconstruction, and 3D reconstruction between GRAFT and corresponding teachers.
Table 3: Teacher sets and training data differ across methods , so the head-to-head in Table 1 is not like-for-like. Teacher scale is per model and counts are the number of distillation teachers. Entries are stated by each source where available and marked unavail when a source does not disclose them. SigLino reports image tokens rather than images, and EUPE additionally trains a from-scratch 1.9B proxy before distillation. GRAFT’s teachers are DUNE’s three, inherited through the warm start, plus the grafted vision–language teacher; its data subtotal counts DataComp-12M and ImageNet-19k and excludes Mapillary and GLDv2, whose sizes are undisclosed.
VCRE <45 px
VCRE <90 px
Median reproj.
Relative pose error
Method
Encoder
AUC ↑
Prec. ↑
AUC ↑
Prec. ↑
error (px) ↓
Trans. (m) ↓
Rot. ( ∘ ) ↓
Prec. ↑
AUC ↑
LoFTR ( Sun et al., 2021 )
CNN
39.7
18.2
61.8
33.5
166.8
2.31
39.4
26.9
9.8
DUSt3R ( Wang et al., 2024 )
ViT-L
45.9
28.7
69.8
50.4
115.8
0.99
7.1
39.4
21.4
MicKey ( Barroso-Laguna et al., 2024 )
ViT-L
57.2
31.2
74.8
49.3
129.5
1.59
26.0
28.3
12.0
MASt3R ( Leroy et al., 2024 )
ViT-L
81.7
63.0
93.3
79.3
48.8
0.37
2.2
74.0
54.7
GRAFT ( S4 , ours)
ViT-B
60.9
48.0
82.2
69.1
73.1
0.64
7.1
27.9
38.8
Appendix
Table 4: Map-free visual relocalization , official leaderboard (test split). The GRAFT encoder is substituted for the MASt3R encoder and the MASt3R decoder is fine-tuned, with the backbone frozen. AUC and precision (Prec.) are percentages, and metric directions are marked in the header. Baseline rows are quoted from Sarıyıldız et al. (2025) , Table 4.