We study how to consolidate the current VITOK progress into a single multi-teacher distillation recipe that jointly preserves global recognition and dense semantics. Our starting point is an AM-RADIO-style student distilled from SigLIP2 and DINOv3-L, where SigLIP2 supplies strong global semantics and DINOv3-L supplies stronger dense features. The central empirical issue is that the same recipe does not optimize all objectives equally well: changes that improve ImageNet-1K kNN accuracy can still degrade ADE20K segmentation. We summarize a progression of modifications that make this trade-off more explicit and more manageable: split adaptor heads for CLS and patch tokens, asymmetric cosine/MSE losses, initialization from a DINOv3-L checkpoint, teacher reweighting, masked image modeling (MIM), and PHI-S feature balancing. The resulting model reaches 83.2 patch kNN and 85.2 CLS kNN, slightly surpassing the DINOv3-L teacher on ImageNet-1K kNN classification, while PHI-S restores ADE20K performance from 46.5/58.1 to 48.5/61.0 mIoU/mAcc, matching the teacher on this dense benchmark. We also summarize negative results: scaling distillation from ImageNet-1K to ImageNet22K does not consistently help, and naively adding extra teachers such as SAM3 or HOG features introduces interference. Rather than claiming a final recipe, this paper distills the current project state into a compact empirical story and a concrete set of lessons for future iterations.
Figures & tables
Figure 1: Motivation for ViTok . We treat SigLIP2 and DINOv3-L as complementary teachers and use MIM as a dense regularizer. The goal is to recover a more balanced encoder that maintains both classification and dense prediction quality.
Figure 2: Overview of the current ViTok recipe. The student encoder receives a masked image, sends CLS and patch tokens through separate teacher-specific adaptor heads, matches normalized teacher targets, and optimizes an auxiliary MIM loss on masked locations. PHI-S is applied only during target construction; inference remains in the original student feature space.
Figure 3: Geometry of PHI-S. Regular channel-wise standardization depends on axis alignment and can overemphasize low-variance directions. PHI-S first rotates the covariance basis and then applies a single isotropic scale, which better preserves balanced dense behavior in our experiments.
Model / setting
Patch
CLS
mIoU
mAcc
DINOv3-L teacher
79.7
85.0
48.5
61.0
ViTok best cls ( s1d0.5 )
83.2
85.2
46.5
58.1
ViTok + PHI-S (IN1K)
81.3
84.6
48.5
61.0
ViTok + PHI-S (IN22K, s1d1 )
80.0
84.5
48.28
60.25
Table 1: Main comparison across the project milestones. The best classification run and the best dense-recovery run are not the same model, which is exactly the tension that motivates PHI-S. Here s / d denote SigLIP/DINO loss weights.
Adaptor / loss design
Patch
CLS
Single adaptor, cosine only
82.2
82.9
Split adaptor, cosine + MSE
82.7
84.1
DINOv3-L teacher
79.7
85.0
Table 2: Adaptor and loss ablation. Splitting CLS and patch heads and assigning different losses to them produces the first major gain.
Setting
Patch
CLS
Split adaptor baseline
82.7
84.1
+ DINOv3-L init, s1d1
82.4
84.8
+ reweight, s0.5d1
81.0
84.5
+ reweight, s1d0.5
83.2
85.2
Table 3: Initialization and teacher reweighting. Starting from DINOv3-L and rebalancing teacher losses moves the student past the teacher on ImageNet-1K kNN. Here s / d denote SigLIP/DINO loss weights.
Setting
mIoU
mAcc
No MIM, no PHI-S
46.3
57.5
+ MIM
46.5
58.1
+ MIM + PHI-S
48.5
61.0
DINOv3-L teacher
48.5
61.0
Table 4: Dense recovery on ADE20K. MIM gives a small gain, but PHI-S is the key intervention that closes the dense-performance gap.
DINO-only setting
Patch
CLS
MIM 0.0, LR 1.5×10−4
79.5
83.0
MIM 0.3, LR 1.5×10−4
67.2
70.0
MIM 0.3, LR 5×10−4
78.8
82.1
MIM 0.3, LR 7.5×10−4
79.2
82.8
Table 5: MIM sensitivity in DINO-only distillation. The auxiliary objective is not uniformly helpful; it becomes viable only after re-tuning optimization.
Figure 4: Project progression. Left: ImageNet-1K kNN improves through structural changes, initialization, and teacher reweighting. Right: ADE20K improves only modestly with MIM but is largely restored by PHI-S; the ImageNet-22K transition remains unstable.
Weight ( s =SigLIP2, d =DINOv3-L)
Patch
CLS
mIoU
mAcc
s1d1
82.4
84.8
47.6
59.2
s1d0.5
83.2
85.2
46.5
58.1
s1d0.3
83.2
85.2
44.9
56.6
s1d0.1
83.1
85.7
42.0
53.7
s1d0.5 + PHI-S
81.3
84.6
48.5
61.0
Table 6: Teacher-balance sweep on IN-1K and ADE20K. Heavier SigLIP2 weighting benefits classification but erodes dense performance. Here s / d denote SigLIP/DINO loss weights.
Figure 5: CLS kNN versus ADE20K mIoU for the teacher-weight sweep. Here s / d denote SigLIP/DINO loss weights. PHI-S improves the trade-off curve rather than simply selecting another point on the original sweep.
Setting
Patch
CLS
mIoU
mAcc
IN22K, no PHI-S ( s1d0.5 )
81.9
84.5
46.0
57.8
IN22K + PHI-S ( s1d0.5 )
80.6
84.3
47.40
58.94
IN22K + PHI-S ( s1d1 )
80.0
84.5
48.28
60.25
+ SAM3
84.0
79.4
47.59
59.82
+ SAM3 + HOG
78.9
69.9
47.31
59.26
Table 7: Scaling and challenge summary. Larger-scale distillation and extra teachers still show interference, even when they improve one side of the trade-off. Here s / d denote SigLIP/DINO loss weights.
PHI-S statistics source
Patch
CLS
Reused IN1K statistics
80.8
84.3
IN22K repeated-sample estimate
80.1
78.4
IN22K one-pass fixed estimate
80.6
84.3
Table 8: PHI-S statistics diagnostics on ImageNet-22K. The corrected one-pass statistics recover the classification branch that was damaged by repeated-sample estimation.
Epochs ( s1d0.5 , s =SigLIP2, d =DINOv3-L)
mIoU
mAcc
30
47.84
59.84
35
47.80
59.71
40
48.24
60.01
50
47.95
59.78
Table 9: ImageNet-22K epoch sweep with PHI-S. Here s / d denote SigLIP/DINO loss weights. Once the corrected statistics are used, the dense metric saturates early rather than improving monotonically with longer training.