Masked image modeling (MIM) methods typically operate in either raw pixel space (reconstructing masked patches) or latent feature space (aligning with a pre-trained teacher). We present MEDiC (Multi-objective Exploration of Distillation from CLIP), a framework that combines both spaces in a single pipeline through three complementary objectives: patch-level token distillation from a frozen CLIP encoder, global CLS alignment, and pixel reconstruction via a lightweight decoder. We conduct a systematic investigation of the design space surrounding this multi-objective framework. First, we show that all three objectives provide complementary information, with the full combination reaching 73.9% kNN accuracy on ImageNet-1K. Second, we introduce hierarchical clustering with relative position bias for evolved masking and find that, despite producing more semantically coherent masks than prior methods, evolved masking does not outperform simple block masking in the teacher-guided distillation setting, a finding we attribute to the teacher's inherent semantic awareness. Third, we reveal that optimal scalar loss weights are extremely fragile, with small perturbations causing drops of up to 17 percentage points in kNN accuracy. Our framework achieves 73.9% kNN and 85.1% fine-tuning accuracy with ViT-Base at 300 epochs.
Figures & tables
Figure 1: Four paradigms in masked image modeling. Top-left: raw-space pixel reconstruction (MAE-style). Top-right: latent-space prediction with discrete visual tokens (BEiT-style). Bottom-left: latent-space distillation at the patch level from a teacher (MaskDistill-style). Bottom-right: MEDiC combines pixel reconstruction with both patch-level and CLS-level distillation from a frozen CLIP teacher.
Figure 2: Masking strategies in MIM. (a-c) Three standard approaches: grid, random, and block masking with different mask ratios. (d) Evolved masking uses attention-guided clustering to produce semantically coherent mask patterns that adapt during training.
Imagenette
Imagewoof
ImageNet-1K
Backbone
Epochs
top-1
top-5
top-1
top-5
top-1
MAE [ 11 ]
ViT-B
800
54.75
89.35
30.29
76.79
9.53
BEiT [ 2 ]
ViT-B
800
30.06
75.69
20.26
70.40
1.93
BootMAE [ 8 ]
ViT-B
800
70.70
94.22
41.74
84.53
19.61
CMAE [ 14 ]
ViT-B
800
77.91
96.33
42.84
85.24
21.17
SimMIM [ 23 ]
ViT-B
400
51.44
87.21
30.16
74.65
8.40
Table 1: kNN classification accuracy using frozen representations. We report Top-1 and Top-5 accuracies on Imagenette, Imagewoof, and ImageNet-1K.
Figure 3: MEDiC achieves strong kNN and fine-tuning performance through multi-objective distillation from CLIP, outperforming methods that operate in either raw or latent space alone.
Method
Backbone
Epochs
kNN
Linear
Fine-tune
MAE †
ViT-L
400
9.53
54.60
83.4
BEiT †
ViT-B
800
1.93
52.10
80.12
BootMAE †
ViT-B
800
19.61
52.07
84.0
CMAE †
ViT-B
800
21.17
60.00
84.4
SimMIM †
ViT-B
400
8.40
58.09
83.8
SemMAE †
ViT-B
800
19.13
54.62
84.5
Table 2: Combined evaluation on ImageNet-1K: kNN (frozen), linear probing, and fine-tuning accuracy. † : publicly available checkpoints. ‡ : our reproduction.
Method
Backbone
Epochs
mIoU
BEiT † [ 2 ]
ViT-B
800
45.6
MAE † [ 11 ]
ViT-B
1600
48.1
BEiT v2 † [ 19 ]
ViT-B
300
52.7
MILAN † [ 12 ]
ViT-B
400
52.7
CAE v2 † [ 24 ]
ViT-B
300
53.4
MaskDistill ‡ [ 18 ]
ViT-B
300
53.8
Table 3: ADE20K semantic segmentation (mIoU %) with UperNet decoder and end-to-end fine-tuning (160K iterations). † : published results. ‡ : our reproduction at 300 epochs.
Latent-Rep
Raw-Pix
Latent-Disc
kNN Top-1
∘
∘
✓
14.1
∘
✓
∘
9.5
✓
∘
∘
68.6
✓
✓
∘
71.4
✓
∘
✓
72.3
✓
✓
✓
73.9
Table 4: Effect of each loss component. Combining all three objectives yields the strongest kNN accuracy.
Figure 4: Loss weight sensitivity. (a) Pixel reconstruction weight (DLW) has a sharp optimum at 0.01; higher values degrade kNN by up to 17 points. (b) CLS alignment weight (CLW) peaks at 0.30 with a sudden drop at 0.20. Both curves reveal the fragility of global scalar weights. The combined optimum (DLW=0.01, CLW=0.30) yields 73.92% kNN and 85.07% fine-tuning. Full sweep data in the Appendix.
Figure 5: Evolved masking across training epochs. For each input image, the top row shows EM-based masks and the bottom row shows HC-based masks. HC produces more spatially coherent groupings that align with semantic content.
Masking Method
kNN Top-1
Block Masking
68.59
Evolved Masking + EM
47.52
Evolved Masking + HC (Ours)
64.56
Table 5: Masking strategy comparison. Despite HC producing more coherent masks (Fig. 5 ), block masking achieves the highest kNN accuracy.
Figure 6: Evolved masking vs. block masking. HC masking is swept across transition rates at two mask ratios. The block masking baseline (68.59%, dashed line) outperforms all evolved configurations. The best HC result (64.93%) leaves a 3.7-point gap.
Objectives
Sparse
Dense
Token + CLS
75.7
74.1
Token + Pixel
71.5
66.9
Token + Pixel + CLS
73.9
69.3
Table 6: Dense vs. sparse encoding comparison (kNN@20). Sparse encoding consistently outperforms dense across objective combinations.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameters
Base Size
Layers
12
Hidden size
768
FFN inner hidden size
3072
Attention heads
12
Layer scale
0.1
Patch size
16×16
Appendix
Table 7: Hyperparameters for pre-training on ImageNet-1K using ViT-Base model.
Hyperparameters
ViT-B/16
Peak learning rate
5e-4
Epochs
100
Warmup epochs
20
Batch size
1024
Adam ϵ
1e-8
Adam β
(0.9, 0.999)
Appendix
Table 8: Hyperparameters for linear-probing on ImageNet-1K.
Hyperparameters
ViT-B/16
Peak learning rate
5e-4
Epochs
100
Warmup epochs
10
Layer-wise learning rate decay
0.65
Batch size
1024
Adam ϵ
1e-8
Appendix
Table 9: Hyperparameters for fine-tuning on ImageNet-1K.
Hyperparameters
ViT-B/16
Relative positional embeddings
✓
Shared relative positional embeddings
✗
Epochs
50
Peak learning rate
0.5e-4
Fine-tuning steps
160K
Batch size
16
Appendix
Table 10: Hyperparameters for semantic segmentation on ADE20K (UperNet, 160K iterations).
Figure 7: Comparison of HC evolved masking methods using γ=2 and γ=4 for scheduling the α values across different pre-training epochs. For each epoch, within each row, the first column represents the evolved mask generated using γ=2 , and the second column shows the evolved mask using γ=4 . Setting γ=4 slows down the transition, leading to more semantically rich masks later in training, thus making the task more challenging over time.
Hyperparameters
ViT-B/16
Mask type
evolved
Mask ratio
0.75
Mask γ
1.7
Mask ζ
0.9
Clustering Method
HC
Min Clusters
10
Appendix
Table 11: Evolved masking hyperparameters for pre-training on ImageNet-1K.
Dec. Loss
DLW
CLW
kNN Top-1
kNN Top-5
Best Epoch
Fine-tune
Pixel Reconstruction Weight (DLW) Variations
L2
0.50
–
61.57
82.21
299
82.70
L2
0.40
–
56.74
79.13
274
83.30
L2
0.30
–
63.26
84.45
299
83.63
L2
0.20
–
62.27
83.11
247
84.00
L2
0.10
–
63.30
84.65
225
84.70
Appendix
Table 12: Full loss weight sweep. Top: pixel reconstruction weight (DLW) with no CLS loss. Middle: CLS alignment weight (CLW) with no pixel loss. Bottom: combined DLW and CLW.
Method
γ
ζ
Mask Ratio
kNN Top-1
kNN Top-5
Best Epoch
BM
–
–
0.50
68.59
88.26
275
Mask Ratio = 0.50
EM
2.0
0.5
0.50
47.52
70.04
200
HC
2.0
0.5
0.50
51.92
75.79
148
HC
1.6
0.5
0.50
62.52
84.10
200
HC
1.4
0.5
0.50
52.91
76.66
100
Appendix
Table 13: Full evolved masking sweep across 26 configurations. BM: block masking, EM: expectation-maximization clustering, HC: hierarchical clustering with relative position bias.