Growing to See: Efficient Pretraining of Self-Supervised Vision Transformers

Amir Mohammad Karimi Mamaghan; Ferdinand Kapl; Andrea Dittadi; Karl Henrik Johansson; Kaitlin Maile; Johannes von Oswald; Stefan Bauer

cs.CVOct 12, 2026

Growing to See: Efficient Pretraining of Self-Supervised Vision Transformers

Authors: Amir Mohammad Karimi Mamaghan, Ferdinand Kapl, Andrea Dittadi, Karl Henrik Johansson, Kaitlin Maile, Johannes von Oswald, Stefan Bauer

Organizations: KTH Royal Institute of Technology · Helmholtz AI, Munich · Technical University of Munich · Munich Center for Machine Learning (MCML) ·

Abstract

Progressive depth growth builds large networks by starting with shallow models and inserting layers during training. Recent advances in large language models have renewed interest in this longstanding approach, demonstrating benefits for training efficiency and reasoning alongside changes in internal computations. However, how growing affects representation quality and computational patterns in self-supervised vision transformers remains unclear. In this work, we systematically investigate progressive depth growth in two popular self-supervised families, namely masked autoencoders (MAE) and DINOv3, and analyze how growing affects downstream performance, intermediate representations, and internal computations. Across global and dense tasks, we show that growing improves downstream performance while reducing training FLOPs by 13.9--24.8%, with ImageNet linear-probe accuracy gains of up to 3.36 percentage points. Growing also improves downstream performance at intermediate layers and reshapes internal computational patterns, introducing recurring alternations between global and localized attention in MAE and earlier local-to-global attention progression in self-distilled DINOv3. Together, these findings establish progressive depth growth as an effective and efficient pretraining strategy for improving downstream performance across global and dense visual tasks with less computation, and advance our understanding of how growing shapes visual computation.

Figures & tables

		Training	ImageNet-1k		ADE20K	NYU
Model	Strategy	FLOPs (%) $\downarrow$	$k$ -NN $\uparrow$	Linear $\uparrow$	mIoU $\uparrow$	RMSE $\downarrow$
MAE ViT-L/16	Baseline	$100.0$	$31.40$	$60.28\pm 0.07$	$26.29\pm 0.03$	$0.6393\pm 0.0009$
	Stacking	$\mathbf{79.6}$	$33.86$	$\mathbf{63.64\pm 0.08}$	$\mathbf{27.73\pm 0.04}$	$\mathbf{0.5855\pm 0.0001}$
	MIDAS		$32.21$	$59.59\pm 0.07$	$25.82\pm 0.08$	$0.6219\pm 0.0004$
	LIDAS		$\mathbf{34.09}$	$61.08\pm 0.08$	$26.54\pm 0.08$	$0.6292\pm 0.0009$
MAE ViT-H/14	Baseline	$100.0$	$36.59$	$66.25\pm 0.10$	$28.76\pm 0.03$	$0.5886\pm 0.0006$

Table 1: Downstream performance and pretraining compute of fixed-depth baseline and grown models across global and dense tasks. Results are reported as mean

\pm

standard deviation across three downstream seeds, while

k

-NN is deterministic. FLOPs are reported relative to the corresponding fixed-depth baseline. Across both MAE and DINOv3, growing improves downstream performance while reducing pretraining compute.

Figure 1: DINOv3 compute–performance trade-off. LIDAS gains over the fixed-depth baseline versus relative pretraining FLOPs for Prop-1–3 (left to right). Accuracy and mIoU gains use the left axis, and RMSE reduction uses the right axis. Allocating more training to deeper stages improves all four metrics while retaining compute savings.

Figure 2: Extended MAE training. Downstream performance of MAE trained with Stacking and its fixed-depth baseline. Stacking outperforms the baseline on all three tasks even after extended training.

Frozen-teacher distillation
	Training	ImageNet-1k		ADE20K	NYU
Model	FLOPs (%) $\downarrow$	$k$ -NN $\uparrow$	Linear $\uparrow$	mIoU $\uparrow$	RMSE $\downarrow$
Frozen teacher	$100.0$	$\mathbf{82.17}$	$\mathbf{83.40\pm 0.04}$	$\mathbf{44.70\pm 0.10}$	$0.4650\pm 0.0001$
Fixed depth	$100.0$	$81.73$	$82.97\pm 0.02$	$43.43\pm 0.10$	$0.4674\pm 0.0006$
MIDAS	$\mathbf{77.0}$	$81.54$	$82.82\pm 0.03$	$43.65\pm 0.20$	$\mathbf{0.4564\pm 0.0007}$
LIDAS	$\mathbf{77.0}$	$81.55$	$82.92\pm 0.03$	$43.71\pm 0.04$	$\underline{0.4579\pm 0.0010}$

Table 2: Distillation from a frozen pretrained teacher. All grown models use Prop-1 schedule. Results are reported as mean

\pm

standard deviation over three downstream seeds, while

k

-NN is deterministic. FLOPs are reported relative to the fixed-depth student. A frozen teacher runs at full depth throughout training, so growing saves less FLOPs than under self-distillation.

Figure 3: Layer-wise downstream performance. Frozen representations of MAE (top) and DINOv3 (bottom), with trained heads fitted separately at each layer. Growing improves performance across global and dense tasks, with larger gains in middle layers than at the final layer.

Figure 4: Spatial attention patterns across depth. Attention distance, spatial entropy, and query–key NMI for MAE (top) and DINOv3 (bottom). Alternating shaded bands mark groups of four layers. MAE Stacking exhibits a recurring four-layer pattern: distance and entropy peak near group starts and NMI peaks near third layers. DINOv3 growing shifts the phase of localized, query-dependent attention and the subsequent broadening of attention toward earlier layers.

Figure 5: Attention between CLS and patch tokens. Patch-to-CLS (left) and CLS-to-patch mass (right) for MAE (top) and DINOv3 (bottom). Alternating shaded bands mark groups of four layers. MAE Stacking reduces patch-to-CLS attention and increases CLS-to-patch attention across much of the middle layers. DINOv3 growing produces an earlier patch-to-CLS peak and CLS-to-patch recovery, with strongly asymmetric attention in the final layers.

Figure 6: Spatial feature similarities across depth for DINOv3. Rows show the fixed-depth baseline, LIDAS, and delta between the baseline and LIDAS, and columns show encoder layers for CLS–patch (left) and patch-to-patch (right) cosine similarities. Red squares mark the anchor patch. LIDAS shows clearer similarity patterns over object regions at earlier layers.

Appendix figures & tables43 assets

Supplementary material from the paper’s appendix.

Appendix

Configuration	ViT-L/16	ViT-H/14
Input resolution	$224\times 224$	$224\times 224$
Patch size	$16\times 16$	$14\times 14$
Encoder depth / width	24 / 1024	32 / 1280
Attention heads / MLP ratio	16 / 4	16 / 4
Decoder depth / width	8 / 512	8 / 512
Decoder attention heads	16	16

Appendix

Table 3: Training hyperparameters of MAE. Encoder depths refer to the final architecture. The decoder configuration is shared by both model sizes. Learning rates are the effective peak values used by the optimizer.

Model	LR schedule	Steps	Peak LR	Warmup	Growth horizon
ViT-L	Cosine	50,000	$2.4\!\times\!10^{-3}$	1,250	50,000
ViT-L	Cosine	500,000	$2.4\!\times\!10^{-3}$	12,500	50,000
ViT-L	WSD	50,000	$1.0\!\times\!10^{-3}$	1,250	50,000
ViT-L	WSD	500,000	$1.0\!\times\!10^{-3}$	12,500	50,000
ViT-H	WSD	50,000	$1.0\!\times\!10^{-3}$	1,250	50,000
ViT-H	WSD	500,000	$1.0\!\times\!10^{-3}$	12,500	50,000

Appendix

Table 4: Training budgets and schedule variants for MAE.

Configuration	Self-distilled, cosine	Self-distilled, WSD	Frozen-teacher, WSD
Optimizer	AdamW	AdamW	AdamW
Adam $(\beta_{1},\beta_{2})$	$(0.9,0.999)$	$(0.9,0.999)$	$(0.9,0.999)$
Batch size	2048	2048	2048
Peak LR	$10^{-3}$	$5\times 10^{-4}$	$5\times 10^{-4}$
Final LR	$10^{-6}$	$10^{-6}$	$10^{-6}$
LR warmup steps	12,500	12,500	12,500

Appendix

Table 5: Training hyperparameters of DINOv3. All columns use ViT-L/16, 125,000 training steps, batch size of 2048, and the same crop and head configurations. Peak learning rates are reported before batch-size scaling and parameter-group multipliers. The WSD Self-distilled column describes the grown reference which uses

1-\sqrt{s}

cooldown. The fixed-depth schedule ablation in Table 14 instead uses linear WSD as its reference.

Figure 7: A simple illustration of growing strategies. Each row illustrates one growing stage with

N=8

and

B=4

. Solid outlines mark the source layers and dashed outlines mark their copies

\widetilde{L}_{j}

, inserted immediately after the source. Stacking copies the final

B

layers, MIDAS the middle

B

-layer block, and LIDAS the central

B

-layer slice. Colors identify the two original four-layer blocks.

Depth transition	Stacking	MIDAS	LIDAS
$4\to 8$	0–3	0–3	0–3
$8\to 12$	4–7	0–3	2–5
$12\to 16$	8–11	4–7	4–7
$16\to 20$	12–15	4–7	6–9
$20\to 24$	16–19	8–11	8–11
$24\to 28$	20–23	8–11	10–13

Appendix

Table 6: Source-layer indices copied at each standard growth event (

B=4

). Indices are zero-based and ranges are inclusive. Each slice is inserted immediately after its source, as in Equation 5 . The final two transitions apply only to ViT-H.

Model	Schedule	Depth 4	8	12	16	20	24
MAE ViT-L	Prop-1	2,380	7,141	14,283	23,806	35,710	50,000
MAE ViT-L	Prop-2	549	2,746	7,691	16,482	30,218	50,000
MAE ViT-L	Prop-3	113	1,020	4,081	11,337	25,509	50,000
DINOv3 ViT-L	Prop-1	5,952	17,856	35,713	59,522	89,283	125,000
DINOv3 ViT-L	Prop-2	1,373	6,867	19,229	41,207	75,547	125,000
DINOv3 ViT-L	Prop-3	283	2,550	10,203	28,343	63,773	125,000

Appendix

Table 7: Cumulative stage endpoints in optimizer steps. Columns correspond to the depth used during the stage ending at that step. The last endpoint is the growth-schedule horizon, not another growth event. The 500k-step MAE runs reuse the 50k Prop-1 endpoints and then continue to step 500,000 at their final depth.

Step interval	Encoder depth	Decoder depth
$[0,2380)$	4	2
$[2380,5000)$	8	2
$[5000,7141)$	8	4
$[7141,14283)$	12	4
$[14283,15000)$	16	4
$[15000,23806)$	16	6

Appendix

Table 8: Joint MAE growth with Stacking for the encoder and LIDAS for the decoder.

Task	Features	Readout
$k$ -NN classification	$\ell_{2}$ -normalized CLS token	Weighted nearest-neighbor vote
Linear / MLP classification	CLS concatenated with mean-pooled patches ( $2D$ )	Linear / MLP classifier
Semantic segmentation	Patch tokens ( $D$ per location)	SyncBatchNorm, dropout, $1\times 1$ convolution
Depth estimation	Patch tokens ( $D$ per location)	$1\times 1$ convolution to 256 depth bins

Appendix

Table 9: Feature extraction for frozen-backbone evaluation.

D

is the backbone width. The backbone’s output LayerNorm is applied at each evaluated layer. Dense features are reshaped to the spatial patch grid.

Setting	Classification	Segmentation	Depth Estimation
Optimizer	AdamW	AdamW	AdamW
Learning rate	Seven-value grid	$4\times 10^{-3}$	$6\times 10^{-4}$
Weight decay	0	$10^{-3}$	$10^{-4}$
Adam betas	$(0.9,0.999)$	$(0.9,0.999)$	$(0.9,0.99)$
Batch size	512	64	64
Training steps	25,000	10,000	2,400

Appendix

Table 10: Hyperparameters used for the downstream heads. The classification learning-rate grid is specified in Section B.1.1 .

MAE ViT-L/16, Prop-1
	Training	ImageNet-1k		ADE20K	NYU
Model	FLOPs (%) $\downarrow$	$k$ -NN $\uparrow$	Linear $\uparrow$	mIoU $\uparrow$	RMSE $\downarrow$
Baseline	$100.0$	$31.40$	$60.28\pm 0.07$	$26.29\pm 0.03$	$0.6393\pm 0.0009$
Stacking	$79.6$	$\underline{33.86}$	$\mathbf{63.64\pm 0.08}$	$\mathbf{27.73\pm 0.04}$	$\mathbf{0.5855\pm 0.0001}$
MIDAS	$79.6$	$32.21$	$59.59\pm 0.07$	$25.82\pm 0.08$	$\underline{0.6219\pm 0.0004}$
LIDAS	$79.6$	$\mathbf{34.09}$	$\underline{61.08\pm 0.08}$	$\underline{26.54\pm 0.08}$	$0.6292\pm 0.0009$

Appendix

Table 11: Ablations on growing strategy with fixed Prop-1 growing schedule. Each family is compared against its own fixed-depth baseline at matched steps and settings. Linear accuracy, mIoU, and RMSE are reported as mean

\pm

standard deviation over three downstream seeds, while

k

-NN is deterministic. FLOPs are total pretraining cost relative to the matched baseline. Bold and underline mark the best and second best within each family.

MAE ViT-L, Stacking
	Training	ImageNet-1k		ADE20K	NYU
Model	FLOPs (%) $\downarrow$	$k$ -NN $\uparrow$	Linear $\uparrow$	mIoU $\uparrow$	RMSE $\downarrow$
Baseline	$100.0$	$31.40$	$60.28\pm 0.07$	$26.29\pm 0.03$	$0.6393\pm 0.0009$
Prop-1	$79.6$	$33.86$	$63.64\pm 0.08$	$27.73\pm 0.04$	$\mathbf{0.5855\pm 0.0001}$
Prop-2	$85.9$	$\mathbf{36.33}$	$\mathbf{64.23\pm 0.05}$	$\mathbf{28.27\pm 0.03}$	$0.6137\pm 0.0002$
Prop-3	$89.7$	$\underline{35.83}$	$\mathbf{64.23\pm 0.05}$	$\underline{28.04\pm 0.03}$	$\underline{0.5963\pm 0.0002}$

Appendix

Table 12: Growth-schedule ablations. The results are shown as mean

\pm

standard deviation over three downstream seeds, while

k

-NN is deterministic. Bold and underline mark the best and second best within each family.

Figure 8: MAE compute–performance trade-off. Downstream gains over the fixed-depth baseline versus relative pretraining FLOPs for Stacking with Prop-1, Prop-2, and Prop-3 (left to right). Accuracy and mIoU gains use the left axis (percentage points), and RMSE reduction uses the right axis (meters). All three schedules improve performance with less compute, while the best schedule depends on the task.

Fixed depth, 50k steps
	Training	ImageNet-1k		ADE20K	NYUv2
Configuration	FLOPs (%) $\downarrow$	$k$ -NN $\uparrow$	Linear $\uparrow$	mIoU $\uparrow$	RMSE $\downarrow$
cosine, 5e-4	$100.0$	$31.82$	$59.62\pm 0.06$	$24.22\pm 0.04$	$0.6557\pm 0.0004$
cosine, 1e-3	$100.0$	$34.77$	$61.60\pm 0.04$	$26.02\pm 0.03$	$0.6226\pm 0.0006$
cosine, 2.4e-3	$100.0$	$31.40$	$60.28\pm 0.07$	$26.29\pm 0.03$	$0.6393\pm 0.0009$
WSD, 5e-4	$100.0$	$34.71$	$61.22\pm 0.06$	$25.80\pm 0.04$	$0.6381\pm 0.0002$

Appendix

Table 13: MAE ViT-L learning-rate and schedule ablations. The first block contains fixed-depth sweep runs, and the second block compares Stacking (Prop-1) with the selected cosine (

2.4\times 10^{-3}

) and WSD (

10^{-3}

) references. Learning rates are effective optimizer peaks. Results are shown as mean

\pm

standard deviation over three downstream seeds, while

k

-NN is deterministic. Bold marks the best performance in each block.

Figure 9: Learning-rate sensitivity of fixed-depth MAE ViT-L trained for 50k steps. Panels show final-layer

k

-NN and linear classification accuracy, ADE20K mIoU, and NYUv2 RMSE under cosine and WSD schedules. Learning rates are effective optimizer peaks.

Figure 10: Learning-rate and schedule sensitivity of fixed-depth DINOv3 ViT-L trained for 125k steps. The horizontal axis shows peak learning rates before batch-size scaling. In the legend, wsd uses linear cooldown, wsd-1sqrt uses

1-\sqrt{s}

cooldown, and wsd-stable has no cooldown. Stars identify the adopted cosine baseline with stochastic depth 0.3. Results are final-layer evaluations without uncertainty bands.

Fixed depth: stochastic depth $0.1\rightarrow 0.3$
	ImageNet-1k		ADE20K	NYUv2
Configuration	$k$ -NN $\uparrow$	Linear $\uparrow$	mIoU $\uparrow$	RMSE $\downarrow$
WSD, linear	$-0.77$	$-0.57$	$-0.71$	$+0.0036$
WSD, $1-\sqrt{s}$	$-0.53$	$-0.50$	$-0.86$	$\mathbf{-0.0010}$
No cooldown	$-1.12$	$-0.88$	$-1.08$	$+0.0051$
Cosine	$\mathbf{+0.23}$	$\mathbf{+0.18}$	$-0.18$	$+0.0003$

Appendix

Table 14: DINOv3 ViT-L configuration ablations. Entries are variant minus the local reference. Fixed-depth stochastic-depth comparisons use a peak learning rate of

5\times 10^{-4}

for WSD variants and

10^{-3}

for cosine. The fixed-depth schedule comparison uses linear WSD as its reference, with a peak learning rate

5\times 10^{-4}

and stochastic depth 0.1. All grown models use Prop-1. Within each strategy, the grown stochastic-depth and layer-wise learning-rate ablations share one reference run: WSD with

5\times 10^{-4}

1-\sqrt{s}

cooldown, stochastic depth 0.1, and layer-wise learning-rate decay 0.9. This reference recipe is maintained across all growth stages. These variants change only the indicated hyperparameter in the final growth stage. The separate grown cooldown comparison replaces

1-\sqrt{s}

with cosine cooldown within WSD, holding the peak learning rate

5\times 10^{-4}

and stochastic depth 0.1 fixed. Bold marks improvement, which is negative for RMSE.

Figure 11: Sensitivity to the iBOT loss weight for DINOv3 ViT-L. Columns show final-layer

k

-NN accuracy, linear accuracy, ADE20K mIoU, and NYUv2 RMSE, and rows show cosine and WSD schedules. MIDAS and LIDAS use Prop-1, with the indicated weight applied only in the final growth stage, and weight 1.0 in earlier stages. Grey fixed-depth curves in the WSD row apply each weight throughout training. Hollow squares denote MIDAS with weight 0.9 throughout all stages as a reference point.

Cosine, peak LR $2.4\times 10^{-3}$
	Training	ImageNet-1k		ADE20K	NYUv2
Configuration	FLOPs (%) $\downarrow$	$k$ -NN $\uparrow$	Linear $\uparrow$	mIoU $\uparrow$	RMSE $\downarrow$
Baseline	$100.0$	$49.96$	$75.24\pm 0.04$	$32.34\pm 0.04$	$0.5074\pm 0.0008$
Stacking, $T_{g}=50$ k	$98.0$	$51.86$	$\mathbf{76.40\pm 0.07}$	$\mathbf{33.27\pm 0.07}$	$\mathbf{0.4981\pm 0.0001}$
Stacking, $T_{g}=500$ k	$79.6$	$\mathbf{57.97}$	$75.14\pm 0.06$	$32.93\pm 0.03$	$0.5040\pm 0.0007$
WSD, peak LR $10^{-3}$

Appendix

Table 15: MAE ViT-L after 500k training steps. Stacking uses Prop-1 with growth horizon

T_{g}

set to 50k or 500k steps. The learning-rate schedule spans the full 500k-step budget in both cases. FLOPs are relative to the 500k-step fixed-depth baseline. Results are shown as mean

\pm

standard deviation over three downstream seeds, while

k

-NN is deterministic. Bold marks the best performance within each group.

WSD: total training budget $125$ k $\rightarrow 160$ k
	Training	ImageNet-1k		ADE20K	NYUv2
Configuration	FLOPs (%) $\downarrow$	$k$ -NN $\uparrow$	Linear $\uparrow$	mIoU $\uparrow$	RMSE $\downarrow$
MIDAS	$100$	$-0.63$	$-0.03$	$\mathbf{+0.13}$	$+0.0070$
LIDAS	$100$	$-0.87$	$-0.15$	$\mathbf{+0.13}$	$+0.0064$
MIDAS: growth horizon $125$ k $\rightarrow 100$ k
Cosine	$78.0$	$\mathbf{+0.29}$	$\mathbf{+0.27}$	$\mathbf{+0.23}$	$+0.0049$

Appendix

Table 16: DINOv3 training-budget and growth-horizon ablations. Metric entries are changes relative to the corresponding 125k-step Prop-1 grown reference, whose growth horizon is 125k. Training extension preserves the original growth boundaries, whereas the 100k-horizon experiment retains 125k training steps and changes those boundaries. Bold marks improvement.

			ImageNet-1k		ADE20K	NYU
Cooldown	Peak LR	Steps	$k$ -NN $\uparrow$	Linear $\uparrow$	mIoU $\uparrow$	RMSE $\downarrow$
Teacher	—	125k	$82.17$	$83.49$	$44.65$	$0.4650$
none	2e-4	125k	$80.78$	$81.94$	$43.07$	$\mathbf{0.4618}$
none	5e-4	125k	$80.46$	$81.69$	$41.35$	$0.4854$
linear, 25k	5e-4	125k	$81.61$	$82.67$	$43.10$	$0.4751$
cosine, 25k	5e-4	125k	$81.46$	$82.55$	$42.98$	$0.4817$

Appendix

Table 17: Fixed-depth DINOv3 ViT-L students distilled from the same frozen teacher. The dagger marks the recipe adopted for growing experiments. All scores are single final-layer evaluations, and three-seed results for the adopted recipe appear in Table 2 . Bold and underline mark the best and second-best student values.

	Training	ImageNet-1k		ADE20K	NYUv2
Model	FLOPs (%) $\downarrow$	$k$ -NN $\uparrow$	Linear $\uparrow$	mIoU $\uparrow$	RMSE $\downarrow$
Baseline	$100.0$	$31.40$	$60.28\pm 0.07$	$26.29\pm 0.03$	$0.6393\pm 0.0009$
Encoder-only growth (Stacking)	$79.6$	$33.86$	$\mathbf{63.64\pm 0.08}$	$\mathbf{27.73\pm 0.04}$	$\mathbf{0.5855\pm 0.0001}$
Joint encoder + decoder growth (Stacking / LIDAS)	$73.3$	$\mathbf{36.30}$	$63.08\pm 0.06$	$26.16\pm 0.03$	$0.6356\pm 0.0002$

Appendix

Table 18: Encoder-only versus joint encoder–decoder growth for MAE ViT-L. Both grown models use the same Stacking Prop-1 encoder schedule, and joint growth additionally uses a LIDAS decoder schedule, as detailed in Table 8 . FLOPs include the encoder and decoder at their actual depths throughout training and are relative to the fixed-depth baseline. Results are reported as mean

\pm

standard deviation over three downstream seeds, while

k

-NN is deterministic. Bold marks the best performance.

Figure 12: Layer-wise downstream performance across MAE growth strategies at 50k training steps. Each row compares the fixed-depth baseline with Stacking, MIDAS, and LIDAS using Prop-1.

Figure 13: Layer-wise downstream performance of MAE ViT-L after 500k training steps. The fixed-depth baseline and Stacking model use cosine learning-rate schedules spanning the full training budget. Stacking uses Prop-1 with growth confined to the first 50k steps and then continues training at full depth.

Figure 14: Layer-wise downstream performance of DINOv3 ViT-L under frozen-teacher distillation and self-distillation. Top: fixed-depth, MIDAS, and LIDAS students trained with a frozen pretrained teacher. Bottom: MIDAS and LIDAS trained under self-distillation, alongside the fixed-depth baseline. All runs use 125k training steps, and all grown models use Prop-1. The black curve represents the same pretrained backbone in both rows: the frozen teacher above and the fixed-depth self-distillation baseline below.

Figure 15: Spatial attention patterns for additional MAE configurations. Columns show normalized attention distance, spatial entropy, and query–key NMI. The first two rows compare the fixed-depth baseline with Stacking, MIDAS, and LIDAS using Prop-1 at both model sizes after 50k steps. The third row compares the ViT-L baseline and Stacking after 500k steps, with growing confined to the first 50k steps. Alternating shaded bands mark groups of four encoder layers.

Figure 16: CLS–patch attention allocation for the MAE configurations in Figure 15 . Columns show the average attention mass assigned by patch queries to CLS and the total attention mass assigned by the CLS query to patches. Rows show ViT-L at 50k steps, ViT-H at 50k steps, and ViT-L at 500k steps. Alternating shaded bands mark four-layer groups.

Figure 17: Propagation of layer-skipping effects. Cells show the relative change in the residual stream at each readout layer (horizontal axis) after skipping a block (vertical axis), compared with the unmodified model. Rows show MAE Stacking and DINOv3 LIDAS and columns show baseline, grown model, and their difference. Growing strengthens early-layer influence and weakens the effects of many late-layer interventions.

Figure 18: Additional layer-skipping comparisons for MAE ViT-L. Rows show MIDAS and LIDAS at 50k steps, followed by Stacking at 500k steps with a 50k growth horizon. All grown models use Prop-1 and cosine scheduling. Columns show the matched fixed-depth baseline, the grown model, and their difference. Each cell measures the relative change in the residual stream at the horizontal readout layer when the block on the vertical axis is bypassed. Red in the difference maps indicates larger effects in the grown model, and blue indicates smaller effects.

Figure 19: Layer-skipping comparisons for MAE ViT-H trained for 50k steps. Rows show Stacking, MIDAS, and LIDAS using Prop-1, each compared with the same fixed-depth baseline. Columns show the matched fixed-depth baseline, the grown model, and their difference. Each cell measures the relative change in the residual stream at the horizontal readout layer when the block on the vertical axis is bypassed. Red in the difference maps indicates larger effects in the grown model, and blue indicates smaller effects.

Figure 20: Spatial attention patterns for DINOv3 ViT-L under self-distillation and frozen-teacher distillation. Columns show normalized attention distance, spatial entropy, and query–key NMI. Top: MIDAS and LIDAS with their fixed-depth self-distillation baseline. Bottom: MIDAS, LIDAS, and fixed-depth students, together with their frozen pretrained teacher. All runs use 125k training steps, and the grown models use Prop-1. The black curve is the same backbone in both rows. Alternating shaded bands mark four-layer groups.

Figure 21: CLS–patch attention allocation for the DINOv3 configurations in Figure 20 . Columns show patch-to-CLS and CLS-to-patch attention mass. The top row shows self-distillation and the bottom row shows frozen-teacher distillation. MIDAS and LIDAS use Prop-1 in both rows. The fixed-depth student appears only in the bottom row, and the black curve represents the fixed-depth self-distillation baseline above and the frozen teacher below. Alternating shaded bands mark four-layer groups.

Figure 22: Additional layer-skipping comparisons for DINOv3 ViT-L. The top row compares cosine self-distillation with MIDAS Prop-3 against its fixed-depth baseline. The remaining rows compare frozen-teacher students grown with MIDAS or LIDAS Prop-1 against the fixed-depth student trained with the same teacher and recipe. Columns show the reference, grown model, and their difference in relative residual-stream change. The skipped block is on the vertical axis and the readout layer is on the horizontal axis. Red denotes larger effects in the grown model and blue denotes smaller effects.

Figure 23: Additional spatial feature similarities across depth for DINOv3. Each triplet of rows shows the fixed-depth baseline, LIDAS with Prop-3, and the delta between the baseline and LIDAS for one image. Columns within each half show encoder layers 4, 8, 12, 16, 20, and 24, with CLS–patch cosine similarity on the left and anchor-to-patch cosine similarity on the right. Red squares mark the anchor used for the patch-to-patch maps. LIDAS shows earlier or more persistent patterns over object regions, especially in the CLS–patch maps.

Figure 24: Examples of input-masking conditions. Rows show a sample from ImageNet-S and ADE20K datasets. From left to right, columns show the original image, random masking of 50% of input patches, the target masked, and the target retained with the remaining image masked. Masked pixels are replaced with the ImageNet channel mean.

Figure 25: MAE robustness to random input masking. Rows show ViT-L at 50k steps, ViT-H at 50k steps, and ViT-L at 500k steps. All grown models use Prop-1, and the 500k-step Stacking run confines growing to the first 50k steps. Columns show ImageNet-S top-1 accuracy and its retention, followed by ADE20K mIoU and its retention. Retention is measured relative to each model’s own clean score.

Figure 26: DINOv3 ViT-L robustness to random input masking. The fixed-depth baseline is compared with MIDAS and LIDAS using Prop-3. Columns show ImageNet-S top-1 accuracy and its retention, followed by ADE20K mIoU and its retention. Retention is measured relative to each model’s own clean score.

MAE ViT-L 50k
	ImageNet-S top-1 $\uparrow$			ADE20K mIoU $\uparrow$
Model	Clean	Target masked	Target retained	Clean	Target masked	Target retained
Baseline	$65.55$	$12.26$ (19%)	$34.20$ (52%)	$26.31$	$19.16$ (73%)	$6.19$ (24%)
Stacking	$\mathbf{68.63}$	$\mathbf{14.86}$ (22%)	$\mathbf{42.58}$ (62%)	$\mathbf{27.79}$	$\mathbf{20.76}$ (75%)	$\mathbf{6.45}$ (23%)
MIDAS	$64.60$	$11.80$ (18%)	$32.20$ (50%)	$25.89$	$18.92$ (73%)	$6.02$ (23%)
LIDAS	$66.23$	$12.62$ (19%)	$37.45$ (57%)	$26.59$	$19.73$ (74%)	$6.31$ (24%)

Appendix

Table 19: Semantic input-masking results. ImageNet-S reports top-1 accuracy and ADE20K reports whole-image mIoU. Target masked replaces the selected semantic region with the ImageNet channel mean, and target retained masks the rest of the image. Parentheses report retention as a percentage of the corresponding clean score. Model configurations match Figures 25 and 26 .

MAE ViT-L, 50k steps
Model	$k$ -NN	Linear	MLP-2	MLP-3	FT
Baseline	$31.40$	$60.28$	$61.71$	$62.01$	$82.24$
Stacking	$\mathbf{33.86}$	$\mathbf{63.70}$	$\mathbf{64.58}$	$\mathbf{64.85}$	$\mathbf{83.77}$
MAE ViT-L, 500k steps
Baseline	$49.96$	$75.18$	$75.41$	$74.81$	$\mathbf{85.71}$
Stacking	$\mathbf{51.86}$	$\mathbf{76.29}$	$\mathbf{76.38}$	$\mathbf{75.71}$	$85.62$

Appendix

Table 20: MAE adaptation on ImageNet-1k, measured by top-1 accuracy (%).

k

-NN uses frozen final-layer CLS features, linear and two- or three-layer MLP probes use frozen final-layer CLS and mean-pooled patch features, and fine-tuning (denoted by FT) updates the full encoder. The results are shown for a single seed. All grown models use Prop-1, with growth horizon of 50k steps for the 500k-step runs. Bold marks the highest accuracy per column within each block.

Figure 27: Layer-wise representation similarity between MAE ViT-L features for the fixed-depth baseline and Stacking at 50k and 500k training steps. Rows show similarity for CLS-only and patch tokens and columns show the model and training step. At 50k, Stacking has broader intermediate-layer CLS alignment. At 500k, both models exhibit strong intermediate alignment, while differences on the final CLS representations remain visible.

Figure 28: Layer-wise representation similarity in DINOv3 for the fixed-depth baseline, MIDAS, and LIDAS. Rows show similarity CLS-only and patch tokens. The three models retain a similar overall progression across depth, with localized differences in alignment. Patch-token similarity extends over a broader depth range than CLS similarity.

Figure 29: Weight similarity between consecutive four-layer groups in MAE ViT-L for the fixed-depth baseline, Stacking, MIDAS, and LIDAS. The axes labeled Block index groups 0–5, corresponding to encoder layers 1–4 through 21–24. The strongest alignment occurs between the final groups for Stacking and between the central groups for MIDAS and LIDAS, matching the locations of their final source–copy pairs.

Figure 30: Weight similarity between consecutive four-layer groups in DINOv3 for the fixed-depth baseline, MIDAS, and LIDAS. Group indices follow Figure 29 . Both grown models exhibit their strongest alignments in the middle blocks.

Figure 31: MAE weight similarity at 50k and 500k training steps. The top row shows Stacking and the bottom row the fixed-depth baseline, and the left and right columns show 50k and 500k steps, respectively. Stacking’s strong alignment among later groups is substantially weaker at 500k, although some structure remains relative to the baseline.

Self-Supervised Visual Representation Learning

Self-supervised visual representation learning trains image, video, and 3D encoders on unlabeled data to produce transferable visual features.

See all 236 papers

Apr 22, 2026

Self-supervised pretraining for an iterative image size agnostic vision transformer

Vision Transformers (ViTs) dominate self-supervised learning (SSL). While they have proven highly effective for large-scale pretraining, they are computationally inefficient and scale poorly with image size. Consequently, foundational models like DINO are constrained to low-resolution processing. A recent foveal-inspired transformer achieves resolution agnosticism by iteratively processing a fixed-size context of multi-zoom patches. This model demonstrated promising results via supervised learning, utilizing a sequential, recurrent-like process without backpropagation through time. To unlock its potential as a foundational backbone, we introduce a novel sequential-to-global SSL framework based on DINO's self-distillation objective. Supported by an efficient integral-image patch extraction method, our approach enables large-scale pretraining for image-size agnostic vision encoders. We achieve competitive performance on ImageNet-1K and downstream classification tasks, maintaining a constant computational budget regardless of input resolution.

Nedyalko Prisadnikov, Danda Pani Paudel, Yuqian Fu +1

INSAIT, Sofia University "St. Kliment Ohridski", Bulgaria

Self-Supervised Pre-Training Vision Transformer Efficient ViTs Self-Supervised Visual Representation Learning

Ask PDF

Feb 28, 2026

Vision-TTT: Efficient and Expressive Visual Representation Learning with Test-Time Training

Learning efficient and expressive visual representation has long been the pursuit of computer vision research. While Vision Transformers (ViTs) gradually replace traditional Convolutional Neural Networks (CNNs) as more scalable vision learners, their applications are plagued by the quadratic complexity of the self-attention mechanism. To address the challenge, we introduce a new linear-time sequence modeling method Test-Time Training (TTT) into vision and propose Vision-TTT, which treats visual sequences as datasets and compresses the visual token sequences in a novel self-supervised learning manner. By incorporating the dual-dataset strategy and Conv2d-based dataset preprocessing, Vision-TTT effectively extends vanilla TTT to model 2D visual correlations with global receptive fields. Extensive experiments show that Vittt-T/S/B achieve

77.7\%,81.8\%,82.7\%

Top-1 accuracy on ImageNet classification and also greatly outperform their counterparts on downstream tasks. At

1280\times1280

resolution, Vittt-T reduces FLOPs by

79.4\%

and runs

4.72\times

faster with

88.9\%

less memory than DeiT-T. These results demonstrate the expressiveness and efficiency of Vision-TTT as a strong candidate for the next-generation generic visual backbone.

Quan Kong, Yanru Xiao, Yuhao Shen +1

Zhejiang University, Hangzhou, China · Amazon Web Services, USA

Visual Representation Learning Efficient ViTs Vision Transformer Test-Time Training Self-Supervised Visual Representation Learning

Ask PDF

Jul 4, 2026

Rethinking Depth Pruning for Vision Transformers: A Heterogeneity-Aware Perspective

While prior studies have successfully compressed vision Transformers (ViTs) through various pruning techniques, most have concentrated on width pruning to achieve significant reductions in model size. Depth pruning, which removes entire layers from a ViT, is notoriously difficult for accuracy recovery despite its potential to deliver higher speedups, limiting the acceleration achieved by existing joint width-and-depth pruning methods. In this work, we reveal that the failure of existing depth pruning methods lies in their neglect of heterogeneity between different layers, and we introduce HetDPT, a heterogeneity-aware depth pruning method that avoids dimension mismatch. Comprehensive experiments on ImageNet-1K, CIFAR-100, COCO, and ADE20K validate our method: HetDPT achieves a 1.58

\times

speedup for DeiT-B while maintaining accuracy and a 1.39

\times

speedup for DeiT-S with nearly no accuracy degradation. Furthermore, when combined with width pruning, HetDPT+ sets a new state-of-the-art record in extreme ViT pruning, enhancing the acceleration ratio from 4.24

\times

to 5.19

\times

for the Isomorphic-Pruning-2.6G configuration while maintaining near-lossless accuracy; our code is available at https://github.com/Efficient-AI-for-All/HetDPT.

Zhenfeng Su, Kang Zhao, Han Bao +4

School of Data Science, The Chinese University of Hong Kong, Shenzhen, China · Huawei Technologies Co., Ltd., Shenzhen, China · Tsinghua University, Beijing, China +1

Vision Transformer Efficient ViTs Neural Network Pruning

Ask PDF

Abstract

Self-Supervised Visual Representation Learning

Explore similar work

Self-supervised pretraining for an iterative image size agnostic vision transformer

Vision-TTT: Efficient and Expressive Visual Representation Learning with Test-Time Training

Rethinking Depth Pruning for Vision Transformers: A Heterogeneity-Aware Perspective