Self-Distillation of Hidden Layers for Self-Supervised Representation Learning

Scott C. Lowe; Anthony Fuller; Sageev Oore; Evan Shelhamer; Graham W. Taylor

cs.CVMar 16, 2026

Self-Distillation of Hidden Layers for Self-Supervised Representation Learning

Authors: Scott C. Lowe, Anthony Fuller, Sageev Oore, Evan Shelhamer, Graham W. Taylor

Organizations: Vector Institute · Carleton University · Dalhousie University · University of British Columbia ·

Abstract

The landscape of self-supervised learning (SSL) is currently dominated by generative approaches (e.g. MAE) that reconstruct raw low-level data, and predictive approaches (e.g. I-JEPA) that predict high-level abstract embeddings. While generative methods are stable due to their reliable training targets based on ground-truth data, they are computationally inefficient for high-redundancy modalities like imagery, and their training objective does not prioritize learning high-level, conceptual features. Conversely, predictive methods often suffer from training instability due to their reliance on the non-stationary targets of final-layer self-distillation. We introduce Bootleg, a method that bridges this divide by tasking the model with predicting latent representations from multiple hidden layers of a teacher. This hierarchical objective forces the model to capture features at varying levels of abstraction simultaneously. We demonstrate Bootleg significantly outperforms comparable baselines (+10% vs. I-JEPA) on frozen probe classification of ImageNet-1K, iNaturalist-21, and VTAB, and semantic segmentation of ADE20K, Cityscapes, and COCO-Stuff.

Figures & tables

Figure 1: Multi-layer self-distillation with Bootleg. The teacher-encoder (blue), student-encoder (green), and predictor (orange) are ViTs, made of repeated transformer blocks. A schematic of a single transformer block is overlaid (bottom right). The teacher-encoder is an EMA of the student-encoder, and processes the full image. The student-encoder sees a subset of the image and must create embeddings of them to facilitate the predictor. The predictor processes the embeddings to predict representations at multiple layers within the teacher-encoder.

Figure 2: Bridging I-JEPA and MAE with targets across hidden layers. Left: We train ViT-S with I-JEPA, except for the target which we change to be a hidden layer (

x

-axis) of the teacher-encoder instead of its final output (blue curve) or in addition to the final output (red). We plot frozen attentive probe top-1 accuracy on IN-1k (

y

-axis) for the respective encoders. Middle: Similar, but for ViT-B to verify at larger scale. Right: We train ViT-S using MAE, except we add EMA for self-distillation and change the target to be a hidden layer of the EMA model instead of input pixels (blue) or in addition to image pixels (green). We compare to a single target with our masking strategy (cyan). In all cases, self-distillation of a hidden target is able to improve on predicting only input or output.

Figure 3: Training stability of Bootleg with multiple hidden-self-distillation targets (purple), final layer only self-distillation (blue) and image generation (green) over 1200 epochs ViT-S pretraining. A kNN monitor runs on 10% of IN-1k’s training set every 10 epochs; we show its accuracy, and effective rank of the 384-d data estimated using the Roy–Vetterli method ( Roy and Vetterli, 2007 ) and a 90% variance explained threshold. Evaluations shown for CLS token (solid) and patch-average (dashed) embeddings.

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Accompanies Fig. 2 of the main text. Bridging I-JEPA and MAE with targets across hidden layers. Left: As in Fig. 2 of the main text, we train ViT-S using with MAE, except we add EMA for self-distillation and change the target to be a hidden layer of the EMA model instead of the input pixels (blue) or in addition to the image pixels (green). We compare to a single target with our masking strategy (cyan), and to a single target but using the L1 loss (yellow). Using the L1 loss is insufficient to prevent training instabilities when using deeper layers as targets. Right: Similar, but for ViT-B with MAE config.

Panel of Figure 5 — Figure 5: Sample masks generated by I-JEPA and Bootleg’s masking strategies. Visible tokens are shown in blue. Prediction masks are shown in green/yellow/orange/red with one colour per mask rectangle. These prediction masks can overlap, leading to brighter shades of orange/yellow. Unused tokens (seen by the teacher-encoder, but not by either the student-encoder or predictor) are shown in black. For both I-JEPA and Bootleg, we show samples from worker-batches of 256 samples, generated with seeds 0,1,2,3. We show the first 4 examples from each batch (grouped horizontally). These samples illustrate how the visible tokens (blue) are always at the top of the image for I-JEPA’s masking, but are better distributed for Bootleg’s masks.

Panel of Figure 6 — Figure 6: Rates at which each token position within the 14 ×14 grid is visible and presented to the student-encoder. Top row: I-JEPA masking. Bottom row: Bootleg masking. We show the change in visibility of the tokens as the per-GPU batch size is increased from 4 (left) to 1024 (right). Red marks on the colorbars indicate the least and most frequent any token is visible for a given plot. Note that token visibility is more consistent across space (a flatter distribution) for Bootleg masks. The I-JEPA masks omit the bottom row and right column entirely, rarely present the centre of the image, and last-in-first-out truncation leads to the bottom four rows being seldom presented when the per-GPU batch size is larger.

Panel of Figure 7 — Figure 7: Token occurrence rates versus per-GPU batch size for I-JEPA and Bootleg masking strategies. Due to stochastic overlap of the predictor masks, the visible tokens need to be truncated to be the same length across all samples on the same GPU. Top left: Fraction of tokens which are visible (shown to the encoder) decreases as the batch size increases. This is because the target length for the visible tokens is the minimum size of the complement of the predictor masks within over all samples in the batch for the GPU worker, and a larger per-GPU batch size leads to more opportunities to generate four predictor rectangles with minimal overlap. Top right: Fraction of tokens predicted at least once across all masked out tokens per sample. Bottom left: Average fraction across the four centre tokens for the rate at which they are visible. Centre tokens infrequently visible for I-JEPA and are visible twice as often for Bootleg. Bottom right: Fraction of tokens within each image used either as a visible token passed to the encoder, or as a target token whose prediction loss is used to update the model. Due to random overlap and truncation, not all patch tokens are used for at least one task, reducing the efficiency of training. For each panel, the standard deviation is shown with error bars, and the min/max values with crosses.

			Rate (%)
(h, w)	Area	Aspect	I-JEPA	Bootleg
(5, 6)	30	0.83	26.5	18.8
(6, 5)	30	1.20	12.6	18.7
(5, 7)	35	0.71	0.0	20.0
(7, 5)	35	1.40	36.5	20.1
(6, 6)	36	1.00	21.8	22.4

Appendix

Table 8: Distribution of shapes of prediction target mask rectangles for I-JEPA and Bootleg. We assume a 14×14 grid of tokens (equivalent to a 224×224 image with 16×16 patch size), and sample 1 million mask sizes.

					IN-1k acc.			ADE20K mIoU
Method	GPU-BS	Seen (%)	Target (%)	Adj. (%)	Patch	CLS	X-Blk	kNN
I-JEPA	64	$27.0\pm 3.8$	$44.5\pm 2.3$	$23.6\pm 3.8$	55.2	N/A	64.7	10.9
	128	$25.6\pm 3.8$	$44.5\pm 2.2$	$22.3\pm 3.8$	55.6	N/A	63.9	10.7
	256	$24.3\pm 3.7$	$44.5\pm 2.2$	$21.1\pm 3.8$	53.4	N/A	62.3	10.5
	512	$23.1\pm 3.7$	$44.5\pm 2.2$	$19.9\pm 3.8$	52.3	N/A	62.6	10.1
	1024	$22.1\pm 3.8$	$44.5\pm 2.1$	$18.9\pm 3.9$	52.7	N/A	62.7	11.2

Appendix

Table 9: Effect of per-GPU batch size (GPU-BS) on model performance for I-JEPA and Bootleg. ViT-S, trained on IN-1k, 300 ep., (final hparams) with 2048 total batch size for all experiments. We alter the number of GPU workers to control the batch size per GPU worker. Rows

\Delta_{\text{max}}

show the maximum difference in performance over the GPU batch sizes for each metric.

	Hyperparameter	Value
Encoder	Drop path	0.0
Data	Input size	224×224
	Cropping	RandCrop(0.3, 1.0)
	Stretch	[ $\nicefrac{{3}}{{4}}$ , $\nicefrac{{4}}{{3}}$ ]
	Interpolation	Bicubic
	Transforms	HFlip( $p\!=\!0.5$ )

Appendix

Table 10: Hyperparameter configuration for IN-1k classification probes.

	Hyperparameter	Value
Encoder	Drop path	0.0
Data	Input size	224×224
	Interpolation	Bicubic
	Augmentations	None
Training	Loss	Cross-entropy
	Optimizer	AdamW

Appendix

Table 12: Hyperparameter configuration for VTAB frozen probes. We sweep an 11×11 grid of learning rates and weight decay values for each probe type.

	Hyperparameter	ViT-S, -B	ViT-L
Encoder	Drop path	0.1	0.2
Decoder	Type	MaskTransformer	MaskTransformer
	Depth	1 block	1 block
Data	Input size	512	512
	Cropping	RandScale(0.5, 2.0)	RandScale(0.5, 2.0)
	Stretch	[1, 1]	[1, 1]

Appendix

Table 19: Hyperparameter configuration for ADE20K segmentation fine-tuning. Learning rate (LR) reference values are shown relative to a batch size of 256 and linearly scaled for the actual batch size (32).

	# tgt	IN-1k Probe
Distillation targets	Layer	Patch	CLS	X-Blk
Block 1:4:12 out	4	62.2	62.4	73.0
Block 4:4:12 out & Tok.	4	60.3	61.6	72.4
Block 1:1:12 out	12	58.5	56.3	72.2
Block 1:1:12 out & Tok.	13	59.2	59.2	72.4
Block 1:1:12 mid, out	24	59.3	59.9	72.8

Appendix

Table 23: Choosing whether to predict the tokenizer’s output. We consider whether to predict the tokenizer output (Tok.) instead of the first block for spaced targets, or in addition to all blocks. We find it is generally disadvantageous to use the tokenizer as a self-distillation target in these scenarios. ViT-S, trained on IN-1k, 300 ep., prototype hparams.

		IN-1k Probe			ADE20K
Epochs	Targets	Patch	CLS	X-Blk	kNN
50	b1, 4, 8, 12	52.9	53.0	66.3	14.3
	b12	55.0	53.8	67.0	15.9
100	b1, 4, 8, 12	59.0	59.4	70.0	17.7
	b12	61.0	60.2	69.9	17.1
200	b1, 4, 8, 12	65.5	66.3	73.2	20.9

Appendix

Table 27: Effect of pretraining duration. ViT-S/16, IN-1k, final hyperparameters.

Figure 8: IN-1k X-Blk top-1 accuracy as a function of pretraining duration for Bootleg (multi-target, blocks

\{1,4,8,12\}

) and the block-12-only ablation. ViT-S/16, IN-1k, final hyperparameters. Each datapoint is a separate run, trained from scratch to the budgeted number of epochs. Bootleg continues to improve through 600 epochs and plateaus; the block-12-only variant peaks near 100 epochs and then degrades with longer training.

		IN-1k Probe			ADE20K
$\tau$	Targets	Patch	CLS	X-Blk	kNN
$0.988$	b1, 4, 8, 12	66.6	67.1	73.6	21.2
	b12	42.1	41.3	60.6	0 1.8
$0.997$	b1, 4, 8, 12	67.5	68.3	74.1	21.5
	b12	51.5	49.3	66.5	0 9.7
$0.9985$	b1, 4, 8, 12	67.8	68.9	74.4	22.0

Appendix

Table 28: Teacher EMA momentum

\tau

sensitivity. ViT-S/16, 300 ep. on IN-1k, final hyperparameters. The default

\tau=0.9985

is highlighted.

		IN-1k Probe			ADE20K
$\beta_{2}$	Targets	Patch	CLS	X-Blk	kNN
$0.9$	b1, 4, 8, 12	67.1	68.2	74.0	21.3
	b12	57.6	54.7	68.9	13.2
$0.95$	b1, 4, 8, 12	67.8	68.9	74.4	22.0
	b12	56.7	54.6	68.6	13.1
$0.999$	b1, 4, 8, 12	67.2	68.5	74.0	22.2

Appendix

Table 29: Optimizer

\beta_{2}

sensitivity. ViT-S/16, 300 ep. on IN-1k, final hyperparameters. The default

\beta_{2}=0.95

is highlighted.

		IN-1k Probe
Arch	Distil. method	Patch	CLS	X-Attn	X-Blk
ViT-S	Baseline	62.2	62.4	71.1	73.0
	+target 4 reg.	62.2	62.5	70.9	72.7

Appendix

Table 30: Effect of using hidden embeddings of register tokens as self-distillation targets, in addition to hidden embeddings of patch tokens. ViT-S, trained 300ep on IN-1k, prototype hparams.

			IN-1k Probe
Arch	Distillation targets	Predictor	Patch	CLS	X-Blk
ViT-S	Block 1:4:12 out	SA	62.2	62.4	73.0
	Block 1:4:12 out	XA, WFM	62.2	61.1	72.8
	Block 1:1:12 res., out	SA	62.4	61.9	73.1
	Block 1:1:12 res., out	XA	54.7	46.5	69.7
	Block 1:1:12 res., out	XA, WFM	62.6	59.1	72.6

Appendix

Table 32: Performance when using cross-attention predictor and weighted feature map (CrossMAE config) instead of self-attention (MAE config). Performance when using either the MAE/I-JEPA method of self-attention (SA) for the predictor network, or the CrossMAE method of cross-attention (XA) against a weighted feature map (WFM). The WFM uses a learnable weighted average of the outputs from the tokenizer and outputs of each transformer block in the encoder as the input to the predictor. ViT-S, 300 ep., prototype hparams.

	# targets	IN-1k Probe
Target standardization	Layers	Patch	CLS	X-Blk
Separately standardized	4	62.2	62.4	73.0
Jointly standardized	4	57.8	57.2	69.8

Appendix

Table 33: We explored whether to standardize the targets with a z-score separately applied to each target embedding vector from each hidden layer used, or with a z-score taken across the concatenation of all targets. ViT-S, trained on IN-1k, 300 ep. prototype hparams.

Figure 9: Inter-layer Pearson correlation matrices for Bootleg ViT-S/16 and ViT-B/16 encoders. Each cell shows the mean correlation between embedding vectors at corresponding spatial positions across 10,000 images. Correlation decays with layer distance, and the final block is highly distinct from early blocks.

Figure 10: Inter-layer CKA similarity matrices. CKA captures shared representational geometry and yields higher values than Pearson correlation for distant layer pairs, but the two measures are strongly rank-correlated ( $\rho\approx 0.89$ ).

Figure 11: Off-diagonal decay of Pearson correlation and CKA as a function of layer distance

k

. CKA decays more slowly, remaining above 0.15 even at the maximum distance, while Pearson correlation drops near zero. Error bars show standard deviation across all layer pairs at each distance.

Figure 12: Pearson correlation of each encoder layer with the four Bootleg distillation target layers (blocks 1, 4, 8, and 12). Each target’s profile peaks at its own block and decays with distance, confirming that the targets capture non-redundant representations.

Figure 13: Azimuthally averaged spatial correlation vs. distance for selected encoder layers. Early layers exhibit localized correlations that decay with distance. Block 12 shows near-uniform high correlation, indicating a globally coherent representation. The post-LayerNorm final output partially restores spatial structure.

Figure 14: Per-block IN-1k kNN top-1 accuracy for ViT-S/16 and ViT-B/16, evaluated at the tokenizer output (tok), every transformer block (b1–b12), and the post-LayerNorm final output (fin). We show patch-averaged embeddings (solid) and CLS embeddings (dashed). All methods track each other closely through roughly block 9 and only diverge in the deepest blocks: Bootleg’s per-block accuracy continues to rise through block 12, while baselines either flatten (MAE, CrossMAE, I-JEPA) or decay (data2vec 2.0).

Figure 15: Per-target MSE loss and target/activation statistics over 1200 epochs of ViT-S/16 pretraining. Left: multi-target Bootleg with targets at blocks 1, 4, 8, 12. Right: single-target b12-only. Rows: per-target MSE loss; top-1 SVD variance fraction of the targets; target excess kurtosis (log y-axis); standard deviation of target token L2-norms; activation kurtosis at the target site (log y-axis). Lines are coloured by target depth on a plasma sweep (yellow for early, purple for deep).

	top-1 SVD frac.		kurtosis
Block	early	late	early	late
1	0.55	0.14	123	16
4	0.65	0.20	166	22
8	0.48	0.10	93	7
12	0.33	0.09	49	4

Appendix

Table 34: Target distribution statistics for multi-target Bootleg (ViT-S/16, 1200 epochs), averaged over epochs

[100,200)

(early training) and

[1000,1100)

(late training). Higher top-1 SVD fraction means more variance concentrated along a single direction (more anisotropic). Higher kurtosis means heavier-tailed activations (more outlier-driven).

Figure 16: Pretraining loss across epoch budgets, ViT-S/16. Lines are coloured by total epoch budget on a viridis sweep (bright yellow = 50 ep, dark purple = 1200 ep). Left: multi-target Bootleg. Right: single-target b12-only.

Figure 17: kNN top-1 accuracy across epoch budgets. Solid: CLS feature; dotted: patch-average feature. Bootleg scales monotonically with budget; b12-only collapses on long schedules.

Figure 18: CLS-feature effective rank ( Roy and Vetterli, 2007 ) across epoch budgets, out of

D=384

Self-Supervised Visual Representation Learning

Self-supervised visual representation learning trains image, video, and 3D encoders on unlabeled data to produce transferable visual features.

See all 234 papers

Scalable Patch-Level Self-Supervised Learning
Oct 7, 2026Maximilian Seitzer, Gabriele Trivigno, Antonín Vobecký +5Self-Supervised Learning Self-Supervised Visual Representation Learning
Meta FAIR
From Alignment to Prediction: A Study of Self-Supervised Learning and Predictive Representation Learning
Apr 15, 2026Mintu Dutta, Ritesh Vyas, Mohendra RoySelf-Supervised Learning Predictive Representation Learning
Department of Information and Communication Technology, School of Technology, Pandit Deendayal Energy University, Gandhinagar, Gujarat 382007, India
Learning to Perceive "Where": Spatial Pretext Tasks for Robust Self-Supervised Learning
May 11, 2026Yang Shen, Yusen Cai, Weronika Hryniewska-Guzik +2Visual Spatial Reasoning Self-Supervised Visual Representation Learning
Nanyang Technological University, Singapore · Warsaw University of Technology, Poland

Oct 7, 2026J/K move · Enter open · S save

Scalable Patch-Level Self-Supervised Learning

Maximilian Seitzer, Gabriele Trivigno, Antonín Vobecký, Seungeun Yi+4

Meta FAIR

Self-Supervised Learning Self-Supervised Visual Representation Learning Representation Learning Multi-View Learning

Open paper AskPDF arXiv

Self-supervised learning (SSL) at scale produces powerful visual representations. However, most scalable SSL methods rely on ad hoc combinations of multiple objectives and stabilization mechanisms. Taking a step back, we ask if we can design a high-performing, yet principled SSL algorithm. Starting from the multi-view assumption, stipulating that task-relevant content is captured by the information common to different views, we construct an information-theoretic objective decomposing into interpretable terms. This derivation yields JEM, a student-teacher method that learns by aligning corresponding patch representations across views, explicitly regularized by information and structure preservation losses. JEM trains stably from 300M to 7B parameters, and, to our knowledge, is the first latent-space patch-level method demonstrated at 7B scale. Across all scales, JEM reaches strong performance on both global and dense probing tasks, on segmentation benchmarks consistently surpassing the DINOv2 algorithm, an influential foundation for today's strongest visual SSL methods. Notably, at 7B parameters, it exceeds the performance of DINOv3 on panoptic segmentation, despite being trained on

12\times

less data without refinement stages. These results demonstrate that we can indeed design an SSL algorithm that learns strong representations, is principled and stable.

Abstract

Self-Supervised Visual Representation Learning

Explore similar work

Scalable Patch-Level Self-Supervised Learning

From Alignment to Prediction: A Study of Self-Supervised Learning and Predictive Representation Learning

Learning to Perceive "Where": Spatial Pretext Tasks for Robust Self-Supervised Learning