The Devil is in the Spectrum Bias: Spectrum-Balanced Feature Matching for Robust Representation Distillation

Kuniaki Saito; Yoshitaka Ushiku

cs.CVSep 28, 2026

The Devil is in the Spectrum Bias: Spectrum-Balanced Feature Matching for Robust Representation Distillation

Authors: Kuniaki Saito, Yoshitaka Ushiku

Organizations: OMRON SINIC X Corporation

Abstract

Large visual foundation models have demonstrated remarkable transferability across a wide range of downstream tasks. To deploy such models efficiently, feature matching has become a popular knowledge distillation approach that transfers teacher representations to smaller student models without requiring labeled data. However, we show that the conventional feature matching objective with L2-distance is inherently biased toward reconstructing dominant spectral directions of the teacher representation, while under-optimizing low-variance directions that often contain task-relevant information. To address this, we propose Spectrum-Balanced Feature Matching, SpecMatch, a simple objective that adaptively emphasizes under-optimized spectral directions while preserving the relative importance of dominant directions. SpecMatch is easy to implement and introduces negligible computational overhead. Extensive experiments on image recognition demonstrate that SpecMatch consistently improves downstream adaptation across diverse tasks, including image classification, anomaly detection, medical image analysis, and domain generalization. In particular, SpecMatch outperforms conventional feature matching in 40 of 42 teacher--student and training-setting combinations, while consistently improving over the original student model in all settings. We further demonstrate that the proposed objective generalizes beyond vision, improving downstream performance across six protein understanding tasks.

Figures & tables

Figure 1: (Left) : Contribution of principal component (PC) groups to downstream tasks. PCs are sorted by explained variance and divided into three equal-sized groups. Different tasks rely on different spectral regions, motivating the preservation of a broad representation spectrum during distillation. (Right) : Variance-normalized reconstruction error of vanilla L2 feature matching. Reconstruction deteriorates toward higher-rank, lower-variance PCs, revealing a bias toward high-variance components that may discard information useful for downstream tasks. We further provide evidence in Fig. 7 that lower-variance PCs do not merely represent noise but can substantially improve performance on certain downstream tasks.

Figure 2: (Left) : Description of the proposed objective. (Right) : The illustration of the error (y-axis) on different principal components (x-axis).

		General		OCR		Fine		Remote		Medical		Anomaly		Generalization
	Average	CIFAR100	CIFAR10	SVHN	GTSRB	NABirds	CUB	Resisc	Eurosat	OrganCMNIST	Camelyon17	VisA	MVTec	iWildCam	FmoW
	# classes	100	10	10	43	555	200	45	10	11	2	2	2	186	62
PE-Core-Large $\rightarrow$ PE-Core-Tiny
Teacher	86.5	93.1	99.4	73.6	93.7	87.3	90.6	96.1	97.4	82.9	90.1	81.9	96.3	79.2	50.1
Student	76.0	76.2	92.1	57.7	81.5	59.7	71.1	88.9	94.6	82.9	89.0	75.8	86.3	71.0	36.5
Logits-KD ( Hinton et al., 2015 )	80.9	79.5	95.1	82.2	96.8	69.9	71.6	93.2	98.4	92.4	88.8	69.2	71.9	74.1	45.3

Table 1: Performance comparison across 14 datasets, including general recognition, OCR, fine-grained recognition, medical image classification, and anomaly detection benchmarks. The cell where the distillation model underperforms the student linear probe is highlighted with red while the best model is highlighted with bold .

Panel of Figure 3 — Figure 3: Difference in reconstruction errors across the principal components. We compute the mean reconstruction error between the teacher representation and the student, and report ${{e}^{\mathrm{FM}}_{k}}-{{e}^{\mathrm{ours}}_{k}}$ normalized by the teacher variance $\lambda_{k}$ . A higher value shows that SpecMatch reconstructs the teacher representation more accurately than the feature matching baseline.

Teacher	Student	Params	Initialization	Method	5-shot	10-shot	Full-shot
PE-Core-G14	ViT-Tiny	5.5M	Scratch	DINO	31.3	37.2	54.6
				FM	48.4	51.6	57.6
				SpecMatch	50.2	52.7	58.4
ConvNEXT-XLarge	ConvNEXT-Tiny	27.8M	Scratch	DINO	36.4	42.9	54.7
				FM	57.6	60.6	67.1
				SpecMatch	61.2	63.9	70.0

Table 2: Experiments on distillation with ImageNet. All models are evaluated with classification accuracy (%) after linear-probing on the corresponding labeled subset.

Panel of Figure 4 — Figure 4: Linear probing with different shots. Our approach consistently improves performance over the base student model while vanilla feature matching sometimes fails in improving as in NABirds full-shot case (a).

$\beta^{-1}$	ImageNet	NAB	CIF100	SVHN
1 (FM)	72.1	60.7	76.2	87.2
2	72.7	62.3	77.5	89.4
3	73.1	64.2	77.6	89.5
4	72.9	64.9	77.8	89.9
5	72.9	64.9	78.2	89.4

Table 3: (Left) : Sensitivity analysis for

\beta^{-1}

\beta=1

corresponds to vanilla feature matching, while

\beta^{-1}=2

and

3

consistently improve performance. (Right) : Comparison among different objectives, including L1, cosine distance, and L2 + whitening. SpecMatch is the best or the second best approach in these results.

Method	ViSA # Anomaly Samples
Method	100	200	500	1000
Teacher	0.898
Student	0.825
FM	0.835	0.851	0.863	0.880
SpecMatch	0.851	0.867	0.873	0.896

Table 4: Study on imbalanced data distribution using PE-Core-G14 and PE-Core-Tiny as teacher and student, respectively. (Left) : Anomaly detection performance (AUROC) on ViSA with different numbers of anomaly samples used for distillation. (Right) : Results on iNaturalist2018 across head, medium, and tail classes.

	Fold: 1195 classes		GO-MF: 489 classes		GO-CC: 320 classes		EC: 585 classes		Cat. Eff. : regression		DeepLoc2: 10 classes
Method	Val	Test	Val	Test	Val	Test	Val	Test	Val	Test	Val	Test
Teacher (ESM-2 650M)	$75.0_{\pm 0.1}$	$73.1_{\pm 0.3}$	$55.6_{\pm 0.2}$	$55.5_{\pm 0.4}$	$45.7_{\pm 0.4}$	$47.6_{\pm 0.2}$	$58.8_{\pm 0.2}$	$59.8_{\pm 0.2}$	$7.5_{\pm 0.4}$	$10.6_{\pm 0.5}$	$71.7_{\pm 0.5}$	$70.9_{\pm 0.1}$
Student (ESM-2 6M)	$49.4_{\pm 0.3}$	$51.2_{\pm 0.2}$	$34.5_{\pm 0.4}$	$36.6_{\pm 0.3}$	$30.4_{\pm 0.5}$	$29.7_{\pm 0.2}$	$38.7_{\pm 0.1}$	$40.1_{\pm 0.3}$	$14.3_{\pm 0.3}$	$17.0_{\pm 0.5}$	$64.3_{\pm 0.1}$	$62.9_{\pm 0.1}$
L1	$52.6_{\pm 0.3}$	$53.1_{\pm 0.3}$	$36.5_{\pm 0.1}$	$37.5_{\pm 0.2}$	$35.2_{\pm 0.1}$	$34.8_{\pm 0.1}$	$38.3_{\pm 0.2}$	$40.9_{\pm 0.4}$	$20.7_{\pm 1.3}$	$22.0_{\pm 0.8}$	$\mathbf{70.1}_{\pm 0.1}$	$68.4_{\pm 0.1}$
L2	$51.9_{\pm 0.2}$	$52.2_{\pm 0.3}$	$36.8_{\pm 0.2}$	$38.0_{\pm 0.2}$	$35.2_{\pm 0.2}$	$35.3_{\pm 0.3}$	$39.9_{\pm 0.4}$	$41.0_{\pm 0.4}$	$18.8_{\pm 1.3}$	$27.0_{\pm 0.8}$	$69.6_{\pm 0.1}$	$68.5_{\pm 0.1}$
SpecMatch	$\mathbf{54.4}_{\pm 0.6}$	$\mathbf{55.6}_{\pm 0.5}$	$\mathbf{38.6}_{\pm 0.2}$	$\mathbf{39.6}_{\pm 0.1}$	$\mathbf{35.8}_{\pm 0.2}$	$\mathbf{35.8}_{\pm 0.2}$	$\mathbf{41.7}_{\pm 0.3}$	$\mathbf{42.0}_{\pm 0.6}$	$\mathbf{21.3}_{\pm 0.6}$	$\mathbf{30.7}_{\pm 1.0}$	$69.9_{\pm 0.3}$	$\mathbf{69.2}_{\pm 0.1}$

Table 5: Protein downstream performance on six PFMBench tasks. Results are scaled by 100 and reported as mean

\pm

standard deviation over three runs. Higher is better for all metrics.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Eigenvalues (x-axis) and reconstruction error.

Figure 6: Reconstruction error along principal components.

Dataset	Train	Test	# Classes
CIFAR-100	50,000	10,000	100
CIFAR-10	50,000	10,000	10
SVHN	73,257	26,032	10
GTSRB	26,640	12,630	43
RESISC45	18,900	6,300	45
EuroSAT	21,600	5,400	10

Appendix

Table 6: Dataset statistics.

Dataset	Task	Classes	Train	Val	Test	Epochs
Fold prediction	classification	1195	13034	1628	1630	20
GO molecular function	multi labels classification	489	22291	2785	2787	20
GO cellular component	multi labels classification	320	11196	1398	1400	20
Enzyme commission number	multi labels classification	585	12928	1615	1616	60
Enzyme catalytic efficiency	regression	-	10363	1290	1298	60
DeepLoc2Multi	multi labels classification	10	21949	2743	2744	50

Appendix

Table 7: Statistics of the protein datasets used in Table 5 .

Role	Model	Source	Pre-training	Input	Params.
Teacher	DINOv2 ViT-g/14	DINOv2	DINOv2	518	1.1B
	PE-Core-G14-448	Meta PE	Perception Encoder	448	1.9B
	CLIP ConvNeXt-XXLarge	OpenCLIP	LAION-2B	256	846.5M
Student	MobileNetV3-Small	timm	IN-1K	224	1.5M
	PE-Core-T16-384	Meta PE	Perception Encoder	384	10M
	ConvNeXt-Tiny	timm	Scratch	224	27.8M

Appendix

Table 8: Overview of teacher and student vision models used in our experiments. Parameter counts correspond to the vision backbone only, excluding classification heads and text encoders.

Panel of Figure 7 — Figure 7: Relationship between linear probe accuracy and the number of used principal components. We measure the linear probe accuracy in using top K% principal components.

Method	CIFAR100	CIFAR10	SVHN	GTSRB	NABirds	CUB	RESISC	EuroSAT	OrganCMNIST	Camelyon17	iWildCam	FMoW
Supervised FT	76.5	93.8	95.2	95.4	50.1	55.8	93.0	98.4	93.4	90.1	58.5	41.0
SpecMatch ( $T$ : PE-Core)	77.6	95.2	89.5	94.6	64.2	72.5	93.6	98.2	90.7	93.9	62.4	42.1
$\Delta$	+1.1	+1.5	-5.7	-0.8	+14.1	+16.7	+0.6	-0.2	-2.7	+3.8	+3.9	+1.1
SpecMatch ( $T$ : DINOv2)	78.1	95.5	82.3	88.1	56.8	69.0	92.8	98.0	91.0	93.4	61.1	39.5
$\Delta$	+1.6	+1.8	-12.9	-7.3	+6.7	+13.2	-0.2	-0.4	-2.4	+3.3	+2.6	-1.5

Appendix

Table 9: Comparison with fully supervised fine-tuning. All methods use the same pretrained MobileNetV3-Small student.

\Delta

denotes the performance difference between SpecMatch and supervised fine-tuning.

Method	Oxford									Paris
	Easy			Medium			Hard			Easy			Medium			Hard
	mAP	P@1	P@10	mAP	P@1	P@10	mAP	P@1	P@10	mAP	P@1	P@10	mAP	P@1	P@10	mAP	P@1	P@10
ConvNEXT-XXlarge $\rightarrow$ ConvNEXT-Tiny
DINO	21.9	41.2	27.7	15.0	45.7	27.3	3.4	17.1	5.3	64.8	95.7	87.7	49.9	97.1	91.3	24.7	68.6	56.0
FM	17.6	29.4	20.9	13.8	30.0	18.7	3.7	10.0	7.0	53.4	87.1	81.0	40.5	88.6	82.7	17.4	54.3	41.6
SpecMatch	19.7	30.9	22.4	15.7	38.6	22.3	4.9	18.6	8.6	55.5	87.1	81.6	43.7	88.6	84.4	21.5	67.1	49.9

Appendix

Table 10: Instance recognition results on the Oxford and Paris datasets. A ConvNeXt-Large model pre-trained with image-text data is used as the teacher, while ConvNeXt-Tiny serves as the student.

Method	CIFAR-10	SVHN	NABirds	CUB	RESISC	ImageNet
FM	94.8	87.6	62.6	71.3	92.6	72.1
Static	95.0	88.3	63.7	71.7	93.6	73.2
Dynamic	95.2	89.5	64.2	72.5	93.6	73.1

Appendix

Table 11: Comparison between static and dynamic weighting.

Explore similar work

Sep 28, 2026cs.CV

PSM: Dataset Distillation Based on Precise Statistical Matching by Difficulty

Dataset distillation (DD) condenses a large original dataset into a small distilled dataset with high training utility. Decoupled statistical matching methods substantially reduce distillation time and memory overhead while achieving strong performance. However, they typically supervise all distilled samples using running statistics estimated from the entire original dataset. These statistics mainly capture the average feature distribution while overlooking differences in sample difficulty, limiting their ability to characterize the difficulty structure of the original data. To address this issue, we propose Precise Statistical Matching (PSM) by difficulty. After pretraining, PSM uses the Global Precision Score (GPS) to estimate image difficulty, ranks the samples within each class, and partitions each class into IPC (images per class) difficulty groups. During distillation, Statistics Updated Again (SUA) updates the teacher's batch normalization (BN) running statistics through forward passes on original samples from each group, providing difficulty-specific supervision for the corresponding distilled batch. Meanwhile, Initial Sample Screening (ISS) initializes distilled samples using original images from the corresponding difficulty group, providing an effective starting point for precise matching. Experiments across multiple datasets and model architectures demonstrate that PSM broadens the difficulty range of distilled samples and improves downstream performance in most evaluated settings. Code will be released.

Hongxu Ma, Guang Li, Shijie Wang +6

Zhejiang University · Hokkaido University · The University of Queensland +3

Diffusion-Based Dataset Distillation Dataset Distillation Batch Normalization

Ask PDF

Apr 24, 2026cs.CV

Distilling Vision Transformers for Distortion-Robust Representation Learning

Self-supervised learning has achieved remarkable success in learning visual representations from clean data, yet remains challenging when clean observations are sparse or not available at all. In this paper, we demonstrate that pretrained vision models can be leveraged to learn distortion-robust representations, which can then be effectively applied to downstream tasks operating on distorted observations. In particular, we propose an asymmetric knowledge distillation framework in which both teacher and student are initialized from the same pretrained Vision Transformer but receive different views of each image: the teacher processes clean images, while the student sees their distorted versions. We introduce multi-level distillation that aligns global embeddings, patch-level features, and attention maps and show that the student is able to approximate clean-image representations despite never directly accessing clean data. We evaluate our approach on image classification tasks across several datasets and under various distortions, consistently outperforming existing alternatives for the same amount of human supervision.

Konstantinos Alexis, Giorgos Giannopoulos, Dimitrios Gunopulos

Department of Informatics and Telecommunications, National and Kapodistrian University of Athens, Greece · National Observatory of Athens, Greece

Self-Supervised Vision Transformers Vision Transformer Knowledge Distillation Self-Supervised Learning

Ask PDF

Jul 18, 2026cs.CV

Dataset Distillation by Influence Matching

We revisit dataset distillation from an outcome-centric perspective. Rather than aligning process surrogates (per-step gradients or training trajectories), Influence Matching (Inf-Match) aligns the final outcome of training: it learns a compact synthetic set whose effect on the converged parameters matches that of the full dataset. Concretely, we introduce a fully differentiable, sample-level influence estimator that quantifies parameter shifts from adding or removing data, without time-consuming inverse-Hessian products or convexity assumptions. The estimator runs in linear time by unrolling the optimization dynamics and applying a first-order Taylor approximation. We then learn the synthetic set by minimizing the mismatch between its influence and that of the real dataset, yielding outcome alignment rather than heuristic process imitation. Inf-Match delivers the best accuracy across standard classification benchmarks. For instance, on Tiny-ImageNet (IPC=10), Inf-Match attains 31.5%, a +4.7% improvement over NCFM. Beyond classification, Inf-Match scales to vision-language distillation on Flickr30K, outperforming strong process-matching baselines. For instance, with 200 to 1000 synthetic samples, our method achieved a leading impressive average on image/text retrieval tasks, higher than NCFM by 2.5%. The code will be released via https://github.com/hrtan/infmatch.

Haoru Tan, Wang Wang, Sitong Wu +5

1HKU · 2CUHK · 3Stanford

Diffusion-Based Dataset Distillation Dataset Distillation Distributional Alignment Influence

Ask PDF