Breaking the Scale Barrier: One-Shot Knowledge Transfer via Frequency Transform

Jianlu Shen; Fu Feng; Yucheng Xie; Jiaqi Lv; Xin Geng

cs.LGMar 8, 2026

Breaking the Scale Barrier: One-Shot Knowledge Transfer via Frequency Transform

Authors: Jianlu Shen, Fu Feng, Yucheng Xie, Jiaqi Lv, Xin Geng

Organizations: School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China

Abstract

Transferring knowledge by fine-tuning large-scale pre-trained networks has become a standard paradigm for downstream tasks, yet the knowledge of a pre-trained model is tightly coupled with monolithic architecture, which restricts flexible reuse across models of varying scales. In response to this challenge, recent approaches typically resort to either parameter selection, which fails to capture the interdependent structure of this knowledge, or parameter prediction using generative models that depend on impractical access to large network collections. In this paper, we identify the low-frequency components of model weights as the concrete carrier of foundational, task-agnostic knowledge, its learngene", and validate this by demonstrating its efficient inheritance by downstream models and tasks. Based on this insight, we propose FRONT (FRequency dOmain kNowledge Transfer), a novel framework that uses the Discrete Cosine Transform (DCT) to isolate the low-frequency learngene". This learngene can be seamlessly adapted to initialize models of arbitrary size via simple truncation or padding, a process that is entirely training-free. For enhanced performance, we propose an optional low-cost refinement process that introduces a spectral regularizer to further improve the learngene's transferability. Extensive experiments demonstrate that FRONT achieves the state-of-the-art performance, accelerates convergence by up to $15\times$ in vision tasks, and reduces training FLOPs by an average of 40.5% in language tasks. Code is available at https://github.com/LUcy0505/FRONT.

Figures & tables

Figure 1: (a) Five different-scaled models initialized from the same DeiT are fine-tuned on the same downstream tasks. (b) A pre-trained DeiT is fine-tuned on five different downstream tasks. We plot the cosine similarity between the low-frequency components of the fine-tuned weights and their original pre-trained state over the training process. Each dashed line represents an individual fine-tuned model, while the solid line indicates their average.

Figure 2: The overview of FRONT and FRONT + . The learngenes are obtained from either FRONT via the open available pre-trained models or FRONT + using source models refined with our frequency regularization. The core process involves transforming weight modules via DCT to extract task-agnostic knowledge. Finally, these learngenes are used to initialize diverse target models through the IDCT, which flexibly accommodate varying architectures via zero-padding or truncation.

				$W_{\text{192}}$ (DeiT-Ti)						$W_{\text{384}}$ (DeiT-S)						$W_{\text{768}}$ (DeiT-B)
				$L_{4}$	$L_{6}$	$L_{8}$	$L_{10}$	$L_{12}$		$L_{4}$	$L_{6}$	$L_{8}$	$L_{10}$	$L_{12}$		$L_{4}$	$L_{6}$	$L_{8}$	$L_{10}$	$L_{12}$
Methods		Epoch	Para.	2.2	3.1	4.0	4.9	5.8	Para.	7.9	11.5	15.0	18.6	22.2	Para.	29.9	44.1	58.3	72.5	86.7
Direct	He-Init	0	0	34.7	40.6	43.7	46.8	48.3	0	42.2	49.4	52.1	53.7	55.5	0	47.9	53.1	54.4	55.0	56.7
	Mimetic	0	0	35.1	40.2	43.2	46.3	48.1	0	43.3	49.1	53.0	54.1	55.6	0	50.2	54.3	56.5	58.5	58.6
	Wt Select	0	3.9	46.1	52.3	55	56.8	58.9	15.0	50.1	56	57.5	61.9	63.3	58.2	56.7	61.1	63.6	64.8	65.9
	Heur-LG	0	1.7	41.5	47.4	50.5	53.5	55.5	6.1	52.3	57.3	61.7	64.4	65.9	22.8	60.5	68.7	72.2	73.6	74.0

Table 1: Performance of models initialized with different depths on ImageNet-1K for 10 epochs. Para.(M) denotes the number of parameters, specified per model size (rows) and as the average transferred during initialization (columns). “Epoch” indicates the extra training epochs for knowledge merging or pre-training. “ / ” denotes failed initialization, and “N/A” indicates that the metric is not applicable. The best result in each column is highlighted in bold, while the second-best is underlined.

				$L_{\text{6}}$
				$W_{\text{384}}$	$W_{\text{576}}$	$W_{\text{768}}$	$W_{\text{1152}}$	$W_{\text{1536}}$
Methods		Epoch	Para.	11.5	25.1	44.1	98.0	173.1
Direct	He-Init	0	0	49.4	51.5	53.1	46.6	31.3
	Mimetic	0	0	49.1	48.0	54.3	47.7	33.0
	Wt Select	0	26.8	48.6	50.7	55.4	/	/
	LiGO	0	10.0	63.6	63.1	69.5	70.0	73.7

Table 2: Performance of initializing models with variable widths on ImageNet-1K.

		DeiT-Ti, 3.0M									DeiT-S, 11.3M
Methods		Para.	Flow.	CUB	Cars	C 10	C 100	Food	iNat.	Aver.	Para.	Flow.	CUB	Cars	C 10	C 100	Food	iNat.	Aver.
Direct	He-Init	0	53.9	26.1	19.9	92.4	68.3	68.4	52.3	54.5	0	57.2	27.3	23.8	94.0	66.5	70.6	54.0	56.2
	Mimetic	0	52.1	35.0	20.5	88.9	63.4	66.9	49.0	53.7	0	57.4	39.6	34.2	91.6	65.7	67.1	52.2	58.3
	Wt Select	2.9	55.0	34.4	21.7	92.5	67.0	67.4	51.2	55.6	11.0	58.3	33.1	28.1	94.1	68.1	69.0	54.2	57.8
	Heur-LG	1.5	64.7	44.6	37.7	94.0	71.1	74.7	57.4	63.5	5.7	69.1	48.0	51.2	95.1	72.8	76.8	59.3	67.5
	Cluster-LG	2.4	62.8	44.8	36.9	94.1	74.5	78.6	56.4	64.0	5.1	65.5	45.4	46.3	95.4	74.6	78.4	60.4	66.6

Table 3: Performance of models on downstream datasets. “Para.(M)” is the average parameter transferred during model initialization.

Panel of Figure 3 — Figure 3: Comparison of different direct initialization methods during full training on DeiT-Ti and DeiT-S.

Task	Scratch	KD	FRONT
SST-2	78.97	77.75	82.44 ( $\uparrow$ 3.47)
MNLI	63.65	64.81	74.08 ( $\uparrow$ 9.27)
MRPC	66.42	64.14	68.46 ( $\uparrow$ 2.04)
CoLA	8.14	9.14	17.38 ( $\uparrow$ 8.24)
QNLI	57.21	58.00	70.86 ( $\uparrow$ 13.65)
QQP	81.50	80.07	84.93 ( $\uparrow$ 4.86)

Table 4: GLUE benchmark results on BERT-S.

	ResNet50			ResNet152
	Rand.	FRONT		Rand.	FRONT
Image.	74.1	76.1	$\uparrow$ 2.0	76.8	77.2	$\uparrow$ 0.4
Flow.	51.3	91.3	$\uparrow$ 40.0	28.0	87.7	$\uparrow$ 59.7
CUB	40.7	68.9	$\uparrow$ 28.2	22.8	68.2	$\uparrow$ 45.4
Cars	48.6	88.2	$\uparrow$ 39.6	29.3	87.4	$\uparrow$ 58.1
C10	94.8	95.6	$\uparrow$ 0.8	95.6	95.7	$\uparrow$ 0.1

Table 5: Performance comparison on ResNet50 and ResNet152.

Panel of Figure 4 — Figure 4: Results of pretraining BRET-S, RoBERTa-S and GPT-S. FRONT can achieve the highest savings in FLOPs with 47.5% for BERT-S, 42.9% for RoBERTa-S, and 31.0% for GPT-S from the Scratch models.

$\gamma_{d}\setminus r$	0.33	0.50	0.67	0.83	1.00
–	60.63	62.74	64.79	65.06	66.19
$\nicefrac{{1}}{{16}}$	66.61	66.64	66.65	66.58	66.60
$\nicefrac{{1}}{{8}}$	66.97	67.22	67.02	67.11	67.13
$\nicefrac{{1}}{{4}}$	67.08	67.58	68.02	67.92	67.88
$\nicefrac{{1}}{{2}}$	66.74	66.92	66.00	66.72	66.94
1	66.57	66.76	66.60	66.69	66.75

Table 6: Top-1 accuracy (%) on ImageNet-1K as a function of

r

and

\gamma_{d}

. And ‘–’ corresponds to a direct application of FRONT without refinement.

$W_{qkv}$	$W_{o}$	$W_{mlp}$	norm	SST-2	MNLI	MRPC	CoLA	QNLI	QQP	STS-B	Avg
				78.97	63.65	66.42	8.14	57.21	81.50	17.27	53.31
✓				82.11	72.14	68.62	4.63	66.95	84.71	43.19	60.34
	✓			82.34	66.70	68.87	10.23	62.57	81.81	42.79	59.33
		✓		80.73	67.29	69.61	12.34	60.97	78.78	36.85	58.08
			✓	80.85	65.69	67.40	14.35	61.76	81.50	42.65	59.17
✓	✓			81.19	72.17	70.34	16.42	69.98	84.82	50.35	63.61

Table 7: Performance comparison of varying module configurations on BERT-B to BERT-S across GLUE benchmarks. The transferable low-frequency knowledge in language models is primarily concentrated in the self-attention weights (

W_{qkv}

and

W_{o}

$W_{qkv}$	$W_{o}$	$W_{mlp}$	norm	Acc (%)
				34.7
✓				46.5
	✓			46.6
		✓		54.8
			✓	47.7
✓	✓			50.1

Table 8: Module-level ablation study on DeiT-Ti (L12 to L4) evaluated on ImageNet-1K. The results indicate that transferable low-frequency knowledge is predominantly concentrated in the MLP weights for vision models.

Figure 5: (a)Achieving superior performance with efficient refinement. (b) Self-attention layer structural patterns illustrated. The figure displays matrices $W_{q}W_{k}^{T}$ (top) and $W_{v}W_{o}$ (bottom) across three DeiT-T configurations: random initialization, pre-trained, and FRONT-initialized models. FRONT can inherit the diagonal property of self-attention layers that only exists in pre-trained ViTs.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Training Settings	Configuration
optimizer	AdamW
base learning rate	Ti: 5e-4 $\mid$ S: 2.5e-4 $\mid$ B: 1.25e-4
warmup learning rate	1e-6
weight decay	0.05
optimizer momentum	0.9
batch size	Ti: 512 $\mid$ S: 256 $\mid$ B: 128

Appendix

Table 9: Hyper-parameters for FRONT’s Retraining on ImageNet-1K.

Dataset	Batch Size	Epochs DeiT/ResNet	Learning Rate	Drop Last	Droppath Rate	Color Jitter	Auto Augment	Random Erase	Scheduler	Optimizer
Oxford Flowers	512	300/100	3e-4	False	0	0.4	rand-m9-mstd0.5-inc1	0.25	cosine	AdamW
CUB-200-2011	512	300/100	3e-4	False	0.1	0		0.25	cosine	AdamW
Stanford Cars	512	300/100	3e-4	False	0.1	0		0.25	cosine	AdamW
CIFAR10	512	300/100	5e-4	True	0.1	0.4		0.25	cosine	AdamW
CIFAR100	512	300/100	5e-4	True	0.1	0.4		0.25	cosine	AdamW
Food101	512	300/100	5e-4	True	0.1	0.4		0.25	cosine	AdamW

Appendix

Table 10: Hyperparameters for neural networks trained on downstream datasets.

Dataset	Classes	Total	Training	Testing
Oxford Flowers ( Nilsback & Zisserman, 2008 )	102	8,189	2,040	6,149
CUB-200-2011 ( Wah et al., 2011 )	200	11,788	5,994	5,794
Stanford Cars ( Gebru et al., 2017 )	196	16,185	8,144	8,041
CIFAR10 ( Krizhevsky & Hinton, 2009 )	10	60,000	50,000	10,000
CIFAR100 ( Krizhevsky & Hinton, 2009 )	100	60,000	50,000	10,000
Food101 ( Bossard et al., 2014 )	101	101,000	75,750	25,250

Appendix

Table 11: Characteristics of downstream datasets.

Methods	Scratch	FRONT	DFT	DWT	PCA	SVD
DeiT-Ti_L12 $\rightarrow$ DeiT-Ti_L6	40.6	63.3	60.8	62.0	57.9	53.0
DeiT-S_L6 $\rightarrow$ DeiT-Ti_L6	40.6	56.9	47.9	55.4	50.2	43.6

Appendix

Table 12: Comparison of different basis transforms for knowledge transfer on ImageNet-1K. Our DCT-based method (FRONT) achieves the highest top-1 accuracy (%) in both depth and width transfer tasks. This validates our design choice.

Select Freq.	DeiT-Ti
Select Freq.	L4	L6	L8	L10	L12
He-Init	34.7	40.6	43.7	46.8	48.3
High	32.6	36.0	35.8	38.9	39.0
Mid	34.0	34.6	37.0	38.1	38.2
Low	48.6	53.7	56.5	58.6	59.6

Appendix

Table 13: Low-frequency knowledge surpasses others.

		DeiT-Ti			DeiT-S
	Ancestry	$L_{\text{6}}$	$L_{\text{8}}$	$L_{\text{12}}$	$L_{\text{4}}$	$L_{\text{8}}$	$L_{\text{12}}$
TLEG ( Xia et al., 2024 )	LeVit-384 (39.1M)	60.5	62.9	65.4	70.5	72.1	73.8
WAVE ( Feng et al., 2025b )	LeVit-384 (39.1M)	63.2	65.4	67.3	72.7	74.1	75.3
FRONT +	LeVit-384 (39.1M)	63.4	65.6	67.5	72.9	74.2	75.4
FRONT +	RegNet-16GF (83.6M)	63.3 ∗	65.2 ∗	67.5	72.4 ∗	74.2	75.3 ∗

Appendix

Table 14: Additional results on different teacher models.

PT. Model	DeiT-Ti
PT. Model	L4	L6	L8	L10	L12
He-Init	34.7	40.6	43.7	46.8	48.3
DeiT-B	29.3	39.2	47.7	51.8	57.1
DeiT-S	34.8	45.5	51.9	56.7	59.7
DeiT-Ti	55.3	63.3	64.4	64.7	65.3

Appendix

Table 15: Pre-trained model’s scale. Pre-trained models who are more similar provide better initialization.

Panel of Figure 6 — Figure 6: Analysis of Module-wise Similarity Distribution and Average Scores for QKV, FC1, and FC2 Across Downstream Tasks

Figure 7: Freqency domain energy distribution comparison between pre-trained and randomly initialized DeiT-Ti/DeiT-S/DeiT-B models after 3D-DCT.

Figure 8: Performance comparisons on ImageNet-1K about depth expansion among FRONT and other learngene methods.

Transfer Learning

Transfer learning studies how models reuse knowledge across tasks, domains, or modalities when target data or computation are limited.

See all 417 papers

A Unified Framework for Knowledge Transfer in Bidirectional Model Scaling
Mar 8, 2026Jianlu Shen, Fu Feng, Jiaze Xu +3Transfer Learning Language Model Scaling
School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China
Pretraining Induces a Reusable Spectral Basis for Downstream Task Adaptation
May 8, 2026Junjie Yu, Yue Wang, Zihan Deng +3Transfer Learning Parameter-Efficient Fine-Tuning
Department of Biomedical Engineering, Southern University of Science and Technology · Department of Psychology, The University of Hong Kong
Wiring Beats Blending: Structure-Aware Compensation for Transformer Downscaling
Aug 3, 2026Ravi Satya Durga Prasad YenugulaLLM Compression Continued Pretraining
Independent Researcher

Mar 8, 2026J/K move · Enter open · S save

A Unified Framework for Knowledge Transfer in Bidirectional Model Scaling

Jianlu Shen, Fu Feng, Jiaze Xu, Yucheng Xie+2

School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China

Transfer Learning Language Model Scaling

Open paper AskPDF arXiv

Transferring pre-trained knowledge from a source model to a target model of a different architectural size is a key challenge for flexible and efficient model scaling. However, current parameter-space methods treat Small-to-Large (S2L) and Large-to-Small (L2S) scaling as separate, incompatible problems, focusing on parameter synthesis and selection, respectively. This fragmented perspective has resulted in specialized tools, hindering a unified, bidirectional framework. In this paper, we propose BoT (Bidirectional knowledge Transfer), the first size-agnostic framework to unify S2L and L2S scaling. Our core insight is to treat model weights as continuous signals, where models of different sizes represent distinct discretizations of the transferable knowledge. This multi-resolution perspective directly casts S2L and L2S scaling as the signal processing operations of upsampling and downsampling, naturally leading to the adoption of the Discrete Wavelet Transform (DWT) and its Inverse (IDWT). BoT leverages the recursive nature of wavelets, using the decomposition level as a dynamic scaling factor to bridge disparate model sizes in a parameter-free and computationally efficient manner. Extensive experiments on DeiT, BERT, and GPT demonstrate significant pre-training FLOPs savings (up to 67.1% for S2L, 52.8% for L2S) and state-of-the-art performance on benchmarks like GLUE and SQuAD.

Abstract

Transfer Learning

Explore similar work

A Unified Framework for Knowledge Transfer in Bidirectional Model Scaling

Pretraining Induces a Reusable Spectral Basis for Downstream Task Adaptation

Wiring Beats Blending: Structure-Aware Compensation for Transformer Downscaling