A Unified Framework for Knowledge Transfer in Bidirectional Model Scaling

Jianlu Shen; Fu Feng; Jiaze Xu; Yucheng Xie; Jiaqi Lv; Xin Geng

cs.LGMar 8, 2026

A Unified Framework for Knowledge Transfer in Bidirectional Model Scaling

Authors: Jianlu Shen, Fu Feng, Jiaze Xu, Yucheng Xie, Jiaqi Lv, Xin Geng

Organizations: School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China

Abstract

Transferring pre-trained knowledge from a source model to a target model of a different architectural size is a key challenge for flexible and efficient model scaling. However, current parameter-space methods treat Small-to-Large (S2L) and Large-to-Small (L2S) scaling as separate, incompatible problems, focusing on parameter synthesis and selection, respectively. This fragmented perspective has resulted in specialized tools, hindering a unified, bidirectional framework. In this paper, we propose BoT (Bidirectional knowledge Transfer), the first size-agnostic framework to unify S2L and L2S scaling. Our core insight is to treat model weights as continuous signals, where models of different sizes represent distinct discretizations of the transferable knowledge. This multi-resolution perspective directly casts S2L and L2S scaling as the signal processing operations of upsampling and downsampling, naturally leading to the adoption of the Discrete Wavelet Transform (DWT) and its Inverse (IDWT). BoT leverages the recursive nature of wavelets, using the decomposition level as a dynamic scaling factor to bridge disparate model sizes in a parameter-free and computationally efficient manner. Extensive experiments on DeiT, BERT, and GPT demonstrate significant pre-training FLOPs savings (up to 67.1% for S2L, 52.8% for L2S) and state-of-the-art performance on benchmarks like GLUE and SQuAD.

Figures & tables

Figure 1: Conceptual illustration of knowledge transfer challenges between mismatched model sizes. (a) Direct fine-tuning from model zoos is often unviable due to size mismatches. (b) Small-to-Large (S2L) methods initialize by mapping or stacking weights from a small model. (c) Large-to-Small (L2S) methods initialize by selecting weights from a large model.

Figure 2: Comparison of cross-architecture initialization frameworks. (a) Previous Large-to-Small (L2S) methods rely on heuristic sampling or layer selection. (b) Previous Small-to-Large (S2L) methods use direct copying or trainable mapping functions. (c) Our proposed BOT framework unifies both directions through a parameter-free, frequency-domain transformation using the 3D-DWT and IDWT.

Methods	Large-to-Small	Small-to-Large	No-Trainability
WS [ 51 ]	✓	$\times$	✓
bert2BERT [ 2 ]	$\times$	✓	✓
LiGO [ 46 ]	$\times$	✓	$\times$
Mango [ 25 ]	$\times$	✓	$\times$
BoT	✓	✓	✓

Table 1: Qualitative comparison of different cross-architecture initialization methods. “No-Trainability” refers to whether the method itself directly transfers knowledge without introducing learnable parameters for the transfer process.

Panel of Figure 3 — Figure 3: Results of pretraining DeiT-S, BERT-S and GPT-S. BoT can achieve the highest savings in FLOPs with 22.0% for DeiT-S, 67.1% for BERT-S, and 58.3% for GPT-S from the Scratch models.

Panel of Figure 4 — Figure 4: Results of pretraining DeiT-B, BRET-B and GPT-B. BoT can achieve the highest savings in FLOPs with 22.0% for DeiT-B, 67.1% for BERT-B, and 58.3% for GPT-B from the Scratch models.

	Methods	C 10	C 100	CUB	Flow.	Food	iNat.	Cars
DeiT-S	Scratch	94.0	66.5	27.3	57.2	70.6	54.0	23.8
	WS	95.8	75.2	57.7	75.8	79.2	61.2	72.1
	BoT	96.0	75.3	61.3	76.3	80.5	61.9	74.6
DeiT-B	Scratch	95.9	73.8	42.2	65.0	78.6	58.2	42.1
	bert2BERT	96.4	74.9	67.1	85.4	82.0	61.1	82.9
	LiGO	97.2	82.2	73.2	94.4	84.4	67.1	87.4

Table 2: Performance comparison on seven downstream tasks. Models are initialized using various methods and directly fine-tuned without extra pre-training. The “Avg” column shows the average accuracy.

	Methods	GLUE Benchmark								SQuAD Benchmark
	Methods	SST-2	MNLI	MRPC	CoLA	QNLI	QQP	STS-B	Avg.	v1.1 (F1/EM)	v2.0 (F1/EM)	Avg.
BERT-S	Scratch	81.19	63.45	69.85	8.02	59.09	81.08	20.12	54.69	18.28 / 9.05	8.13 / 3.72	9.80
	WS	81.77	74.26	69.61	10.21	77.72	86.43	26.33	60.90	32.99 / 23.38	17.11 / 11.65	21.28
	BoT	85.67	74.64	70.34	17.05	81.97	88.13	74.20	70.29	69.63 / 57.87	35.77 / 29.71	48.25
BERT-B	Scratch	80.85	63.31	68.38	2.06	61.19	79.20	11.44	52.35	18.95 / 9.62	9.23 / 4.55	10.59
	bert2BERT	81.19	71.44	69.61	6.09	75.27	85.84	23.44	58.98	31.58 / 22.20	16.25 / 11.20	20.31

Table 3: Performance comparison on the GLUE and SQuAD benchmarks based on BERT. Models are initialized and then directly fine-tuned without additional pre-training. We report accuracy for most tasks, along with Matthews Correlation Coefficient (MCC) for CoLA and Pearson correlation for STS-B. SQuAD performance is measured by F1 scores and Exact Match (EM). The “Avg” columns show the average scores for each benchmark.

	Methods	Savings		GLUE Benchmark								SQuAD Benchmark
	Methods	FLOPs	Walltime	SST-2	MNLI	MRPC	CoLA	QNLI	QQP	STS-B	Avg.	v1.1 (F1/EM)	v2.0 (F1/EM)	Avg.
BERT-S	Scratch	0.0%	0.0%	87.50	77.74	77.94	28.47	85.54	88.93	83.01	75.59	76.12 / 66.02	37.86 / 32.43	53.11
	WS	33.0%	41.7%	89.45	77.67	77.70	28.00	85.14	88.93	82.15	75.58	78.07 / 68.41	39.43 / 34.27	55.04
	BoT	52.8%	66.8%	89.33	78.34	77.47	27.54	85.69	88.95	82.62	75.71	78.79 / 68.87	39.52 / 34.23	55.35
BERT-B	Scratch	0%	0.0%	91.51	83.28	84.31	43.34	89.93	90.64	85.47	81.21	84.81 / 75.33	45.43 / 40.81	61.59
	bert2BERT	33.3%	32.4%	90.48	83.60	84.80	50.47	90.19	90.44	87.03	82.43	84.83 / 75.86	45.51 / 41.07	61.82

Table 4: Performance comparison on GLUE and SQuAD benchmarks after pre-training from initialization. This table shows the final performance of models that were first pre-trained and then fine-tuned. “Avg.” columns show the average scores for each benchmark.

Figure 5: Ablation study on the choice of wavelet family. Lower validation loss indicates better performance.

Figure 6: Visualization of structured knowledge within self-attention layers for (a) DeiT-S and (b) DeiT-B. BoT inherits the strong diagonal structure present in pre-trained models.

Figure 7: Visualization of knowledge extracted by BoT. All networks operate directly after initialization without any additional training or fine-tuning.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Wavelet	Length	Key Characteristics	Visualization
haar	2	Discontinuous, piecewise-constant. Has the most compact support. Highly efficient. Optimal for: BERT (L2S)
db2	4	Daubechies family. Orthogonal and compact, but asymmetric. Optimal for: GPT (S2L)
db4	8	Daubechies family. Smoother than db2 due to longer filter and more vanishing moments.
sym8	16	Symlet family. “Least asymmetric” modification of Daubechies wavelets. Orthogonal with compact support.
coif3	18	Coiflet family. Nearly symmetric with vanishing moments for both scaling and wavelet functions. Optimal for: GPT (L2S)
bior3.3	8 (Decomp.)	Biorthogonal family. Symmetric (linear phase) by relaxing orthogonality. Uses separate analysis/synthesis filters.

Appendix

Table 5: Overview of Wavelet Families Used in Ablation Study. The ”Visualization” column shows the wavelet function (

\psi

), which determines how high-frequency details are captured.

Config	DeiT-Ti	DeiT-S	DeiT-B	DeiT-L
# layers	3	6	12	24
# hidden	192	384	768	1024
# heads	3	6	12	16
input size	224	224	224	224
patch size	16	16	16	16

Appendix

Table 6: The structures of DeiT models used in our experiments.

Training Settings	Configuration
optimizer	AdamW
base learning rate	S: 2.5e-4 $\mid$ B: 1.25e-4
warmup learning rate	1e-6
weight decay	0.05
optimizer momentum	0.9
batch size	S: 256 $\mid$ B: 128

Appendix

Table 7: Hyper-parameters for all baseline on ImageNet-1K.

Dataset	Classes	Total	Training	Testing
Oxford Flowers [ 24 ]	102	8,189	2,040	6,149
CUB-200-2011 [ 43 ]	200	11,788	5,994	5,794
Stanford Cars [ 10 ]	196	16,185	8,144	8,041
CIFAR10 [ 18 ]	10	60,000	50,000	10,000
CIFAR100 [ 18 ]	100	60,000	50,000	10,000
Food101 [ 1 ]	101	101,000	75,750	25,250

Appendix

Table 8: Characteristics of downstream datasets.

Dataset	Batch Size	Epoch	Learning Rate	Drop Last	Warmup Epochs	Droppath Rate	Color Jitter	Auto Augment	Random Rrase	Mixup	Cutmix	Scheduler	Optimizer
Oxford Flowers	512	300	3e-4	False	0	0	0.4	rand-m9-mstd0.5-inc1	0.25	0	0	cosine	AdamW
CUB-200-2011	512	300	3e-4	False	0	0.1	0		0.25	0	0	cosine	AdamW
Stanford Cars	512	300	3e-4	False	0	0.1	0		0.25	0	0	cosine	AdamW
CIFAR10	512	300	5e-4	True	0	0.1	0.4		0.25	0	0	cosine	AdamW
CIFAR100	512	300	5e-4	True	0	0.1	0.4		0.25	0	0	cosine	AdamW
Food101	512	300	5e-4	True	0	0.1	0.4		0.25	0	0	cosine	AdamW

Appendix

Table 9: Hyper-parameters for neural networks trained on downstream datasets.

Config	BERT-Ti	BERT-S	BERT-B	BERT-L
# layers	3	6	12	24
# hidden	192	384	768	1024
# heads	3	6	12	16
# vocab	30522	30522	30522	30522
seq. length	128	128	128	128

Appendix

Table 10: The structures of BERT models.

Training Setting	Configuration
Optimizer	AdamW
Adam $\epsilon$	1e-6
Learning Rate	2e-4
Weight Decay	0.1
Learning Rate Schedule	Linear Decay
Warmup Steps	0

Appendix

Table 11: Hyper-parameters for BERT and RoBERTa pre-training.

Config	RoBERTa-S	RoBERTa-B	GPT-S	GPT-B
# layers	6	12	6	12
# hidden	384	768	384	768
# heads	6	12	6	12
# vocab	50265	50265	50257	50257
seq. length	128	128	1024	1024

Appendix

Table 12: The structures of RoBERTa and GPT-2 models used in our experiments. RoBERTa shares a similar architecture to BERT but uses a different vocabulary.

Training Setting	Configuration
Optimizer	AdamW
Adam $\epsilon$	1e-8
Adam $\beta_{1},\beta_{2}$	0.9, 0.95
Learning Rate	1e-3
Weight Decay	0.1
Learning Rate Schedule	Cosine Decay

Appendix

Table 13: Hyper-parameters for GPT-2 pre-training.

Model	Methods	GLUE Benchmark								SQuAD Benchmark
Model	Methods	SST-2	MNLI	MRPC	CoLA	QNLI	QQP	STS-B	Avg	v1.1 (EM)	v1.1 (F1)	v2.0 (EM)	v2.0 (F1)	Avg
RoBERTa-S	Scratch	88.19	75.30	76.47	21.27	85.19	88.48	82.87	73.97	7.99	18.02	3.39	8.16	9.39
	WS	88.42	75.26	78.43	25.04	83.53	88.12	80.84	74.23	23.75	32.72	12.79	18.04	21.83
	DWT	88.42	76.08	78.43	23.27	85.58	88.62	82.81	74.74	42.58	54.22	20.26	26.87	35.98
RoBERTa-B	Scratch	81.65	31.82	69.61	2.16	61.58	77.50	15.69	48.57	8.80	18.76	3.92	8.86	10.09
	bert2BERT	81.19	61.91	70.59	10.51	59.86	79.94	20.86	54.98	13.64	24.08	5.39	10.62	13.43

Appendix

Table 14: Performance comparison on the GLUE and SQuAD benchmarks for RoBERTa. Models are initialized and then directly fine-tuned without additional pre-training. The “Avg” columns show the average scores for each benchmark.

Figure 8: (a) BERT

\leftrightarrow

GPT; (b) Generalization to LLaMA; (c) Generalization to RoBERTa; (d) Scalability on Tiny and Large.

Model	Methods	Savings		GLUE Benchmark								SQuAD Benchmark
Model	Methods	FLOPs	Walltime	SST-2	MNLI	MRPC	CoLA	QNLI	QQP	STS-B	Avg	v1.1 (f1/em)	v2.0 (f1/em)	Avg
RoBERTa-S	Scratch	0.0%	0.0%	89.56	76.64	79.41	20.39	84.90	88.32	82.59	74.54	76.2 / 65.8	37.5 / 32.0	52.9
	WS	32.4%	32.3%	88.50	76.85	78.19	21.62	83.54	88.15	81.43	74.04	75.3 / 64.7	37.6 / 32.1	52.4
	BoT	35.7%	35.8%	89.33	76.78	76.47	23.18	84.61	88.71	82.93	74.57	76.7 / 66.3	37.9 / 32.4	53.3
RoBERTa-B	Scratch	0.0%	0.0%	90.71	82.59	83.33	45.87	89.75	90.38	85.44	81.15	84.4 / 74.7	43.6 / 38.8	60.4
	bert2BERT	7.2%	6.6%	91.63	82.00	86.03	41.04	89.00	90.21	85.20	80.73	83.8 / 74.2	41.5 / 36.6	59.0

Appendix

Table 15: Performance comparison on GLUE and SQuAD benchmarks for RoBERTa after pre-training from initialization. This table shows the final performance of models that were first pre-trained and then fine-tuned. “Avg” columns show the average scores for each benchmark.

Method	CIFAR-10	CIFAR-100	CUB200	Flowers102	Food-101	iNat-2019	Cars	Avg.
Random Uniform	95.13	74.73	62.53	69.01	73.94	51.92	52.11	68.85
Random Gaussian	96.87	80.14	73.91	90.63	83.19	65.02	79.92	81.67
Zero Padding (BoT)	97.24	82.38	74.08	94.71	84.44	67.14	88.64	84.37

Appendix

Table 16: Ablation on padding strategies for high-frequency coefficients in IDWT-based S2L transfer. Zero padding consistently yields the best average accuracy. We report top-1 accuracy (%) on seven downstream datasets after direct fine-tuning.

Transfer Learning

Transfer learning studies how models reuse knowledge across tasks, domains, or modalities when target data or computation are limited.

See all 417 papers

Breaking the Scale Barrier: One-Shot Knowledge Transfer via Frequency Transform
Mar 8, 2026Jianlu Shen, Fu Feng, Yucheng Xie +2Transfer Learning
School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China
Wiring Beats Blending: Structure-Aware Compensation for Transformer Downscaling
Aug 3, 2026Ravi Satya Durga Prasad YenugulaLLM Compression Continued Pretraining
Independent Researcher
Constraint-based Pre-training: From Structured Constraints to Scalable Model Initialization
Apr 16, 2026Fu Feng, Yucheng Xie, Ruixiao Shi +2
School of Computer Science and Engineering, Southeast University, Nanjing, China and the Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China

Mar 8, 2026J/K move · Enter open · S save

Breaking the Scale Barrier: One-Shot Knowledge Transfer via Frequency Transform

Jianlu Shen, Fu Feng, Yucheng Xie, Jiaqi Lv+1

School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China

Transfer Learning

Open paper AskPDF arXiv

Transferring knowledge by fine-tuning large-scale pre-trained networks has become a standard paradigm for downstream tasks, yet the knowledge of a pre-trained model is tightly coupled with monolithic architecture, which restricts flexible reuse across models of varying scales. In response to this challenge, recent approaches typically resort to either parameter selection, which fails to capture the interdependent structure of this knowledge, or parameter prediction using generative models that depend on impractical access to large network collections. In this paper, we identify the low-frequency components of model weights as the concrete carrier of foundational, task-agnostic knowledge, its

learngene", and validate this by demonstrating its efficient inheritance by downstream models and tasks. Based on this insight, we propose FRONT (FRequency dOmain kNowledge Transfer), a novel framework that uses the Discrete Cosine Transform (DCT) to isolate the low-frequency

learngene". This learngene can be seamlessly adapted to initialize models of arbitrary size via simple truncation or padding, a process that is entirely training-free. For enhanced performance, we propose an optional low-cost refinement process that introduces a spectral regularizer to further improve the learngene's transferability. Extensive experiments demonstrate that FRONT achieves the state-of-the-art performance, accelerates convergence by up to

15\times

in vision tasks, and reduces training FLOPs by an average of 40.5% in language tasks. Code is available at https://github.com/LUcy0505/FRONT.

Abstract

Transfer Learning

Explore similar work

Breaking the Scale Barrier: One-Shot Knowledge Transfer via Frequency Transform

Wiring Beats Blending: Structure-Aware Compensation for Transformer Downscaling

Constraint-based Pre-training: From Structured Constraints to Scalable Model Initialization