Memory-Efficient Transfer Learning with Fading Side Networks via Masked Dual Path Distillation

Yutong Zhang; Jiaxin Chen; Honglin Chen; Kaiqi Zheng; Shengcai Liao; Hanwen Zhong; Weixin Li; Yunhong Wang

cs.CVApr 10, 2026

Memory-Efficient Transfer Learning with Fading Side Networks via Masked Dual Path Distillation

Authors: Yutong Zhang, Jiaxin Chen, Honglin Chen, Kaiqi Zheng, Shengcai Liao, Hanwen Zhong, Weixin Li, Yunhong Wang

Organizations: State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, China · School of Computer Science and Engineering, Beihang University, China · College of Information Technology, United Arab Emirates University, United Arab Emirates

Abstract

Memory-efficient transfer learning (METL) approaches have recently achieved promising performance in adapting pre-trained models to downstream tasks. They avoid applying gradient backpropagation in large backbones, thus significantly reducing the number of trainable parameters and high memory consumption during fine-tuning. However, since they typically employ a lightweight and learnable side network, these methods inevitably introduce additional memory and time overhead during inference, which contradicts the ultimate goal of efficient transfer learning. To address the above issue, we propose a novel approach dubbed Masked Dual Path Distillation (MDPD) to accelerate inference while retaining parameter and memory efficiency in fine-tuning with fading side networks. Specifically, MDPD develops a framework that enhances the performance by mutually distilling the frozen backbones and learnable side networks in fine-tuning, and discard the side network during inference without sacrificing accuracy. Moreover, we design a novel feature-based knowledge distillation method for the encoder structure with multiple layers. Extensive experiments on distinct backbones across vision/language-only and vision-and-language tasks demonstrate that our method not only accelerates inference by at least 25.2% while keeping parameter and memory consumption comparable, but also remarkably promotes the accuracy compared to SOTA approaches. The source code is available at https://github.com/Zhang-VKk/MDPD.

Figures & tables

Figure 1: Illustration on our method. Part (I) shows the pipeline of Masked Dual Path Distillation. During training, the backbone and side networks mutually act as teacher and student for effective distillation, where side networks are abandoned after training for inference efficiency. Part (II) displays Hierarchical Feature-based Distillation that adopt distinct strategies across different layers. It particularly selects a set of masked features, making them learnable with guidance of backbones in deep layers.

Method	Params.	Mem.	QPS	BERT+BUTD			Params.	Mem.	QPS	ResNeXt+BiGRU			Params.	Mem.	QPS	ViT+Text Transf.
Method	(M) $\!\downarrow$	(G) $\!\downarrow$	$\!\uparrow$	I-T $\!\uparrow$	T-I $\!\uparrow$	Rsum $\!\uparrow\$	(M) $\!\downarrow$	(G) $\!\downarrow$	$\!\uparrow$	I-T $\!\uparrow$	T-I $\!\uparrow$	Rsum $\!\uparrow\$	(M) $\!\downarrow$	(G) $\!\downarrow$	$\!\uparrow$	T-V $\!\uparrow$	V-T $\!\uparrow$	Rsum $\!\uparrow$
Fully-FT	109.5	9.9	24.14	79.7	62.1	513.5	90.9	174.4	30.80	85.6	70.2	539.0	151.3	48.8	44.07	42.8	42.1	389.2
Partially	0.8	1.0	24.19	74.8	57.3	485.5	2.1	14.9	31.26	75.2	58.2	505.8	0.7	7.6	44.48	36.4	37.0	353.9
Adapter [ 40 ]	2.6	8.8	22.44	79.1	60.5	511.3	3.5	176.8	28.65	66.8	62.9	493.3	5.2	41.2	39.17	38.3	39.6	364.3
LoRA [ 41 ]	1.1	8.8	23.37	78.8	59.6	508.2	–	–	–	–	–	–	1.3	40.8	41.55	38.8	39.9	366.8
BitFit [ 106 ]	0.9	8.6	23.70	77.3	57.8	503.9	2.2	168.0	30.53	83.4	67.4	530.6	0.1	42.0	43.12	38.1	40.6	370.8

Table 1: Comparison results with PETL (Middle) and METL (Bottom) approaches across distinct network architectures. Best results are in bold , and second best results are underlined .

Method	Params.	Memory (G) $\downarrow$		Time	CoLA	SST-2	MRPC	QQP	MNLI	QNLI	RTE	STS-B	Mean
Method	(%) $\downarrow$	Train	Test	(ms) $\downarrow$	CoLA	SST-2	MRPC	QQP	MNLI	QNLI	RTE	STS-B	Mean
Fully-FT	100	17.6	0.86	232.5	62.8	93.9	91.9	89.9	86.2	92.5	74.1	90.3	85.2
Adapter [ 40 ]	1.63	13.0	0.87	247.3	64.4	94.2	88.9	88.9	86.4	93.1	75.1	91.1	85.3
LoRA [ 41 ]	1.71	12.6	0.86	238.1	63.3	94.3	90.1	89.0	86.3	93.2	75.5	90.9	85.3
BitFit [ 106 ]	0.13	10.7	0.86	236.9	61.8	94.3	91.0	88.7	85.6	93.1	67.6	90.8	84.1
Prompt [ 55 ]	0.03	22.2	0.87	291.4	0	90.3	74.6	88.5	82.5	92.5	59.5	90.1	72.2

Table 2: Comparison results with PETL (Top) and METL (Middle, Bottom) methods on GLUE. Ours ♡ and Ours ♠ indicate our method implemented based on UniPT and SHERL, respectively. Best results are in bold , and second best results are underlined .

				Natural							Specialized				Structured
Method	Params. (M) $\downarrow$	Memory (G) $\downarrow$	FPS $\uparrow$	Cifar100	Caltech101	DTD	Flower102	Pets	SVHN	Sun397	Camelyon	EuroSAT	Resisc45	Retinopathy	Clevr-Count	Clevr-Dist	DMLab	KITTI-Dist	dSpr-Loc	dSpr-Ori	sNORB-Azim	sNORB-Ele	Mean
Fully-FT	85.8	6.1	89.21	68.9	87.7	64.3	97.2	86.9	87.4	38.8	79.7	95.7	84.2	73.9	56.3	58.6	41.7	65.5	57.5	46.7	25.7	29.1	68.9
BitFiT	0.10	3.8	87.95	72.8	87.0	59.2	97.5	85.3	59.9	51.4	78.7	91.6	72.9	69.8	61.5	55.6	32.4	55.9	66.6	40.0	15.7	25.1	65.2
VPT	0.53	5.6	72.10	78.8	90.8	65.8	98.0	88.3	78.1	49.6	81.8	96.1	83.4	68.4	68.5	60.0	46.5	72.8	73.6	47.9	32.9	37.8	72.0
RS-Bypass.	0.42	2.4	47.53	64.5	88.8	73.2	99.4	90.6	63.5	57.2	85.5	95.2	82.4	75.2	70.4	61.0	40.2	66.8	79.2	52.6	26.0	49.3	72.3
Adapter	0.16	3.9	82.78	69.2	90.1	68.0	98.8	89.9	82.8	54.3	84.0	94.9	81.9	75.5	80.9	65.3	48.6	78.3	74.8	48.5	29.9	41.6	73.9

Table 3: Comparison results with PETL (Top) and METL (Bottom) methods on VTAB-1K. Best results are in bold , and second best results are underlined .

Method	Params.	Mem.	QPS	Flickr30K			MSCOCO1K			MSCOCO5K			Params	Mem.	QPS	MSR-VTT			MSVD
Method	(M) $\!\downarrow$	(G) $\!\downarrow$	$\!\uparrow$	I-T $\!\uparrow\$	T-I $\!\uparrow\$	Rsum $\!\uparrow\$	I-T $\!\uparrow\$	T-I $\!\uparrow\$	Rsum $\!\uparrow\$	I-T $\!\uparrow\$	T-I $\!\uparrow\$	Rsum $\!\uparrow\$	(M) $\!\downarrow$	(G) $\!\downarrow$	$\!\uparrow$	T-V $\!\uparrow\$	V-T $\!\uparrow\$	Rsum $\!\uparrow\$	T-V $\!\uparrow\$	V-T $\!\uparrow\$	Rsum $\!\uparrow\$
Fully-FT	201.2	176.8	-	85.6	73.3	546.6	83.1	71.7	542.7	64.2	51.2	468.9	151.3	48.8	-	42.8	42.1	389.2	45.2	57.1	425.5
LST [ 84 ]	9.7	24.4	-	82.1	66.5	529.5	78.2	64.8	525.8	57.8	43.1	434.5	11.2	32.0	-	37.0	37.8	356.7	35.5	55.4	407.2
UniPT [ 59 ]	12.4	24.4	8.61	84.8	69.1	537.4	80.6	67.5	532.9	61.1	45.9	445.3	9.6	13.6	27.03	38.9	39.3	361.3	40.9	59.7	432.1
SHERL [ 20 ]	11.3	24.4	10.24	86.1	71.1	542.3	81.8	69.2	537.5	62.5	47.3	450.8	9.6	13.6	32.53	39.2	40.6	363.7	40.9	60.2	429.7
Ours ♡	12.4	25.1	14.71	85.9	70.7	541.2	81.3	68.8	536.4	61.9	46.8	448.1	9.7	14.0	43.65	40.2	40.7	365.8	42.3	60.0	435.7

Table 4: Comparison results with METL approaches across various architectures and distinct VL tasks. Ours ♡ and Ours ♠ denote our implementations based on UniPT and SHERL, respectively. Best results are in bold , and second best results are underlined .

Method	Params.	Memory	QPS	Flickr30K			MSCOCO1K
Method	(M) $\downarrow$	(G) $\downarrow$	$\uparrow$	I-T $\uparrow$	T-I $\uparrow$	Rsum $\uparrow$	I-T $\uparrow$	T-I $\uparrow$	Rsum $\uparrow$
Baseline (UniPT) [ 59 ]	12.4	24.4	8.61	84.8	69.1	537.4	80.6	67.5	532.9
Logits-based (S $\to$ B)	12.4	24.7	14.71	81.2	66.1	529.7	77.2	65.0	526.2
Feature-based (B $\to$ S)	12.4	24.7	14.71	81.9	66.5	531.5	78.3	65.9	528.1
Feature-based (S $\to$ B)	12.4	24.7	14.71	78.9	63.0	518.2	73.1	60.9	508.3
Ours (B $\leftrightarrows$ S)	12.4	25.1	14.71	84.9	69.5	538.1	80.2	67.7	532.7

Table 5: Ablation study on various distillation strategies that are applied to distinct structures ( i.e. ‘logits-based’ and ‘feature-based’) with different distillation directions, where ‘S’ and ‘B’ refer to side network and backbone, respectively. Best results are in bold , and the second best ones are underlined .

Method	Fading	Params.	Memory	QPS	Flickr30K			MSCOCO1K
Method	Side Network	(M) $\downarrow$	(G) $\downarrow$	$\uparrow$	I-T $\uparrow$	T-I $\uparrow$	Rsum $\uparrow$	I-T $\uparrow$	T-I $\uparrow$	Rsum $\uparrow$
Fully-FT	$\times$	201.2	176.8	-	85.6	73.3	546.6	83.1	71.7	542.7
LST [ 84 ]	$\times$	9.7	24.4	-	82.1	66.5	529.5	78.2	64.8	525.8
UniPT [ 59 ]	$\times$	12.4	24.4	8.61	84.8	69.1	537.4	80.6	67.5	532.9
Ours ♡	$\times$	12.4	25.1	7.98	86.2	70.9	541.9	81.8	69.4	537.6
Ours ♡	✓	12.4	25.1	14.71	85.9	70.7	541.2	81.3	68.8	536.4

Table 6: Ablation study on fading the side network during inference on distinct datasets for the ITR task. Best results are in bold , and the second best ones are underlined .

Task	Model	Learning Rate	Optimizer ( $\beta_{1}$ , $\beta_{2}$ , Weight Decay)	Batch Size	Total Epochs	Warmup Strategy
ITR	VSE $\infty$	$5\times 10^{-4}$	$0.9,0.999,1\times 10^{-2}$	$112$	$25$	linear
VTR	CLIP4Clip	$1\times 10^{-4}$	$0.9,0.98,1\times 10^{-2}$	$128$	$5$	cosine
VQA	CLIP-ViL	$5\times 10^{-4}$	$0.9,0.999,1\times 10^{-2}$	$256$	$5$	linear
GQA	CLIP-ViL	$1\times 10^{-4}$	$0.9,0.999,1\times 10^{-2}$	$256$	$5$	linear
VG	MDETR	$5\times 10^{-4}$	$0.9,0.999,0$	$8$	$10$	linear

Table A : Detailed Hyper-parameters of MDPD on ITR, VTR, VQA, GQA, and VG tasks. Among them, AdamW is adopted as the optimizer uniformly.

Figure A : Ablation study on the effect of the mask rate

\lambda

in our method. ( Left ) The Rsum (%) performance on Flickr30K and MSCOCO1K datasets. ( Right ) The R@1 (%) performance for sentence retrieval (”I-T”) and image retrieval (”T-I”) on Flickr30K and MSCOCO1K datasets.

Generation Blocks	Params.	Memory	Flickr30K			MSCOCO1K			MSCOCO5K
Generation Blocks	(M) $\downarrow$	(G) $\downarrow$	I-T $\uparrow$	T-I $\uparrow$	Rsum $\uparrow$	I-T $\uparrow$	T-I $\uparrow$	Rsum $\uparrow$	I-T $\uparrow$	T-I $\uparrow$	Rsum $\uparrow$
Self-Attention	12.6	25.4	84.7	69.5	537.4	79.8	67.2	531.5	60.4	46.4	444.9
Convolution	12.4	25.1	85.9	70.7	541.2	81.3	68.8	536.4	61.9	46.8	448.1

Table B : Ablation results (%) on the generation blocks for feature transformation in the student network on distinct datasets for ITR task. The best results are highlighted in bold .

Layers		Params.	Memory	Flickr30K			MSCOCO1K			MSCOCO5K
Layers		(M) $\downarrow$	(G) $\downarrow$	I-T $\uparrow$	T-I $\uparrow$	Rsum $\uparrow$	I-T $\uparrow$	T-I $\uparrow$	Rsum $\uparrow$	I-T $\uparrow$	T-I $\uparrow$	Rsum $\uparrow$
Imitation	1	12.4	24.9	83.4	68.5	536.8	78.6	65.8	529.1	59.1	44.7	441.3
	1,2	12.4	24.9	83.7	68.8	536.7	78.9	66.1	529.2	59.7	44.3	441.6
	5,6	12.4	24.9	84.6	69.2	537.4	79.6	67.1	531.2	60.5	45.6	443.8
	1 $\sim$ 6	12.4	25.1	84.5	69.5	537.6	79.4	67.3	530.9	60.8	45.5	444.0
	All	12.4	25.1	84.9	69.7	538.3	80.0	67.8	532.6	61.4	45.8	445.7

Table C : Ablation results (%) on adopting Hierarchical Feature-based Distillation strategy across shallow (Top) , deep (Middle) , and all (Bottom) layers on distinct datasets for ITR task. The best results are highlighted in bold . and the second best results are underlined .

Memory-Efficient Fine-Tuning

Memory-efficient fine-tuning reduces the GPU, device, and communication resources needed to adapt large models while preserving task performance.

See all 61 papers

MP-ISMoE: Mixed-Precision Interactive Side Mixture-of-Experts for Efficient Transfer Learning
Apr 10, 2026Yutong Zhang, Zimeng Wu, Shangcai Liao +2Transfer Learning Memory-Efficient Fine-Tuning
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China · School of Computer Science and Engineering, Beihang University, Beijing, China · College of Information Technology, United Arab Emirates University, Al Ain, Abu Dhabi, United Arab Emirates
Breaking the Scale Barrier: One-Shot Knowledge Transfer via Frequency Transform
Mar 8, 2026Jianlu Shen, Fu Feng, Yucheng Xie +2Transfer Learning
School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China
Meta-Learning Hyperparameters for Parameter Efficient Fine-Tuning
Mar 2, 2026Zichen Tian, Yaoyao Liu, Qianru SunParameter-Efficient Fine-Tuning Vision Foundation Model Adaptation
Singapore Management University · University of Illinois Urbana-Champaign

Apr 10, 2026J/K move · Enter open · S save

MP-ISMoE: Mixed-Precision Interactive Side Mixture-of-Experts for Efficient Transfer Learning

Yutong Zhang, Zimeng Wu, Shangcai Liao, Shujiang Wu+1

State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China · School of Computer Science and Engineering, Beihang University, Beijing, China · College of Information Technology, United Arab Emirates University, Al Ain, Abu Dhabi, United Arab Emirates

Transfer Learning Memory-Efficient Fine-Tuning Parameter-Efficient Fine-Tuning Mixture-of-Experts Models Mixed-Precision Quantization

Open paper AskPDF arXiv

Parameter-efficient transfer learning (PETL) has emerged as a pivotal paradigm for adapting pre-trained foundation models to downstream tasks, significantly reducing trainable parameters yet suffering from substantial memory overhead caused by gradient backpropagation during fine-tuning. While memory-efficient transfer learning (METL) circumvents this challenge by bypassing backbone gradient computation via lightweight small side networks, its stringent memory constraint severely limits learning capacity of side networks, thereby significantly compromising performance. To address these limitations, we propose a novel Mixed-Precision Interactive Side Mixture-of-Experts framework (MP-ISMoE). Specifically, we first propose a Gaussian Noise Perturbed Iterative Quantization (GNP-IQ) scheme to quantize weights into lower-bits while effectively decreasing quantization errors. By leveraging memory conserved from GNP-IQ, we subsequently employ Interactive Side Mixture-of-Experts (ISMoE) to scaling up side networks without sacrificing overall memory efficiency. Different from conventional mixture-of-experts, ISMoE learns to select optimal experts by interacting with salient features from frozen backbones, thus suppressing knowledge forgetting and boosting performance. Extensive experiments across diverse vision-language and language-only tasks demonstrate that MP-ISMoE remarkably promotes accuracy compared to state-of-the-art METL approaches, while maintaining comparable parameter and memory efficiency.

Abstract

Memory-Efficient Fine-Tuning

Explore similar work

MP-ISMoE: Mixed-Precision Interactive Side Mixture-of-Experts for Efficient Transfer Learning

Breaking the Scale Barrier: One-Shot Knowledge Transfer via Frequency Transform

Meta-Learning Hyperparameters for Parameter Efficient Fine-Tuning