MP-ISMoE: Mixed-Precision Interactive Side Mixture-of-Experts for Efficient Transfer Learning

Yutong Zhang; Zimeng Wu; Shangcai Liao; Shujiang Wu; Jiaxin Chen

cs.LGApr 10, 2026

MP-ISMoE: Mixed-Precision Interactive Side Mixture-of-Experts for Efficient Transfer Learning

Authors: Yutong Zhang, Zimeng Wu, Shangcai Liao, Shujiang Wu, Jiaxin Chen

Organizations: State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China · School of Computer Science and Engineering, Beihang University, Beijing, China · College of Information Technology, United Arab Emirates University, Al Ain, Abu Dhabi, United Arab Emirates

Abstract

Parameter-efficient transfer learning (PETL) has emerged as a pivotal paradigm for adapting pre-trained foundation models to downstream tasks, significantly reducing trainable parameters yet suffering from substantial memory overhead caused by gradient backpropagation during fine-tuning. While memory-efficient transfer learning (METL) circumvents this challenge by bypassing backbone gradient computation via lightweight small side networks, its stringent memory constraint severely limits learning capacity of side networks, thereby significantly compromising performance. To address these limitations, we propose a novel Mixed-Precision Interactive Side Mixture-of-Experts framework (MP-ISMoE). Specifically, we first propose a Gaussian Noise Perturbed Iterative Quantization (GNP-IQ) scheme to quantize weights into lower-bits while effectively decreasing quantization errors. By leveraging memory conserved from GNP-IQ, we subsequently employ Interactive Side Mixture-of-Experts (ISMoE) to scaling up side networks without sacrificing overall memory efficiency. Different from conventional mixture-of-experts, ISMoE learns to select optimal experts by interacting with salient features from frozen backbones, thus suppressing knowledge forgetting and boosting performance. Extensive experiments across diverse vision-language and language-only tasks demonstrate that MP-ISMoE remarkably promotes accuracy compared to state-of-the-art METL approaches, while maintaining comparable parameter and memory efficiency.

Figures & tables

Figure 1: Diagram on the proposed framework. Part (I) presents Mixed-Precision Interactive Side Mixture-of-Experts for universal METL based on side networks. We first quantize most of weights in pre-trained backbone before fine-tuning, while preserving trainable Layer Normalization with full-precision. Subsequently, we introduce a learnable parallel Interactive Side Mixture-of-Experts (ISMoE) structure, connected to backbone via downsampling module. During fine-tuning phase, Gaussian Noise Perturbed Iterative Quantization (GNP-IQ) strategy introduces Gaussian noise perturbations into weights to mitigate quantization error. Part (II) illustrates detailed structure of $l^{th}$ layer of MP-ISMoE. We adopt MoE to scale up side network, design representative feature for each expert to measure correlation with backbone, and select Topk experts based on the routing Probability and Correlation Score.

Method	Params.	Mem.	Flickr30K			MSCOCO1K			MSCOCO5K			Params.	Mem.	MSR-VTT			MSVD
Method	(M) $\!\downarrow$	(G) $\!\downarrow$	I-T $\!\uparrow\$	T-I $\!\uparrow\$	Rsum $\!\uparrow\$	I-T $\!\uparrow\$	T-I $\!\uparrow\$	Rsum $\!\uparrow\$	I-T $\!\uparrow\$	T-I $\!\uparrow\$	Rsum $\!\uparrow\$	(M) $\!\downarrow$	(G) $\!\downarrow$	T-V $\!\uparrow\$	V-T $\!\uparrow\$	Rsum $\!\uparrow\$	T-V $\!\uparrow\$	V-T $\!\uparrow\$	Rsum $\!\uparrow\$
Fully-FT	201.2	176.8	85.6	73.3	546.6	83.1	71.7	542.7	64.2	51.2	468.9	151.3	48.8	42.8	42.1	389.2	45.2	57.1	425.5
LST	9.7	24.4	82.1	66.5	529.5	78.2	64.8	525.8	57.8	43.1	434.5	11.2	32.0	37.0	37.8	356.7	35.5	55.4	407.2
UniPT	12.4	24.4	84.8	69.1	537.4	80.6	67.5	532.9	61.1	45.9	445.3	9.6	13.6	38.9	39.3	361.3	40.9	59.7	432.1
SHERL	11.3	24.4	86.1	71.1	542.3	81.8	69.2	537.5	62.5	47.3	450.8	9.6	13.6	39.2	40.6	363.7	40.9	60.2	429.7
Ours †	12.9	25.5	86.5	71.3	543.1	81.9	69.1	538.7	62.2	47.1	449.1	10.1	14.5	39.9	41.1	365.5	42.0	60.4	435.6

Table 1: Comparison results (%) with METL approaches across various architectures and distinct VL tasks, in terms of amount of learnable parameters, memory usage, and other task-specific metrics. The best results are highlighted in bold , and the second best results are underlined .

Method	Params.	Memory ( G) $\downarrow$		CoLA	SST-2	MRPC	QQP	MNLI	QNLI	RTE	STS-B	Avg.
Method	(%) $\downarrow$	Train	Test	CoLA	SST-2	MRPC	QQP	MNLI	QNLI	RTE	STS-B	Avg.
Fully-FT	100	17.6	0.86	62.8	93.9	91.9	89.9	86.2	92.5	74.1	90.3	85.2
Adapter	1.63	13.0	0.87	64.4	94.2	88.9	88.9	86.4	93.1	75.1	91.1	85.3
LoRA	1.71	12.6	0.86	63.3	94.3	90.1	89.0	86.3	93.2	75.5	90.9	85.3
BitFit	0.13	10.7	0.86	61.8	94.3	91.0	88.7	85.6	93.1	67.6	90.8	84.1
Prompt	0.03	22.2	0.87	0	90.3	74.6	88.5	82.5	92.5	59.5	90.1	72.2

Table 2: Comparison results with PETL (Top) and METL (Bottom) methods on GLUE benchmark, with T5-base/large . We report the number of learnable parameters and memory usage as efficiency metrics, and accuracy, F1 score, Matthew’s Correlation, and Pearson-Spearman Correlation as performance indicators. The best results are highlighted in bold , and the second best results are underlined .

GNP-IQ	ISMoE	Params. (M) $\downarrow$	Memory (G) $\downarrow$	Flickr30K			MSCOCO1K			MSCOCO5K
GNP-IQ	ISMoE	Params. (M) $\downarrow$	Memory (G) $\downarrow$	I-T $\uparrow$	T-I $\uparrow$	Rsum $\uparrow$	I-T $\uparrow$	T-I $\uparrow$	Rsum $\uparrow$	I-T $\uparrow$	T-I $\uparrow$	Rsum $\uparrow$
		12.4	24.4	84.8	69.1	537.4	80.6	67.5	532.9	61.1	45.9	445.3
✓		12.5	18.4	83.7	68.1	534.8	78.9	66.3	530.4	59.9	44.7	442.7
	✓	12.7	32.4	86.9	71.7	544.3	82.5	69.8	540.4	63.4	47.7	450.9
✓	✓	12.9	25.5	86.5	71.3	543.1	81.9	69.1	538.7	62.2	47.1	449.1

Table 3: Ablation results (%) of the main components using VSE

\infty

on ITR tasks. The best results are highlighted in bold , and the second best results are underlined .

Weight Precision	Gaussian Noise	Params. (M) $\downarrow$	Memory (G) $\downarrow$	Flickr30K			MSCOCO1K			MSCOCO5K
Weight Precision	Gaussian Noise	Params. (M) $\downarrow$	Memory (G) $\downarrow$	I-T $\uparrow$	T-I $\uparrow$	Rsum $\uparrow$	I-T $\uparrow$	T-I $\uparrow$	Rsum $\uparrow$	I-T $\uparrow$	T-I $\uparrow$	Rsum $\uparrow$
Full-Pre		12.4	24.4	84.8	69.1	537.4	80.6	67.5	532.9	61.1	45.9	445.3
Low-Pre		12.4	14.8	81.6	66.4	529.3	77.5	64.6	524.7	58.4	43.0	433.8
Mixed-Pre		12.4	15.4	82.1	66.7	531.4	77.8	65.0	526.9	58.8	43.5	437.2
Mixed-Pre	✓	12.5	18.4	83.7	68.1	534.8	78.9	66.3	530.4	59.9	44.7	442.7

Table 4: ( Top ) Ablation results (%) of GNP-IQ with various backbone weights precision (Full-, Mixed-, and Low-Precision), w/ or w/o Gaussian noise perturbation. ( Bottom ) Ablation results (%) of ISMoE w/ or w/o MoE structure and measuring correlation between networks. All experiments are conducted on ITR tasks with VSE

\infty

. The best results are highlighted in bold , and the second best results are underlined .

Transfer Learning

Transfer learning studies how models reuse knowledge across tasks, domains, or modalities when target data or computation are limited.

See all 417 papers

Memory-Efficient Transfer Learning with Fading Side Networks via Masked Dual Path Distillation
Apr 10, 2026Yutong Zhang, Jiaxin Chen, Honglin Chen +5Memory-Efficient Fine-Tuning Efficient Neural Network Inference
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, China · School of Computer Science and Engineering, Beihang University, China · College of Information Technology, United Arab Emirates University, United Arab Emirates
GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMs
May 21, 2026Jianing Deng, Song Wang, Dongwei Wang +4Mixed-Precision Quantization LLM Quantization
University of Pittsburgh · University of Central Florida · University of Arizona +1
Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guarantees
Apr 7, 2026Mohammed Nowaz Rabbani Chowdhury, Kaoutar El Maghraoui, Hsinyu Tsai +4Mixture-of-Experts Quantization Neural Network Quantization
Rensselaer Polytechnic Institute · IBM Research

Apr 10, 2026J/K move · Enter open · S save

Memory-Efficient Transfer Learning with Fading Side Networks via Masked Dual Path Distillation

Yutong Zhang, Jiaxin Chen, Honglin Chen, Kaiqi Zheng+4

State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, China · School of Computer Science and Engineering, Beihang University, China · College of Information Technology, United Arab Emirates University, United Arab Emirates

Memory-Efficient Fine-Tuning Efficient Neural Network Inference Knowledge Distillation Feature-Based Knowledge Distillation

Open paper AskPDF arXiv

Memory-efficient transfer learning (METL) approaches have recently achieved promising performance in adapting pre-trained models to downstream tasks. They avoid applying gradient backpropagation in large backbones, thus significantly reducing the number of trainable parameters and high memory consumption during fine-tuning. However, since they typically employ a lightweight and learnable side network, these methods inevitably introduce additional memory and time overhead during inference, which contradicts the ultimate goal of efficient transfer learning. To address the above issue, we propose a novel approach dubbed Masked Dual Path Distillation (MDPD) to accelerate inference while retaining parameter and memory efficiency in fine-tuning with fading side networks. Specifically, MDPD develops a framework that enhances the performance by mutually distilling the frozen backbones and learnable side networks in fine-tuning, and discard the side network during inference without sacrificing accuracy. Moreover, we design a novel feature-based knowledge distillation method for the encoder structure with multiple layers. Extensive experiments on distinct backbones across vision/language-only and vision-and-language tasks demonstrate that our method not only accelerates inference by at least 25.2% while keeping parameter and memory consumption comparable, but also remarkably promotes the accuracy compared to SOTA approaches. The source code is available at https://github.com/Zhang-VKk/MDPD.

Abstract

Transfer Learning

Explore similar work

Memory-Efficient Transfer Learning with Fading Side Networks via Masked Dual Path Distillation

GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMs

Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guarantees