Organizations: State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China · School of Computer Science and Engineering, Beihang University, Beijing, China · College of Information Technology, United Arab Emirates University, Al Ain, Abu Dhabi, United Arab Emirates
Parameter-efficient transfer learning (PETL) has emerged as a pivotal paradigm for adapting pre-trained foundation models to downstream tasks, significantly reducing trainable parameters yet suffering from substantial memory overhead caused by gradient backpropagation during fine-tuning. While memory-efficient transfer learning (METL) circumvents this challenge by bypassing backbone gradient computation via lightweight small side networks, its stringent memory constraint severely limits learning capacity of side networks, thereby significantly compromising performance. To address these limitations, we propose a novel Mixed-Precision Interactive Side Mixture-of-Experts framework (MP-ISMoE). Specifically, we first propose a Gaussian Noise Perturbed Iterative Quantization (GNP-IQ) scheme to quantize weights into lower-bits while effectively decreasing quantization errors. By leveraging memory conserved from GNP-IQ, we subsequently employ Interactive Side Mixture-of-Experts (ISMoE) to scaling up side networks without sacrificing overall memory efficiency. Different from conventional mixture-of-experts, ISMoE learns to select optimal experts by interacting with salient features from frozen backbones, thus suppressing knowledge forgetting and boosting performance. Extensive experiments across diverse vision-language and language-only tasks demonstrate that MP-ISMoE remarkably promotes accuracy compared to state-of-the-art METL approaches, while maintaining comparable parameter and memory efficiency.
Figures & tables
Figure 1: Diagram on the proposed framework. Part (I) presents Mixed-Precision Interactive Side Mixture-of-Experts for universal METL based on side networks. We first quantize most of weights in pre-trained backbone before fine-tuning, while preserving trainable Layer Normalization with full-precision. Subsequently, we introduce a learnable parallel Interactive Side Mixture-of-Experts (ISMoE) structure, connected to backbone via downsampling module. During fine-tuning phase, Gaussian Noise Perturbed Iterative Quantization (GNP-IQ) strategy introduces Gaussian noise perturbations into weights to mitigate quantization error. Part (II) illustrates detailed structure of lth layer of MP-ISMoE. We adopt MoE to scale up side network, design representative feature for each expert to measure correlation with backbone, and select Topk experts based on the routing Probability and Correlation Score.
Method
Params.
Mem.
Flickr30K
MSCOCO1K
MSCOCO5K
Params.
Mem.
MSR-VTT
MSVD
(M) ↓
(G) ↓
I-T \!\uparrow\
T-I \!\uparrow\
Rsum \!\uparrow\
I-T \!\uparrow\
T-I \!\uparrow\
Rsum \!\uparrow\
I-T \!\uparrow\
T-I \!\uparrow\
Rsum \!\uparrow\
(M) ↓
(G) ↓
T-V \!\uparrow\
V-T \!\uparrow\
Rsum \!\uparrow\
T-V \!\uparrow\
V-T \!\uparrow\
Rsum \!\uparrow\
Fully-FT
201.2
176.8
85.6
73.3
546.6
83.1
71.7
542.7
64.2
51.2
468.9
151.3
48.8
42.8
42.1
389.2
45.2
57.1
425.5
LST
9.7
24.4
82.1
66.5
529.5
78.2
64.8
525.8
57.8
43.1
434.5
11.2
32.0
37.0
37.8
356.7
35.5
55.4
407.2
UniPT
12.4
24.4
84.8
69.1
537.4
80.6
67.5
532.9
61.1
45.9
445.3
9.6
13.6
38.9
39.3
361.3
40.9
59.7
432.1
SHERL
11.3
24.4
86.1
71.1
542.3
81.8
69.2
537.5
62.5
47.3
450.8
9.6
13.6
39.2
40.6
363.7
40.9
60.2
429.7
Ours †
12.9
25.5
86.5
71.3
543.1
81.9
69.1
538.7
62.2
47.1
449.1
10.1
14.5
39.9
41.1
365.5
42.0
60.4
435.6
Table 1: Comparison results (%) with METL approaches across various architectures and distinct VL tasks, in terms of amount of learnable parameters, memory usage, and other task-specific metrics. The best results are highlighted in bold , and the second best results are underlined .
Method
Params.
Memory ( G) ↓
CoLA
SST-2
MRPC
QQP
MNLI
QNLI
RTE
STS-B
Avg.
(%) ↓
Train
Test
Fully-FT
100
17.6
0.86
62.8
93.9
91.9
89.9
86.2
92.5
74.1
90.3
85.2
Adapter
1.63
13.0
0.87
64.4
94.2
88.9
88.9
86.4
93.1
75.1
91.1
85.3
LoRA
1.71
12.6
0.86
63.3
94.3
90.1
89.0
86.3
93.2
75.5
90.9
85.3
BitFit
0.13
10.7
0.86
61.8
94.3
91.0
88.7
85.6
93.1
67.6
90.8
84.1
Prompt
0.03
22.2
0.87
0
90.3
74.6
88.5
82.5
92.5
59.5
90.1
72.2
Table 2: Comparison results with PETL (Top) and METL (Bottom) methods on GLUE benchmark, with T5-base/large . We report the number of learnable parameters and memory usage as efficiency metrics, and accuracy, F1 score, Matthew’s Correlation, and Pearson-Spearman Correlation as performance indicators. The best results are highlighted in bold , and the second best results are underlined .
GNP-IQ
ISMoE
Params. (M) ↓
Memory (G) ↓
Flickr30K
MSCOCO1K
MSCOCO5K
I-T ↑
T-I ↑
Rsum ↑
I-T ↑
T-I ↑
Rsum ↑
I-T ↑
T-I ↑
Rsum ↑
12.4
24.4
84.8
69.1
537.4
80.6
67.5
532.9
61.1
45.9
445.3
✓
12.5
18.4
83.7
68.1
534.8
78.9
66.3
530.4
59.9
44.7
442.7
✓
12.7
32.4
86.9
71.7
544.3
82.5
69.8
540.4
63.4
47.7
450.9
✓
✓
12.9
25.5
86.5
71.3
543.1
81.9
69.1
538.7
62.2
47.1
449.1
Table 3: Ablation results (%) of the main components using VSE ∞ on ITR tasks. The best results are highlighted in bold , and the second best results are underlined .
Weight Precision
Gaussian Noise
Params. (M) ↓
Memory (G) ↓
Flickr30K
MSCOCO1K
MSCOCO5K
I-T ↑
T-I ↑
Rsum ↑
I-T ↑
T-I ↑
Rsum ↑
I-T ↑
T-I ↑
Rsum ↑
Full-Pre
12.4
24.4
84.8
69.1
537.4
80.6
67.5
532.9
61.1
45.9
445.3
Low-Pre
12.4
14.8
81.6
66.4
529.3
77.5
64.6
524.7
58.4
43.0
433.8
Mixed-Pre
12.4
15.4
82.1
66.7
531.4
77.8
65.0
526.9
58.8
43.5
437.2
Mixed-Pre
✓
12.5
18.4
83.7
68.1
534.8
78.9
66.3
530.4
59.9
44.7
442.7
Table 4: ( Top ) Ablation results (%) of GNP-IQ with various backbone weights precision (Full-, Mixed-, and Low-Precision), w/ or w/o Gaussian noise perturbation. ( Bottom ) Ablation results (%) of ISMoE w/ or w/o MoE structure and measuring correlation between networks. All experiments are conducted on ITR tasks with VSE ∞ . The best results are highlighted in bold , and the second best results are underlined .
Memory-efficient transfer learning (METL) approaches have recently achieved promising performance in adapting pre-trained models to downstream tasks. They avoid applying gradient backpropagation in large backbones, thus significantly reducing the number of trainable parameters and high memory consumption during fine-tuning. However, since they typically employ a lightweight and learnable side network, these methods inevitably introduce additional memory and time overhead during inference, which contradicts the ultimate goal of efficient transfer learning. To address the above issue, we propose a novel approach dubbed Masked Dual Path Distillation (MDPD) to accelerate inference while retaining parameter and memory efficiency in fine-tuning with fading side networks. Specifically, MDPD develops a framework that enhances the performance by mutually distilling the frozen backbones and learnable side networks in fine-tuning, and discard the side network during inference without sacrificing accuracy. Moreover, we design a novel feature-based knowledge distillation method for the encoder structure with multiple layers. Extensive experiments on distinct backbones across vision/language-only and vision-and-language tasks demonstrate that our method not only accelerates inference by at least 25.2% while keeping parameter and memory consumption comparable, but also remarkably promotes the accuracy compared to SOTA approaches. The source code is available at https://github.com/Zhang-VKk/MDPD.
Yutong Zhang, Jiaxin Chen, Honglin Chen +5
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, China · School of Computer Science and Engineering, Beihang University, China · College of Information Technology, United Arab Emirates University, United Arab Emirates
Mixture-of-Experts Large Language Models (MoE-LLMs) achieve strong performance but incur substantial memory overhead due to massive expert parameters. Mixed-precision quantization mitigates this cost by allocating expert-wise bit-widths based on their importance, approaching the accuracy-memory Pareto frontier and enabling extreme low-bit quantization. However, existing methods rely on layer-wise importance estimation and overlook router shifts induced by quantization, resulting in suboptimal allocation and routing. In this work, we propose Global Expert-level Mixed-precision Quantization (GEMQ) to overcome these limitations via (1) a global linear-programming formulation that captures model-wide expert importance based on quantization error analysis, and (2) efficient router fine-tuning to adapt routing to quantized experts. These components are integrated into a progressive quantization framework that iteratively refines importance estimation and allocation. Experiments demonstrate that GEMQ significantly reduces memory and accelerates inference with minimal accuracy degradation. Source code is available at https://github.com/jndeng/GEMQ .
Jianing Deng, Song Wang, Dongwei Wang +4
University of Pittsburgh · University of Central Florida · University of Arizona +1
Sparse Mixture-of-Experts (MoE) allows scaling of language and vision models efficiently by activating only a small subset of experts per input. While this reduces computation, the large number of parameters still incurs substantial memory overhead during inference. Post-training quantization has been explored to address this issue. Because uniform quantization suffers from significant accuracy loss at low bit-widths, mixed-precision methods have been recently explored; however, they often require substantial computation for bit-width allocation and overlook the varying sensitivity of model performance to the quantization of different experts. We propose a theoretically grounded expert-wise mixed precision strategy that assigns bit-width to each expert primarily based on their change in routers l2 norm during training. Experts with smaller changes are shown to capture less frequent but critical features, and model performance is more sensitive to the quantization of these experts, thus requiring higher precision. Furthermore, to avoid allocating experts to lower precision that inject high quantization noise, experts with large maximum intra-neuron variance are also allocated higher precision. Experiments on large-scale MoE models, including Switch Transformer and Mixtral, show that our method achieves higher accuracy than existing approaches, while also reducing inference cost and incurring only negligible overhead for bit-width assignment.
Mohammed Nowaz Rabbani Chowdhury, Kaoutar El Maghraoui, Hsinyu Tsai +4