Organizations: Institute for Artificial Intelligence · School of Integrated Circuits · Sun Yat-sen University · Southeast University · Beijing Advanced Innovation Center for Integrated Circuits
Mixture-of-Experts (MoE) architecture has been widely adopted in large language models (LLMs) since it reduces the computational cost through sparse expert activation. However, the massive parameter scale of MoE models hinders their deployment on resource-constrained edge devices. Although existing mixed-precision quantization reduces the model size, they are inherently static, fixing the bit-width of each expert across decoding steps. Such static schemes yield suboptimal performance as they overlook the dynamic variation in expert importance during decoding. Furthermore, they fail to adapt to the dynamic latency constraint driven by the fluctuating system utilization. To address these challenges, we propose MatMoE, a dynamic mixed-precision quantization framework for efficient MoE inference. MatMoE features: (1) a Matryoshka MoE PTQ algorithm that enables experts to operate flexibly at multiple precisions within a stringent memory budget; (2) a floor-and-compensation quantization method that eliminates redundant memory accesses in the Matryoshka representation; and (3) a dynamic bit-width allocation strategy that determines the precision for each expert based on their routing scores to satisfy the dynamic latency requirement. We implement our MatMoE system atop llama.cpp for both CPU and GPU platforms. Extensive evaluations across three modern MoE models and challenging benchmarks demonstrate that MatMoE achieves 4.0%∼18.3% higher accuracy than state-of-the-art static mixed-precision method. Compared to llama.cpp, MatMoE delivers 1.3×∼3.7× speedup on Raspberry Pi 5, and 1.2×∼1.3× speedup on NVIDIA RTX 4090 and Jetson AGX Orin.
Figures & tables
Figure 1. MoE architecture and the comparison between the static and our proposed dynamic mixed-precision quantization. Static method assigns each expert a fixed bit-width offline, regardless of the input token. In contrast, our method dynamically assigns higher precision to more important activated experts and lower precision to less important ones.
Granularity
Quant. scheme
Efficiency
Any-Precision LLM ( Park et al., 2024 )
Model
Non-uniform (codebook)
Low
DP-LLM ( Kwon et al., 2025 )
Layer
Non-uniform (codebook)
Low
Matryoshka Quant. ( Nair et al., 2025 )
Layer
Uniform (extra bit rounding)
Low
Ours
Expert
Uniform (floor-and-compensation)
High
Table 1. Comparison of existing multi-scale quantization works with our proposed method.
Figure 2. The routing scores and expert ranking variations in a single layer of Qwen1.5-MoE-A2.7B ( Team, 2024 ) for the first sample in the GSM8K-CoT benchmark ( Cobbe et al., 2021 ) .
Figure 3. Overview of MatMoE.
Figure 4. (a) Distribution of the model weights quantized to 4-bit and the truncated 2-bit weights. (b) Distribution of the residual between the 4-bit weights and the 2-bit weights.
GSM8K-CoT
HumanEval
MBPP
Matryoshka Quant. ( Nair et al., 2025 )
21.9
14.6
17.0
Floor
0.0
0.0
0.0
Floor-and-Compensation
24.5
15.2
15.8
Table 2. Accuracy impact of different quantization schemes.
Figure 7
Figure 6. System implementation of MatMoE.
GSM8K-CoT
HumanEval
MBPP
Qwen
FP16
62.9
32.9
39.4
DeepSeek
FP16
39.4
28.0
43.2
OLMoE
FP16
41.0
31.1
27.8
Table 4. Accuracy of FP16 models.
Dataset
GSM8K-CoT
HumanEval
MBPP
Target Prec.
2.25
2.50
2.75
2.25
2.50
2.75
2.25
2.50
2.75
Qwen
PMQ
32.2
41.6
48.8
17.7
17.1
23.8
20.0
23.2
22.0
MatMoE
50.6
51.5
52.4
26.8
28.7
28.7
30.8
30.8
31.0
DeepSeek
PMQ
18.4
25.9
31.1
10.4
12.8
17.7
25.2
24.4
27.6
MatMoE
29.8
29.6
31.1
16.5
18.3
18.3
25.6
25.4
26.2
OLMoE
PMQ
9.9
15.6
22.8
3.7
3.0
3.7
1.4
1.6
2.6
Table 5. Accuracy comparison under 3-bit memory budget.
Dataset
GSM8K-CoT
HumanEval
MBPP
Target Prec.
2.25
2.50
2.75
3.00
3.25
3.50
3.75
2.25
2.50
2.75
3.00
3.25
3.50
3.75
2.25
2.50
2.75
3.00
3.25
3.50
3.75
Qwen
PMQ
32.2
41.6
48.8
56.0
52.8
56.0
57.3
17.7
17.1
23.8
20.1
18.9
25.0
30.5
20.0
23.2
22.0
27.0
30.8
32.6
34.8
MatMoE
50.6
52.1
55.3
55.7
56.5
56.9
56.9
31.1
34.2
34.2
33.5
33.5
33.5
35.4
31.4
34.2
36.0
36.2
36.4
36.4
36.6
DeepSeek
PMQ
18.4
25.9
31.2
34.7
34.8
38.9
39.3
10.4
12.8
17.7
22.6
18.9
25.0
25.6
25.2
24.4
27.6
30.6
33.4
34.4
36.4
MatMoE
32.1
36.2
36.2
36.6
36.6
36.5
36.8
25.0
26.2
26.8
26.2
26.6
26.6
25.6
39.6
41.6
42.6
41.0
42.2
41.8
42.4
OLMoE
PMQ
9.9
15.6
22.8
27.7
23.5
28.4
31.5
3.7
3.0
3.7
3.7
6.1
6.7
9.8
1.4
1.6
2.6
6.8
6.4
5.2
13.4
Table 6. Accuracy comparison under 4-bit memory budget.
Figure 7. Latency comparison on Raspberry Pi 5 across three models at different target precisions.
Figure 8. Latency comparison on NVIDIA RTX 4090 and Jetson Orin across three models at different target precisions.
Figure 14
Target precision
2.25
2.50
2.75
3.00
3.25
3.50
3.75
PMQ + Llama.cpp
TPOT
76.6
77.8
79.1
80.3
81.5
82.7
83.9
Accuracy
9.9
15.6
22.8
27.7
23.5
28.4
31.5
PMQ + Our kernels
TPOT
24.9
25.6
26.6
27.3
28.4
29.2
30.0
Accuracy
9.9
15.6
22.8
27.7
23.5
28.4
31.5
Matryoshka ( Nair et al., 2025 ) + Dynamic
TPOT
30.1
30.3
30.4
30.5
30.7
30.8
31.0
Accuracy
26.1
31.1
34.9
35.6
37.0
37.5
38.9
Table 7. Latency and accuracy comparison under 4-bit memory budget. We evaluate OLMoE on GSM8k-CoT on Raspberry Pi 5 with 4-thread inference.
Mixture-of-Experts Large Language Models (MoE-LLMs) achieve strong performance but incur substantial memory overhead due to massive expert parameters. Mixed-precision quantization mitigates this cost by allocating expert-wise bit-widths based on their importance, approaching the accuracy-memory Pareto frontier and enabling extreme low-bit quantization. However, existing methods rely on layer-wise importance estimation and overlook router shifts induced by quantization, resulting in suboptimal allocation and routing. In this work, we propose Global Expert-level Mixed-precision Quantization (GEMQ) to overcome these limitations via (1) a global linear-programming formulation that captures model-wide expert importance based on quantization error analysis, and (2) efficient router fine-tuning to adapt routing to quantized experts. These components are integrated into a progressive quantization framework that iteratively refines importance estimation and allocation. Experiments demonstrate that GEMQ significantly reduces memory and accelerates inference with minimal accuracy degradation. Source code is available at https://github.com/jndeng/GEMQ .
Jianing Deng, Song Wang, Dongwei Wang +4
University of Pittsburgh · University of Central Florida · University of Arizona +1
Mixture-of-Experts (MoE) has become a practical architecture for scaling LLM capacity while keeping per-token compute modest, but deploying MoE models on a single, memory-limited GPU remains difficult because expert weights dominate the HBM footprint. Existing expert offloading and prefetching systems reduce the resident set, yet they often pay expert-loading costs on the critical path when activation becomes dense. Post-training quantization (PTQ) lowers the footprint without transfers, but prevailing pipelines fix expert bit-widths offline and assume routing remains stable, even though MoE expert utilization is heavy-tailed and the hot set can shift across workloads. We present DynaExq, a runtime-aware mixed-precision serving system that treats single-GPU MoE inference under a hard HBM envelope as an online, budget-constrained precision allocation problem. The key insight is to keep the experts that dominate runtime traffic resident at higher precision, while maintaining a low-precision fallback for the remaining experts, so the system can reduce transfer volume and avoid the waiting latency that limits offloading and prefetching under dense activation. DynaExq estimates long-horizon expert hotness from router traces, selects a per-layer high-precision resident set via a budget-feasible top-n rule, and applies promotions and demotions asynchronously through stable expert handles so the forward pass always executes on a fully materialized expert version. Across Qwen3-MoE-30B/80B and six benchmarks, DynaExq improves accuracy over static PTQ on Qwen3-80B (73.09% to 77.57%) under comparable device-memory budgets and achieves up to 2.73x higher throughput than offloading/prefetch baselines at batch size 32.
Kexin Chu, Dawei Xiang, Zixu Shen +3
University of Connecticut · University of California, Santa Cruz · Independent Researcher
Sparse Mixture-of-Experts (MoE) allows scaling of language and vision models efficiently by activating only a small subset of experts per input. While this reduces computation, the large number of parameters still incurs substantial memory overhead during inference. Post-training quantization has been explored to address this issue. Because uniform quantization suffers from significant accuracy loss at low bit-widths, mixed-precision methods have been recently explored; however, they often require substantial computation for bit-width allocation and overlook the varying sensitivity of model performance to the quantization of different experts. We propose a theoretically grounded expert-wise mixed precision strategy that assigns bit-width to each expert primarily based on their change in routers l2 norm during training. Experts with smaller changes are shown to capture less frequent but critical features, and model performance is more sensitive to the quantization of these experts, thus requiring higher precision. Furthermore, to avoid allocating experts to lower precision that inject high quantization noise, experts with large maximum intra-neuron variance are also allocated higher precision. Experiments on large-scale MoE models, including Switch Transformer and Mixtral, show that our method achieves higher accuracy than existing approaches, while also reducing inference cost and incurring only negligible overhead for bit-width assignment.
Mohammed Nowaz Rabbani Chowdhury, Kaoutar El Maghraoui, Hsinyu Tsai +4