Organizations: Institute for Artificial Intelligence · School of Integrated Circuits · Sun Yat-sen University · Southeast University · Beijing Advanced Innovation Center for Integrated Circuits
Mixture-of-Experts (MoE) architecture has been widely adopted in large language models (LLMs) since it reduces the computational cost through sparse expert activation. However, the massive parameter scale of MoE models hinders their deployment on resource-constrained edge devices. Although existing mixed-precision quantization reduces the model size, they are inherently static, fixing the bit-width of each expert across decoding steps. Such static schemes yield suboptimal performance as they overlook the dynamic variation in expert importance during decoding. Furthermore, they fail to adapt to the dynamic latency constraint driven by the fluctuating system utilization. To address these challenges, we propose MatMoE, a dynamic mixed-precision quantization framework for efficient MoE inference. MatMoE features: (1) a Matryoshka MoE PTQ algorithm that enables experts to operate flexibly at multiple precisions within a stringent memory budget; (2) a floor-and-compensation quantization method that eliminates redundant memory accesses in the Matryoshka representation; and (3) a dynamic bit-width allocation strategy that determines the precision for each expert based on their routing scores to satisfy the dynamic latency requirement. We implement our MatMoE system atop llama.cpp for both CPU and GPU platforms. Extensive evaluations across three modern MoE models and challenging benchmarks demonstrate that MatMoE achieves 4.0%∼18.3% higher accuracy than state-of-the-art static mixed-precision method. Compared to llama.cpp, MatMoE delivers 1.3×∼3.7× speedup on Raspberry Pi 5, and 1.2×∼1.3× speedup on NVIDIA RTX 4090 and Jetson AGX Orin.
Figures & tables
Figure 1. MoE architecture and the comparison between the static and our proposed dynamic mixed-precision quantization. Static method assigns each expert a fixed bit-width offline, regardless of the input token. In contrast, our method dynamically assigns higher precision to more important activated experts and lower precision to less important ones.
Granularity
Quant. scheme
Efficiency
Any-Precision LLM ( Park et al., 2024 )
Model
Non-uniform (codebook)
Low
DP-LLM ( Kwon et al., 2025 )
Layer
Non-uniform (codebook)
Low
Matryoshka Quant. ( Nair et al., 2025 )
Layer
Uniform (extra bit rounding)
Low
Ours
Expert
Uniform (floor-and-compensation)
High
Table 1. Comparison of existing multi-scale quantization works with our proposed method.
Figure 2. The routing scores and expert ranking variations in a single layer of Qwen1.5-MoE-A2.7B ( Team, 2024 ) for the first sample in the GSM8K-CoT benchmark ( Cobbe et al., 2021 ) .
Figure 3. Overview of MatMoE.
Figure 4. (a) Distribution of the model weights quantized to 4-bit and the truncated 2-bit weights. (b) Distribution of the residual between the 4-bit weights and the 2-bit weights.
GSM8K-CoT
HumanEval
MBPP
Matryoshka Quant. ( Nair et al., 2025 )
21.9
14.6
17.0
Floor
0.0
0.0
0.0
Floor-and-Compensation
24.5
15.2
15.8
Table 2. Accuracy impact of different quantization schemes.
Figure 7
Figure 6. System implementation of MatMoE.
GSM8K-CoT
HumanEval
MBPP
Qwen
FP16
62.9
32.9
39.4
DeepSeek
FP16
39.4
28.0
43.2
OLMoE
FP16
41.0
31.1
27.8
Table 4. Accuracy of FP16 models.
Dataset
GSM8K-CoT
HumanEval
MBPP
Target Prec.
2.25
2.50
2.75
2.25
2.50
2.75
2.25
2.50
2.75
Qwen
PMQ
32.2
41.6
48.8
17.7
17.1
23.8
20.0
23.2
22.0
MatMoE
50.6
51.5
52.4
26.8
28.7
28.7
30.8
30.8
31.0
DeepSeek
PMQ
18.4
25.9
31.1
10.4
12.8
17.7
25.2
24.4
27.6
MatMoE
29.8
29.6
31.1
16.5
18.3
18.3
25.6
25.4
26.2
OLMoE
PMQ
9.9
15.6
22.8
3.7
3.0
3.7
1.4
1.6
2.6
Table 5. Accuracy comparison under 3-bit memory budget.
Dataset
GSM8K-CoT
HumanEval
MBPP
Target Prec.
2.25
2.50
2.75
3.00
3.25
3.50
3.75
2.25
2.50
2.75
3.00
3.25
3.50
3.75
2.25
2.50
2.75
3.00
3.25
3.50
3.75
Qwen
PMQ
32.2
41.6
48.8
56.0
52.8
56.0
57.3
17.7
17.1
23.8
20.1
18.9
25.0
30.5
20.0
23.2
22.0
27.0
30.8
32.6
34.8
MatMoE
50.6
52.1
55.3
55.7
56.5
56.9
56.9
31.1
34.2
34.2
33.5
33.5
33.5
35.4
31.4
34.2
36.0
36.2
36.4
36.4
36.6
DeepSeek
PMQ
18.4
25.9
31.2
34.7
34.8
38.9
39.3
10.4
12.8
17.7
22.6
18.9
25.0
25.6
25.2
24.4
27.6
30.6
33.4
34.4
36.4
MatMoE
32.1
36.2
36.2
36.6
36.6
36.5
36.8
25.0
26.2
26.8
26.2
26.6
26.6
25.6
39.6
41.6
42.6
41.0
42.2
41.8
42.4
OLMoE
PMQ
9.9
15.6
22.8
27.7
23.5
28.4
31.5
3.7
3.0
3.7
3.7
6.1
6.7
9.8
1.4
1.6
2.6
6.8
6.4
5.2
13.4
Table 6. Accuracy comparison under 4-bit memory budget.
Figure 7. Latency comparison on Raspberry Pi 5 across three models at different target precisions.
Figure 8. Latency comparison on NVIDIA RTX 4090 and Jetson Orin across three models at different target precisions.
Figure 14
Target precision
2.25
2.50
2.75
3.00
3.25
3.50
3.75
PMQ + Llama.cpp
TPOT
76.6
77.8
79.1
80.3
81.5
82.7
83.9
Accuracy
9.9
15.6
22.8
27.7
23.5
28.4
31.5
PMQ + Our kernels
TPOT
24.9
25.6
26.6
27.3
28.4
29.2
30.0
Accuracy
9.9
15.6
22.8
27.7
23.5
28.4
31.5
Matryoshka ( Nair et al., 2025 ) + Dynamic
TPOT
30.1
30.3
30.4
30.5
30.7
30.8
31.0
Accuracy
26.1
31.1
34.9
35.6
37.0
37.5
38.9
Table 7. Latency and accuracy comparison under 4-bit memory budget. We evaluate OLMoE on GSM8k-CoT on Raspberry Pi 5 with 4-thread inference.