DuoRate: Two-Rate Per-Expert Precision Allocation for Mixture-of-Experts Serving

Mengqi Guo; Zhe Wei; Yuan Yuan; Jiunn Bin Lim; Boyi Pan; Michael Bi Mi

cs.LGOct 9, 2026

DuoRate: Two-Rate Per-Expert Precision Allocation for Mixture-of-Experts Serving

Authors: Mengqi Guo, Zhe Wei, Yuan Yuan, Jiunn Bin Lim, Boyi Pan, Michael Bi Mi

Organizations: Huawei Technologies Ltd.

Abstract

Serving a Mixture-of-Experts (MoE) model pays for expert precision twice: the resident footprint pays for every expert the router can reach, while each batch-one decode step pays only for the experts it selects. These are two different rate functionals over the same per-expert widths, yet published per-expert MoE allocators constrain only the first. We call the difference the store-read gap; in the released BitsMoE checkpoint, 2.54 bits per weight (bpw) of storage become 7.50 bpw of decode traffic. We propose DuoRate, which stores once and reads per workload. Offline, it stores one uniform tier of a prefix-readable weight stream and bakes a distortion tensor. Online, it profiles a workload's own prefill without labels and solves a Lagrangian rate-distortion allocation under the decode functional; routing frequency provably cancels from every per-expert decision, and the solve costs one dot product per expert and a bisection. A deployment therefore keeps a single artifact that prefill reads at the stored rate and decode reads at the workload's own rate. On Qwen3-30B-A3B, DuoRate reading 2.3 bpw exceeds every 2-bit-class baseline by 6.3 to 47.0 points of average accuracy, and at 2.9 bpw it matches 4-bit formats that read 1.4-1.6 bpw more. The advantage is largest under long decoding: on GSM8K, DuoRate at 2.9 bpw matches or exceeds every 4-bit format and trails BF16 by only 0.25 points, and at 2.6 bpw it still matches or exceeds the 4.5-bit NF4 and FP4. Against controls in its own format, the allocation alone is worth 0.07-0.11 bpw of read rate. Code will be released upon publication.

Figures & tables

Figure 1: The DuoRate pipeline. Stage 1 (offline, once) stores one uniform tier of a prefix-readable stream and bakes the distortion tensor

S

. Stage 2 (per workload) profiles the workload’s own prefill without labels, scores every expert and tier by one dot product, and takes a per-expert Lagrangian

\arg\min

with one shared

\lambda

, from which

f_{e}

cancels; a bisection on

\lambda

meets the read budget

B

. Serving : prefill pays

R_{\text{store}}

, decode pays

R_{\text{read}}

Figure 2: Frequency cancellation (Proposition 1 , Corollary 1 ); illustrative values. (a) For two experts eightfold apart in

f

, supporting lines of slope

-\lambda

d_{e}

and

-\lambda f_{e}

f_{e}d_{e}

touch the same tier, so the

\arg\min

does not depend on

f

. (b) On the read set

\{\mathrm{p2},\mathrm{p3},\mathrm{p4}\}

the thresholds fix one family of tier maps; routing frequencies

f

and

g

only change where the bisection stops.

Method	BPW	PPL $\downarrow$	Accuracy $\uparrow$ (%)			Avg. $\uparrow$
	Read	WikiText-2	MMLU-Pro	MMLU	GSM8K
BF16	16.0	8.27	56.86	83.30	91.25	77.14
NVFP4	4.5	8.44	56.79	83.09	90.25	76.71
NF4	4.5	8.41	56.79	82.11	89.50	76.13
FP4	4.5	8.62	56.43	81.75	88.25	75.48
MXFP4	4.3	8.58	56.71	82.53	90.25	76.50

Table 1: Results on Qwen3-30B-A3B-Base. BPW is the rate read per activated expert weight at batch-one decode (for DuoRate , the read budget); DuoRate stores

4.3

. ∗ BitsMoE stores

2.5

;

7.5

is its read rate. GSM8K is strict match; Avg. is the mean of MMLU-Pro, MMLU and GSM8K. Bold is the best in each group.

Figure 3: Average accuracy (MMLU-Pro, MMLU, GSM8K) against the rate read per activated weight at batch-one decode. (a) Every row of Table 1 . (b) The top-left corner: DuoRate at

2.9~\text{bpw}

sits with the 4-bit formats, which read

1.4

–

1.6~\text{bpw}

more. Hollow markers are stored rates joined to read rates: BitsMoE stores

2.54

and reads

7.50

; DuoRate stores

4.27

and reads

2.9

Figure 4: What the allocation alone is worth (WikiText-2). (a) Perplexity against the budgeted read rate: DuoRate , random mixtures of two adjacent tiers (every draw), the interpolated uniform-tier null (dashed) and tables re-solved from a disjoint split (hollow). Brackets: gap to the strongest mixture; arrows: extra read rate the mixture needs to match (dotted: extrapolated); blue ticks: each table at its stored rate. (b) Gap to the null placed at either rate, with

95\%

bootstrap intervals.

Method	Resident		Decode, batch 1
	Store ( bpw )	GiB $\downarrow$	Read ( bpw )	MB/token $\downarrow$
BF16	16.0000	54.000	16.0000	3623.9
BitsMoE 2-bit	2.5425	8.581	7.5008	1698.9
DuoRate $2.3$	4.2704	14.413	2.3000 ‡	520.9 ‡
DuoRate $2.6$	4.2704	14.413	2.6000 ‡	588.9 ‡
DuoRate $2.9$	4.2704	14.413	2.9000 ‡	656.8 ‡

Table 2: Resident footprint and batch-one decode traffic of the expert weights, implied by each artifact’s stored and read rates. ‡ Read budget; the realized decode traffic on GSM8K is lower.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Unit	Rate constraint	Distortion signal	Frequency enters	Store $\neq$ read	Paired
MC-MoE / MC#	expert	unweighted, per block	output error $\times$ freq. α gate β	distortion prior	$\times$	$\times$
MxMoE	expert $\times$ proj. pair	memory, per layer	block-output error	cost term only	$\times$	$\times$
GEMQ	expert, global	unweighted, global	leave-one-out, CE-gradient	implicit at most	$\times$	$\times$
BitsMoE	spectral component	unweighted, per layer	spectral energy $\times$ analytic $D(b)$	importance weight	$\times$	$\times$
MODE	expert, per layer	unweighted, per layer	modality freq. $\times$ KL	distortion weight	$\times$	$\times$
AlphaQ	projection matrix	unweighted, global	analytic weight-domain	none	$\times$	$\times$

Appendix

Table S1: Objective-function audit of per-expert MoE allocators, read from the published objectives ( Huang et al., 2025b ; Huang et al., 2026 ; Duanmu et al., 2025 ; Deng et al., 2026 ; Zhao et al., 2026 ; Chen et al., 2026 ; Yang et al., 2026a ; Chitty-Venkata et al., 2025 ; Chu et al., 2025 ; Yi et al., 2023 ; Tang et al., 2024 ) . Rows are restricted to methods whose objectives we verified in full.

Figure S1: Where routing frequency survives (Proposition 2 in the main paper); illustrative values. (a) Divided by

f_{e}

, a bit costs

\lambda_{\mathrm{s}}/(Ef_{e})

under the stored functional but a constant

\lambda

under the read one. (b) Two experts with the same per-domain curves but different mixtures

\pi_{d\mid e}

take different tiers at one

\lambda

Figure S2: The stored stream (schematic). (a) One expert matrix in plane order, widths proportional to bits per weight: metadata read at every depth, base codes

q

, then planes

b_{3},\dots,b_{6}

; tier p

M

is a contiguous prefix ending at

R(M)

. DuoRate stores p4 and decode reads a p2, p3 or p4 prefix. (b) Equation ( S1 ): plane

k

moves the reconstruction by

\pm s/2^{k-1}

Field	bpw	Provenance
Base code, two bits per weight	2.000000	measured
Group record ( lo , s ), 8 bits per group of 32	0.250000	measured
Two 16-entry codebooks per matrix	0.000326	measured count, spec word
Sparse residual record, one per row	0.020074	measured count, spec word
Total, p2	2.270399
Published floor	2.2704

Appendix

Table S2: The floor of our ladder, recomputed field by field from the released base checkpoint over the

28{,}991{,}029{,}248

expert weights. Measured : count from the checkpoint; spec : word length from the format specification. Rows are rounded individually; the total is summed before rounding.

Method	Store	Read	PPL $\downarrow$	MMLU-Pro	MMLU	GSM8K	Avg.
BF16	16.000	16.000	8.27	56.86	83.30	91.25	77.14
Uniform p2	2.2704	2.2704	10.08	52.64	80.00	88.00	73.55
Uniform p3	3.2704	3.2704	8.69	55.14	81.96	88.50	75.20
Uniform p4	4.2704	4.2704	8.37	57.21	82.67	–	–
DuoRate	4.2704	2.900	9.07	56.00	81.47	91.00	76.16
DuoRate	4.2704	2.600	9.46	53.86	80.56	89.50	74.64

Appendix

Table S3: Stored and read rates of the uniform tiers of our ladder and of the DuoRate rows of Table 1 in the main paper, with BF16 and BitsMoE for reference. A uniform tier stores and reads the same width; DuoRate stores p4, and its Read is the budget each benchmark’s table was solved to. Uniform p4 was not scored on GSM8K.

Domain	Budget	$R_{\mathrm{store}}$	$R^{\mathrm{cal}}_{\mathrm{read}}$	$R^{\mathrm{dec}}_{\mathrm{read}}$	$R^{\mathrm{dec}}-R^{\mathrm{cal}}$
GSM8K	2.6	2.464	2.589	2.559	$-0.029$
	2.9	2.691	2.879	2.818	$-0.061$
MMLU	2.6	2.567	2.587	—	—
	2.9	2.822	2.880	—	—
WikiText-2	2.6	2.659	2.600	—	—
	2.9	2.917	2.899	—	—

Appendix

Table S4: Rates of the

2.6

and

2.9~\text{bpw}

read tables, recomputed from their tier histograms and the ladder of Eq. (1) in the main paper.

R_{\mathrm{store}}

is the unweighted rate of the read table itself, not of the resident p4 tier;

R^{\mathrm{cal}}_{\mathrm{read}}

is the budgeted rate under prefill routing;

R^{\mathrm{dec}}_{\mathrm{read}}

uses decode-step routing and exists only for GSM8K, since the log-likelihood benchmarks have no decode phase.

Figure S3: Per-document perplexity change of DuoRate on WikiText-2, sorted over the

62

documents. (a, b) Against a random mixture of two adjacent tiers at the table’s read rate (least favourable draw); (c) against the interpolated null at the stored rate, the weakest comparison in the upper block of Table S5 . Dashed: the aggregate change; open circles: exact ties, excluded from the sign test.

Arm	Rate definition	$\Delta$	Wins / losses, 95% CI	$p$
Read $2.6$	budgeted	$-1.46\%$	57 / 5, $[-1.90,-1.05]$	$3.1\times 10^{-12}$
	stored	$-0.59\%$	43 / 19, $[-1.06,-0.11]$	$0.0032$
Read $2.9$	budgeted	$-1.18\%$	48 / 14, $[-1.64,-0.74]$	$1.7\times 10^{-5}$
	stored	$-0.92\%$	46 / 16, $[-1.39,-0.47]$	$0.00018$
with the calibration split disjoint from the scored documents
Read $2.6$	budgeted	$-1.28\%$	56 / 6, $[-1.67,-0.90]$	$3.0\times 10^{-11}$

Appendix

Table S5: The WikiText-2 read tables against an interpolated uniform-tier null at their own rate, placed at the budgeted read rate and at the stored rate. Lower block: tables re-solved from the WikiText-2 validation split. Intervals are percentile bootstraps on the aggregate perplexity.

Comparison	Benchmark / rate	$\Delta$	Wins / losses, 95% CI	$p$
against a random mixture of the same tiers , WikiText-2
DuoRate $2.6$	mixture at $2.600$	$-1.35\%$	48 / 8, $[-1.76,-0.94]$	$4.7\times 10^{-8}$
	mixture at $2.700$	$-0.07\%$	32 / 23, $[-0.55,+0.47]$	$0.28$
DuoRate $2.9$	mixture at $2.900$	$-1.04\%$	45 / 9, $[-1.43,-0.64]$	$7.3\times 10^{-7}$
	mixture at $2.950$	$-0.28\%$	34 / 21, $[-0.73,+0.24]$	$0.10$
against uniform p2 ( $2.27~\text{bpw}$ )

Appendix

Table S6: Paired comparisons of the DuoRate read tables against controls in the same format. Top: a random mixture of two adjacent tiers at the table’s read rate and at a dearer one (worst of the draws). Bottom: the uniform tiers below and above the

2.9~\text{bpw}

table. Perplexity rows report document wins/losses, a bootstrap interval and an exact sign test; accuracy rows report exact McNemar.

Figure S4: Tier histograms of the

2.6

and

2.9~\text{bpw}

read tables. Each bar splits the

6{,}144

experts into p2 / p3 / p4; every read tier is a byte prefix of the stored p4 table (top). Right: the p4 count and the unweighted mean of

R(M_{e})

, the

R_{\mathrm{store}}

column of Table S4 .

Figure S5: Rate of each of the

48

layers in the

2.9~\text{bpw}

read tables, weighted by the calibration routing counts (what a decode step pays) and unweighted over the

128

experts of the layer. Dotted lines mark the p2 and p3 tiers; the dashed line is the read budget.

Figure S6: Per-field rates of the released BitsMoE 2-bit checkpoint, recomputed from its safetensors headers. (a) The expert segment stores

2.5425~\text{bpw}

; a batch-one decode step charges the FP16 shared basis to the

k{=}8

selected experts rather than all

E{=}128

, and with the never-fetched fields refunded reads

7.5008~\text{bpw}

. (b) Stored side fields; hatched parts (

0.041618~\text{bpw}

) are never fetched.

as published in Zhao et al. (2026) , their protocol
Method	Bits	PPL $\downarrow$	Accuracy $\uparrow$ (%)							Avg.
			HellaS.	MathQA	MMLU	Openb.	WinoG.	GSM8K	HumanE.
FP16	16	10.24	81.35	60.03	78.77	45.00	72.85	83.47	56.10	68.22
HQQ	2	23.65	63.05	33.17	48.82	36.40	60.85	22.67	11.59	39.51
GPTQ ( Frantar et al., 2023 )	2	15.63	70.16	24.89	39.17	39.40	60.62	4.32	0.00	34.08
MiLo	2	21.53	62.82	31.83	44.51	35.40	59.98	19.64	7.93	37.44

Appendix

Table S7: The published Qwen3-30B-A3B-Base results of Zhao et al. (2026) (upper half, C4 perplexity, zero-shot) with our rows under our protocol (lower half, WikiText-2 perplexity). The halves are not commensurable. Bits is the published label in the upper half; in the lower half it is store / read where the two differ.

Benchmark	Arm	Published	Ours	$\Delta$
aligned: 0-shot on both sides, full split on both sides
OpenBookQA (acc_norm)	unquantized	45.00	44.20	$-0.80$
	2-bit	43.40	43.20	$-0.20$
WinoGrande (acc)	unquantized	72.85	73.24	$+0.39$
	2-bit	72.93	72.14	$-0.79$
aligned, but generative and underpowered

Appendix

Table S8: Our re-scoring of the released BitsMoE checkpoint and of the unquantized model against that paper’s published numbers. Aligned : same shot count and same full split on both sides. Our unquantized MMLU figure is the full test set; our 2-bit figure is the

25

-per-subject sample of Table 1 in the main paper.

Target	Domain	Store-only: read	overspend	Read-only: store	shortfall
3.60	WikiText-2	4.718	$+31.1\%$	3.572	$-0.028$
	GSM8K	4.568	$+26.9\%$	3.522	$-0.078$
	code	4.411	$+22.5\%$	3.560	$-0.040$
3.20	WikiText-2	4.092	$+27.9\%$	3.176	$-0.024$
	GSM8K	3.972	$+24.1\%$	3.151	$-0.049$
	code	3.874	$+21.1\%$	3.156	$-0.044$

Appendix

Table S9: Constraining one rate and reading the other, on the additive instance. Store-only overspends the read target on all nine cells; read-only never exceeds the stored target.

Figure S7: Online recalibration on a

150

-document code corpus from a prose-calibrated allocation at

3.2~\text{bpw}

(earlier additive instance). (a) Experts changing precision at each step; after the third, fewer than

5\%

move. (b) Code-corpus perplexity: updating from the stream’s own unlabelled inputs closes

95.3\%

of the gap to an allocation calibrated offline on code.

Budget	Distortion functional	$R_{\mathrm{store}}$	$R_{\mathrm{read}}$	MMLU
2.600	none (rate-matched control)	2.600	2.595	81.26
	activation-weighted squared error	2.600	2.598	80.98
	decision-aligned	2.600	2.600	81.19
2.900	none	2.900	2.881	81.54
	squared error	2.900	2.912	80.42
	decision-aligned	2.845	2.900	81.47

Appendix

Table S10: The distortion functional at four budgets on the predecessor instance, MMLU accuracy (

n=1{,}425

). The control is a rate-matched random assignment between two adjacent tiers.

Figure S8: Five distortion functionals at a realized MMLU read rate of

3.000~\text{bpw}

(earlier tier family). (a) WikiText-2 perplexity; (b) MMLU-Pro accuracy, nearly its mirror image. The decision-aligned functional used by DuoRate is the only arm above activation-weighted squared error on MMLU-Pro at a nominal

p<0.05

Figure S9: The MC-MoE objective inside our bisection, with the same distortion table and cap (WikiText-2). (a) At matched read rate, DuoRate ’s perplexity relative to MC-MoE’s: the sign test detects the difference, the

95\%

interval includes zero. (b) The same tables at their stored rates: MC-MoE stores

0.29

and

0.43~\text{bpw}

less. Neither dominates.

Mixture-of-Experts Quantization

Mixture-of-Experts quantization compresses expert weights and computations to reduce memory use and speed up inference while preserving model quality.

See all 32 papers

Oct 15, 2026

DPA-MoE: A Runtime Precision Control Framework for MoE Inference

Mixture-of-Experts (MoE) models reduce computation by routing each token to a small set of experts, but each selected expert is still used at a fixed precision. We present DPA-MoE, a runtime precision control framework that makes routed experts any-precision. DPA-MoE encodes expert feed-forward weights as prefix-decodable chunks using SharedScale AnyBCQ, profiles per-expert reconstruction residuals offline, and allocates chunks at inference time using live router gate scores. This post-routing allocation lets a single encoded MoE checkpoint operate at multiple effective weight budgets without retraining or re-encoding. Experiments on Qwen3-30B-A3B and DeepSeekMoE-16B show that DPA-MoE improves KL divergence and downstream accuracy over uniform chunk allocation at matched budgets, with modest steady-state throughput overhead in our prototype.

Jaeyun Kim, Hoonki Lee, Susie Shin +3

Agency for Defense Development · Yonsei University

Mixture-of-Experts Inference Adaptive Inference Mixture-of-Experts Quantization

Ask PDF

Jun 3, 2026

AlphaQ: Calibration-Free Bit Allocation for Mixture-of-Experts Quantization

Mixture-of-Experts (MoE) architectures scale model capacity through sparse expert activation, but their deployment remains memory-bound because all expert weights must reside in memory. Mixed-precision quantization can substantially reduce this footprint by assigning different bit-widths to different experts. Existing approaches, however, typically rely on calibration data to estimate expert importance and determine bit allocation. For frontier MoE LLMs, the original training data, and hence the true training distribution, is proprietary and inaccessible. As a result, calibration sets are inevitably imperfect surrogates, and this can misestimate expert utilization and lead to suboptimal bit allocation. Motivated by the substantial cross-expert quality variability observed in modern MoE models, and by the success of Heavy-Tailed Self-Regularization (HT-SR) theory at predicting neural network model quality without access to training or testing data, we propose AlphaQ, a calibration-free bit-allocation method for MoE quantization. AlphaQ draws on HT-SR theory and follows a simple principle: experts with more heavy-tailed weight spectra are typically better trained and hence should receive higher bit-widths, while experts with weaker heavy-tailed structure can be quantized more aggressively. AlphaQ operationalizes this principle by measuring expert-wise spectral heavy-tailedness and solving a budget-constrained optimization problem that minimizes total quantization error under a global bit-budget constraint. Across several MoE models, AlphaQ consistently outperforms calibration-based baselines under matched bit budgets. Notably, on Qwen1.5-MoE, AlphaQ achieves near full-precision accuracy with an average expert precision of only 3.5 bits, while delivering more than 4

\times

memory compression. Our code is available at https://github.com/Superone77/AlphaQ.

Wanqi Yang, Yuexiao Ma, Alexander Conzelmann +4

Max Planck Institute for Intelligent Systems · ELLIS Institute Tübingen · Tübingen AI Center +5

Mixed-Precision Quantization Mixture-of-Experts Language Models Mixture-of-Experts Quantization Neural Network Quantization

Ask PDF

Apr 9, 2026

Alloc-MoE: Budget-Aware Expert Activation Allocation for Efficient Mixture-of-Experts Inference

Mixture-of-Experts (MoE) has become a dominant architecture for scaling large language models due to their sparse activation mechanism. However, the substantial number of expert activations creates a critical latency bottleneck during inference, especially in resource-constrained deployment scenarios. Existing approaches that reduce expert activations potentially lead to severe model performance degradation. In this work, we introduce the concept of activation budget as a constraint on the number of expert activations and propose Alloc-MoE, a unified framework that optimizes budget allocation coordinately at both the layer and token levels to minimize performance degradation. At the layer level, we introduce Alloc-L, which leverages sensitivity profiling and dynamic programming to determine the optimal allocation of expert activations across layers. At the token level, we propose Alloc-T, which dynamically redistributes activations based on routing scores, optimizing budget allocation without increasing latency. Extensive experiments across multiple MoE models demonstrate that Alloc-MoE maintains model performance under a constrained activation budget. Especially, Alloc-MoE achieves

1.15\times

prefill and

1.34\times

decode speedups on DeepSeek-V2-Lite at half of the original budget.

Baihui Liu, Kaiyuan Tian, Wei Wang +3

National Key Laboratory of Parallel and Distributed Computing, College of Computer Science and Technology, National University of Defense Technology

Mixture-of-Experts Inference Mixture of Experts Efficient Language Model Inference

Ask PDF

Abstract

Mixture-of-Experts Quantization

Explore similar work

DPA-MoE: A Runtime Precision Control Framework for MoE Inference

AlphaQ: Calibration-Free Bit Allocation for Mixture-of-Experts Quantization

Alloc-MoE: Budget-Aware Expert Activation Allocation for Efficient Mixture-of-Experts Inference