DuoRate: Two-Rate Per-Expert Precision Allocation for Mixture-of-Experts Serving

Mengqi Guo; Zhe Wei; Yuan Yuan; Jiunn Bin Lim; Boyi Pan; Michael Bi Mi

cs.LGOct 9, 2026

DuoRate: Two-Rate Per-Expert Precision Allocation for Mixture-of-Experts Serving

Authors: Mengqi Guo, Zhe Wei, Yuan Yuan, Jiunn Bin Lim, Boyi Pan, Michael Bi Mi

Organizations: Huawei Technologies Ltd.

Abstract

Serving a Mixture-of-Experts (MoE) model pays for expert precision twice: the resident footprint pays for every expert the router can reach, while each batch-one decode step pays only for the experts it selects. These are two different rate functionals over the same per-expert widths, yet published per-expert MoE allocators constrain only the first. We call the difference the store-read gap; in the released BitsMoE checkpoint, 2.54 bits per weight (bpw) of storage become 7.50 bpw of decode traffic. We propose DuoRate, which stores once and reads per workload. Offline, it stores one uniform tier of a prefix-readable weight stream and bakes a distortion tensor. Online, it profiles a workload's own prefill without labels and solves a Lagrangian rate-distortion allocation under the decode functional; routing frequency provably cancels from every per-expert decision, and the solve costs one dot product per expert and a bisection. A deployment therefore keeps a single artifact that prefill reads at the stored rate and decode reads at the workload's own rate. On Qwen3-30B-A3B, DuoRate reading 2.3 bpw exceeds every 2-bit-class baseline by 6.3 to 47.0 points of average accuracy, and at 2.9 bpw it matches 4-bit formats that read 1.4-1.6 bpw more. The advantage is largest under long decoding: on GSM8K, DuoRate at 2.9 bpw matches or exceeds every 4-bit format and trails BF16 by only 0.25 points, and at 2.6 bpw it still matches or exceeds the 4.5-bit NF4 and FP4. Against controls in its own format, the allocation alone is worth 0.07-0.11 bpw of read rate. Code will be released upon publication.

Figures & tables

Figure 1: The DuoRate pipeline. Stage 1 (offline, once) stores one uniform tier of a prefix-readable stream and bakes the distortion tensor

S

. Stage 2 (per workload) profiles the workload’s own prefill without labels, scores every expert and tier by one dot product, and takes a per-expert Lagrangian

\arg\min

with one shared

\lambda

, from which

f_{e}

cancels; a bisection on

\lambda

meets the read budget

B

. Serving : prefill pays

R_{\text{store}}

, decode pays

R_{\text{read}}

Figure 2: Frequency cancellation (Proposition 1 , Corollary 1 ); illustrative values. (a) For two experts eightfold apart in

f

, supporting lines of slope

-\lambda

d_{e}

and

-\lambda f_{e}

f_{e}d_{e}

touch the same tier, so the

\arg\min

does not depend on

f

. (b) On the read set

\{\mathrm{p2},\mathrm{p3},\mathrm{p4}\}

the thresholds fix one family of tier maps; routing frequencies

f

and

g

only change where the bisection stops.

Method	BPW	PPL $\downarrow$	Accuracy $\uparrow$ (%)			Avg. $\uparrow$
	Read	WikiText-2	MMLU-Pro	MMLU	GSM8K
BF16	16.0	8.27	56.86	83.30	91.25	77.14
NVFP4	4.5	8.44	56.79	83.09	90.25	76.71
NF4	4.5	8.41	56.79	82.11	89.50	76.13
FP4	4.5	8.62	56.43	81.75	88.25	75.48
MXFP4	4.3	8.58	56.71	82.53	90.25	76.50

Table 1: Results on Qwen3-30B-A3B-Base. BPW is the rate read per activated expert weight at batch-one decode (for DuoRate , the read budget); DuoRate stores

4.3

. ∗ BitsMoE stores

2.5

;

7.5

is its read rate. GSM8K is strict match; Avg. is the mean of MMLU-Pro, MMLU and GSM8K. Bold is the best in each group.

Figure 3: Average accuracy (MMLU-Pro, MMLU, GSM8K) against the rate read per activated weight at batch-one decode. (a) Every row of Table 1 . (b) The top-left corner: DuoRate at

2.9~\text{bpw}

sits with the 4-bit formats, which read

1.4

–

1.6~\text{bpw}

more. Hollow markers are stored rates joined to read rates: BitsMoE stores

2.54

and reads

7.50

; DuoRate stores

4.27

and reads

2.9

Figure 4: What the allocation alone is worth (WikiText-2). (a) Perplexity against the budgeted read rate: DuoRate , random mixtures of two adjacent tiers (every draw), the interpolated uniform-tier null (dashed) and tables re-solved from a disjoint split (hollow). Brackets: gap to the strongest mixture; arrows: extra read rate the mixture needs to match (dotted: extrapolated); blue ticks: each table at its stored rate. (b) Gap to the null placed at either rate, with

95\%

bootstrap intervals.

Method	Resident		Decode, batch 1
	Store ( bpw )	GiB $\downarrow$	Read ( bpw )	MB/token $\downarrow$
BF16	16.0000	54.000	16.0000	3623.9
BitsMoE 2-bit	2.5425	8.581	7.5008	1698.9
DuoRate $2.3$	4.2704	14.413	2.3000 ‡	520.9 ‡
DuoRate $2.6$	4.2704	14.413	2.6000 ‡	588.9 ‡
DuoRate $2.9$	4.2704	14.413	2.9000 ‡	656.8 ‡

Table 2: Resident footprint and batch-one decode traffic of the expert weights, implied by each artifact’s stored and read rates. ‡ Read budget; the realized decode traffic on GSM8K is lower.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Unit	Rate constraint	Distortion signal	Frequency enters	Store $\neq$ read	Paired
MC-MoE / MC#	expert	unweighted, per block	output error $\times$ freq. α gate β	distortion prior	$\times$	$\times$
MxMoE	expert $\times$ proj. pair	memory, per layer	block-output error	cost term only	$\times$	$\times$
GEMQ	expert, global	unweighted, global	leave-one-out, CE-gradient	implicit at most	$\times$	$\times$
BitsMoE	spectral component	unweighted, per layer	spectral energy $\times$ analytic $D(b)$	importance weight	$\times$	$\times$
MODE	expert, per layer	unweighted, per layer	modality freq. $\times$ KL	distortion weight	$\times$	$\times$
AlphaQ	projection matrix	unweighted, global	analytic weight-domain	none	$\times$	$\times$

Appendix

Table S1: Objective-function audit of per-expert MoE allocators, read from the published objectives ( Huang et al., 2025b ; Huang et al., 2026 ; Duanmu et al., 2025 ; Deng et al., 2026 ; Zhao et al., 2026 ; Chen et al., 2026 ; Yang et al., 2026a ; Chitty-Venkata et al., 2025 ; Chu et al., 2025 ; Yi et al., 2023 ; Tang et al., 2024 ) . Rows are restricted to methods whose objectives we verified in full.

Figure S1: Where routing frequency survives (Proposition 2 in the main paper); illustrative values. (a) Divided by

f_{e}

, a bit costs

\lambda_{\mathrm{s}}/(Ef_{e})

under the stored functional but a constant

\lambda

under the read one. (b) Two experts with the same per-domain curves but different mixtures

\pi_{d\mid e}

take different tiers at one

\lambda

Figure S2: The stored stream (schematic). (a) One expert matrix in plane order, widths proportional to bits per weight: metadata read at every depth, base codes

q

, then planes

b_{3},\dots,b_{6}

; tier p

M

is a contiguous prefix ending at

R(M)

. DuoRate stores p4 and decode reads a p2, p3 or p4 prefix. (b) Equation ( S1 ): plane

k

moves the reconstruction by

\pm s/2^{k-1}

Field	bpw	Provenance
Base code, two bits per weight	2.000000	measured
Group record ( lo , s ), 8 bits per group of 32	0.250000	measured
Two 16-entry codebooks per matrix	0.000326	measured count, spec word
Sparse residual record, one per row	0.020074	measured count, spec word
Total, p2	2.270399
Published floor	2.2704

Appendix

Table S2: The floor of our ladder, recomputed field by field from the released base checkpoint over the

28{,}991{,}029{,}248

expert weights. Measured : count from the checkpoint; spec : word length from the format specification. Rows are rounded individually; the total is summed before rounding.

Method	Store	Read	PPL $\downarrow$	MMLU-Pro	MMLU	GSM8K	Avg.
BF16	16.000	16.000	8.27	56.86	83.30	91.25	77.14
Uniform p2	2.2704	2.2704	10.08	52.64	80.00	88.00	73.55
Uniform p3	3.2704	3.2704	8.69	55.14	81.96	88.50	75.20
Uniform p4	4.2704	4.2704	8.37	57.21	82.67	–	–
DuoRate	4.2704	2.900	9.07	56.00	81.47	91.00	76.16
DuoRate	4.2704	2.600	9.46	53.86	80.56	89.50	74.64

Appendix

Table S3: Stored and read rates of the uniform tiers of our ladder and of the DuoRate rows of Table 1 in the main paper, with BF16 and BitsMoE for reference. A uniform tier stores and reads the same width; DuoRate stores p4, and its Read is the budget each benchmark’s table was solved to. Uniform p4 was not scored on GSM8K.

Domain	Budget	$R_{\mathrm{store}}$	$R^{\mathrm{cal}}_{\mathrm{read}}$	$R^{\mathrm{dec}}_{\mathrm{read}}$	$R^{\mathrm{dec}}-R^{\mathrm{cal}}$
GSM8K	2.6	2.464	2.589	2.559	$-0.029$
	2.9	2.691	2.879	2.818	$-0.061$
MMLU	2.6	2.567	2.587	—	—
	2.9	2.822	2.880	—	—
WikiText-2	2.6	2.659	2.600	—	—
	2.9	2.917	2.899	—	—

Appendix

Table S4: Rates of the

2.6

and

2.9~\text{bpw}

read tables, recomputed from their tier histograms and the ladder of Eq. (1) in the main paper.

R_{\mathrm{store}}

is the unweighted rate of the read table itself, not of the resident p4 tier;

R^{\mathrm{cal}}_{\mathrm{read}}

is the budgeted rate under prefill routing;

R^{\mathrm{dec}}_{\mathrm{read}}

uses decode-step routing and exists only for GSM8K, since the log-likelihood benchmarks have no decode phase.

Figure S3: Per-document perplexity change of DuoRate on WikiText-2, sorted over the

62

documents. (a, b) Against a random mixture of two adjacent tiers at the table’s read rate (least favourable draw); (c) against the interpolated null at the stored rate, the weakest comparison in the upper block of Table S5 . Dashed: the aggregate change; open circles: exact ties, excluded from the sign test.

Arm	Rate definition	$\Delta$	Wins / losses, 95% CI	$p$
Read $2.6$	budgeted	$-1.46\%$	57 / 5, $[-1.90,-1.05]$	$3.1\times 10^{-12}$
	stored	$-0.59\%$	43 / 19, $[-1.06,-0.11]$	$0.0032$
Read $2.9$	budgeted	$-1.18\%$	48 / 14, $[-1.64,-0.74]$	$1.7\times 10^{-5}$
	stored	$-0.92\%$	46 / 16, $[-1.39,-0.47]$	$0.00018$
with the calibration split disjoint from the scored documents
Read $2.6$	budgeted	$-1.28\%$	56 / 6, $[-1.67,-0.90]$	$3.0\times 10^{-11}$

Appendix

Table S5: The WikiText-2 read tables against an interpolated uniform-tier null at their own rate, placed at the budgeted read rate and at the stored rate. Lower block: tables re-solved from the WikiText-2 validation split. Intervals are percentile bootstraps on the aggregate perplexity.

Comparison	Benchmark / rate	$\Delta$	Wins / losses, 95% CI	$p$
against a random mixture of the same tiers , WikiText-2
DuoRate $2.6$	mixture at $2.600$	$-1.35\%$	48 / 8, $[-1.76,-0.94]$	$4.7\times 10^{-8}$
	mixture at $2.700$	$-0.07\%$	32 / 23, $[-0.55,+0.47]$	$0.28$
DuoRate $2.9$	mixture at $2.900$	$-1.04\%$	45 / 9, $[-1.43,-0.64]$	$7.3\times 10^{-7}$
	mixture at $2.950$	$-0.28\%$	34 / 21, $[-0.73,+0.24]$	$0.10$
against uniform p2 ( $2.27~\text{bpw}$ )

Appendix

Table S6: Paired comparisons of the DuoRate read tables against controls in the same format. Top: a random mixture of two adjacent tiers at the table’s read rate and at a dearer one (worst of the draws). Bottom: the uniform tiers below and above the

2.9~\text{bpw}

table. Perplexity rows report document wins/losses, a bootstrap interval and an exact sign test; accuracy rows report exact McNemar.

Figure S4: Tier histograms of the

2.6

and

2.9~\text{bpw}

read tables. Each bar splits the

6{,}144

experts into p2 / p3 / p4; every read tier is a byte prefix of the stored p4 table (top). Right: the p4 count and the unweighted mean of

R(M_{e})

, the

R_{\mathrm{store}}

column of Table S4 .

Figure S5: Rate of each of the

48

layers in the

2.9~\text{bpw}

read tables, weighted by the calibration routing counts (what a decode step pays) and unweighted over the

128

experts of the layer. Dotted lines mark the p2 and p3 tiers; the dashed line is the read budget.

Figure S6: Per-field rates of the released BitsMoE 2-bit checkpoint, recomputed from its safetensors headers. (a) The expert segment stores

2.5425~\text{bpw}

; a batch-one decode step charges the FP16 shared basis to the

k{=}8

selected experts rather than all

E{=}128

, and with the never-fetched fields refunded reads

7.5008~\text{bpw}

. (b) Stored side fields; hatched parts (

0.041618~\text{bpw}

) are never fetched.

as published in Zhao et al. (2026) , their protocol
Method	Bits	PPL $\downarrow$	Accuracy $\uparrow$ (%)							Avg.
			HellaS.	MathQA	MMLU	Openb.	WinoG.	GSM8K	HumanE.
FP16	16	10.24	81.35	60.03	78.77	45.00	72.85	83.47	56.10	68.22
HQQ	2	23.65	63.05	33.17	48.82	36.40	60.85	22.67	11.59	39.51
GPTQ ( Frantar et al., 2023 )	2	15.63	70.16	24.89	39.17	39.40	60.62	4.32	0.00	34.08
MiLo	2	21.53	62.82	31.83	44.51	35.40	59.98	19.64	7.93	37.44

Appendix

Table S7: The published Qwen3-30B-A3B-Base results of Zhao et al. (2026) (upper half, C4 perplexity, zero-shot) with our rows under our protocol (lower half, WikiText-2 perplexity). The halves are not commensurable. Bits is the published label in the upper half; in the lower half it is store / read where the two differ.

Benchmark	Arm	Published	Ours	$\Delta$
aligned: 0-shot on both sides, full split on both sides
OpenBookQA (acc_norm)	unquantized	45.00	44.20	$-0.80$
	2-bit	43.40	43.20	$-0.20$
WinoGrande (acc)	unquantized	72.85	73.24	$+0.39$
	2-bit	72.93	72.14	$-0.79$
aligned, but generative and underpowered

Appendix

Table S8: Our re-scoring of the released BitsMoE checkpoint and of the unquantized model against that paper’s published numbers. Aligned : same shot count and same full split on both sides. Our unquantized MMLU figure is the full test set; our 2-bit figure is the

25

-per-subject sample of Table 1 in the main paper.

Target	Domain	Store-only: read	overspend	Read-only: store	shortfall
3.60	WikiText-2	4.718	$+31.1\%$	3.572	$-0.028$
	GSM8K	4.568	$+26.9\%$	3.522	$-0.078$
	code	4.411	$+22.5\%$	3.560	$-0.040$
3.20	WikiText-2	4.092	$+27.9\%$	3.176	$-0.024$
	GSM8K	3.972	$+24.1\%$	3.151	$-0.049$
	code	3.874	$+21.1\%$	3.156	$-0.044$

Appendix

Table S9: Constraining one rate and reading the other, on the additive instance. Store-only overspends the read target on all nine cells; read-only never exceeds the stored target.

Figure S7: Online recalibration on a

150

-document code corpus from a prose-calibrated allocation at

3.2~\text{bpw}

(earlier additive instance). (a) Experts changing precision at each step; after the third, fewer than

5\%

move. (b) Code-corpus perplexity: updating from the stream’s own unlabelled inputs closes

95.3\%

of the gap to an allocation calibrated offline on code.

Budget	Distortion functional	$R_{\mathrm{store}}$	$R_{\mathrm{read}}$	MMLU
2.600	none (rate-matched control)	2.600	2.595	81.26
	activation-weighted squared error	2.600	2.598	80.98
	decision-aligned	2.600	2.600	81.19
2.900	none	2.900	2.881	81.54
	squared error	2.900	2.912	80.42
	decision-aligned	2.845	2.900	81.47

Appendix

Table S10: The distortion functional at four budgets on the predecessor instance, MMLU accuracy (

n=1{,}425

). The control is a rate-matched random assignment between two adjacent tiers.

Figure S8: Five distortion functionals at a realized MMLU read rate of

3.000~\text{bpw}

(earlier tier family). (a) WikiText-2 perplexity; (b) MMLU-Pro accuracy, nearly its mirror image. The decision-aligned functional used by DuoRate is the only arm above activation-weighted squared error on MMLU-Pro at a nominal

p<0.05

Figure S9: The MC-MoE objective inside our bisection, with the same distortion table and cap (WikiText-2). (a) At matched read rate, DuoRate ’s perplexity relative to MC-MoE’s: the sign test detects the difference, the

95\%

interval includes zero. (b) The same tables at their stored rates: MC-MoE stores

0.29

and

0.43~\text{bpw}

less. Neither dominates.

Mixture-of-Experts Quantization

Mixture-of-Experts quantization compresses expert weights and computations to reduce memory use and speed up inference while preserving model quality.

See all 32 papers

DPA-MoE: A Runtime Precision Control Framework for MoE Inference
Oct 15, 2026Jaeyun Kim, Hoonki Lee, Susie Shin +3Mixture-of-Experts Inference Adaptive Inference
Agency for Defense Development · Yonsei University
AlphaQ: Calibration-Free Bit Allocation for Mixture-of-Experts Quantization
Jun 3, 2026Wanqi Yang, Yuexiao Ma, Alexander Conzelmann +4Mixed-Precision Quantization Mixture-of-Experts Language Models
Max Planck Institute for Intelligent Systems · ELLIS Institute Tübingen · Tübingen AI Center +5
Alloc-MoE: Budget-Aware Expert Activation Allocation for Efficient Mixture-of-Experts Inference
Apr 9, 2026Baihui Liu, Kaiyuan Tian, Wei Wang +3Mixture-of-Experts Inference Mixture of Experts
National Key Laboratory of Parallel and Distributed Computing, College of Computer Science and Technology, National University of Defense Technology

Oct 15, 2026J/K move · Enter open · S save

DPA-MoE: A Runtime Precision Control Framework for MoE Inference

Jaeyun Kim, Hoonki Lee, Susie Shin, Hyunmin Kim+2

Agency for Defense Development · Yonsei University

Mixture-of-Experts Inference Adaptive Inference Mixture-of-Experts Quantization

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) models reduce computation by routing each token to a small set of experts, but each selected expert is still used at a fixed precision. We present DPA-MoE, a runtime precision control framework that makes routed experts any-precision. DPA-MoE encodes expert feed-forward weights as prefix-decodable chunks using SharedScale AnyBCQ, profiles per-expert reconstruction residuals offline, and allocates chunks at inference time using live router gate scores. This post-routing allocation lets a single encoded MoE checkpoint operate at multiple effective weight budgets without retraining or re-encoding. Experiments on Qwen3-30B-A3B and DeepSeekMoE-16B show that DPA-MoE improves KL divergence and downstream accuracy over uniform chunk allocation at matched budgets, with modest steady-state throughput overhead in our prototype.

Abstract

Mixture-of-Experts Quantization

Explore similar work

DPA-MoE: A Runtime Precision Control Framework for MoE Inference

AlphaQ: Calibration-Free Bit Allocation for Mixture-of-Experts Quantization

Alloc-MoE: Budget-Aware Expert Activation Allocation for Efficient Mixture-of-Experts Inference