Every Expert Counts: ExactMoE for Memory-Efficient W4A16 Inference

Amjad Saab

cs.LGAug 15, 2026

Every Expert Counts: ExactMoE for Memory-Efficient W4A16 Inference

Authors: Amjad Saab

Organizations: Appendture Dubai, United Arab Emirates

Abstract

Sparse mixture-of-experts (MoE) language models reduce arithmetic by activating only a small subset of experts per token, yet deployment still requires storing and moving the full expert bank. We present ExactMoE, an inference design that applies symmetric group-128 four-bit weight quantization only to routed experts, stores those experts in kernel-native MARLIN form in pinned host memory, and executes all selected experts through a configurable GPU-resident slot cache and fused grouped MoE kernels. The router, attention, embeddings, normalization layers, and language-model head remain in BF16. "Exact" refers to complete expert availability and an unchanged top-k routing procedure: no expert is pruned, substituted, or forced to execute on the CPU. It does not imply numerical identity with the BF16 model. On OLMoE-1B-7B-0924-Instruct, evaluated on a single NVIDIA L4, a 16-slot configuration reduces peak reserved GPU memory from 14.168 to 1.836 GiB (87.04%) while retaining 81.85% of BF16 decode throughput. A fully resident 64-slot configuration reaches 31.923 tokens/s versus 21.662 tokens/s for BF16 while reserving 4.061 GiB. Across 12,450 zero-shot multiple-choice questions, ExactMoE obtains 70.3534% normalized accuracy versus 70.8996% for BF16, retaining 99.23% of the baseline accuracy. In a matched 16-token ablation, fused grouped execution is 1.97x as fast as a sequential W4 reference. These results identify a practical memory-transfer-throughput frontier for complete-expert MoE inference.

Figures & tables

Figure 1: Overview of ExactMoE . Routed expert projections are quantized and packed once into the representation consumed by MARLIN. During inference, the original BF16 router selects experts, tokens are grouped by selected expert, and active experts execute in cache-sized fused waves. The pinned host pool retains every packed expert; GPU cache misses transfer packed weights and scales without FP16 expert materialization.

Method	Slots	Decode	TTFT	Reserved	Hit rate	H2D
		(tok/s)	(ms)	(GiB)	(%)	(GiB)
BF16	–	21.66	79.03	14.17	–	0.00
ExactMoE	8	15.76	340.77	1.48	38.8	30.04
ExactMoE	16	17.73	290.42	1.84	56.3	21.44
ExactMoE	32	21.66	210.74	2.58	81.4	9.13
ExactMoE	64	31.92	33.92	4.06	resident	0.00

Table 1: Persistent-warm OLMoE runtime medians. H2D is median host-to-device traffic per 128-token request. Full residency preloads all experts before timing.

Figure 2: Measured OLMoE capacity frontier from the archived Colab run. Larger caches reduce expert transfers and improve decode throughput at the cost of additional GPU memory. Dashed lines show the matched BF16 reference. Capacity 64 is fully resident; its zero hit rate in raw counters means cache lookup accounting is bypassed, not that experts are absent.

Method	Wiki PPL	Dolly PPL	MC acc. norm	MC acc. raw
BF16	17.8185	11.1014	70.8996%	59.0602%
ExactMoE W4A16	18.5047	10.9399	70.3534%	58.5542%

Table 2: OLMoE quality from the same checkpoint. Perplexities use the custom immutable manifests described in Section 4.3 .

Task	$n$	BF16	W4A16	$\Delta$	McNemar $p$
ARC-Easy	570	58.42%	58.77%	$+0.35$	0.8877
PIQA	1,838	75.68%	75.73%	$+0.05$	1.0000
HellaSwag	10,042	70.73%	70.03%	$-0.71$	0.0027
Aggregate	12,450	70.90%	70.35%	$-0.55$	0.0112

Table 3: Paired normalized multiple-choice accuracy. Deltas are W4A16 minus BF16 in percentage points.

Figure 3: Paired normalized multiple-choice accuracy from the archived OLMoE evaluation. ExactMoE W4A16 is close to BF16 on all three tasks; the 12,450-example aggregate retains 99.23% of BF16 normalized accuracy with an absolute change of

-0.546

percentage points.

Batch	BF16 tok/s	W4A16 tok/s	BF16 reserved GiB	W4A16 reserved GiB
1	32.14	15.74	14.15	1.82
2	46.64	28.26	14.16	1.83
4	58.07	47.39	14.18	1.85
8	65.25	90.02	14.68	1.86

Table 4: Single-sweep aggregate generation throughput. ExactMoE uses the 16-slot configuration. This is batched generation, not continuous serving.

Mixture-of-Experts Inference

Mixture-of-experts inference studies how to run sparsely activated expert models efficiently while preserving their quality.

See all 177 papers

FluxMoE: Decoupling Expert Residency for High-Performance MoE Serving
Apr 3, 2026Qingxiu Liu, Yongchao He, Runhan Jiang +4Expert Offloading GPU Acceleration
CUHK · Unaffiliated · SCITIX +1
Speculating Experts Accelerates Inference for Mixture-of-Experts
Mar 9, 2026Vivan Madan, Prajwal Singhania, Abhinav Bhatele +2Mixture-of-Experts Inference LLM Inference Acceleration
University of Maryland · TogetherAI
GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMs
May 21, 2026Jianing Deng, Song Wang, Dongwei Wang +4Mixed-Precision Quantization LLM Quantization
University of Pittsburgh · University of Central Florida · University of Arizona +1

Apr 3, 2026J/K move · Enter open · S save

FluxMoE: Decoupling Expert Residency for High-Performance MoE Serving

Qingxiu Liu, Yongchao He, Runhan Jiang, Zion Wang+3

CUHK · Unaffiliated · SCITIX+1

Expert Offloading GPU Acceleration Mixture of Experts Mixture-of-Experts Inference Memory-Efficient Inference LLM Inference Acceleration

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) models have become mainstream for scaling language models to hundreds of billions of expert parameters. Despite sparse expert activation, existing inference engines keep all experts GPU-resident, crowding out the key-value cache in large-batch, long-output offline workloads. We present FluxMoE, which decouples experts from physical GPU residency and adapts their footprint to available memory through a new expert paging abstraction. FluxMoE combines PagedTensor for transparent remapping, a bandwidth-balanced hierarchy spanning losslessly compressed GPU memory and host DRAM, and a budget-aware residency planner. Unlike CPU-GPU co-inference and whole-layer offloading, FluxMoE streams weights on demand while keeping expert computation on GPUs. We implement FluxMoE atop vLLM and evaluate it on three MoE models. For GLM-4.5 on 8

\times

H20 GPUs, FluxMoE delivers up to 7.2

\times

vLLM's throughput and 79.0% lower average Time-Per-Output-Token (TPOT), without measurable model-quality loss using lossless compression. For Mixtral-8

\times

7B-Instruct on 2

\times

L40S GPUs, where weight-resident vLLM cannot fit, FluxMoE delivers 4.3

\times

KTransformers's throughput and 29.1% lower average TPOT.

Abstract

Mixture-of-Experts Inference

Explore similar work

FluxMoE: Decoupling Expert Residency for High-Performance MoE Serving

Speculating Experts Accelerates Inference for Mixture-of-Experts

GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMs