Every Expert Counts: ExactMoE for Memory-Efficient W4A16 Inference

Amjad Saab

cs.LGAug 15, 2026

Every Expert Counts: ExactMoE for Memory-Efficient W4A16 Inference

Authors: Amjad Saab

Organizations: Appendture Dubai, United Arab Emirates

Abstract

Sparse mixture-of-experts (MoE) language models reduce arithmetic by activating only a small subset of experts per token, yet deployment still requires storing and moving the full expert bank. We present ExactMoE, an inference design that applies symmetric group-128 four-bit weight quantization only to routed experts, stores those experts in kernel-native MARLIN form in pinned host memory, and executes all selected experts through a configurable GPU-resident slot cache and fused grouped MoE kernels. The router, attention, embeddings, normalization layers, and language-model head remain in BF16. "Exact" refers to complete expert availability and an unchanged top-k routing procedure: no expert is pruned, substituted, or forced to execute on the CPU. It does not imply numerical identity with the BF16 model. On OLMoE-1B-7B-0924-Instruct, evaluated on a single NVIDIA L4, a 16-slot configuration reduces peak reserved GPU memory from 14.168 to 1.836 GiB (87.04%) while retaining 81.85% of BF16 decode throughput. A fully resident 64-slot configuration reaches 31.923 tokens/s versus 21.662 tokens/s for BF16 while reserving 4.061 GiB. Across 12,450 zero-shot multiple-choice questions, ExactMoE obtains 70.3534% normalized accuracy versus 70.8996% for BF16, retaining 99.23% of the baseline accuracy. In a matched 16-token ablation, fused grouped execution is 1.97x as fast as a sequential W4 reference. These results identify a practical memory-transfer-throughput frontier for complete-expert MoE inference.

Figures & tables

Figure 1: Overview of ExactMoE . Routed expert projections are quantized and packed once into the representation consumed by MARLIN. During inference, the original BF16 router selects experts, tokens are grouped by selected expert, and active experts execute in cache-sized fused waves. The pinned host pool retains every packed expert; GPU cache misses transfer packed weights and scales without FP16 expert materialization.

Method	Slots	Decode	TTFT	Reserved	Hit rate	H2D
		(tok/s)	(ms)	(GiB)	(%)	(GiB)
BF16	–	21.66	79.03	14.17	–	0.00
ExactMoE	8	15.76	340.77	1.48	38.8	30.04
ExactMoE	16	17.73	290.42	1.84	56.3	21.44
ExactMoE	32	21.66	210.74	2.58	81.4	9.13
ExactMoE	64	31.92	33.92	4.06	resident	0.00

Table 1: Persistent-warm OLMoE runtime medians. H2D is median host-to-device traffic per 128-token request. Full residency preloads all experts before timing.

Figure 2: Measured OLMoE capacity frontier from the archived Colab run. Larger caches reduce expert transfers and improve decode throughput at the cost of additional GPU memory. Dashed lines show the matched BF16 reference. Capacity 64 is fully resident; its zero hit rate in raw counters means cache lookup accounting is bypassed, not that experts are absent.

Method	Wiki PPL	Dolly PPL	MC acc. norm	MC acc. raw
BF16	17.8185	11.1014	70.8996%	59.0602%
ExactMoE W4A16	18.5047	10.9399	70.3534%	58.5542%

Table 2: OLMoE quality from the same checkpoint. Perplexities use the custom immutable manifests described in Section 4.3 .

Task	$n$	BF16	W4A16	$\Delta$	McNemar $p$
ARC-Easy	570	58.42%	58.77%	$+0.35$	0.8877
PIQA	1,838	75.68%	75.73%	$+0.05$	1.0000
HellaSwag	10,042	70.73%	70.03%	$-0.71$	0.0027
Aggregate	12,450	70.90%	70.35%	$-0.55$	0.0112

Table 3: Paired normalized multiple-choice accuracy. Deltas are W4A16 minus BF16 in percentage points.

Figure 3: Paired normalized multiple-choice accuracy from the archived OLMoE evaluation. ExactMoE W4A16 is close to BF16 on all three tasks; the 12,450-example aggregate retains 99.23% of BF16 normalized accuracy with an absolute change of

-0.546

percentage points.

Batch	BF16 tok/s	W4A16 tok/s	BF16 reserved GiB	W4A16 reserved GiB
1	32.14	15.74	14.15	1.82
2	46.64	28.26	14.16	1.83
4	58.07	47.39	14.18	1.85
8	65.25	90.02	14.68	1.86

Table 4: Single-sweep aggregate generation throughput. ExactMoE uses the 16-slot configuration. This is batched generation, not continuous serving.

Mixture-of-Experts Inference

Mixture-of-experts inference studies how to run sparsely activated expert models efficiently while preserving their quality.

See all 177 papers

Apr 3, 2026

FluxMoE: Decoupling Expert Residency for High-Performance MoE Serving

Mixture-of-Experts (MoE) models have become mainstream for scaling language models to hundreds of billions of expert parameters. Despite sparse expert activation, existing inference engines keep all experts GPU-resident, crowding out the key-value cache in large-batch, long-output offline workloads. We present FluxMoE, which decouples experts from physical GPU residency and adapts their footprint to available memory through a new expert paging abstraction. FluxMoE combines PagedTensor for transparent remapping, a bandwidth-balanced hierarchy spanning losslessly compressed GPU memory and host DRAM, and a budget-aware residency planner. Unlike CPU-GPU co-inference and whole-layer offloading, FluxMoE streams weights on demand while keeping expert computation on GPUs. We implement FluxMoE atop vLLM and evaluate it on three MoE models. For GLM-4.5 on 8

\times

H20 GPUs, FluxMoE delivers up to 7.2

\times

vLLM's throughput and 79.0% lower average Time-Per-Output-Token (TPOT), without measurable model-quality loss using lossless compression. For Mixtral-8

\times

7B-Instruct on 2

\times

L40S GPUs, where weight-resident vLLM cannot fit, FluxMoE delivers 4.3

\times

KTransformers's throughput and 29.1% lower average TPOT.

Qingxiu Liu, Yongchao He, Runhan Jiang +4

CUHK · Unaffiliated · SCITIX +1

Expert Offloading GPU Acceleration Mixture of Experts Mixture-of-Experts Inference Memory-Efficient Inference LLM Inference Acceleration

Ask PDF

Mar 9, 2026

Speculating Experts Accelerates Inference for Mixture-of-Experts

Mixture-of-Experts (MoE) models have gained popularity as a means of scaling the capacity of large language models (LLMs) while maintaining sparse activations and reduced per-token compute. However, in memory-constrained inference settings, expert weights must be offloaded to CPU, creating a performance bottleneck from CPU-GPU transfers during decoding. We propose an expert prefetching scheme that leverages currently computed internal model representations to speculate future experts, enabling memory transfers to overlap with computation. Across multiple MoE architectures, we demonstrate that future experts can be reliably predicted by these internal representations. We also demonstrate that executing speculated experts generally maintains downstream task accuracy, thus preserving more effective compute-memory overlap by eliminating the need to re-fetch true router-selected experts. Integrated into an optimized inference engine, our approach achieves up to 14% reduction in time per output token (TPOT) over on-demand loading of experts from CPU memory. For MoEs where speculative execution alone yields suboptimal accuracy, we further examine lightweight estimators that improve expert prediction hit rates, thereby reducing performance degradation. Our code is released in open-source at https://github.com/axonn-ai/yalis/tree/offload_prefetch.

Vivan Madan, Prajwal Singhania, Abhinav Bhatele +2

University of Maryland · TogetherAI

Mixture-of-Experts Inference LLM Inference Acceleration Mixture-of-Experts Language Models

Ask PDF

May 21, 2026

GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMs

Mixture-of-Experts Large Language Models (MoE-LLMs) achieve strong performance but incur substantial memory overhead due to massive expert parameters. Mixed-precision quantization mitigates this cost by allocating expert-wise bit-widths based on their importance, approaching the accuracy-memory Pareto frontier and enabling extreme low-bit quantization. However, existing methods rely on layer-wise importance estimation and overlook router shifts induced by quantization, resulting in suboptimal allocation and routing. In this work, we propose Global Expert-level Mixed-precision Quantization (GEMQ) to overcome these limitations via (1) a global linear-programming formulation that captures model-wide expert importance based on quantization error analysis, and (2) efficient router fine-tuning to adapt routing to quantized experts. These components are integrated into a progressive quantization framework that iteratively refines importance estimation and allocation. Experiments demonstrate that GEMQ significantly reduces memory and accelerates inference with minimal accuracy degradation. Source code is available at https://github.com/jndeng/GEMQ .

Jianing Deng, Song Wang, Dongwei Wang +4

University of Pittsburgh · University of Central Florida · University of Arizona +1

Mixed-Precision Quantization LLM Quantization Mixture-of-Experts Language Models LLM Inference Acceleration Mixture-of-Experts Quantization Post-Training Quantization+1

Ask PDF

Abstract

Mixture-of-Experts Inference

Explore similar work

FluxMoE: Decoupling Expert Residency for High-Performance MoE Serving

Speculating Experts Accelerates Inference for Mixture-of-Experts

GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMs