FORGE: Procedural Weight Generation for Sparse MoE Inference

Chanyoung Moon; Travis Pence; Daisuke Yamada; Jiaqi Mo; Yanxin Zhang; Vikas Singh; Karthikeyan Sankaralingam

cs.AROct 12, 2026

FORGE: Procedural Weight Generation for Sparse MoE Inference

Authors: Chanyoung Moon, Travis Pence, Daisuke Yamada, Jiaqi Mo, Yanxin Zhang, Vikas Singh, Karthikeyan Sankaralingam

Organizations: University of Wisconsin-Madison Madison, Wisconsin, USA · Korea University Seoul, Republic of Korea · NVIDIA Santa Clara, California, USA

Abstract

We introduce FORGE (Fast On-chip Rotor Generation for Experts), a procedural weight generation system for Sparse Mixture-of-Experts (MoE) inference. Rather than storing and loading every expert weight tensor from HBM, we show how a compact accelerator inside each Streaming Multiprocessor can regenerate them on-the-fly from $O(\log^2 d)$ rotor parameters per projection. Per replaced layer, gigabytes of stored expert weights are approximated with under a megabyte of seeds. This addresses the dual memory bottleneck of autoregressive Sparse MoE decode, where per-expert FFN operations are HBM-bandwidth-bound at small routed-token counts and the aggregate expert footprint forces multi-node Expert Parallelism (EP) with all-to-all traffic. We realize the approach as the Rotor Generation Unit (RGU), a fixed-function datapath that generates FFN weight tiles without competing with Tensor Core execution. With 30% of the MoE FFN layers replaced across five Sparse MoE models, we estimate 1.3-2.8x higher system throughput at matched GPU count and 1.10-1.15x lower per-user decode latency at short context. In exchange, downstream accuracy on three model families falls below that of the same model fine-tuned without rotor replacement, and the replacement ratio controls this tradeoff. FORGE also expands KV cache capacity by 1.2-2.6x. It is agnostic to the bit width at which weights are stored, so it can compose with quantization. The RGU is fully implemented/verified in SystemVerilog at ~0.9% SM area overhead (0.032 mm^2/SM, 7nm).

Figures & tables

Figure 1. FORGE replaces HBM-resident expert weights with on-chip procedural regeneration. (Top) Training adds an offline rotor-fitting step that compresses each dense expert into a rotor-parameter seed $\theta$ . (Bottom) At inference, the RGU inside each SM materializes weight tiles on-the-fly from seeds alone, reclaiming HBM capacity for an expanded KV cache.

Model	$E$	$K$	MoE%	Seed/layer	Comp.
DeepSeek-v3.2	256	8	94.5	594 KB	$19{,}065\times$
GLM-5	256	8	98.2	509 KB	$19{,}065\times$
Qwen3-235B	128	8	96.6	276 KB	$8{,}738\times$
Qwen3-30B	128	8	95.0	221 KB	$2{,}731\times$
Qwen3-Coder-Next	512	10	98.0	776 KB	$2{,}081\times$

Table 1. Target Sparse MoE model configurations.

E

K

: total and activated experts; Seed: the FORGE parameters that replace one layer’s experts.

Figure 2. Rotor 3-stage projection pipeline (

n{=}4

c{=}16

Figure 3.

(n{=}4,k{=}2)

block, iteration

t{=}0

. Light/dark shading marks the two independent row pairs induced by axis

(p,q)

Figure 4. Mapping an on-chip generated Supertile to a standard GEMM Thread Block Tile.

Model	Slowdown	Bank conflicts	Instr.
		(gen / base)
DeepSeek-v3.2	38.8 $\times$	1,314,929 / 15,165	148.7 $\times$
GLM-5	33.8 $\times$	1,026,936 / 10,981	147.5 $\times$
Qwen3-235B	45.1 $\times$	868,634 / 2,919	116.9 $\times$
Qwen3-30B	33.7 $\times$	644,941 / 5	76.3 $\times$
Qwen3-Coder-Next	25.8 $\times$	124,711 / 1	80.3 $\times$

Table 2. SIMT weight generation against the baseline expert GEMM.

Figure 5. SM integration. The RGU bypasses HBM by generating weight tiles directly into SMEM.

Panel of Figure 6 — Figure 6. Baseline vs. FORGE Tiled-GEMM pipelines.

Figure 7. RGU microarchitecture datapath.

Figure 8. Steady-state PE pipeline timing.

Component	Specification
Baseline GPU	H100 SXM5 (132 SMs, 3.35 TB/s HBM3)
Kernel Profiling	cuBLASLt FP8, CUDA 13.2, NCU
Target Models	DeepSeek-v3.2, GLM-5, Qwen3-235B, Qwen3-30B, Qwen3-Coder-Next
Logic Synthesis	Synopsys DC + ASAP7 ( Clark et al., 2016 ) (7nm finFET, RVT, TT)
SRAM Modeling	CACTI ( Balasubramonian et al., 2017 ) 32 nm scaled to 7 nm via ( Stillmaker and Baas, 2017 )
System-Level Sim	LIMINAL ( Davies et al., 2025 ) analytical per-layer compose (Attn + KV + MoE + Comm + kernel launch)

Table 3. System Configuration and Analyzed Workloads. Details baseline GPU parameters (H100), Target LLMs, and RTL synthesis environments.

Model	$n$	8 tok/expert	16 tok/expert
DeepSeek-v3.2	11	2.33 $\times$	1.29 $\times$
GLM-5	11	2.47 $\times$	1.40 $\times$
Qwen3-235B	10	3.23 $\times$	1.67 $\times$
Qwen3-30B	9	2.28 $\times$	1.13 $\times$
Qwen3-Coder-Next	9	2.34 $\times$	1.19 $\times$

Table 4. Per-MoE-FFN Kernel Speedup (FP8, RGU,

m{=}2

). Baseline: cuBLASLt FP8 GEMM on H100.

Model	$T{=}4\text{K}$	$T{=}32\text{K}$	$T{=}128\text{K}$
DeepSeek-v3.2	1.10 $\times$	1.09 $\times$	1.07 $\times$
GLM-5	1.12 $\times$	1.11 $\times$	1.08 $\times$
Qwen3-235B	1.15 $\times$	1.12 $\times$	1.08 $\times$
Qwen3-30B	1.15 $\times$	1.14 $\times$	1.11 $\times$
Qwen3-Coder-Next	1.15 $\times$	1.13 $\times$	1.09 $\times$

Table 5. Full-Model Decode Latency Speedup (FP8, RGU,

B{=}1

r{=}0.3

). Baseline: TP+EP with minimum GPUs to fit weights. FORGE replaces

30\%

of the MoE layers; kept baseline MoE layers still pay EP all-to-all.

Figure 9. Full-model decode latency (FP8, B=1, T=4096, r=0.3).

Figure 10. Context-length ablation (matched GPU count). (a) Decode speedup narrows as KV cache access dilutes the MoE FFN contribution. (b) Freed HBM grows max concurrent users per GPU sharply at long context. (c) The user-count gain offsets the narrowing decode speedup, so TPS gains hold or widen.

Figure 11. System throughput scaling across replacement ratios (r) (T=4096). Accuracy is evaluated at

r{=}0.3

only;

r{=}0.7

is a what-if point (Section 5.5 ).

Component	Value	Notes
Logic Area	0.017 $mm^{2}$	PE array ${\sim}$ 69%
SRAM Area	0.015 $mm^{2}$	28.875 KB, 4-bank
Total / SM	0.032 $mm^{2}$
vs one SM (3.475 $mm^{2}$ ( Locuza, 2022 ) )	${\sim}$ 0.9%
Clock (ASAP7)	1.80 GHz	WNS $-65$ ps @ 2 GHz
Logic Power (0.7 V)	35.9 mW	3.4 switch + 32.5 internal

Table 6. 7 nm PPA Summary for the FP8 dual-pair RGU.

ERNIE-21B
Config	WikiText $\downarrow$	ARC-C $\uparrow$	HellaSwag $\uparrow$
Baseline	8.37	58.00	61.10
Baseline + LoRA	5.93	58.96	62.50
FORGE	7.16	57.25	59.30
Qwen3-30B
Baseline	9.40	52.60	59.50

Table 7. Model performance with

r{=}0.3

on ERNIE-21B , Qwen3-30B , and Qwen3-Coder-Next on WikiText, ARC-C, and HellaSwag. Baseline + LoRA is the reference for FORGE. The gap between the two is the accuracy cost of rotor replacement.

Mixture-of-Experts Inference

Mixture-of-experts inference studies how to run sparsely activated expert models efficiently while preserving their quality.

See all 177 papers

MawForge: Memory-Bounded Expert Materialization for Local Mixture-of-Experts Inference
Jun 17, 2026Craig OpieExpert Offloading On-Device Language Model Inference
Holocron Security, Inc.
Every Expert Counts: ExactMoE for Memory-Efficient W4A16 Inference
Aug 15, 2026Amjad SaabMixture-of-Experts Inference Mixture-of-Experts Language Models
Appendture Dubai, United Arab Emirates
Hardware-Native Joint Sparse-Quantization for Trillion-Scale Mixture-of-Experts
Sep 29, 2026Kwanhee Lee, Namhoon Lee, Dan AlistarhMixture-of-Experts Quantization Mixture-of-Experts Inference
POSTECH & ISTA · POSTECH · ISTA

Jun 17, 2026J/K move · Enter open · S save

MawForge: Memory-Bounded Expert Materialization for Local Mixture-of-Experts Inference

Craig Opie

Holocron Security, Inc.

Expert Offloading On-Device Language Model Inference Mixture-of-Experts Inference Memory-Efficient Inference

Open paper AskPDF arXiv

Sparse Mixture-of-Experts (MoE) language models separate total parameter count from per-token active computation, but local inference systems often still require the full model, key-value cache, runtime buffers, and operatingsystem headroom to fit in fast memory. MawForge tests a different systems hypothesis: local MoE serving can be made practical on constrained unified-memory machines by storing the full model on disk, keeping common tensors resident, and materializing routed expert tensors into a bounded execution cache on demand. The central finding is that MawForge is effective as a bounded execution mechanism and measurement substrate for local MoE inference, but not as a cache-maximization policy. Performance depends on balancing expert reuse against resident footprint, KV-cache size, quantization, route locality, and macOS memory pressure.

Abstract

Mixture-of-Experts Inference

Explore similar work

MawForge: Memory-Bounded Expert Materialization for Local Mixture-of-Experts Inference

Every Expert Counts: ExactMoE for Memory-Efficient W4A16 Inference

Hardware-Native Joint Sparse-Quantization for Trillion-Scale Mixture-of-Experts