Systematic Exploration of Multi-core Architectures for Efficient LLM Serving using WaferAI-SIM

Tianhao Zhu; Dahu Feng; Erhu Feng; Yubin Xia

cs.AROct 7, 2025

Systematic Exploration of Multi-core Architectures for Efficient LLM Serving using WaferAI-SIM

Authors: Tianhao Zhu, Dahu Feng, Erhu Feng, Yubin Xia

Organizations: Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University · Department of Precision Instrument, Tsinghua University

Abstract

With the widespread adoption of Large Language Models (LLMs), the demand for high-performance LLM inference services continues to grow. Multi-core AI accelerators, such as Groq, Graphcore IPU, and Cerebras WSE, provide promising platforms for LLM serving, but their distributed memory systems require careful coordination between hardware configuration and serving policies. Otherwise, mismatched tensor partitioning, data placement, and memory management can substantially underutilize compute and communication resources. To address these challenges, we present WaferAI-SIM, a multi-level simulation framework that combines transaction-level simulation with an analytical performance model. WaferAI-SIM enables simulator-driven co-design of LLM serving strategies and multi-core accelerator architectures, targeting the early design stage in which emerging platforms are not yet broadly available for empirical serving studies. It captures how LLM serving policies interact with compute-core count, memory hierarchy, and interconnect topology, enabling architecture-aware exploration beyond GPU-centric assumptions. We evaluate representative LLMs across a range of chip configurations and serving scenarios. Across the evaluation, WaferAI-SIM reports 1.32 $\times$ --6.03 $\times$ latency improvements, where the lower endpoint comes from ring-based placement at TP=16 over the placement baselines, and the upper endpoint comes from K-dimension TP over MN-dimension TP for Qwen3_4B at TP=4 with sequence length 256. For LLM serving, our findings provide guidance for co-designing hardware architectures and serving strategies for multi-core AI accelerators across diverse LLM workloads.

Figures & tables

Simulator	LLM Serving Support			Computation Simulation		Memory Simulation		New Architecture Exploration
Simulator	P/D Disagg	P/D Fusion	KV Cache Mgmt	Micro-Arch	Perf/Profiler	TLM/Event	Perf/Profiler	Routing	Network	Hybrid-bonding	Dataflow
RapidChiplet [ 37 ]	✗	✗	✗	✗	✗	✗	✗	✓	✗	✓	✓
h21lm [ 38 ]	✗	✗	✗	✗	✓	✗	✓	✗	✗	✓	✗
LLMServingSim [ 39 ]	✓	✓	✓	✗	❍ Real-Machine	✗	❍ Real-Machine	✗	✗	✗	✓
astra-sim [ 40 ]	✗	✗	✗	✗	✓	✗	✓	✗	✓	✗	✓
vidur [ 41 ]	✓	✓	❍ Partial	✗	❍ Real-Machine	✗	❍ Real-Machine	✗	✗	✗	✓

TABLE I: Comparison of different simulators for AI accelerators: This table compares various simulation tools based on their support for LLM serving (including PD disaggregation/fusion, and KV cache management), computational modeling (architectural vs. performance model), memory modeling (event vs. performance model), and capabilities for exploring new architectural design spaces (scale up network, on chip routing, hybrid bonding, dataflow).

	Tensor Partition	Core Placement	Memory Management	Request Scheduling	PD Disaggregation	PD Fusion	Target Platform
T10 [ 27 ]	AllGather	Linear, 2D Mesh	SRAM	Not mentioned	No	No	IPU
WaferLLM [ 26 ]	AllGather, AllReduce	Interleaved Linear, 2D Mesh	SRAM	Not mentioned	No	No	Cerebras WSE
WSC-LLM [ 29 ]	AllReduce	2D Mesh	HBM	Yes	Fixed / DP-prioritized	No	Wafer-scale Chip
Ours	AllGather, AllReduce, AllGather+AllReduce	Interleaved Linear, Ring, 2D Mesh	SRAM+HBM	Yes	Simulation-guided / workload-aware	Yes	Multi-core Chip

TABLE II: Comparison of different methods for LLM inference.

Fig. 2: The overall design of WaferAI-SIM: Computing/memory/router sub-systems with tracing and scheduler models.

	Input Tensor	Weight Tensor	Output Tensor	Total Communication	Max Hop
Input-only Partition	$\frac{input\_size}{num}$	$weight\_size$	$\frac{output\_size}{num}$	0	0
1-D Partition (M/N)	$\frac{input\_size}{num}$	$\frac{weight\_size}{num}$	$\frac{output\_size}{num}$	$\frac{num-1}{num}\times\begin{pmatrix}K\times N\end{pmatrix}$	$1\sim 2$
1-D Partition (K)	$\frac{input\_size}{num}$	$\frac{weight\_size}{num}$	$\frac{output\_size}{num}$	$2\times\frac{num-1}{num}\times\begin{pmatrix}M\times N\end{pmatrix}$	$1\sim 2$
2-D Partition	$\frac{input\_size}{R_{num}\times C_{num}}$	$\frac{weight\_size}{R_{num}\times C_{num}}$	$\frac{output\_size}{R_{num}\times C_{num}}$	$(R_{num}-1)\times(2\times\frac{C_{num}-1}{C_{num}}\times\frac{M\times N}{C_{num}\times C_{num}}+\frac{K\times N}{C_{num}\times R_{num}})$	$1\sim 2$

TABLE III: Communication and memory cost of different tensor partition strategies. Input/Weight/Output Tensor , Total Communication , and Max Hop denote per-core memory cost, total inter-core transferred data during GEMM, and maximum transfer hops, respectively;

Num

R_{num}

, and

C_{num}

denote the numbers of total, row, and column partitions.

Parameter	Large-core	Small-core
# of compute cores	64	256
Core frequency	500 MHz	500 MHz
Systolic array size	$32{\times}32$ – $128{\times}128$	$32{\times}32$ – $64{\times}64$
Vector unit (64 ALUs/lane)	32–128 lanes	32–64 lanes
SRAM per core	8–128 MB	8–48 MB
SRAM bandwidth per core	scaled with SA	scaled with SA

TABLE IV: Chip configuration space for evaluation.

Model	MAPE (%)	Max error (%)
Qwen3_4B	4.48	6.70
Qwen3_8B	5.43	8.10
Qwen3_30B-A3B	7.15	10.40
Llama-3.1-8B	5.52	8.40
Llama-3.2-3B	3.88	5.60
Overall	5.29	10.40

TABLE V: Simulation accuracy of WaferAI-SIM relative to Ascend-NPU-910B hardware. MAPE and maximum error are computed across all tested batch sizes and decode sizes for each model.

Explore similar work

Frontier: Towards Comprehensive and Accurate LLM Inference Simulation
May 20, 2026Yicheng Feng, Xin Tan, Yangtao Deng +3LLM Inference Optimization User Simulation
The Chinese University of Hong Kong · Anuttacon · StepFun
AGENTSERVESIM: A Hardware-aware Simulator for Multi-Turn LLM Agent Serving
Jun 8, 2026Rakibul Hasan Rajib, Mengxin Zheng, Qian LouLarge Language Model Agents Hardware-Aware
University of Central Florida
Characterizing Parallelism Strategies in LLM Inference: Fundamental Compute-Communication Trade-offs
Oct 4, 2026Javad Mirzaei, Jeebak MitraLLM Inference Optimization
Office of CTO Dell Technologies Inc. Canada

May 20, 2026, cs.DCJ/K move · Enter open · S save

Frontier: Towards Comprehensive and Accurate LLM Inference Simulation

Yicheng Feng, Xin Tan, Yangtao Deng, Yimin Jiang+2

The Chinese University of Hong Kong · Anuttacon · StepFun

LLM Inference Optimization User Simulation Frontiers Autoregressive Rollout Pareto Frontier

Open paper AskPDF arXiv

Modern LLM serving is no longer homogeneous or monolithic. Production systems now combine disaggregated execution, complex parallelism, runtime optimizations, and stateful workloads such as reasoning, agents, and RL rollouts. Simulation is attractive for exploring this growing design space, yet existing simulators lack the architectural completeness and decision-grade fidelity it demands. Their monolithic-replica abstractions are ill-suited to disaggregated serving, while average-case analytical proxies can distort SLA predictions and even reverse optimization conclusions. We present Frontier, a discrete-event simulator for modern LLM inference serving. Frontier features a disaggregated abstraction. It captures the structure and dynamics of modern serving systems by modeling co-location, Prefill-Decode Disaggregation (PDD), and Attention-FFN Disaggregation (AFD) with role-specific cluster workers, incorporating key runtime optimizations (e.g., CUDA Graphs, speculative decoding) within the scheduler-batch-engine loop, and supporting stateful requests for emerging workloads. It further provides accurate and generalizable predictions of computation, communication, and memory costs across diverse serving scenarios with complex workload compositions. On 16-H800 GPU testbed, Frontier achieves an average throughput error below 4%. Compared with state-of-the-art simulators, it reduces end-to-end latency error from 44.9% to 6.4% under co-location and from 51.7% to 2.6% under disaggregation. It scales to over 1K GPUs on commodity CPUs and enables new use cases such as SLA-dependent Pareto frontier exploration, heterogeneous disaggregated allocation, agentic reasoning scheduling validation, and RL post-training reconfiguration. We release Frontier at https://github.com/NetX-lab/Frontier.

Abstract

Explore similar work

Frontier: Towards Comprehensive and Accurate LLM Inference Simulation

AGENTSERVESIM: A Hardware-aware Simulator for Multi-Turn LLM Agent Serving

Characterizing Parallelism Strategies in LLM Inference: Fundamental Compute-Communication Trade-offs