Wafer-scale accelerators offer massive compute capacity through hundreds of thousands of processing elements (PEs) interconnected across a single wafer, but realizing this potential requires efficient application mapping. Their scale makes programming these systems and mapping applications onto them challenging. Existing approaches rely on template-based mappings to maintain productivity, but these templates restrict the mapping design space. In particular, they impose fixed geometric layouts over the PE grid, often leaving compute and memory resources underutilized. In this work, we present WaferTetris, a system that removes these geometric restrictions through freeform mapping. Its Tetris-like strategy flexibly packs computation across the wafer, improving PE utilization and usable memory capacity while preserving PE-task code. We develop a recurrence-based throughput model that captures branch-and-join dependencies to guide spatial parallelism, and an architecture model with per-color gates that enable existing FPGA place-and-route algorithms to enforce WSE routing constraints. We evaluate WaferTetris on Cerebras WSE-3 using LLM decoding and scientific stencil benchmarks. For LLM decoding, it achieves up to 4.42× the system throughput of a template-based automatic mapping baseline on the same fabric size. Compared with the manually optimized WaferLLM, WaferTetris supports 1.8× the context length for LLaMA3.1-8B (80k versus 44k), while delivering equal or higher per-user throughput at matched context lengths, up to 1.10×.
Figures & tables
Figure 2. We present WaferTetris, an automated mapping system that improves resource utilization on a wafer-scale accelerator.
Figure 3. CSL code for the mappings in Figures 2(b) and 2(c) . A layout decides where each task runs and how data moves between PEs; the PE task in (a) is the same under both layouts. In (c), island0 and island1 specify the PE-task parameters for each island, including M_local and N_local .
Figure 4. A slowest-unit objective reads only L and reports 6 at U4 , while the actual interval of 12 is modeled by the recurrence graph.
Figure 5. Routing-resource mismatch between a conventional FPGA routing model and the WSE for the same logical communication. Markers 1 and 2 denote the I/O-access and router-semantics mismatches, respectively.
Figure 6. Overview of the WaferTetris mapping workflow.
Compute
FPGAs
WSEs
Component scales
~10 6 CLBs
~10 6 PEs
Multi-in multi-out functions
Yes
Yes
Communication
FPGAs
WSEs
Interconnect nodes
Switchbox
Router
Data channels
Hardware tracks
Virtual colors
Distributing data mechanisms
Fan-out
Broadcast
Table 1. Correspondence between WSE resources and their FPGA abstractions in our architecture model.
Figure 7. WSE-specific architecture abstraction. WaferTetris represents PEs as custom CLBs and router colors as subset-switch tracks guarded by per-color gates, exposing WSE compute and routing constraints to existing P&R tools.
Figure 8. (a) The P&R time scales exponentially with the fabric size. WaferTetris improves scalability via floorplanning. (b) Different floorplanning strategies could affect throughput results for LLaMA3.1-8B (sequence length = 4k).
Figure 9. An example showing how CU.POKer and WaferTetris map a LLaMA layer onto a 50×75 PE region. Matching colors denote the same operators across the graph and both placements; white cells indicate unused PEs. Panel (a) shows the branch-and-join operator graph of one LLaMA layer. Panels (b) and (c) show the corresponding CU.POKer rectangular-island placement and WaferTetris freeform placement on the same 50 by 75 PE region. Matching colors denote the same operators.
Figure 10. System throughput and PE utilization for LLM decoding on WSE-3. TPS = tokens/s. Three model panels compare CU.POKer and WaferTetris for Qwen2.5-7B, LLaMA3.1-8B, and LLaMA2-13B at sequence lengths of 1K, 2K, and 4K. Each panel shows throughput bars above PE-utilization curves. WaferTetris achieves 3.24 to 4.42 times the throughput where both methods run; CU.POKer runs out of memory for LLaMA2-13B at 4K.
Figure 11. Stencil throughput and PE utilization on WSE-3. GLUP/s = billions of lattice updates per second. Gray-Scott and elastic-wave panels compare CU.POKer with WaferTetris at Z dimensions of 50, 100, and 200. Throughput bars sit above PE-utilization curves. WaferTetris improves throughput by approximately 1.26 times for Gray-Scott and 1.18 to 1.21 times for elastic wave. PE utilization is constant across the Z dimensions.
Figure 12. Per-user decoding throughput and PE utilization compared with WaferLLM ( He et al., 2025 ) . Two panels compare WaferLLM and WaferTetris for LLaMA3.1-8B and LLaMA2-13B with WSE-2 and WSE-3 fabric budgets. Each panel contains throughput bars and PE-utilization curves. Speedup annotations compare the WSE-3 results. Unavailable configurations are marked OOM in red.
Figure 13. Mapping-unit latency under rectangle and freeform P&R on WSE-3. Three line-chart panels compare rectangle and freeform mapping latency. CheckerBoard and MeshGEMV use square matrix dimensions 400, 800, 1600, 3200, and 6400; MeshGEMM uses dimensions 80, 160, 240, and 320. The curves remain close except at the smallest GEMV configuration.
Figure 14. Predicted and measured local latency across logical island sizes for two workloads per kernel. The proposed latency models effectively capture the communication bound for larger islands. Lines show predicted latency and markers show measurements for GEMV, Softmax, and RMSNorm. Latency initially decreases as islands grow, then flattens or increases. Mean absolute percentage errors over the displayed measurements are 4.2, 6.5, and 0.1 percent, respectively.
Figure 15. Ablation of the performance objective and mapping geometry for two LLMs at a 4k context on WSE-3. The recurrence model improves throughput prediction and mapping selection; freeform mapping further improves measured throughput and PE utilization. Each model panel shows the two performance objectives, and within each objective the rectangular and freeform geometries. Under the slowest-unit objective, throughput is predicted at 85.2k and 161.0k tokens per second for LLaMA3.1-8B but measured at only 28.2k and 49.3k, while the rectangular LLaMA2-13B mapping runs out of memory and the freeform one is predicted at 131.0k against a measured 43.8k. Under the recurrence objective, the predictions of 64.2k and 93.7k for LLaMA3.1-8B and 47.1k and 72.8k for LLaMA2-13B match measurements of 70.1k, 94.7k, 48.2k and 76.8k within 8.4 percent. Freeform raises PE utilization from 59.8 to 98.0 percent and from 72.6 to 98.0 percent on LLaMA3.1-8B, and from 95.5 to 98.0 percent and from 88.8 to 98.0 percent on LLaMA2-13B.
Stage
1k
2k
4k
Profiling (one-time)
90.00 (shared)
Parallelism selection
0.20
0.25
0.43
Placement
12.26
12.42
12.35
Routing
0.50
0.58
0.30
CSL compilation
23.89
23.82
24.26
Per-mapping total
36.84
37.07
37.34
Table 2. LLaMA2-13B mapping time in minutes. Profiling is shared across configurations; per-mapping totals exclude profiling and simulation.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Full fit
LOOCV
Operation
n
k
R2
MAPE
R2
MAPE
P95 APE
GEMV
88
6
0.999
2.1%
0.998
2.3%
6.8%
Softmax
36
3
1.000
10.5%
1.000
11.3%
23.0%
RMSNorm
30
3
1.000
0.2%
1.000
0.2%
0.8%
Stencil
18
4
0.948
10.1%
0.898
12.8%
21.4%
Appendix
Table 3. Full-fit and held-out accuracy of the mapping-unit latency models. n is the number of profiled configurations and k the number of fitted coefficients. P95 APE is computed over LOOCV predictions.
Figure 16. PE replication as an optimization beyond P&R. Legend entries indicate the original PE counts; replicated mappings use 17 and 65 PEs. The dashed line marks unchanged performance. The first two panels illustrate shortening communication by copying PE2 near PE4. The third panel shows the original measurements as two lines. Speedup ranges from 0.98 to 1.14 for the 16-PE baseline and from 0.93 to 1.08 for the 64-PE baseline. Unreported configurations are omitted.
Large Language Models (LLMs) incur severe memory-access overheads due to the cost of loading large weight matrices and key-value (KV) caches. Analog In-Memory Computing (AIMC) is an emerging solution that enables computation directly on static weights stored in memory, substantially reducing energy and latency. 3D-IMC further stacks memory arrays into vertical tiers, increasing capacity but revealing a new challenge: how weights are placed in memory determines which matrix-vector multiplications (MVMs) can execute concurrently, affecting latency and throughput. The concurrency aspect of 3D-IMC weight-mapping remains underexplored in the literature, leading to sub-optimal utilization of compute resources on 3D-IMC chips. In this context we propose CAMP (Context-Aware Mapping and Placement), an inference-context-aware weight mapping algorithm for 3D-IMC-based LLM accelerators. CAMP captures the potential resource contention resulting from the architecture, targeted network, and runtime context in a conflict matrix, enabling throughput-optimized weight mapping. This mapping is refined through a profiling-driven feedback loop that efficiently prunes the search space and returns optimized mappings within practical runtimes. CAMP is an offline, software-based solution requiring no hardware changes and improving 3D-IMC compute utilization from 42.4% to 60.3%, delivering up to 2.0x higher 3D-IMC throughput and 1.74x higher end-to-end throughput over state-of-the-art mapping algorithms.
Transformer blocks are prevalent in large language model (LLM) but present deployment challenges due to their challenging computational and memory demands. While prior work has typically optimized attention mechanisms or feed-forward networks (FFNs) separately, few hardware (HW) architecture have jointly addressed both components with co-designed hardware acceleration. We present ELiTeFormer (Efficient Linear Ternary Transformer), the first Transformer model architecture that unifies hybrid linear attention with ultra-low-precision (ternary) linear projections, specifically co-designed for field-programmable gate array (FPGA) deployment. ELiTeFormer achieves 10x model weight compression and 12.8x key-value (KV) cache compression compared to LLaMA 3, while maintaining competitive accuracy (31.9% on the MMLU benchmark, within 3.0% of BitNet b1.58). Our key architectural contribution is a novel processing element (PE) micro-architecture that eliminates all multiplications in ternary linear projections through bitmasking operations, significantly reducing resource utilization by completely avoiding dedicated digital signal processing (DSP) blocks. We simulate, synthesize, and deploy ELiTeFormer targeting a Xilinx VCK5000 Versal board using high-level synthesis (HLS) flows. Block-level simulations show 9.6x speedup for FFN operations and 4.4x speedup for attention compared to standard implementations. End-to-end deployment achieves up to 3.9x lower latency and 3.2x better energy efficiency than LLaMA 3 on an NVIDIA A100 graphics processing unit (GPU) at long context lengths. This represents the first FPGA realization combining linear attention with ternary quantization, demonstrating the viability of algorithm-architecture co-design for next-generation LLM acceleration.
Victor Agostinelli, Nicolas Bohm Agostini, Antonino Tumeo
Oregon State University Corvallis, Oregon, USA · Pacific Northwest National Laboratory Richland, Washington, USA
With the widespread adoption of Large Language Models (LLMs), the demand for high-performance LLM inference services continues to grow. Multi-core AI accelerators, such as Groq, Graphcore IPU, and Cerebras WSE, provide promising platforms for LLM serving, but their distributed memory systems require careful coordination between hardware configuration and serving policies. Otherwise, mismatched tensor partitioning, data placement, and memory management can substantially underutilize compute and communication resources. To address these challenges, we present WaferAI-SIM, a multi-level simulation framework that combines transaction-level simulation with an analytical performance model. WaferAI-SIM enables simulator-driven co-design of LLM serving strategies and multi-core accelerator architectures, targeting the early design stage in which emerging platforms are not yet broadly available for empirical serving studies. It captures how LLM serving policies interact with compute-core count, memory hierarchy, and interconnect topology, enabling architecture-aware exploration beyond GPU-centric assumptions. We evaluate representative LLMs across a range of chip configurations and serving scenarios. Across the evaluation, WaferAI-SIM reports 1.32×--6.03× latency improvements, where the lower endpoint comes from ring-based placement at TP=16 over the placement baselines, and the upper endpoint comes from K-dimension TP over MN-dimension TP for Qwen3_4B at TP=4 with sequence length 256. For LLM serving, our findings provide guidance for co-designing hardware architectures and serving strategies for multi-core AI accelerators across diverse LLM workloads.
Tianhao Zhu, Dahu Feng, Erhu Feng +1
Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University · Department of Precision Instrument, Tsinghua University