RPU -- A Reasoning Processing Unit

Matthew Adiletta; Gu-Yeon Wei; David Brooks

cs.ARFeb 20, 2026

RPU -- A Reasoning Processing Unit

Authors: Matthew Adiletta, Gu-Yeon Wei, David Brooks

Organizations: Harvard University

Abstract

Large language model (LLM) inference performance is increasingly bottlenecked by the memory wall. While GPUs continue to scale raw compute throughput, they struggle to deliver scalable performance for memory bandwidth bound workloads. This challenge is amplified by emerging reasoning LLM applications, where long output sequences, low arithmetic intensity, and tight latency constraints demand significantly higher memory bandwidth. As a result, system utilization drops and energy per inference rises, highlighting the need for an optimized system architecture for scalable memory bandwidth. To address these challenges we present the Reasoning Processing Unit (RPU), a chiplet-based architecture designed to address the challenges of the modern memory wall. RPU introduces: (1) A Capacity-Optimized High-Bandwidth Memory (HBM-CO) that trades capacity for lower energy and cost; (2) a scalable chiplet architecture featuring a bandwidth-first power and area provisioning design; and (3) a decoupled microarchitecture that separates memory, compute, and communication pipelines to sustain high bandwidth utilization. Simulation results show that RPU performs up to 45.3x lower latency and 18.6x higher throughput over an H100 system at ISO-TDP on Llama3-405B.

Figures & tables

Fig. 1: RPU provides higher memory bandwidth than H100, which is required for low-latency decoding. Even up to BS=32, arithmetic intensity remains low, but requires the RPU to execute kernels which straddle the roofline.

Fig. 2: Power and utilization characterization of H100 using NVML. Left: Power trace during distributed inference (4xH100) of Llama3-70B (Batch=32). Right: Isolated kernel profiling for memory bandwidth utilization across batch sizes and matrix dimensions (BF16).

Fig. 3: Isolated kernel profiling for power consumption and energy efficiency across batch sizes and matrix dimensions (BF16).

Fig. 4: Memory technology landscape comparing bandwidth per capacity versus latency per token with 100% capacity utilization for dense LLMs. A technology gap exists in the Goldilocks range for low-latency inference.

Fig. 5: Tradeoffs in HBM-CO memories, illustrating that high-BW/Cap memories are up to

\sim

2.5x more energy efficient than an HBM3e device, but

\sim

1.8x the higher cost per GB.

Fig. 6: Proposed RPU system architecture for low-latency LLM inference, featuring a chiplet-based design that rebalances compute, memory, and network resources. Each level of the hierarchy, from core micro-architecture to compute units, packages, and ring-station scale-up, is co-optimized for energy-efficient, cost-effective, and scalable memory bandwidth.

Fig. 7: Vector-Tile weight streaming dataflow and VMM procedure following a stripe-based execution. Arrows in Weight Streaming Dataflow indicate how activations and weights are moved into TMAC unit – activations are broadcast across columns while weights are element-wise moved. The arrows in the VMM Procedure indicates the order tiles are processed – column-wise first until eight rows of tiles are processed, then the next column starts, proceeding until all the columns in a stripe are completed.

Panel of Figure 8 — Fig. 8: Simulation of one CU in a 64-CU system running Llama3-8B. Top: Batch size 1, seq len 16k. Bottom: Batch size 32, seq len 8k. Each timeline shows memory, compute, and network utilization, buffer usage, and power. Memory power dominates total system power, highlighting the RPU’s unique power provisioning. The plot also shows that batch size 32 generates tokens $\sim$ 13× slower than batch size 1, primarily due to sequential KV$ computations.

Fig. 9: Pareto frontier of HBM-CO memories for Llama3-405B inference on 64-CU RPU, annotated by stepwise changes in optimal HBM-CO. These represent the set of HBM-CO chiplets useful for a memory-chiplet ecosystem.

Fig. 10: RPU with 64 CUs running Llama4-Maverick showing batch size versus sequence length, comparing optimal HBM-CO BW/Cap and slowdown relative to BS=1, Seq Len=8K. Slowdown sub-metrics indicate the fraction of capacity used for KV cache versus active parameters and total capacity.

Fig. 11: Top: Strong scaling for Llama models under ISO TDP vs H100. Bottom: Output tokens per second per query (8xH200 from [ 19 ] ) and bandwidth utilization versus batch size.

Fig. 12: Energy and cost analysis for scales of CUs running Llama3-405B at batch size 1. Top: Energy per inference and optimal memory selection. Bottom: Normalized system cost.

Fig. 13: Speedup and energy-per-inference of an RPU versus and H100, sweeping batch size for Llama models with 8k prefill 2k decode.

Fig. 14: A comparison of leading hardware platforms. Speculative decoding throughput for Llama3-70B based on published data [ 57 , 52 , 18 , 64 ] .

AI Accelerator Inference

AI accelerator inference studies hardware and software for running neural networks faster and more efficiently across data-center, edge, and mobile devices.

See all 102 papers

Understand and Accelerate Memory Processing Pipeline for Large Language Model Inference
Mar 30, 2026Zifan He, Rui Ma, Yizhou Sun +1LLM Inference LLM Inference Acceleration
Department of Computer Science, University of California, Los Angeles · Microsoft Research Asia
Ragged Paged Attention: A High-Performance and Flexible LLM Inference Kernel for TPU
Apr 16, 2026Jevin Jiang, Ying Chen, Blake A. Hechtman +2Efficient Transformer Inference AI Accelerator Inference
Google
BOOST: Concurrent Access to Host Memory and HBM to Accelerate LLM Inference
Sep 11, 2026Anish Saxena, Jae Hyung Ju, Hritvik Taneja +4LLM Inference Efficiency LLM Inference Acceleration
Georgia Tech Atlanta, USA · Nvidia Research Westford, USA · Nvidia Research & Stanford Santa Clara, USA

Mar 30, 2026J/K move · Enter open · S save

Understand and Accelerate Memory Processing Pipeline for Large Language Model Inference

Zifan He, Rui Ma, Yizhou Sun, Jason Cong

Department of Computer Science, University of California, Los Angeles · Microsoft Research Asia

LLM Inference LLM Inference Acceleration FPGA Acceleration

Open paper AskPDF arXiv

Modern large language models (LLMs) increasingly depends on efficient long-context processing and generation mechanisms, including sparse attention, retrieval-augmented generation (RAG), and compressed contextual memory, to support complex reasoning. We show that these optimizations can be unified into a four-step memory processing pipeline: Prepare Memory, Compute Relevancy, Retrieval, and Apply to Inference. Through systematic profiling, we identify a 22%-97% memory processing overhead in LLM inference and strong heterogeneity in its computational characteristics. Motivated by this insight, we argue that heterogeneous systems are well-suited to accelerate memory processing and thus end-to-end inference. We demonstrate this approach on a GPU-FPGA system by offloading sparse, irregular, and memory-bounded operations to FPGAs while retaining compute-intensive operations on GPUs. Evaluated on an AMD MI210 GPU and an Alveo U55C FPGA, our system is up to

2.2\times

faster and achieves up to

4.7\times

less energy across multiple LLM inference optimizations than the GPU baseline (similar results hold on NVIDIA A100). These results establish heterogeneous systems as a practical direction for efficient LLM memory processing and inform future heterogeneous hardware design.

Abstract

AI Accelerator Inference

Explore similar work

Understand and Accelerate Memory Processing Pipeline for Large Language Model Inference

Ragged Paged Attention: A High-Performance and Flexible LLM Inference Kernel for TPU

BOOST: Concurrent Access to Host Memory and HBM to Accelerate LLM Inference