Analog processing-using-memory (PUM; a.k.a. in-memory computing) makes use of electrical interactions inside memory arrays to perform bulk matrix-vector multiplication (MVM) operations. However, many popular matrix-based kernels need to execute non-MVM operations, which analog PUM cannot directly perform. To retain its energy efficiency, analog PUM architectures augment memory arrays with CMOS-based domain-specific fixed-function hardware to provide complete kernel functionality, but the difficulty of integrating such specialized CMOS logic with memory arrays has largely limited analog PUM to being an accelerator for machine learning inference, or for closely related kernels. An opportunity exists to harness analog PUM for general-purpose computation: recent works have shown that memory arrays can also perform Boolean PUM operations, albeit with very different supporting hardware and electrical signals than analog PUM. We propose DARTH-PUM, a general-purpose hybrid PUM architecture that tackles key hardware and software challenges to integrating analog PUM and digital PUM. We propose optimized peripheral circuitry, coordinating hardware to manage and interface between both types of PUM, an easy-to-use programming interface, and low-cost support for flexible data widths. These design elements allow us to build a practical PUM architecture that can execute kernels fully in memory, and can scale easily to cater to domains ranging from embedded applications to large-scale data-driven computing. We show how three popular applications (AES encryption, convolutional neural networks, large language models) can map to and benefit from DARTH-PUM, with speedups of 59.4x, 14.8x, and 40.8x over an analog+CPU baseline.
Figures & tables
Figure 1. Matrix–vector multiply with analog PUM.
Figure 2. Bit-slicing matrix values.
Figure 3. Number representations for the range [-3, +3].
Figure 4. Boolean PUM operation with OSCAR NOR .
Figure 5. Data layout for digital PUM pipeline.
Figure 6. Motivating architectures.
Figure 7. Throughput of AES-128 encryption with digital (D), analog (A), and naive hybrid (H) PUM with the OSCAR and ideal logic families, normalized to D with OSCAR.
Figure 8. DARTH-PUM architecture.
Figure 9. Unoptimized matrix–vector multiply with DARTH-PUM using ACE and DCE (gray components are unused).
Figure 10. Sequence of operations for MVM.
Figure 11. Steps for parasitic compensation scheme.
API Call
Description
Application-Agnostic Calls
allocVACore()
Allocates a vACore on an HCT, based on elementSize and bits per cell. Initializes the HCTs with the shift-and-add \SIUnitSymbolMicro op sequence.
setMatrix()
Allocate and store a matrix with the required number of HCTs.
execMVM()
Execute MVM between matrix and vector.
updateCol()
Updates a column of a matrix on HCTs.
updateRow()
Updates a row of a matrix on HCTs.
Table 1. List of DARTH-PUM library calls.
Figure 12. Dataflow for AES encryption.
1 Digital Compute Element
Number of Pipelines
64
Pipeline Depth
64 arrays
ReRAM Array Size
64 × 64
1 Analog Compute Element
Number of Arrays
64
ReRAM Array Size
64 × 64
Table 2. Hybrid compute tile configuration.
DCE
Area ( \SIUnitSymbolMicrom 2 )
ACE
Area ( \SIUnitSymbolMicrom 2 )
ReRAM Array
240
ReRAM Array
240
Pipeline Control
74000
Input Buffers
27000
IO Ctrl
9600
Row Periphery
13000
Decode & Drive
280
SAR/RAMP ADC
600/3800
Pipeline Select
64
Sample & Hold
62
HCT
Area ( \SIUnitSymbolMicrom 2 )
HCT
Area ( \SIUnitSymbolMicrom 2 )
Table 3. Area and power for HCT hardware.
Figure 13. Throughput normalized to Baseline. AppAccel-AES uses an AES-NI accelerator (yellow), while the other AppAccel configurations use analog PUM accelerators (red).
Figure 14. Kernel latency breakdown for AES, normalized to Baseline.
Figure 15. Per-layer speedup for ResNet-20 during inference, normalized to Baseline.
Figure 16. Energy savings, normalized to Baseline (y-axis in log scale). AppAccel-AES uses an AES-NI accelerator (yellow), while the other AppAccel configurations use state-of-the-art analog PUM accelerators (red).
Figure 17. Comparison of SAR ADCs vs. ramp ADCs, normalized to Baseline: SAR.
Figure 18. Iso-area comparisons to state-of-the-art GPU.
High-performance Host processors can integrate Processing-In-Memory (PIM) devices, which can accelerate memory-intensive kernels of Machine Learning (ML) models, including Large Language Models (LLMs), by leveraging the large memory bandwidth available at PIM cores. However, Host processor needs consecutive elements distributed across DRAM banks, while PIM cores need consecutive elements within their local banks. This necessitates data rearrangements in ML kernel execution that pose significant performance and programmability challenges, further exacerbated by the need to support diverse PIM devices. Current compilation approaches lack systematic optimization for diverse ML kernels and multiple PIM devices, and may largely ignore data rearrangement costs during the compute code optimization step. We show that data rearrangements and compute code optimization are interdependent, and need to be jointly optimized during the tuning process. Therefore, we design DCC, the first data-centric ML compiler for PIM systems that jointly co-optimizes data rearrangements and compute code in a unified tuning process. DCC integrates a multi-layer PIM abstraction to support multiple PIM backends. DCC enables effective co-optimization of data partitioning strategies with compute loop partitioning schemes. DCC applies PIM-specific code optimizations, and leverages a fast and accurate performance prediction model to select the bestperforming code schedule for a given kernel on a target PIM architecture. Our evaluations in various individual ML kernels show that DCC achieves up to 7.68x speedup (2.21x average) on HBM-PIM, and up to 13.17x speedup (3.92x average) on AttAcc PIM, over GPU-only execution. In end-to-end LLM inference, DCC on AttAcc accelerates GPT-3 and LLaMA-2 by 4.52x average (up to 7.71x in LLaMA-2) over GPU. DCC is open-sourced at https://github.com/SPIN-Research-Group/DCC.
Peiming Yang, Sankeerth Durvasula, Ivan Fernandez +4
University of Toronto & Vector Institute · Barcelona Supercomputing Center · ETH Zürich +2
Large language model (LLM) inference performance is increasingly bottlenecked by the memory wall. While GPUs continue to scale raw compute throughput, they struggle to deliver scalable performance for memory bandwidth bound workloads. This challenge is amplified by emerging reasoning LLM applications, where long output sequences, low arithmetic intensity, and tight latency constraints demand significantly higher memory bandwidth. As a result, system utilization drops and energy per inference rises, highlighting the need for an optimized system architecture for scalable memory bandwidth. To address these challenges we present the Reasoning Processing Unit (RPU), a chiplet-based architecture designed to address the challenges of the modern memory wall. RPU introduces: (1) A Capacity-Optimized High-Bandwidth Memory (HBM-CO) that trades capacity for lower energy and cost; (2) a scalable chiplet architecture featuring a bandwidth-first power and area provisioning design; and (3) a decoupled microarchitecture that separates memory, compute, and communication pipelines to sustain high bandwidth utilization. Simulation results show that RPU performs up to 45.3x lower latency and 18.6x higher throughput over an H100 system at ISO-TDP on Llama3-405B.
Analog in-memory computing (AIMC) speeds up neural-network inference by doing the arithmetic directly inside a memory array, instead of shuttling weights back and forth between memory and a processor. This saves energy, but the physical devices that store the weights are imperfect: programming errors, electrical noise, limited-resolution converters, and outright broken cells all distort the computation, and every physical chip is distorted in its own way. A designer with several such chips available faces an uncomfortable choice: run all of them and combine the answers (safe, but wasteful of energy), or trust a single chip blindly (cheap, but with no guarantee on how often it is wrong). This paper introduces RACE-AIMC (Risk-Aware Certified Ensemble for AIMC), a framework that resolves this choice with statistics rather than guesswork. Offline, RACE-AIMC studies a pool of physical accelerators, picks the single best one for a given energy budget, and computes a mathematically exact upper bound on how often that accelerator will be wrong when it chooses to answer. Online, only that one accelerator is switched on; a lightweight check decides whether to accept its answer or defer to a fallback. In our simulations using a noisy weight mapping and multiple independent test runs, every certified bound stayed under a 10% error target (mean bound 7.83% +- 0.89%, with 70.88% +- 0.98% of inputs answered directly). The resulting system matches the accuracy of a clean digital baseline while cutting modeled energy use by 69.02% relative to always running every accelerator in the pool.