Understand and Accelerate Memory Processing Pipeline for Large Language Model Inference

Zifan He; Rui Ma; Yizhou Sun; Jason Cong

cs.DCMar 30, 2026

Understand and Accelerate Memory Processing Pipeline for Large Language Model Inference

Authors: Zifan He, Rui Ma, Yizhou Sun, Jason Cong

Organizations: Department of Computer Science, University of California, Los Angeles · Microsoft Research Asia

Abstract

Modern large language models (LLMs) increasingly depends on efficient long-context processing and generation mechanisms, including sparse attention, retrieval-augmented generation (RAG), and compressed contextual memory, to support complex reasoning. We show that these optimizations can be unified into a four-step memory processing pipeline: Prepare Memory, Compute Relevancy, Retrieval, and Apply to Inference. Through systematic profiling, we identify a 22%-97% memory processing overhead in LLM inference and strong heterogeneity in its computational characteristics. Motivated by this insight, we argue that heterogeneous systems are well-suited to accelerate memory processing and thus end-to-end inference. We demonstrate this approach on a GPU-FPGA system by offloading sparse, irregular, and memory-bounded operations to FPGAs while retaining compute-intensive operations on GPUs. Evaluated on an AMD MI210 GPU and an Alveo U55C FPGA, our system is up to $2.2\times$ faster and achieves up to $4.7\times$ less energy across multiple LLM inference optimizations than the GPU baseline (similar results hold on NVIDIA A100). These results establish heterogeneous systems as a practical direction for efficient LLM memory processing and inform future heterogeneous hardware design.

Figures & tables

Figure 1: The GPU-FPGA heterogeneous system (1 MI210 + 1 Alveo U55C) can provide $1.2-1.8\times$ speedup and $1.3-4.7\times$ energy cost reduction consistently over a wide range of long-context LLM inference optimizations. “SA-R” stands for SeerAttention-R and “DSA” stands for DeepSeek Attention.

LLM Opt.	Related Works	Prepare Memory	Compute Relevancy	Retrieval	Apply to Inference
Sparse Attention	DeepSeek Attention ( Liu et al., 2025 )	Linear Projections + RoPE	Multi-headed Inner Product	Top- $k$	Fine-grain Sparse Attention
	SeerAttention-R ( Gao et al., 2026a )	Linear Projections + Pooling	Inner Product	Top- $k$ / Threshold	Block Sparse Attention
	LServe ( Yang et al., 2025b )	Page-wise Min/Max Pooling	Inner Product + Max Reduction	Top- $k$	Block Sparse Attention
Retrieval Augmented Generation (RAG)	Two-stage RAG ( Moreira et al., 2024 )	Embedding Model / Tokenization	Inner Product / BM25 + Reranker	Top- $k$	Append to query
	Fixed-Sentence RAG ( Trivedi et al., 2023 )	Tokenization	BM25	Top- $k$	Append to query
	Dynamic RAG ( Su et al., 2024 ; Jiang et al., 2023 )	Tokenization	BM25	Top- $k$	Append to query

Table 1: Summary of LLM inference optimizations and the computations in their memory processing pipeline.

Figure 2: Four-Step Memory Processing Pipeline in LLMs: Prepare Memory preprocesses and structures raw memory for efficient access; Compute Relevancy assigns relevance scores to memory entries with respect to the input query; Retrieval extracts the most relevant memory based on these scores; and Apply to Inference integrates retrieved content and input into intermediate outputs, used in the rest operations in LLMs to produce tokens.

Figure 3: Percentage of latency spent on memory processing for sparse attention methods. With 1M tokens, memory processing can take 22%–81% of the decoding time.

Figure 4: Percentage of latency on memory processing for RAG using the Wikipedia dump ( Su et al., 2024 ) . For two-stage RAG, reranking is time consuming, leading to a high percentage at 500K with a slow increment as the document count grows.

Figure 5: Left: Percentage of latency on memory processing for parameterized memory (Titans/HMT, LaCT). Right: Percentage of latency on memory processing for MemAgent.

LLM Opt.		Prepare Memory	Compute Relevancy	Retrieval	Apply to Inference	Rest Ops in LLM
Sparse Attention	Arith. Intensity	$10$ – $100$	$1$ – $10$	1	$10$ – $100$	$1$ – $10$
	Access Pattern	Regular	Regular	Irregular	Regular	Regular
	Data Requirement	Local Memory	Local Memory	Across Memories	Local Memory	Local Memory
RAG	Arith. Intensity	$1$ – $100$	$1$ – $10$	1	$0$	$>100$
	Access Pattern	Irregular (tokenizer)	Irregular	Irregular	Regular	Regular
		Regular (embeddings)

Table 2: Summary of LLM inference optimizations and computational properties of their memory processing pipelines for single batch. We show arithmetic intensity (FLOPs/byte, higher means more compute-bound) with only orders of magnitude (details in Appendix B ). “Local Memory” denotes independent computation across memory entries, and “Regular” indicates consecutive data access.

Figure 6: Kernel mapping and data communication on the GPU-FPGA system. (a) Sparse attention and RAG employ general setup, where the GPU prepares the memory and apply to the inference and the FPGA executes an efficient fused kernel for compute relevancy and retrieval. (b) For MemAgent, we utilize prefill-decode disaggregation where the FPGA operates LLM decoding and the GPU handles prefilling. (c) For Memory as Context, we update the data mapping for locality: memory on the FPGA, retrieved memory delivered from the FPGA to the GPU, and output is directly streamed to Prepare Memory.

Figure 7: Architecture of the FPGA kernel for General Setup. A streaming dataflow design connecting two modules: (a) Compute Relevancy uses fast SRAM (BRAM+URAM) and HBM to store compressed keys, where an inner-product engine consumes queries and computes scores; (b) Retrieval performs partial and weighted sum across query heads and feeds results to a top-

k

retriever that continuously maintains a running top-

k

list.

Figure 8: End-to-end speedup of the GPU-FPGA heterogeneous system over the baseline for sparse attention mechanisms.

Figure 9: Speedup for the memory processing steps deployed on the GPU-FPGA heterogeneous system for sparse attentions.

Figure 10: Left: End-to-end speedup of the GPU-FPGA system over the baseline for RAG. Right: Speedup of memory processing for the single stage RAG (DRAGIN/FLARE/FS-RAG) and two stage RAG. The reranker of the two stage RAG is executed on GPU.

Figure 11: Left: End-to-end latency of the GPU-FPGA system vs. the baseline for Memory as Context method. Right: Latency for memory processing in Memory as Context. Similar to Titans, we use a linear projection on the current segment for query generation.

Figure 12: Left: End-to-end latency of the GPU-FPGA heterogeneous system vs. the GPU-centric system for MemAgent. Right: Latency for memory processing in MemAgent.

Category	Method	GPU-FPGA	Baseline	Max	Geomean
		(J/req. or J/tok)	(J/req. or J/tok)	Improve	Improve
Sparse Attn.	DSA	15.86	25.62	$1.69\times$	$1.61\times$
	SA-R (Thres.)	0.30	0.34	$1.26\times$	$1.14\times$
	SA-R (Top- $k$ )	0.32	0.36	$1.21\times$	$1.11\times$
	LServe	0.29	0.43	$1.62\times$	$1.43\times$
RAG	DRAGIN	328.16	362.57	$1.34\times$	$1.10\times$

Table 3: The energy efficiency and the improvement of the GPU-FPGA system over the baseline. DSA stands for DeepSeek Attention, and SA-R denotes for SeerAttention-R.

Category	Method	BS=1	BS=2	BS=4	BS=8	BS=32
Sparse Attn.	SA-R (Thres.)	1.12 $\times$	1.21 $\times$	1.33 $\times$	1.47 $\times$	1.60 $\times$
	DSA	1.02 $\times$	1.05 $\times$	1.09 $\times$	1.14 $\times$	1.15 $\times$
	SA-R (Top- $k$ )	1.07 $\times$	1.15 $\times$	1.24 $\times$	1.30 $\times$	1.32 $\times$
	LServe	1.19 $\times$	1.34 $\times$	1.51 $\times$	1.67 $\times$	1.83 $\times$
RAG	DRAGIN	1.14 $\times$	1.20 $\times$	1.29 $\times$	1.44 $\times$	1.92 $\times$
	FLARE	1.19 $\times$	1.26 $\times$	1.38 $\times$	1.55 $\times$	2.11 $\times$

Table 4: Geomean speedup of the GPU-FPGA system over the GPU-centric baseline across batch sizes for various LLM optimization approaches.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 13: Arithmetic intensity (FLOPs/byte) of memory processing pipeline and the rest operations in LLM inference for sparse attention, single-stage RAG, Memory as Context, and TTT/LaCT. For RAG, prepare memory is a one-time operation and will be amortized with multiple queries.

Figure 14: Arithmetic intensity of memory processing pipeline and the rest operations in LLM inference for two-stage RAG. Each stage has a compute relevancy and retrieval step.

Figure 15: (a) Standard cross device data transfer using memory copy for heterogeneous system. (b) PCIe P2P data transfer to bypass system DRAM accesses.

Figure 16: Transfer latency on PCIe bus against the transfer data size. For transferring indices (KB) and single-token KV embeddings (MB) are in the order of microseconds, which is negligible compared to the latency of memory processing.

Device	Model	Memory
CPU	AMD EPYC 7v13 @ 2.5GHz (7 nm)	1 TB (51.2 GB/s)
GPU	AMD Instinct MI210 (6 nm)	64 GB (1.6 TB/s)
FPGA	AMD Alveo U55C (16 nm)	16 GB (460 GB/s)

Appendix

Table 5: Hardware Specification

Figure 17: FPGA kernel architecture for Memory as Context method. The kernel is a dataflow design with each module fully data driven compute a single operation. Past memory embeddings are cached in the HBM and the segment embeddings are loaded from the CPU directly for each incoming segment.

Figure 18: FPGA kernel architecture for MemAgent. Following the design in prior works ( Zeng et al., 2024 ; He et al., 2025c ; He et al., 2025b ) , we design the kernel specialized to LLM decoding, where the KV cache is delivered from the GPU through PCIe. Linear projections are executed in INT4 to align with the weight precision, and the rest operations (attention, SwiGLU, LayerNorm) are calculated at FP32 to maintain accuracy of the model.

Figure 19: End-to-end latency of each sparse attention mechanism on the baseline and the GPU-FPGA system with respect to the sequence length.

Figure 20: Latency of memory processing in each sparse attention mechanism with respect to the sequence length.

Figure 21: End-to-end latency of each RAG system on the baseline system and GPU-FPGA system with respect to the document counts.

Figure 22: Latency of memory processing in single stage RAG using BM25 as the retrieval heuristic (DRAGIN, FLARE, FS-RAG) and two stage RAG with respect to the document counts.

Figure 23: Energy efficiency of sparse attention mechanisms in Joule per token.

Figure 24: Energy efficiency of RAG systems in Joule per request.

Figure 25: The relative speedup of end-to-end inference of DeepSeek V3.2 Exp with DeepSeek Attention when deployed on MI210, A100, MI210 + U55C, and A100 + U55C. A100 is generally faster in LLM inference than MI210. When integrating U55C with A100, the GPU-FPGA heterogeneous system can still speed up the inference.

Figure 26: The relative speedup of memory processing in DeepSeek Attention when deployed on MI210, A100, MI210 + U55C, and A100 + U55C. Even with MI210, the GPU-FPGA heterogeneous system can still outperform A100.

LLM Inference

Large language model inference studies how models generate outputs efficiently, reliably, and securely across hardware and workloads.

See all 268 papers

NeuroPrefetcher: Storage-Aware Sparse LLM Inference via Delta Prefetching
Aug 23, 2026Nobel Dhar, Md Romyull Islam, Xuechen Zhang +4Parallel LLM Inference LLM Inference
Kennesaw State University Marietta, USA · Memory Solutions Lab, Samsung Semiconductor, Inc. San Jose, California, USA
Prima.cpp: Fast 30-70B LLM Inference on Heterogeneous and Low-Resource Home Clusters
Apr 7, 2025Zonghang Li, Tao Li, Wenjiao Feng +8On-Device Language Model Inference LLM Inference
MBZUAI · UESTC · City University of Hong Kong +1
InferScale: GPU-Native KV Injection for Personalized LLM Serving
Jul 29, 2026Peter Li, Prashant PandeyLLM Serving Memory-Augmented Language Models
Northeastern University Boston, Massachusetts, USA

Aug 23, 2026J/K move · Enter open · S save

NeuroPrefetcher: Storage-Aware Sparse LLM Inference via Delta Prefetching

Nobel Dhar, Md Romyull Islam, Xuechen Zhang, Gongjin Sun+3

Kennesaw State University Marietta, USA · Memory Solutions Lab, Samsung Semiconductor, Inc. San Jose, California, USA

Parallel LLM Inference LLM Inference LLM Inference Acceleration On-Device Language Model Inference

Open paper AskPDF arXiv

Deploying large language models on edge devices is increasingly limited by a widening gap between model size and available memory. Existing approaches such as quantization, smaller models, and offloading can raise the effective memory limit, but they still assume that the model can be compressed or partitioned to fit within some budget. We target the harder model-exceeds-memory setting, in which the model remains larger than resident memory throughout execution and storage becomes an active source of weights on the critical path. We observe that MLP activity during autoregressive decoding has strong temporal locality: approximately 82-85% of active neurons persist from one token to the next. This means that most sparse weights needed for the current token are already resident, and only the newly needed rows must be fetched from storage. We present NeuroPrefetcher, a storage-backed LLM inference system that exploits this property through predictive delta prefetching. After layer 0, a single GPU-resident predictor, occupying 2.86% of base model parameters, predicts sparse activity for all downstream MLP layers in one forward pass. The runtime compares these predictions against resident GPU buffers and issues application-scheduled NVMe reads only for incoming delta rows, replacing reactive operating-system demand paging with explicit, model-aware weight movement. On real unified-memory edge hardware, NeuroPrefetcher achieves 7.9-12.0x speedup over llama.cpp across constrained memory budgets.

Abstract

LLM Inference

Explore similar work

NeuroPrefetcher: Storage-Aware Sparse LLM Inference via Delta Prefetching

Prima.cpp: Fast 30-70B LLM Inference on Heterogeneous and Low-Resource Home Clusters

InferScale: GPU-Native KV Injection for Personalized LLM Serving