FLINT: Efficiently Leveraging High Bandwidth Flash for Capacity-Scalable LLM Inference Acceleration

Geraldo F. Oliveira; Arash Tavakkol; Xiangyu Zhu; Ahmet Caner Yüzügüler; Vamanan Arulchelvan; Lukas Cavigelli; Renzo Andri; Mohammad Sadrosadati

cs.ARAug 25, 2026

FLINT: Efficiently Leveraging High Bandwidth Flash for Capacity-Scalable LLM Inference Acceleration

Authors: Geraldo F. Oliveira, Arash Tavakkol, Xiangyu Zhu, Ahmet Caner Yüzügüler, Vamanan Arulchelvan, Lukas Cavigelli, Renzo Andri, Mohammad Sadrosadati,

Organizations: Huawei Technologies Switzerland AG · Huawei Technologies Co., Ltd. · ETH Zürich · HUST

Abstract

LLM inference is increasingly constrained by accelerator memory capacity rather than compute throughput. This constraint is especially acute in single-accelerator and small-node inference systems, where limited on-package memory capacity restricts the size of deployable models. HBF is an emerging 3D-stacked NAND flash technology that provides multi-terabyte near-accelerator capacity, making it a promising capacity tier for storing LLM weights. However, existing HBF-based proposals face three adoption challenges: they (1) rely on coarse-grained static prefetching for LLM weights aiming to hide the microsecond-level read latency of the NAND flash device while maximizing HBF's read throughput, (2) expose NAND flash management tasks (e.g., refresh operations) to the accelerator-visible critical inference path, and (3) miss optimization opportunities to specialize and optimize the flash-management mechanisms to the workload behavior. Our goal is to design an efficient HBF substrate that integrates HBF as a memory-capacity tier alongside HBM while addressing these three challenges. To this end, we propose FLINT, a workload-driven HBF substrate for capacity-scalable LLM inference. FLINT introduces three mechanisms: (1) a hardware burst-buffer controller that dynamically coalesces and pipelines HBF reads aiming to utilize existing NAND flash buffers while sustaining high HBF bandwidth, (2) a phantom-plane refresh mechanism, which removes refresh from the critical inference path by moving refresh-related NAND flash operations outside the read foreground back via low-cost resource duplication, and (3) a read-only FTL, which replaces SSD-class support for arbitrary writes with a compact table that translates logical weight bursts to physical HBF locations.

Figures & tables

Figure 1: Overview of an HBF -enabled xPU package.

Figure 2: Overview of a NAND flash die organization.

Figure 3: Cost of a minimum-fit HBM -only deployment at 128K tokens of context for various batch sizes: (top) execution time breakdown, (bottom) memory traffic imbalance.

Figure 4: GPU packages needed to hold the model and meet the

50\text{\,}\mathrm{ms}

TPOT SLO at a context size of 128K tokens.

Figure 5: HBF channel usage under the static H 3 prefetcher.

Figure 6: Refresh-induced stall per decode step.

	Page-level L2P ( $1\text{\,}\mathrm{GB}$ DRAM)	ECC engine ( $<\!$ 1\text{,}{\mathrm{mm}}^{2}$$ )	Refresh sched. (counters)	Bad-block map ( $\mathrm{K}\mathrm{B}$ table)	Garbage coll. ( $530\text{\,}\mathrm{\%}$ BW)	Wear leveling (P/E ctrs.)	Out-of-pl. buf. ( $\mathrm{M}\mathrm{B}$ SRAM)	Erase-blk. alloc. ( $\mathrm{K}\mathrm{B}$ table)	Write journal ( $\mathrm{M}\mathrm{B}$ SRAM)
Read path	✓	✓	✓	✓
Write path	✓	✓	✓	✓	✓	✓	✓	✓	✓

Table 1: Major SSD -class FTL components.

Figure 7: Overview of FLINT burst-buffer controller.

Figure 8: Phantom-plane refresh controller.

GPU	B200 (Blackwell) [ 21 ] ; $148$ SMs at $2.5\text{\,}\mathrm{GHz}$ ; $11\,000$ fp16 MACs/ SM /cycle; MFU $0.5$ ; L2 $126\text{\,}\mathrm{MB}$ at $21\text{\,}\mathrm{TB}\text{/}\mathrm{s}$
HBM3e	$192\text{\,}\mathrm{GB}$ , $8\times$ 1\text{,}\mathrm{TB}\text{/}\mathrm{s}$$ stacks [ 23 ] ; calibrated $0.70$ sustained/peak; row hit/miss $70$ / $105\text{\,}\mathrm{ns}$ ; $20\text{\,}\mathrm{GB}$ KV reserve
HBF	Gen-1 SLC [ 33 , 34 ] ; $4\text{\,}\mathrm{TB}$ , $8\times$ ( $512\text{\,}\mathrm{GB}$ , $1\text{\,}\mathrm{TB}\text{/}\mathrm{s}$ ) stacks; $16$ dies/stack, $32$ planes/die, $256$ blocks/plane, $1024$\times$4\text{\,}\mathrm{kB}$ pages/block; $t_{R}/t_{\text{PROG}}/t_{\text{ERASE}}=$ 2\text{,}\mathrm{\SIUnitSymbolMicro s} $/$ 50\text{,}\mathrm{\SIUnitSymbolMicro s} $/$ 3\text{,}\mathrm{ms}$$
Reliability	read-disturb $1\text{\times}{10}^{6}$ senses/block; $1\text{\times}{10}^{5}$ P/E cycles/block; $>\!1$ yr end-of-life retention; in-place block refresh $54.2\text{\,}\mathrm{ms}$ over $512$ planes/stack
Staging	latency-hiding buffer (LHB) $2\text{\,}\mathrm{MB}$ /slot, $5000$ -cycle min. residency; $2\text{\,}\mathrm{MB}$ burst ( $512$ page-senses); weights block-striped across stacks; D2D $1\text{\,}\mathrm{TB}\text{/}\mathrm{s}$ /stack [ 41 ]
Interconnect	NVLink, $18\times$ 450\text{,}\mathrm{GB}\text{/}\mathrm{s}$$ per direction per GPU ; $800\text{\,}\mathrm{ns}$ /hop; per-layer barrier convergence
SSD	PCIe Gen5 $\times$ 4 [ 55 ] ; $14\text{\,}\mathrm{GB}\text{/}\mathrm{s}$ ; $40\text{\,}\mathrm{\SIUnitSymbolMicro s}$ read

Table 2: Simulated system configuration.

Figure 9: Attained HBF bandwidth (left y-axis, bars) and re-fetch percentage (right y-axis, lines) for H 3 and FLINT.

Figure 10: Per-GPU decode throughput.

Figure 11: Per-token decode energy.

Figure 12: Capacity scaling at 128K context size. (top) Per-user TPOT vs. GPU packages, FLINT (blue) against the capacity-lifted HBM oracle (yellow). (bottom) GPU packages needed to serve a total decode batch within the SLO; dashed lines mark the HBM min-fit GPU-package count

N

Figure 13: Decode throughput overhead due to read-disturb-induced refresh operations.

Figure 14: Total decode tokens FLINT serves before the always-on block wears out, per SLC P/E budget (left y-axis), and the corresponding lifetime at a 50-tok/s decode rate (right y-axis).

Structure (size)	Area
HBF die: phantom-plane, 1 plane per die	$+3.1\%$
HBF base die ( ${\mathrm{mm}}^{2}$ at $22\text{\,}\mathrm{nm}$ ):
Burst translation table ( $1\text{\,}\mathrm{MB}$ )	1.29
Cache-request queue $+$ cl-to-page map ( $256\text{\,}\mathrm{kB}$ )	0.35
Migration bitmaps ( $16{\times}$ 32\text{,}\mathrm{kB}$$ )	0.71
Program queues ( $16{\times}$ 0.5\text{,}\mathrm{MB}$$ )	10.5

Table 3: FLINT area overhead.

LLM Inference Acceleration

LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.

See all 1,101 papers

LLM Inference in a Flash!
Sep 14, 2026Sebastian Zhao, Minseo Kim, Coleman Hooper +5LLM Quantization Compute-in-Memory
UC Berkeley · ICSI · LBNL
Characterizing High Bandwidth Flash for LLM Serving
Sep 30, 2026Zack Yu, Chloe Wong, Coleman Hooper +7LLM Inference Efficiency LLM Serving
University of California, Berkeley Berkeley, California, USA · FuriosaAI Seoul, South Korea · University of California, Berkeley; ICSI; LBNL Berkeley, California, USA
BOOST: Concurrent Access to Host Memory and HBM to Accelerate LLM Inference
Sep 11, 2026Anish Saxena, Jae Hyung Ju, Hritvik Taneja +4LLM Inference Efficiency LLM Inference Acceleration
Georgia Tech Atlanta, USA · Nvidia Research Westford, USA · Nvidia Research & Stanford Santa Clara, USA

Sep 14, 2026J/K move · Enter open · S save

LLM Inference in a Flash!

Sebastian Zhao, Minseo Kim, Coleman Hooper, Luca Manolache+4

UC Berkeley · ICSI · LBNL

LLM Quantization Compute-in-Memory LLM Inference Acceleration KV-Cache Compression

Open paper AskPDF arXiv

Large Language Models (LLMs) have shown impressive capabilities across a range of natural language processing tasks, and LLM inference has emerged as a critical workload for enabling downstream applications. The demands of serving LLM inference are becoming increasingly challenging as requests shift toward longer sequences and heavier inference, driven by retrieval-augmented generation, inference-time compute scaling, and long-context applications. Additionally, these challenges are compounded by hardware trends, as memory capacity and communication bandwidth are not scaling as fast as increases in workload complexity. Compute-in-Flash is a promising solution to address memory bandwidth limitations by moving computation close to memory, and to exploit the large capacity of SSD technologies. However, it is challenging to deploy LLMs on these systems as they lack support for high-precision floating point operations and have limited write endurance. In our work, we aim to address these challenges by designing inference algorithms to enable LLM inference on Flash compute-in-memory devices. We present an end-to-end integer-only quantization approach to eliminate expensive floating-point computations. To address the limited write endurance, we design a dictionary-based KV cache compression strategy based on sparse dictionary coding that represents each KV vector as a linear combination of static dictionary vectors. These algorithmic improvements enable us to exploit the benefits of Compute-in-Flash for both model weights and KV cache, and to minimize expensive data transfer operations. Across Llama-3.1-8B and Qwen-2.5-7B, our combined method exhibits limited accuracy degradation while reducing dynamic KV cache traffic by 15

\times

Abstract

LLM Inference Acceleration

Explore similar work

LLM Inference in a Flash!

Characterizing High Bandwidth Flash for LLM Serving

BOOST: Concurrent Access to Host Memory and HBM to Accelerate LLM Inference