AXELRAM: Quantize Once, Never Dequantize

Yasushi Nishida

cs.LGApr 3, 2026

AXELRAM: Quantize Once, Never Dequantize

Authors: Yasushi Nishida

Organizations: Axelidea Inc.

Abstract

We propose AXELRAM, a smart SRAM macro architecture that computes attention scores directly from quantized KV cache indices without dequantization. The key enabler is a design-time fixed codebook: orthogonal-transform-based quantization concentrates each coordinate's distribution to N(0,1/d), so the optimal quantizer depends only on dimension d and bit-width b, not on input data. The asymmetric path design -- transform on write, table-lookup on read with no inverse transform -- reduces per-query multiplications by 102.4x (a mathematical identity). Through multi-seed evaluation (10 seeds x 3 models), we discover that sign pattern sensitivity causes catastrophic PPL spikes (Delta > 50) on certain models (Qwen2.5-3B), while others (LLaMA-3.1-8B) are fully stable. This phenomenon extends SpinQuant's observation of rotation variance in weight quantization to the KV cache domain, where the effect is qualitatively more severe. We trace the root cause to layer-wise norm heterogeneity and propose a gradient-free sign pattern selection (200 candidates, 8 calibration samples, one-time) that eliminates catastrophic spikes with zero additional hardware cost. All source code is available at https://github.com/Axelidea/AXELRAM.

Figures & tables

Fig. 1: Quantize Once, Never Dequantize. Conventional rotation-based quantization (top) must dequantize and inverse-rotate each stored key before computing attention— $T\times O(d\log d)$ per query. AXELRAM (bottom) computes attention entirely within the rotated transform domain: the query is rotated once, and scores are computed via pre-computed codebook products with no dequantization. The orthogonal invariance $\langle\mathbf{q},\mathbf{k}\rangle=\langle R\mathbf{q},R\mathbf{k}\rangle$ makes this possible, yielding $102.4\times$ fewer multiplications.

Fig. 2: AXELRAM smart SRAM macro. Write path (left): norm extraction, FWHT butterfly network (448 add/sub, zero multipliers), Lloyd-Max comparator quantization (896 comparators), writing 3-bit indices + FP16 norm to SRAM. Read path (right): pre-computed table lookup (128 parallel reads), adder tree (127 adders), norm scaling (1 multiplication). Pre-computation (top center, once per query): table generation with 1024 multiplications. The fixed codebook ROM (30 bytes) is shared by both paths across all $d{=}128$ dimensions.

Fig. 3: Read path detail. Phase 1 (once per query): FWHT rotation of query, table generation ( $d\times 2^{b}$ multiplications). Phase 2 (repeated $T$ times): table lookup, adder tree, norm scaling (1 multiplication per key). Total: 5,120 multiplications for $T{=}4096$ versus 524,288 conventional ( $102.4\times$ reduction).

Model		2-bit	3-bit	4-bit
LLaMA-3.1-8B (FP16: 6.98)	mean $\pm$ std	+0.10 $\pm$ 0.01	+0.02 $\pm$ 0.00	+0.00 $\pm$ 0.00
LLaMA-3.1-8B (FP16: 6.98)	worst	+0.11	+0.02	+0.01
Qwen2.5-3B (FP16: 8.29)	mean $\pm$ std	+8.13 $\pm$ 16.9	+9.64 $\pm$ 14.9	+1.10 $\pm$ 0.85
Qwen2.5-3B (FP16: 8.29)	worst	+58.4	+51.0	+2.98
Qwen3-8B (FP16: 9.07)	mean $\pm$ std	+12.2 $\pm$ 2.51	+0.48 $\pm$ 0.29	+0.04 $\pm$ 0.01
Qwen3-8B (FP16: 9.07)	worst	+17.5	+1.09	+0.05

TABLE I: Sign pattern sensitivity: PPL increase (

\Delta

) over FP16 baseline. Mean

\pm

std over 10 seeds. Worst = maximum

\Delta

observed.

	SpinQuant [ 2 ]	This work
Domain	Weight+Act. (W4A4)	KV cache
Max variance	6 pts (accuracy)	$>$ 50 pts (PPL $\Delta$ )
Catastrophic failure	Not reported	$\Delta>50$
Model dependency	Not analyzed	Norm heterogeneity
Mitigation	Cayley (gradient)	Selection (grad-free)
HW cost	N/A (software)	Zero

TABLE II: Comparison with SpinQuant’s rotation sensitivity findings.

Model	Bits	Default worst	Optimized	Reduction
LLaMA-3.1-8B	2	+0.11	+0.10	–
	3	+0.02	+0.02	–
	4	+0.01	+0.00	–
Qwen2.5-3B	2	+58.43	+0.82	99%
	3	+51.00	+0.58	99%
	4	+2.98	+0.25	92%

TABLE III: Effect of sign pattern optimization. Default worst : maximum

\Delta

across 10 random seeds. Optimized : deterministic result with selected signs. Reduction : spike elimination rate.

	Default	Optimized
Sign ROM content	Random seed-derived	Calibration-selected
Sign ROM size	$d$ bits/layer	$d$ bits/layer
Transform circuit	FWHT + sign flip	FWHT + sign flip
Codebook ROM	30 bytes	30 bytes
Read path	Table lookup + add	Table lookup + add
Additional circuit	–	None

Table 7

Attention Quantization

Attention quantization reduces the precision and memory cost of attention computations while preserving model quality.

See all 37 papers

OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond
May 19, 2026Zunhai Su, Rui Yang, Chao Zhang +12KV Caching Rotation-Based Quantization
Tsinghua University · Meituan LongCat Team · The University of Hong Kong +3
Bridging KV-Cache Quantization and Linear Attention: From Theory to Pretrained Weight Migration
Oct 8, 2026Kaicheng Xiao, Liran Dong, Haotian Li +1Memory-Augmented Neural Networks KV-Cache Quantization
The Chinese University of Hong Kong
QuantMLA: Function-Aligned Dual-Path Quantization for Low-Bit MLA KV Caching
Sep 29, 2026Zunhai Su, Yuxuan Sun, Jianchao Tan +5LLM Inference Acceleration Attention Quantization
The University of Hong Kong · Meituan LongCat Team · South China University of Technology +1

May 19, 2026J/K move · Enter open · S save

OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond

Zunhai Su, Rui Yang, Chao Zhang, Yaxiu Liu+11

Tsinghua University · Meituan LongCat Team · The University of Hong Kong+3

KV Caching Rotation-Based Quantization Memory-Efficient Inference LLM Inference Acceleration KV-Cache Quantization Low-Bit Quantization+1

Open paper AskPDF arXiv

The rapid advancement toward long-context reasoning and multi-modal intelligence has made the memory footprint of the Key-Value (KV) cache a dominant memory bottleneck for efficient deployment. While the established per-channel quantization effectively accommodates intrinsic channel-wise outliers in Key tensors, its efficacy diminishes under extreme compression. In this work, we revisit the inherent limitations of the per-channel quantization paradigm from both empirical and theoretical perspectives. Our analysis identifies Token Norm Imbalance (TNI) as the primary bottleneck to quantization fidelity. We demonstrate that TNI systematically amplifies errors when shared quantization parameters are required to span token groups exhibiting substantial norm disparities. Instead of relying on intricate quantization pipelines (e.g., TurboQuant), we propose OScaR (Omni-Scaled Canalized Rotation), an accurate and lightweight KV cache compression framework for X-LLMs (i.e., text-only, multi-modal, and omni-modal LLMs). Advancing the per-channel paradigm, OScaR employs Canalized Rotation followed by Omni-Token Scaling to mitigate TNI-induced sequence-dimensional variance both effectively and efficiently, further supported by our optimized system design and CUDA kernels. Extensive evaluations across X-LLMs show that OScaR consistently outperforms existing methods and achieves near-lossless performance under INT2 quantization, establishing it as a robust, low-complexity, and universal framework that defines a new Pareto front. Compared with the BF16 FlashDecoding-v2 baseline, our OScaR implementation achieves a notable up to 3.0x speedup in decoding, reduces memory footprint by 5.3x, and increases throughput by 4.1x. The code for OScaR is publicly available at https://github.com/ZunhaiSu/OScaR-KV-Quant.

Abstract

Attention Quantization

Explore similar work

OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond

Bridging KV-Cache Quantization and Linear Attention: From Theory to Pretrained Weight Migration

QuantMLA: Function-Aligned Dual-Path Quantization for Low-Bit MLA KV Caching