RoLaQ: Joint Rotation and Range Calibration for 2-Bit KV Caches in Multi-Head Latent Attention

Shiyu Ji; Peng Ling; Chen Lin; Shupeng Li; Yixuan Wang; Qingfu Zhu; Wanxiang Che

cs.LGOct 14, 2026

RoLaQ: Joint Rotation and Range Calibration for 2-Bit KV Caches in Multi-Head Latent Attention

Authors: Shiyu Ji, Peng Ling, Chen Lin, Shupeng Li, Yixuan Wang, Qingfu Zhu, Wanxiang Che

Organizations: Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology, China · Tsinghua University, China · Baidu Inc., China

Abstract

Multi-head latent attention (MLA) compresses the content of a key-value cache into a shared latent per token, making low-bit storage attractive for long-context inference. At two bits, errors in this representation perturb both attention matching and value readout, with effects that depend on error direction and reconstruction scale. We introduce RoLaQ (Rotation-calibrated Latent Quantization), an offline method for joint rotation and range calibration. RoLaQ refines a shared rotation within subspaces defined by the frozen key and value projections, and jointly learns quantization ranges whose thresholds and reconstruction gains are coupled to control the shared signal's response. Prediction matching and attention-output supervision guide calibration with model weights frozen. Across three MLA models and six benchmarks, RoLaQ achieves the highest average accuracy among the compared 2-bit methods both without and with token protection. On DeepSeek-V2-Lite, an SGLang implementation provides 3.96 times the BF16 KV capacity under the same memory budget and 2.51 times its decode throughput with 256K input tokens.

Figures & tables

Figure 1: (a) MLA shares a content cache between keys and values. (b,c) On one A100 80GB, RoLaQ gives DeepSeek-V2-Lite

3.96\times

the BF16 capacity at equal memory budgets and

2.51\times

its decode throughput with four concurrent requests and 256K inputs.

Figure 2: Two observations on shared-latent quantization. (a) K/V energy in the leading half of the joint spectrum (mean

\pm

layer std.; dashed: 50%). (b) Prediction KL at 8K with OSCAR’s rotation and threshold fixed, normalized by its value at

g_{0}=1.16

(vertical dashed line). DS, ML, and CPM denote DeepSeek-V2-Lite, Moonlight, and MiniCPM3.

Figure 3: Overview of RoLaQ . (a) A shared 2-bit content cache supplies both K/V paths. The dashed box groups the read-side rotation with the projections that absorb it. (b)

H_{K}+H_{V}

defines two spectral subspaces for learning the shared rotation. (c) The learned threshold determines the four-level assignments and gain

g(T)=1/C(T)

. (d) A frozen BF16 teacher guides joint calibration through prediction KL and attention-output loss.

DeepSeek-V2-Lite
Method	Knowledge & QA			Math	Code		Avg.
Method	MMLU	CMMLU	TriviaQA	GSM8K	HumanEval	MBPP
BF16	58.0	64.1	65.2	37.4	28.0	43.4	49.4
RTN	40.5 / 44.9	45.8 / 51.8	41.5 / 59.6	7.0 / 15.1	12.2 / 18.9	17.6 / 28.0	27.4 / 36.4
QuaRot	53.2 / 53.8	58.6 / 60.6	60.4 / 63.4	25.5 / 30.5	25.6 / 25.0	33.8 / 38.6	42.9 / 45.3
KVarN	49.9 / 53.7	56.0 / 59.1	57.8 / 63.4	23.6 / 27.9	20.1 / 26.2	32.6 / 37.0	40.0 / 44.6

Table 1: Downstream accuracy (%) with 2-bit content caches. Each pair reports w.o. / w. token protection (without / with 4 sink and 16 recent tokens retained in BF16). BF16 is a shared reference; bold marks the best 2-bit scores at the displayed precision within each setting.

DeepSeek-V2-Lite
Method	S1	S2	S3	MK1	MK2	MK3	MV	MQ	Avg.
BF16	100.0	100.0	96.0	94.0	88.0	47.0	63.5	89.0	84.7
RTN	86.0	55.0	21.0	23.0	0.0	0.0	33.0	25.2	30.4
QuaRot	100.0	98.0	80.0	54.0	35.0	5.0	64.2	72.0	63.5
KVarN	100.0	97.0	77.0	75.0	58.0	7.0	50.5	74.5	67.4
OSCAR	100.0	89.0	70.0	52.0	7.0	1.0	55.8	62.5	54.7

Table 2: Retrieval accuracy (%) on the eight RULER retrieval tasks with 4 sink and 16 recent tokens retained in BF16. Avg. denotes the task mean; bold marks the best 2-bit scores at the displayed precision.

(a) Calibration components
	DeepSeek		Moonlight
Configuration	MMLU	MBPP	MMLU	MBPP
Initialization	52.0	39.4	67.0	48.4
Rotation only	54.5	41.6	67.2	48.8
Range only	52.3	40.0	67.4	48.4
Rotation + Range	55.9	44.0	67.9	49.2

Table 3: Ablations of calibration components and objectives. (a) MMLU and MBPP accuracy (%) for different component configurations. (b) Six-task average accuracy (%) for different calibration objectives.

		DeepSeek		Moonlight
Rotation	Params.	$S\downarrow$	Avg. $\uparrow$	$S\downarrow$	Avg. $\uparrow$
Full (Exp.)	$d^{2}$	0.664	47.5	0.559	62.0
Full (Cayley)	$d^{2}$	0.661	47.5	0.592	62.3
Random subspaces	$d^{2}/2$	0.628	47.4	0.550	61.9
$H_{K}{-}H_{V}$	$d^{2}/2$	0.624	47.1	0.579	62.2
$H_{K}{+}H_{V}$	$d^{2}/2$	0.622	47.7	0.508	62.3

Table 4: Rotation search spaces. Params. counts trainable rotation parameters per layer;

S

is validation KL relative to OSCAR. Avg. is six-task mean accuracy (%).

Figure 4: Effect of quantizer response on DeepSeek-V2-Lite. Prediction KL to BF16 is normalized by its value at

\kappa=1

for each context length. Using a checkpoint calibrated with full rotation and unit response, we vary the gain at inference while keeping the rotation and thresholds fixed.

Figure 5: Effect of token protection on six-task mean accuracy (%). The horizontal axis gives the number of sink + recent content tokens retained in BF16.

Figure 6: Serving performance of DeepSeek-V2-Lite on an A100 80GB GPU. (a) KV token capacity. (b,c) Decode throughput for long inputs and successive output intervals, with four concurrent requests. (d) P99 time to first token (TTFT) under increasing concurrency. OSCAR and RoLaQ share the INT2 kernels, so their overlapping curves use timings measured with RoLaQ under the same workloads.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Type	Layers	$d$	$d_{r}$
DeepSeek-V2-Lite	MoE	27	512	64
Moonlight-16B-A3B	MoE	27	512	64
MiniCPM3-4B	Dense	62	256	32

Appendix

Table 5: Model configurations;

d

and

d_{r}

are the content-latent and RoPE-key dimensions.

Benchmark	Shots	Metric
MMLU	5	Accuracy
CMMLU	5	Accuracy
TriviaQA	5	Exact match
GSM8K	5	Exact match
HumanEval	0	pass@1
MBPP	3	pass@1

Appendix

Table 6: Downstream evaluation settings.

Setting	Value
Calibration data	Pile, 1,430,800 tokens
Sequence length	512–4,096 tokens
Optimizer / updates	Adam / 427
Rotation learning rate	$10^{-3}$
Threshold learning rate	$5\times 10^{-3}$
Learning-rate schedule	Cosine

Appendix

Table 7: Calibration hyperparameters.

Figure 7: Cumulative K/V projection energy in the common joint eigenbasis. Solid and dashed curves denote keys and values, respectively; shading shows one standard deviation across layers. The diagonal gives the isotropic reference. The vertical line marks the equal-dimensional split used in Figure 2 a.

Method	Knowledge & QA			Math	Code		Avg.
Method	MMLU	CMMLU	TriviaQA	GSM8K	HumanEval	MBPP	Avg.
DeepSeek-V2-Lite
BF16	58.0	64.1	65.2	37.4	28.0	43.4	49.4
RTN	43.0	49.6	55.0	13.3	18.9	26.0	34.3
QuaRot	53.1	60.1	62.4	30.3	27.4	36.2	44.9
KVarN	52.8	59.0	62.6	29.5	24.4	39.8	44.7

Appendix

Table 8: Accuracy (%) with 4 sink and 8 recent BF16 tokens.

Model	Seed	MMLU	CMMLU	TriviaQA	GSM8K	HumanEval	MBPP	Avg.
DS	42	55.1	61.5	63.8	31.6	26.8	42.4	46.9
DS	43	55.3	61.5	64.1	32.5	26.2	42.2	47.0
DS	44	54.8	61.6	64.3	32.3	23.2	40.8	46.2
	std	0.21	0.04	0.24	0.47	1.96	0.87	0.44
ML	42	67.1	74.8	63.0	65.7	38.4	47.2	59.4
ML	43	67.6	74.8	63.0	67.0	42.7	48.8	60.7

Appendix

Table 9: Per-seed accuracy (%) without protection. Avg. averages six tasks; std denotes sample standard deviation.

Model	Seed	MMLU	CMMLU	TriviaQA	GSM8K	HumanEval	MBPP	Avg.
DS	42	55.6	62.5	64.4	34.1	29.9	43.2	48.3
DS	43	55.4	61.8	64.2	33.8	26.8	41.4	47.2
DS	44	55.7	62.1	64.5	33.5	25.6	41.8	47.2
	std	0.15	0.34	0.16	0.30	2.20	0.95	0.61
ML	42	67.3	75.9	64.7	71.3	47.0	47.4	62.3
ML	43	67.7	75.6	64.0	68.8	42.7	51.0	61.6

Appendix

Table 10: Per-seed accuracy (%) with 4 sink and 8 recent BF16 tokens.

Model	Seed	MMLU	CMMLU	TriviaQA	GSM8K	HumanEval	MBPP	Avg.
DS	42	55.9	62.5	64.8	34.6	26.8	44.0	48.1
DS	43	55.5	61.6	64.8	34.3	29.3	42.8	48.0
DS	44	55.9	62.1	65.0	33.6	25.0	40.6	47.0
	std	0.25	0.45	0.14	0.54	2.14	1.72	0.60
ML	42	67.9	76.2	65.7	70.4	46.3	49.2	62.6
ML	43	67.8	76.1	65.2	70.1	44.5	51.6	62.5

Appendix

Table 11: Per-seed accuracy (%) with 4 sink and 16 recent BF16 tokens.

Range	Validation KL $S\downarrow$
Free gain	0.606 $\pm$ 0.031
$\kappa=1$	0.527 $\pm$ 0.008

Appendix

Table 12: Range calibration with

H_{K}+H_{V}

on MiniCPM3.

S

is validation KL relative to OSCAR (lower is better); values are mean

\pm

sample standard deviation over three seeds.

Figure 8: Learned per-layer thresholds with

\kappa=1

for all rotations and seeds. The dashed line is the Lloyd–Max threshold used by OSCAR.

(a) Counting both bags
Question	Mitchell is making nachos for his family. He buys two bags of chips with 55 chips each. If his family has five members, how many chips does each person get if they all get the same number?
Answer	22
BF16	22
QuaRot	Each person gets 55 / 5 = <<55/5=11>>11 chips. #### 11
OSCAR	There are 55 chips in each bag. If there are five people, each person gets 55 / 5 = <<55/5=11>>11 chips. #### 11
RoLaQ	Mitchell buys 2 * 55 = <<2*55=110>>110 chips. There are 5 members in his family, so each person gets 110 / 5 = <<110/5=22>>22 chips. #### 22

Appendix

Table 13: GSM8K examples on DeepSeek-V2-Lite. (a) Both bags must be counted. (b) The available line must be subtracted. We show BF16 final answers and the quantized models’ solutions. All quantized methods retain 4 sink and 16 recent tokens in BF16; RoLaQ uses calibration seed 42.

Method	KV tokens	GiB	Ratio
BF16	1,151,705	33.36	$1.00\times$
OSCAR	4,564,012	33.33	$3.96\times$
RoLaQ	4,564,012	33.33	$3.96\times$

Appendix

Table 15: Allocated KV capacity in Figure 6 (a). GiB measures KV memory; ratios use BF16 as the reference.

Long input
Input length	BF16	RoLaQ	Speedup
1K	366.1	323.6	$0.88\times$
2K	350.9	310.1	$0.88\times$
4K	318.4	308.5	$0.97\times$
8K	271.6	275.4	$1.01\times$
16K	210.5	250.2	$1.19\times$

Appendix

Table 16: Decode throughput (tok/s) underlying Figure 6 (b,c). Both workloads use four concurrent requests. Speedups relative to BF16 are computed from unrounded measurements.

Concurrent requests	BF16	RoLaQ
1	1.40	1.44
2	2.83	2.86
4	5.60	5.73
8	11.19	11.46
16	22.28	22.97
32	44.60	46.16

Appendix

Table 17: P99 TTFT (s) underlying Figure 6 (d). Each request has 16K input and 1K output tokens.

KV-Cache Quantization

Key-value cache quantization compresses transformer attention state to reduce memory use and bandwidth during long-context inference.

See all 100 papers

QuantMLA: Function-Aligned Dual-Path Quantization for Low-Bit MLA KV Caching
Sep 29, 2026Zunhai Su, Yuxuan Sun, Jianchao Tan +5LLM Inference Acceleration Attention Quantization
The University of Hong Kong · Meituan LongCat Team · South China University of Technology +1
RoPE-Aware Bit Allocation for KV-Cache Quantization
Jun 23, 2026Fengfeng Liang, Yuechen Zhang, Jiaya JiaKV Caching LLM Inference Acceleration
1Hong Kong University of Science and Technology · 2The Chinese University of Hong Kong · 3MiMo, Xiaomi Corporation
MLAfication: Efficiently Enabling High-Fidelity MLA for Modern LLMs
Oct 12, 2026Zile Wang, Zhide Xie, Tao JiMulti-Head Latent Attention LLM Inference Acceleration
Fudan University

Sep 29, 2026J/K move · Enter open · S save

QuantMLA: Function-Aligned Dual-Path Quantization for Low-Bit MLA KV Caching

Zunhai Su, Yuxuan Sun, Jianchao Tan, Tao Zhang+4

The University of Hong Kong · Meituan LongCat Team · South China University of Technology+1

LLM Inference Acceleration Attention Quantization KV-Cache Quantization Multi-Head Latent Attention Post-Training Quantization Low-Bit Quantization+1

Open paper AskPDF arXiv

Multi-Head Latent Attention (MLA) enables expressive multi-head attention with compact caches for its content and decoupled RoPE paths, yet cache memory still scales linearly with context length and batch size. In this work, we establish a systematic model of MLA's dual-path quantization errors, characterizing their distinct effects on attention-output distortion and explaining the pronounced amplification of RoPE-path errors. Guided by this analysis, we introduce QuantMLA, a function-aligned framework for low-bit dual-path quantization. We derive path-specific transformation spaces that preserve full-precision computation while remaining fully fusible into model parameters offline, eliminating online transformation overhead. Within these spaces, QuantMLA learns path-specific transformations with function-aligned objectives: attention-output reconstruction captures the content path's coupled matching and aggregation errors, while positional QK reconstruction preserves the RoPE-induced component of the attention logits and admits a theoretical bound on output distortion. Across four MLA model families, QuantMLA enables, to our knowledge, the first reported joint INT4 caching of the content and RoPE caches with minimal accuracy degradation. Further compressing the content cache to INT2 while retaining the RoPE key cache at INT4 maintains competitive performance on challenging reasoning and code benchmarks. We develop a native low-bit MLA attention kernel that integrates unpacking and dequantization directly into attention computation. The physical cache layout provides 3.59x compression at 128K context, while a cache-pressure serving workload achieves 5.168x higher whole-job output throughput than BF16. The code will be released upon acceptance.

Abstract

KV-Cache Quantization

Explore similar work

QuantMLA: Function-Aligned Dual-Path Quantization for Low-Bit MLA KV Caching

RoPE-Aware Bit Allocation for KV-Cache Quantization

MLAfication: Efficiently Enabling High-Fidelity MLA for Modern LLMs