FlagRot: Selective Rotation for Low-Bit KV Caches

Kunming Shao; Xijie Huang; Jierun Chen; Jiaqi Shao; Yanli Wang; Jiangnan Yu; Qiwei Li; Wei Zhang

cs.CLNov 2, 2026

FlagRot: Selective Rotation for Low-Bit KV Caches

Authors: Kunming Shao, Xijie Huang, Jierun Chen, Jiaqi Shao, Yanli Wang, Jiangnan Yu, Qiwei Li, Wei Zhang,

Organizations: The Hong Kong University of Science and Technology · Meta · Sun Yat-sen University · Wuhan University

Abstract

Long-context inference is bound by the KV cache, which 2-bit codes could shrink about sixfold if their error stays in check. The usual remedy rotates keys before quantizing, assuming rotation always helps. However, we observe that it does not: rotation flattens a spiky piece of a key but can give a flat one a new peak, and both coexist within one key. We therefore propose FlagRot, which stores each 32-channel piece rotated or as is, whichever has the smaller calibrated attention error, and rotates the query instead when reading. We further show that how often to decide follows the quantization axis: per-channel caches such as KIVI need one decision per head, at no write cost, while per-token caches decide per key, with the choice stored in existing sign bits. At 2.5 bits per value, FlagRot lowers the twelve-model KL divergence of KIVI from 18.7 to 12.4 x 10^-3, below RotateKV on 9 of 12 models. Served as a vLLM plugin, it raises RULER at 128k tokens by +2.9 points and the LongBench v1 mean by +0.4 to +1.9 points on the three of four models where 2-bit KIVI loses accuracy. It also lowers the error of 4-bit integer and FP4 caches.

Figures & tables

Figure 1: On real cached keys, rotation helps spiky sub-vectors and hurts many flat ones. (a) Two 32-dim key sub-vectors of Llama-3.1-8B before and after

\mathbf{H}_{32}

, with crest factor and change of the query-weighted INT4 error. (b) Share of sub-vectors whose error rises under

\mathbf{H}_{32}

, by decile of the crest factor. (c) The selections of FlagRot within one key and their shares. (d) Whole-head rotation of NVFP4 caches, per model.

Figure 2: FlagRot . (a) At cache write, each 32-dim key sub-vector keeps

I

\mathbf{H}_{32}

, or the whole head takes

\mathbf{H}_{128}

, whichever has the lower query-weighted error; the per-token branch first subtracts the calibrated key mean, and the per-channel branch decides once per completed 32-token key group. (b) INT2 pages of 80 and 97 bytes per token and KV head (

D{=}128

), with the flags in the sign bits of the scales and 4 sink and 128 recent tokens in BF16. (c) Attention rotates the query into each basis once per step; the flags select the partial dot products, and keys are never rotated back.

(a) Equal size: INT2 at 2.5 bits per value
Cache	KL	Held-out	FlagRot-KIVI L/H
KIVI	18.7	17.6	11/1
RotateKV, clip 0.8	25.6	26.5	12/0
RotateKV, clip 1.0	18.1	19.5	9/2
FlagRot-auto (ours)	17.8	17.8	12/0
FlagRot-KIVI (ours)	12.4	12.1	–

Table 1: Simulated KL

\times 10^{3}

(

\downarrow

) with groups of 32 and S4W128, geometric mean over twelve models and over the eight held-out ones. (a) Caches of equal size; L/H: models on which FlagRot-KIVI is significantly lower / higher than the row (paired 95% CI). (b) Each encoder without and with selection (+ sel.); percentages relative to the row above, at

b{+}0.5

bits per value with FP8 or FP16 scale and minimum (

{}^{\dagger}b{+}1

). Models significantly lower with selection: 10–12 of 12 per cell.

Figure 3: Simulated KL

\times 10^{3}

with S4W128. (a) Geometric mean over the four development models against nominal bits per value; RTN-auto: RTN with the centering of FlagRot-auto . (b, c) Context length; 8k, 16k and 32k use 8, 6 and 4 windows. (d) WUSH-KV’s fitted transform at 32k, without and with selection and RoPE-shifted calibration.

Benchmark	Model	BF16 $\to$ KIVI $\to$ Ours	Fixed-mode gain	Per-key gain
RULER 128k	Llama-3.1-8B-It	73.7 $\,\to\,$ 62.3 $\,\to\,$ 65.2	+2.5 [+0.9, +4.1]	+2.9 [+1.2, +4.5]
LongBench v1	Qwen3-8B	49.35 $\,\to\,$ 47.28 $\,\to\,$ 47.70	+0.72 [+0.08, +1.33]	+0.42 [ $-$ 0.20, +1.04]
	Llama-3.1-8B-It	49.70 $\,\to\,$ 47.38 $\,\to\,$ 48.24	+0.47 [ $-$ 0.17, +1.10]	+0.86 [+0.23, +1.48]
	Qwen3-32B	44.97 $\,\to\,$ 42.80 $\,\to\,$ 44.73	–	+1.94 [+1.22, +2.66]
	Qwen3.8-27B ∗	53.49 $\,\to\,$ 53.82 $\,\to\,$ 53.25	$-$ 0.74 [ $-$ 1.19, $-$ 0.30]	$-$ 0.57 [ $-$ 0.99, $-$ 0.17]

Table 2: vLLM, 2.5 bits/value (FP8 scale and minimum, 80 B/token/KV head), S4W128 on H200. Path: BF16

\to

KIVI-only

\to

FlagRot-KIVI (per-key), scores 0–100. Gains: points over KIVI-only [nominal paired 95% CI], same kernel and layout. Blue: CI above zero; –: not run. LongBench: 21-task macro-score, 4,750 items/model; RULER: 1,300 items; BF16-relative intervals: Table 18 . Scores and contrasts rounded independently. ∗ Hybrid model: KV cache in 16 of 64 layers.

	Equal-budget gain
Endpoint	vs. FlagRot- auto	vs. KIVI-only	vs. BF16
Llama-It	+18.5	+2.9	$-$ 8.4
RULER 128k	[+16.5, +20.4]	[+1.2, +4.5]	[ $-$ 10.0, $-$ 6.8]
Qwen3-8B	+1.7	+0.4	$-$ 1.6
LongBench	[+1.0, +2.4]	[ $-$ 0.2, +1.0]	[ $-$ 2.2, $-$ 1.1]
Llama-It	+1.8	+0.9	$-$ 1.5

Table 3: 2.5-bit FlagRot-KIVI in vLLM: score differences in points [nominal paired 95% CI]; all 2-bit caches at 80 B/token/KV head. Blue: CI above zero. Llama-It: Llama-3.1-8B-Instruct.

Group	INT4 ( $-$ %)	MXFP4 ( $-$ %)	NVFP4 ( $-$ %)
Dev. (4)	55 [51, 61]	69 [65, 75]	46 [38, 53]
Larger (4)	58 [13, 69]	62 [30, 69]	33 [17, 56]
Held-out (7)	56 [41, 57]	70 [63, 89]	45 [42, 55]
All (15)	56 [13, 69]	67 [30, 89]	44 [17, 56]

Table 4: 4-bit caches without BF16 tokens: reduction of KL (%) with FlagRot relative to RTN, median over each group of models with the range in brackets. Significant (95% CI) on 15 of 15 models per format. Qwen2.5-7B uses static key centering in both caches; per-model values are in Table 25 .

(a) LongBench v1: points added by selection
Compared with	HF	vLLM
Qwen3-8B
Centered RTN	+3.70 [+1.77, +5.66]	+3.81 [+1.91, +5.75]
Fixed mode	+2.74 [+1.01, +4.52]	+1.48 [ $-$ 0.15, +3.08] ‡
Llama-3.1-8B-Instruct
Centered RTN	+5.20 [+3.27, +7.24]	+3.69 [+1.71, +5.69]

Table 5: Selection and centering on per-token INT2 caches (2.5-bit served layout;

g{=}32

, S4W128). Paired score differences [95% CI]. (a) Per-key selection minus the named baseline; HF: HuggingFace simulation. (b) Qwen3-8B on H20. Bold: positive CIs or Holm-significant means. Experiment settings, ‡ fixed-baseline differences and AIME scores: Appendix A.6 .

Variant	KL	Held-out	Worse
FlagRot-auto (per-token keys)
Full, weighted by $C_{q}$	17.8	17.8	–
Plain reconstruction error	29.2	30.4	12/12
Per-piece candidates only	23.3	23.4	11/12
Whole-head candidate only	21.0	21.2	12/12
No selection	43.4	45.2	12/12

Table 6: Ablation at INT2 with S4W128: KL

\times 10^{3}

, geometric mean over twelve models and over the eight held-out ones. Worse: models significantly above the full method (paired 95% CI).

	$\tau^{2}$ , 3 domains		SWE-V
vs BF16	passˆ1	passˆ4	resolved
FP8	$-$ 2.6	$-$ 4.1	$-$ 4.8
	[ $-$ 4.7, $-$ 0.5]	[ $-$ 6.8, $-$ 1.5]	[ $-$ 9.0, $-$ 0.6]
FlagRot ⋆	$-$ 0.7	$-$ 2.5	$-$ 2.8
	[ $-$ 2.9, +1.6]	[ $-$ 4.9, $-$ 0.1]	[ $-$ 7.0, +1.4]
BF16 rerun	$-$ 0.8 to +0.6	$-$ 0.7 to $-$ 0.3	$-$ 3.0

Table 7: Qwen3.8-27B on

\tau^{2}

(16 trials per task, 278 tasks pooled) and all 500 SWE-bench Verified instances: differences in points (95% CI); gray: CI below zero. BF16 rerun : for

\tau^{2}

, range over three 8-vs-8 seed splits; for SWE, a second run. ⋆ S4W128.

Qwen3-8B
		tok/s at TPOT $\leq$
Cache	Seqs	40 ms	60 ms	80 ms
BF16	23	707	707	707
FP8	46	1,494	1,494	1,494
FlagRot-auto	142	1,905	2,150	2,175
Qwen3-32B

Table 8: Decoding throughput at a 32k-token context on one H200 (no prefix caching): generated tokens/s of the best batch within each time-per-output-token budget. Seqs: most 32k sequences that fit in the cache. FlagRot-auto : per-token INT2 at 2.5 bits per value.

		TPOT (ms) at batch				Prefill	Cap.
Cache	Bits	64	32	8	1	(tok/s)	(M)
BF16	16	29.1	22.0	17.1	14.9	12,031	0.93
KIVI-only	3.0	35.3	26.1	19.1	15.9	11,518	2.29
FlagRot-KIVI	3.0	40.7	29.0	19.7	16.0	10,893	2.29
KIVI-only	2.5	33.2	24.8	18.5	15.5	11,792	2.75
FlagRot-KIVI	2.5	38.7	27.8	19.1	15.7	11,172	2.75

Table 9: Pure decoding of Qwen3.8-27B in vLLM on one H200 (8192-token context, no prefix caching): time per output token, prefill throughput and KV capacity in tokens at equal memory. Caches at S4W128; KIVI rows at 2.5 bits store an FP8 scale and minimum; capacity as reported by vLLM, whose pages on this hybrid model also hold the linear-attention state; BF16 was re-measured in each session within 1%.

Appendix figures & tables31 assets

Supplementary material from the paper’s appendix.

Appendix

				QK-
Model	Layers	$H_{kv}$	$D$	norm	INT2	Suite
Development models
Llama-3.1-8B	32	8	128	no	✓	✓
Qwen3-8B	36	8	128	yes	✓	✓
Mistral-7B-v0.3	32	8	128	no	✓	✓
Qwen3-14B	40	8	128	yes	✓	✓

Appendix

Table 10: Models: attention layers with a KV cache, KV heads

H_{kv}

and head dimension

D

. All fifteen enter the 4-bit study (Table 4 ). INT2 : the INT2 cohort; Suite : the 4-bit baseline and protected-token comparisons (Appendix C.2 ).

$B$	Context	FlagRot-KIVI	FlagRot-auto	Ratio
8	16k	258	97	2.7
8	32k	467	162	2.9
32	16k	880	318	2.8
32	32k	1721	600	2.9
64	16k	1723	592	2.9
64	32k	3402	1161	2.9

Appendix

Table 11: Decode attention of one layer in

\mu

s (Llama-3.1-8B shapes, H20, CUDA graphs, median), FlagRot-KIVI in the packed 2.5-bit layout; Ratio : FlagRot-KIVI over FlagRot-auto.

(a) 12 models, INT2, 4 sink + 128 recent BF16 tokens				(b) WUSH-KV setting, 12 models
Cache	Bits	KL	Lower/higher	Cache	KL	L/H
RTN	2.5	75.4		WUSH-KV transform	35.7
RTN, centered	2.5	43.4		+ selection on it	26.7	12/0
Fixed mode, centered	2.5	27.4	0/11 FA	Head selection, centered	24.8	11/0
FlagRot-auto	2.5 (80 B)	17.8	12/0 cRTN	FlagRot-auto	78.8	1/11
KIVI	3.0 (97 B)	18.4

Appendix

Table 12: Simulated KL

\times 10^{3}

(

\downarrow

), geometric mean over models on all 2048-token WikiText-2 test windows. Bits: nominal bits per value with metadata (KIVI: FP16 scale and minimum; § FP8; others: 16 bits per group); in parentheses, page bytes per token and KV head in our kernels (97 B: 3.03 bits at

D{=}128

). Lower/higher (L/H): models whose paired 95% CI puts the KL below / above the reference (cRTN: centered RTN; FA: FlagRot-auto; FK: FlagRot-KIVI; RKV: RotateKV clip 1.0); the rest are not separated. (b) Setting of Table 15 with QuEST-2, centering statistics from 4 of its FineWeb sequences; vs WUSH-KV. (c) vs RotateKV.

	Per-token		Per-token, centered			Per-channel, 2.5 bits		Per-channel, 3.0 bits		RotateKV (our simulation)
Model	RTN	FlagRot	RTN	Fixed	FlagRot-auto	KIVI	FlagRot-KIVI	KIVI	FlagRot-KIVI	g32, 0.8	g32, 1.0	g128, 0.8
Llama-3.1-8B	115	25.5	39.8	27.7	17.2	18.5	12.7	18.8	12.8	16.1	13.8	14.7
Qwen3-8B	83.9	44.7	54.9	39.6	25.7	29.8	18.6	30.0	19.0	43.0	28.0	29.0
Mistral-7B-v0.3	44.5	13.7	33.9	21.8	13.8	17.2	9.3	17.0	9.3	14.9	8.9	11.1
Qwen3-14B	56.4	29.6	35.1	22.8	17.1	21.3	13.1	21.1	13.4	31.9	17.3	19.1
Qwen3.8-27B	27.0	21.1	22.5	19.3	18.7	19.6	14.2	20.0	14.3	26.2	20.6	21.8

Appendix

Table 13: Per-model KL

\times 10^{3}

of INT2 caches at S4W128 (all WikiText-2 test windows). Centered : selective key centering; per-channel 2.5 bits: FP8 scale and minimum; RotateKV columns: group size, clip ratio (2.5 bits for g32). Bold: lowest per model.

Calibration variant	$\Delta$ (%)	max	L/H/T
FlagRot-KIVI (3.0-bit layout)
1 window	+0.4	1.2	0/0/6
2 windows	$-$ 0.7	1.5	3/0/3
8 windows	$-$ 0.1	0.7	0/0/6
16 windows	+0.2	1.8	0/0/6
FineWeb-Edu instead of WikiText-2	+2.1	3.7	0/4/2

Appendix

Table 14: Calibration sensitivity at INT2 with S4W128: change of the geometric-mean KL over six models (four development models, Llama-3.2-3B and Phi-4) relative to the default of four WikiText-2 windows, largest change on one model (max, %), and models significantly lower / higher / tied (paired 95% CI). ∗ Development models only.

			FlagRot
Model	Head	WUSH-KV	-H	-C K	-C	-C $8{\times}8$
Llama-3.1-8B	2.03	1.78	1.37	1.53	1.44	1.36
Qwen3-8B	14.59	3.57	7.84	2.34	2.14	1.93
Mistral-7B-v0.3	1.11	1.03	0.70	0.85	0.77	0.72
Qwen3-14B	2.99	2.61	1.74	1.72	1.56	1.42
Qwen3.8-27B	5.08	4.70	3.38	3.62	3.16	3.26

Appendix

Table 15: WUSH-KV’s published setting: QuEST 4-bit, one scale per token and head, S16W128, 128 FineWeb-Edu calibration sequences. KL

\times 10^{3}

; bold: best. FlagRot -C builds on WUSH-KV’s transform; -C K : keys only; -C

8{\times}8

: with FlagRot -L candidates. Qwen2.5-7B is excluded (all methods exceed 340).

S4W128
	INT2			INT3
Group size $g$	128	64	32	128	64	32
RTN	223	106	75.4	21.1	13.8	10.5
FlagRot	54.1	37.0	28.5	8.1	6.1	4.6
FlagRot-C	30.7	25.3	21.9	5.0	4.6	4.2
KIVI	44.7	28.4	18.4	5.8	4.4	3.2

Appendix

Table 16: Rate and distortion: geometric-mean KL

\times 10^{3}

over twelve models. Bits per value:

b+16/g

for per-token caches,

b+32/g

for KIVI; without BF16 tokens, KIVI keeps the incomplete key group in BF16.

			Llama-3.1-8B					Qwen3-8B
			KL	$\Delta$ vs RTN (%)				KL	$\Delta$ vs RTN (%)
Text	Len.	Win.	RTN	FlagRot	FlagRot-C	KIVI	FlagRot-KIVI	RTN	FlagRot	FlagRot-C	KIVI	FlagRot-KIVI
WikiText-2	2k	141/146	115	$-$ 78	$-$ 83	$-$ 84	$-$ 89	83.9	$-$ 47	$-$ 55	$-$ 64	$-$ 77
PG-19	2k	77	88.6	$-$ 77	$-$ 81	$-$ 89	$-$ 92	55.6	$-$ 46	$-$ 58	$-$ 67	$-$ 78
PG-19	8k	77	266	$-$ 87	$-$ 89	$-$ 94	$-$ 96	132	$-$ 44	$-$ 45	$-$ 80	$-$ 87
PG-19	32k	24	554	$-$ 90	$-$ 70	$-$ 96	$-$ 97	292	$-$ 28	+366	$-$ 89	$-$ 92

Appendix

Table 17: INT2 caches at S4W128 on other text, calibration frozen on four WikiText-2 training windows: RTN KL

\times 10^{3}

and the change (%) from it. Len./Win. : window length and windows per model. Bold / gray: significantly below / above RTN (paired bootstrap, 95% CI).

Model	Task	Items	BF16	KIVI-only	FlagRot-KIVI	Fixed	Per-key
				$\Delta$ vs BF16		Selection over KIVI-only
Qwen3-8B	LongBench v1	4,750	49.35	$-$ 2.07	$-$ 1.65	+0.72	+0.42
				[ $-$ 2.69, $-$ 1.47]	[ $-$ 2.19, $-$ 1.12]	[+0.08, +1.33]	[ $-$ 0.20, +1.04]
Llama-3.1-8B-It	LongBench v1	4,750	49.70	$-$ 2.32	$-$ 1.46	+0.47	+0.86
				[ $-$ 2.96, $-$ 1.70]	[ $-$ 2.03, $-$ 0.91]	[ $-$ 0.17, +1.10]	[+0.23, +1.48]
Qwen3-32B	LongBench v1	4,750	44.97	$-$ 2.17	$-$ 0.23	–	+1.94

Appendix

Table 18: FlagRot-KIVI in vLLM at 2.5 bits per value (FP8 scale and minimum, 80 B per token and KV head) with S4W128 on H200: BF16 score (%) and paired differences in points with nominal 95% CIs. Selection: FlagRot-KIVI in its fixed and per-key mode minus KIVI-only, same kernel and layout; –: not run.

Benchmark	Model	BF16 $\to$ KIVI $\to$ Ours	Fixed-mode gain	Per-key gain
RULER 128k	Llama-3.1-8B-It	73.7 $\,\to\,$ 61.8 $\,\to\,$ 65.9	+3.9 [+2.3, +5.5]	+4.2 † [+2.6, +5.8]
LongBench v1	Qwen3-8B	49.35 $\,\to\,$ 47.39 $\,\to\,$ 48.45	+0.5 [ $-$ 0.1, +1.1]	+1.05 [+0.47, +1.64]
	Llama-3.1-8B-It	49.70 $\,\to\,$ 47.55 $\,\to\,$ 48.21	+0.3 [ $-$ 0.3, +1.0]	+0.66 [+0.03, +1.30]
	Qwen3.8-27B	53.49 $\,\to\,$ 53.63 $\,\to\,$ 53.18	$-$ 1.0 [ $-$ 1.4, $-$ 0.6]	$-$ 0.45 [ $-$ 0.89, 0.00]
	Qwen3-32B	44.97 $\,\to\,$ 43.25 $\,\to\,$ 44.65	–	+1.40 [+0.72, +2.07]
AIME24+25	Qwen3-8B	71.5 $\,\to\,$ 68.5 $\,\to\,$ 68.5	+2.1 [ $-$ 0.2, +4.4]	0.0 [ $-$ 2.7, +2.7]

Appendix

Table 19: FlagRot-KIVI in vLLM, 3.0 bits/value (FP16 scale and minimum), S4W128 (H200; retail H20). Path: BF16

\to

KIVI-only

\to

FlagRot-KIVI (per-key), scores 0–100. Gains: points over KIVI-only, same kernel [nominal paired 95% CI]. Fixed: decided once from calibration data; per-key: decided as each key is written. Blue: CI above zero. † Holm (per-key)

p<0.001

on RULER,

p=1

on AIME/retail, one family; LongBench CIs descriptive. Metrics: RULER 13-subtask string-match mean (1,300 items); LongBench 21-task macro-score (4,750/model); AIME pass@1 (60 problems

\times

8 samples); retail DB-only pass 1 (114 tasks

\times

4 trials). Scores/contrasts rounded independently; –: not run.

Endpoint	KIVI-only	Fixed	Per-key
Llama-It, RULER 128k	61.8	+3.9	+4.2
		[+2.3, +5.5]	[+2.5, +5.7]
Llama-It, LongBench	47.5	+0.3	+0.7
		[ $-$ 0.3, +1.0]	[+0.0, +1.3]
Qwen3-8B, LongBench	47.4	+0.5	+1.1
		[ $-$ 0.1, +1.1]	[+0.5, +1.6]

Appendix

Table 20: Fixed and per-key selection on the 3.0-bit per-channel cache in vLLM (FP16 scales and minima; H200, same kernel, layout and protection): score minus KIVI-only in points (nominal 95% CI). Llama-It: Llama-3.1-8B-Instruct.

H200, 114 tasks $\times$ 16 trials
Cache	pass^1	$\Delta$	pass^4	msg/sim	limit
BF16	27.8		10.8	51.3	7.0%
KIVI-only	20.4	$-$ 7.3	6.3	58.5	11.5%
		[ $-$ 12.0, $-$ 2.7]
FlagRot-KIVI	22.1	$-$ 5.7	5.4	62.0	12.3%
		[ $-$ 10.4, $-$ 0.9]

Appendix

Table 21: Qwen3-8B on

\tau^{2}

-bench retail, INT2 caches at S4W128: database-state pass^1 and pass^4 (%), paired

\Delta

of pass^1 from BF16 (points, nominal 95% CI), messages per simulation and share hitting the step limit. KIVI rows: 3.0-bit layout; fixed mode as in Table 20 . Pooled: seeds 300, 304–307. –: not recorded.

Qwen3-8B
		$\Delta$ vs BF16
Task	BF16	FP8	RTN	FlagRot	-auto	$\Delta\Delta$
LongBench v1 ‡	49.4	+0.1	$-$ 11.0	$-$ 4.4	$-$ 3.3	+7.7
LongBench v2	32.4	0.0	$-$ 7.8	$-$ 2.8	$-$ 2.8	+5.0
RULER 32k	73.2	+0.7	$-$ 32.7	$-$ 10.7	$-$ 8.8	+23.9
AIME24	75.0	+0.4	$-$ 35.4	$-$ 9.6	$-$ 1.7	+33.7

Appendix

Table 22: Per-token INT2 caches in vLLM on all tasks: BF16 score (%) and paired differences in points. RTN, FlagRot and -auto (FlagRot-auto) use S4W128; FP8: vLLM’s FP8 cache with BF16 queries.

\Delta\Delta

: -auto minus RTN on the same items. ‡ H200 (21-task mean); all others H20.

Figure 4: Closed-loop

\tau^{2}

-bench retail with prefix caching and S4W128: generated tokens/s, completed and successful simulations per hour, relative to BF16.

Agents	Cache	tok/s	sims/h	succ./h	msg/sim	$\times$ BF16
Qwen3-8B, H200
64	BF16	1354	3257	861	49.3	1.00
	FP8	1598	3435	866	57.4	1.05
	FlagRot, INT4	1508	3211	843	53.3	0.99
	KIVI-only	906	2153	387	53.8	0.66
	FlagRot-KIVI, fixed	972	1892	369	61.2	0.58

Appendix

Table 23: Closed-loop

\tau^{2}

-bench retail serving with prefix caching; INT2 and INT4 caches at S4W128, KIVI rows in the packed 2.5-bit layout; FP8: vLLM’s FP8 cache with BF16 queries. sims/h , succ./h : completed and successful simulations per hour;

\times

BF16: sims/h relative to BF16.

Per-token INT2
Cache	Qwen3-8B	Llama-3.1-8B-It
BF16 (score)	49.90	50.13
RTN	$-$ 11.24	$-$ 11.74
RTN, centered	$-$ 5.92	$-$ 6.97
Fixed mode	$-$ 5.24	$-$ 4.94
Fixed mode, centered	$-$ 4.97	$-$ 4.19

Appendix

Table 24: HuggingFace study: BF16 score (%) and differences from BF16 in points on 16 LongBench v1 tasks (50 items each); INT2,

g{=}32

, S4W128 unless noted.

Development models
	INT4 ( $g{=}32$ )					MXFP4					NVFP4
Model	KL	Head	Fixed	FlagRot	-L	KL	Head	Fixed	FlagRot	-L	KL	Head	Fixed	FlagRot	-L
Llama-3.1-8B	20.2	$-$ 36.8	$-$ 40.3	$-$ 54.2	$-$ 72.7	95.1	$-$ 70.7	$-$ 68.3	$-$ 72.7	$-$ 86.7	26.8	$-$ 30.4	$-$ 33.1	$-$ 53.4	$-$ 72.1
Qwen3-8B	43.5	$-$ 28.4	$-$ 44.2	$-$ 56.4	$-$ 73.2	140.3	+41.5	$-$ 49.9	$-$ 65.0	$-$ 78.1	38.4	+214.9	$-$ 12.3	$-$ 37.8	$-$ 59.2
Mistral-7B-v0.3	7.1	$-$ 33.6	$-$ 37.6	$-$ 51.0	$-$ 69.2	30.6	$-$ 62.0	$-$ 58.6	$-$ 65.5	$-$ 83.3	10.6	$-$ 35.2	$-$ 32.6	$-$ 51.1	$-$ 71.1
Qwen3-14B	25.1	$-$ 46.3	$-$ 49.0	$-$ 61.4	$-$ 75.5	113.3	$-$ 56.0	$-$ 59.3	$-$ 75.0	$-$ 86.0	24.2	+31.6	$-$ 12.8	$-$ 40.2	$-$ 61.8

Appendix

Table 25: 4-bit KV caches without BF16 tokens: KL

\times 10^{3}

without rotation and relative KL change (%) of each rotation. Head : whole-head Hadamard; Fixed : fixed mode; -L : FlagRot -L. Bold: best and significantly below zero; gray: significantly above zero (95% CI). Qwen2.5-7B without key centering.

	INT4									NVFP4
Model	RTN	TurboQ	Smooth	Head	Base	Fixed	FlagRot	-L	-C	RTN	4/6	4/6+FR	FlagRot	-L
bits/elem.	4.50	4.9–7.1	4.50	4.50	4.50	4.50	4.50	4.62	4.50	4.50	4.50	4.50	4.50	4.71
Llama-3.1-8B	20.2	16.6	18.4	12.7	7.8	12.0	9.2	5.5	7.0	26.8	21.0	10.5	12.5	7.5
Qwen3-8B	43.5	27.3	33.2	31.2	14.8	24.3	19.0	11.7	13.3	38.4	33.4	19.3	23.9	15.7
Mistral-7B-v0.3	7.1	6.0	6.7	4.7	3.2	4.4	3.5	2.2	2.8	10.6	8.4	4.3	5.2	3.1
Qwen3-14B	25.1	17.2	21.2	13.5	10.9	12.8	9.7	6.1	9.7	24.2	20.1	12.0	14.5	9.2

Appendix

Table 26: Recent KV-cache quantizers without BF16 tokens: KL

\times 10^{3}

; bold: best per group. TurboQ : TurboQuant in vLLM; Smooth : SmoothAttention; Head : whole-head Hadamard; Base : calibrated base alone; -L / -C : FlagRot -L/ FlagRot -C; 4/6 : Four Over Six; FR : FlagRot . Qwen2.5-7B without key centering.

	$S{=}4$ , $W{=}128$					$W{=}128$ only
Model	RTN	Fixed	FlagRot	FlagRot -C	FlagRot -L	RTN	Fixed	FlagRot	FlagRot -C
Llama-3.1-8B	2.30	1.52	1.31	1.28	1.05	3.22	1.92	1.83	1.71
Qwen3-8B	4.05	2.30	2.09	1.93	1.60	10.00	5.01	3.96	7.14
Mistral-7B-v0.3	1.34	0.78	0.70	0.71	0.52	1.46	0.83	0.77	0.79
Qwen3-14B	2.71	1.69	1.40	1.46	1.25	7.36	3.74	2.78	5.56
Qwen3.8-27B	3.38	2.92	2.88	2.92	2.53	3.80	3.34	2.89	3.46

Appendix

Table 27: Per-model KL

\times 10^{3}

of INT4 caches with BF16 sink and recent tokens; geometric means for other windows exclude Qwen2.5-7B (no key centering; every per-token 4-bit cache exceeds 600).

Task	BF16	$\Delta$ FlagRot [95% CI]	BF16	$\Delta$ FlagRot [95% CI]	BF16	$\Delta$ FlagRot [95% CI]
	Qwen3-8B		Llama-3.1-8B-It		Qwen3.8-27B
RULER (32k)	73.2	$-$ 0.1 [ $-$ 1.1, +0.9]	86.1	$-$ 0.2 [ $-$ 0.8, +0.3]	88.2	+0.1 [ $-$ 0.1, +0.3]
LongBench v2	32.4	+0.4 [ $-$ 1.0, +1.8]	27.8	+1.4 [+0.2, +2.6]	50.7	$-$ 0.6 [ $-$ 3.2, +2.2]
LongBench v1 ¶	49.2	0.0 [ $-$ 0.3, +0.3]	49.6	$-$ 0.1 [ $-$ 0.5, +0.2]	54.3	$-$ 0.2 [ $-$ 0.4, +0.1]
AIME24	75.0	$-$ 2.5 [ $-$ 6.7, +1.2]	–	–	89.6	+2.5 [ $-$ 2.1, +8.3]
AIME25	69.6	$-$ 4.6 [ $-$ 11.2, +1.2]	–	–	82.1	+1.3 [ $-$ 1.7, +4.2]

Appendix

Table 28: FlagRot (INT4, S4W128) in vLLM: BF16 score (%) and paired

\Delta

from BF16 in points. Bold / gray: CI above / below zero. † BF16 below 20%. ‡ Database-state reward. § 100-instance subset. ¶ Mean of 16 English tasks on H200, with its own BF16 run. –: not evaluated.

	H200			H20
KV cache concurrent agents	64	128	256	64	128	256
BF16 †	1381	1402	637	482	166	134
FP8 (FlashAttention-3) †	1468	1608	1744	589	654	706
vLLM INT4 †	391	484	577	256	288	320
FP8 (FlashInfer)	1660	2051	2197	768	819	994
FlagRot , no BF16 tokens	1808	2051	2398	737	781	852

Appendix

Table 29: Serving

\tau^{2}

-bench retail agents with Qwen3-8B and INT4 caches: generated tokens/s. ⋆ S4W128. † Earlier run block of the same protocol.

INT4
Method	bits	KL
RTN	4.50	16.72
TurboQuant (vLLM)	4.9–7.1	14.43
SmoothAttention	4.50	14.53
Whole-head Hadamard	4.50	11.05
Calibrated base	4.50	7.02

Appendix

Table 30: Recent KV quantizers without BF16 tokens: geometric-mean KL

\times 10^{3}

over 12 models; bits : per cached element. Per model: Table 26 .

BF16 sinks $S$	0	0	4	16
BF16 window $W$	0	128	128	128
RTN	16.72	4.09	2.78	2.70
Fixed mode	10.15	2.56	1.88	1.87
FlagRot	7.88	2.08	1.48	1.46
FlagRot -C	6.30	2.23	1.45	1.42
FlagRot -L	4.81	1.46	1.16	1.14

Appendix

Table 31: INT4 caches with

S

sink and

W

recent tokens in BF16: geometric-mean KL

\times 10^{3}

over 12 models (all but Qwen2.5-7B); per model in Table 27 .

Cache	Qwen3-8B	Llama-3.1-8B-It	Qwen3.8-27B
BF16	49.15	49.62	54.27
FP8	$-$ 0.1	$-$ 0.7	+0.1
	[ $-$ 0.6, +0.3]	[ $-$ 1.3, $-$ 0.2]	[ $-$ 0.1, +0.3]
vLLM INT4	$-$ 1.0	$-$ 1.1	+0.3
	[ $-$ 1.6, $-$ 0.5]	[ $-$ 1.7, $-$ 0.4]	[ $-$ 0.1, +0.6]
RTN ⋆	+0.2	$-$ 0.6	0.0

Appendix

Table 32: LongBench v1 (16 English tasks), INT4 caches on H200: BF16 score and paired differences in points (95% CI, items resampled within tasks). Bold / gray: CI above / below zero. ⋆ S4W128; vs RTN ⋆ : FlagRot ⋆ minus RTN ⋆ .

Baseline	Source	Check
vLLM INT4	vLLM kernels	bit-exact
TurboQuant	vLLM kernels	bit-exact
Four Over Six	vLLM CUDA code	transcribed
KIVI	official code	bit-exact
SmoothAttention	QServe code	same calibration
QuaRot, SpinQuant	official code	PPL within 1.1%

Appendix

Table 33: How each external baseline was obtained and checked.

4-bit KV cache	WUSH-KV paper	Our evaluation
No transform	11.36	11.34
Hadamard	9.77	9.77
WUSH	9.73	9.74

Appendix

Table 34: WikiText-2 perplexity of the WUSH-KV evaluation on Qwen3-8B (QuEST, one scale per token and head, 16 sink and 128 recent tokens, FineWeb-Edu calibration); BF16: 9.72 in the paper, 9.73 here.

With QK-norm
Quantized	K+V	K	V	K	K	K
Group scale	E4M3	E4M3	E4M3	FP32	E4M3	FP32
Grid	E2M1	E2M1	E2M1	E2M1	unif.	unif.
Qwen3-8B	+210	+228	$-$ 3	+264	$-$ 32	$-$ 27
Qwen3-14B	+31	+34	+4	+46	$-$ 48	$-$ 44
Gemma-3-27B	+82	+93	+1	+97	$-$ 1	$-$ 3

Appendix

Table 35: Source of the NVFP4 sign flip: relative KL change (%) of whole-head Hadamard rotation without BF16 tokens; the unquantized cache stays in BF16 (groups of 16; unif. : uniform 15-level grid). Bold / gray: significantly below / above zero (95% CI).

	QK-	ch.			Head rot. (%)
Model	norm	ratio	crest	kurt.	MXFP4	NVFP4
Llama-3.1-8B	no	7.1	4.91	8.5	$-$ 71	$-$ 30
Qwen3-8B	yes	8.2	4.98	9.9	+41	+215
Mistral-7B-v0.3	no	7.1	4.72	7.8	$-$ 62	$-$ 35
Qwen3-14B	yes	7.6	4.81	9.0	$-$ 56	+32
Qwen3-32B	yes	6.5	4.62	8.0	$-$ 52	+26

Appendix

Table 36: Post-RoPE key statistics on two validation windows (median over layers and KV heads) and the relative KL change of whole-head rotation. ch. ratio : largest/median channel RMS; crest : max/RMS of a key vector; kurt. : its kurtosis over channels.

Figure 5: (a) Agreement of the per-token decision whether a sub-vector takes a block transform with its majority decision (mean over layers, heads and sub-vectors; FlagRot -L, INT4, test windows; 0.5: even split). (b) Per-token transforms on values, keys or both (relative KL change w.r.t. no rotation; FlagRot -L, validation windows of the development models). (c) Share of tokens per head-level selection for keys.

Static
	INT4		NVFP4
Selection	W0	S4W128	W0	S4W128
RTN (KL)	19.8	2.40	22.8	3.29
K only	$-$ 41	$-$ 37	$-$ 23	$-$ 37
K and V (fixed mode)	$-$ 42	$-$ 39	$-$ 23	$-$ 37
Per token, K only

Appendix

Table 37: Transform selection rules on the test windows of the four development models: RTN geometric-mean KL

\times 10^{3}

and mean KL change (%) from RTN, with the same candidates in every row. Bold: significantly below RTN on each of the four models (95% CI).

Uniform
Decision	bits	INT4	MXFP4	NVFP4
Every block	0.000	$-$ 37.0	$-$ 51.5	+4.9
Whole head	0.000	$-$ 35.1	$-$ 39.1	+42.4
Static (one selection per vector or head)
Block bits	0.000	$-$ 37.4	$-$ 54.5	$-$ 8.6
Block bits $\|$ whole head	0.000	$-$ 42.2	$-$ 59.2	$-$ 23.0

Appendix

Table 38: Design space of the rotation decision (four development models, 40 WikiText-2 validation windows): mean KL change (%) from no rotation. bits : extra bits per element to store the selection (

r_{b}{=}32

D{=}128

KV-Cache Quantization

Key-value cache quantization compresses transformer attention state to reduce memory use and bandwidth during long-context inference.

See all 115 papers

Output-Aware Rotation for INT2 KV-Cache Quantization
Aug 3, 2026Vincent-Daniel Yun, Woosang Lim, Minsoo Cheong +4LLM Inference KV Caching
University of Southern California · Seoul National University · Inha University
Block-GTQ: RoPE-Block Rate Allocation for KV-Cache Quantization
Jun 23, 2026Fengfeng Liang, Yuechen Zhang, Jiaya JiaKV Caching LLM Inference Acceleration
Hong Kong University of Science and Technology · The Chinese University of Hong Kong · MiMo, Xiaomi Corporation
RateQuant: Optimal Mixed-Precision KV Cache Quantization via Rate-Distortion Theory
Apr 22, 2026Fei Zuo, Zikang Zhou, Hao Cong +2LLM Inference Efficiency Mixed-Precision Quantization
1BA TechWorks (BMW Group) · 2National University of Singapore · 3Tsinghua University

Aug 3, 2026J/K move · Enter open · S save

Output-Aware Rotation for INT2 KV-Cache Quantization

Vincent-Daniel Yun, Woosang Lim, Minsoo Cheong, Sunwoo Lee+3

University of Southern California · Seoul National University · Inha University

LLM Inference KV Caching Rotation-Based Quantization Attention Quantization KV-Cache Quantization Low-Bit Quantization+1

Open paper AskPDF arXiv

The key-value (KV) cache has become a major memory and bandwidth bottleneck in long-context large language model inference, making ultra-low-bit quantization increasingly important. However, existing rotation-based INT2 methods optimize cache statistics or proxy errors before the complete attention readout, even though the model is ultimately affected by the error propagated through attention and the output projection

W_O

. To address this mismatch, we propose OptR, an output-aware rotation method that minimizes post-

W_O

attention-output error. OptR decomposes the post-

W_O

attention-output error into key- and value-induced terms and learns per-head orthogonal corrections through the full INT2 quantization and attention path. OptR further applies an attention-equivalent key reparameterization to reduce large channel-wise offsets without changing the softmax distribution. Across three models and five reasoning and coding benchmarks, OptR consistently improves both QuaRot and OSCAR and strengthens long-context retrieval, while preserving the paged KV-cache format with negligible inference overhead.

Abstract

KV-Cache Quantization

Explore similar work

Output-Aware Rotation for INT2 KV-Cache Quantization

Block-GTQ: RoPE-Block Rate Allocation for KV-Cache Quantization

RateQuant: Optimal Mixed-Precision KV Cache Quantization via Rate-Distortion Theory