LoopVQ: Sub 1-bit KV Cache Quantization for Looped Transformers

Minsoo Cheong; Sungjoo Yoo

cs.LGNov 3, 2026

LoopVQ: Sub 1-bit KV Cache Quantization for Looped Transformers

Authors: Minsoo Cheong, Sungjoo Yoo

Organizations: Seoul National University

Abstract

Looped transformers are a parameter-efficient alternative to standard transformers for language modeling. By reapplying the same layers across multiple loop steps, recent looped language models match the performance of non-looped models several times their size. While this architecture effectively reduces parameter memory, each loop step still stores its own keys and values, so the key-value (KV) cache grows linearly with the number of loop steps and becomes the main memory bottleneck. We observe that the KV cache of looped models is highly redundant along the loop axis and that the loop steps differ substantially in their sensitivity to quantization. Based on these observations, we propose $\textbf{LoopVQ}$ , a KV cache vector quantization method for looped transformers that codes channels of multiple loop steps jointly, and fine-tunes the codebooks to match the output of the full-precision model. At 0.5 bits per element, LoopVQ outperforms existing KV cache vector quantization methods for standard LLMs on general, long chain-of-thought reasoning, and long-context retrieval tasks with Ouro and Huginn models. Its design is orthogonal to these methods, and combined with TaSQ, it retains much of the BF16 accuracy even at 0.25 bits per element. We also implement LoopVQ in SGLang, where it nearly halves the decode latency at a 32K-token context and raises the peak serving throughput by up to $4.3\times$ and $6.1\times$ on the 1.4B and 2.6B Ouro models compared to the BF16 baseline.

Figures & tables

Figure 2: The KV cache of a looped LM is redundant along the loop axis. (a) Pre-RoPE keys of one head of Ouro-1.4B at each loop step. (b) Mean absolute correlation between (loop step, channel) pairs within a 16-channel group, averaged over all groups, heads, and layers of Ouro-1.4B. (c) $k$ -means reconstruction error at equal bitwidth when each 8-bit code spans $G$ channels of one loop step (channel) or $G/4$ channels across four loop steps (loop), both with plain $k$ -means.

	Quantized loop step
Bitwidth	1	2	3	4
0.5 bit	0.069	0.018	0.013	0.030
0.25 bit	0.123	0.029	0.026	0.065

Table 1: KL divergence from the full-precision output of Ouro-1.4B when only one loop step is quantized (loop-axis grouping, Fisher-weighted

k

-means codebooks, sequences not used for calibration).

Figure 3: Overview of LoopVQ. Each loop step of a looped LM writes its own KV cache. LoopVQ codes

G/T_{c}

channels across

T_{c}

loop steps, so one index serves all of them at the same bitwidth as coding each loop step separately, and fine-tunes the codebooks against the full-precision output through the loop. At inference, each loop step reads the shared indices through its own slice

\mathcal{B}_{t}

of the codebook.

Figure 4: KL divergence from the full-precision Ouro-1.4B during fine-tuning, starting from the

k

-means codebooks (circles). One iteration processes one calibration sequence (48 iterations = 4 epochs).

Figure 5: Serving LoopVQ in SGLang. The KV cache keeps the sink and recent tokens in bf16 at every loop step, and all other tokens as codebook indices shared across loop steps. At loop step

t

, the LoopVQ attention kernel loads

\mathcal{B}_{t}

, the slice of the codebook entries for loop step

t

, into shared memory once and looks up the keys and values of each token from its indices in registers; the bf16 kernel reads the bf16 keys and values of loop step

t

directly. Both attend with the query

q^{(t)}

of the new token, and their outputs are merged. The key and value of each new token are written in bf16, and after every 8 generated tokens the oldest recent page is encoded into indices.

Model	Bitwidth	Method	Bits (K/V)	GSM8K	MATH500	HumanEval	MBPP	Avg.
Ouro-1.4B		BF16	16/16	78.77	77.20	73.17	73.54	75.67
	0.5 bit	CQ	0.5/0.5	46.78	17.80	14.63	50.00	32.30
		TaSQ	0.508/0.5	57.32	38.80	24.39	53.97	43.62
		LoopVQ	0.5/0.5	75.82	69.40	63.41	64.55	68.29
		TaSQ+ LoopVQ	0.508/0.5	77.56	75.20	70.12	71.43	73.58
	0.25 bit	CQ	0.25/0.25	42.99	12.20	10.98	46.56	28.18

Table 2: General performance under low-bit KV cache quantization. All results use greedy decoding. Bold denotes the best quantized result; higher is better.

Model	Bitwidth	Method	Bits (K/V)	AIME’24	AIME’25	GPQA-D	HumanEval+	MBPP+	Avg.
Ouro-1.4B-Thinking		BF16	16/16	47.78 $\pm$ 5.09	32.22 $\pm$ 5.09	45.12 $\pm$ 2.28	84.55 $\pm$ 0.70	72.49 $\pm$ 1.32	56.43 $\pm$ 1.54
	0.5 bit	CQ	0.5/0.5	1.11 $\pm$ 1.92	2.22 $\pm$ 1.92	11.78 $\pm$ 1.05	27.85 $\pm$ 0.70	35.27 $\pm$ 2.31	15.65 $\pm$ 0.76
		TaSQ	0.508/0.5	4.44 $\pm$ 1.92	5.56 $\pm$ 1.92	12.12 $\pm$ 2.31	35.16 $\pm$ 2.14	40.12 $\pm$ 2.72	19.48 $\pm$ 0.99
		LoopVQ	0.5/0.5	38.89 $\pm$ 5.09	31.11 $\pm$ 3.85	40.57 $\pm$ 1.27	75.81 $\pm$ 1.27	67.64 $\pm$ 1.19	50.80 $\pm$ 1.35
		TaSQ+ LoopVQ	0.508/0.5	45.56 $\pm$ 8.39	34.44 $\pm$ 5.09	42.76 $\pm$ 3.04	81.50 $\pm$ 1.53	70.11 $\pm$ 0.70	54.87 $\pm$ 2.08
	0.25 bit	CQ	0.25/0.25	0.00 $\pm$ 0.00	1.11 $\pm$ 1.92	11.62 $\pm$ 3.07	25.61 $\pm$ 2.79	33.86 $\pm$ 0.00	14.44 $\pm$ 0.92

Table 3: Long CoT reasoning performance under low-bit KV cache quantization. All results are averaged over three sampling seeds (mean

\pm

std). Bold denotes the best quantized result; higher is better.

Figure 6: RULER needle-in-a-haystack (NIAH) accuracy of Ouro-1.4B and Ouro-2.6B versus context length under 0.5- and 0.25-bit KV cache quantization (8-bit codes,

G{=}16

and

32

). Each point averages the eight NIAH tasks over three seeds.

Figure 7: End-to-end throughput of LoopVQ on Ouro-1.4B-Thinking with one RTX 6000 Ada GPU, versus batch size up to the largest batch that fits in memory.

Figure 8: Latency of LoopVQ at 0.5 and 0.25 bit on Ouro-1.4B-Thinking with one RTX 6000 Ada GPU. (a) Prefill latency and (b) decode latency per token at batch size 1, each with the change relative to BF16.

			GSM8K		HumanEval
Base	+ Coupling	+ FT	0.5	0.25	0.5	0.25
CQ			46.8	43.0	14.6	11.0
	✓		75.4	57.5	61.6	23.8
		✓	48.7	42.5	14.6	11.0
	✓	✓	75.8	58.9	63.4	25.6
TaSQ			57.3	44.9	24.4	12.2

Table 4: Loop-axis coupling and loop-aware fine-tuning (FT) applied to CQ and TaSQ on Ouro-1.4B at 0.5 and 0.25 bits per element. With both, CQ and TaSQ become LoopVQ and TaSQ+LoopVQ. BF16: 78.8 (GSM8K) and 73.2 (HumanEval).

Figure 9: Accuracy of Ouro-1.4B at 0.25 bit with every method calibrated on a different corpus. GPQA is the calibration set of the main experiments.

	GSM8K			HumanEval
Sequences	8	16	32	8	16	32
LoopVQ	57.5	58.9	60.8	22.6	25.6	28.0
TaSQ+LoopVQ	69.3	71.0	71.9	49.4	55.5	59.8

Table 5: Accuracy of Ouro-1.4B at 0.25 bit versus the number of 2,048-token GPQA calibration sequences (16 in the main experiments).

Figure 10: Generation length on GPQA-Diamond of Ouro-1.4B-Thinking under 0.5-bit KV cache quantization, with the share of problems that reach the 32K-token budget.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 11: Latency of LoopVQ at 0.5 and 0.25 bit on Ouro-1.4B-Thinking with one A100 (40 GB) GPU. (a) Prefill latency and (b) decode latency per token at batch size 1, each with the change relative to BF16.

Figure 12: End-to-end throughput of LoopVQ on Ouro-1.4B-Thinking with one A100 (40 GB) GPU, versus batch size up to the largest batch that fits in memory.

Figure 13: Latency of LoopVQ at 0.5 and 0.25 bit on Ouro-2.6B-Thinking with one RTX 6000 Ada GPU. (a) Prefill latency and (b) decode latency per token at batch size 1, each with the change relative to BF16.

Figure 14: End-to-end throughput of LoopVQ on Ouro-2.6B-Thinking with one RTX 6000 Ada GPU, versus batch size up to the largest batch that fits in memory.

Figure 15: GPU time of LoopVQ and BF16 on Ouro-1.4B-Thinking with one RTX 6000 Ada GPU, by kernel type. (a) Decode time per token at batch size 1. (b) Decode time per generated token at batch size 16 with 2K-token prompts. (c) Prefill time. Attention includes the bf16 window, the LoopVQ kernel, and the merge of their splits.

Figure 16: Generation length on GPQA-Diamond of Ouro-1.4B-Thinking and Ouro-2.6B-Thinking under 0.5- and 0.25-bit KV cache quantization, with the share of problems that reach the 32K-token budget.

Model	Bitwidth	Method	Single-doc QA			Multi-doc QA			Summarization			Few-shot			Synthetic		Code		Avg.
Model	Bitwidth	Method	NrtvQA	Qasper	MF-en	HotpotQA	2WikiMQA	Musique	GovReport	QMSum	MultiNews	TREC	TriviaQA	SAMSum	PCount	PRe	Lcc	RB-P	Avg.
Ouro-1.4B		BF16	7.81	23.56	34.75	33.17	19.75	13.15	26.94	20.52	23.08	64.50	88.57	36.18	1.50	32.50	19.99	28.72	29.67
	0.5 bit	CQ	4.93	10.38	22.10	20.18	16.35	8.08	16.88	17.44	15.72	44.00	79.12	33.07	1.50	4.50	19.68	27.95	21.37
		TaSQ	7.02	13.39	25.05	24.90	17.46	9.22	18.38	17.88	16.58	47.00	81.79	34.07	1.00	4.50	21.85	27.87	23.00
		LoopVQ	7.31	19.29	30.71	32.63	19.23	12.82	23.73	18.94	21.06	58.50	87.80	35.61	2.00	16.00	22.09	29.60	27.33
		TaSQ+ LoopVQ	8.11	22.84	33.34	34.56	18.89	12.60	26.07	19.68	22.15	64.00	89.03	36.09	1.50	23.00	22.16	30.55	29.04

Appendix

Table 6: LongBench performance of Ouro-1.4B and Ouro-2.6B under low-bit KV cache quantization on the 16 English tasks, with the official metric of each task. All results use greedy decoding. Bold denotes the best quantized result; higher is better.

KV-Cache Quantization

Key-value cache quantization compresses transformer attention state to reduce memory use and bandwidth during long-context inference.

See all 109 papers

LoopKV: Loop-Aware KV Cache Compression for Serving Looped Transformers
Nov 3, 2026Tianle Zhou, Yang Zhang, Tianyu Wu +3KV-Cache Compression LLM Inference Acceleration
Harvard University · University of California San Diego · University of Chicago
ResidualQuant: KV Cache Quantization for Looped Transformers with 2-Bit Residuals
Oct 7, 2026Heejun Kim, Junyoung Lee, SangLyul Cho +3KV-Cache Quantization Efficient Transformer Inference
KAIST · Yonsei University · Seoul National University
FlashLoop: Fast and Memory-Efficient Looped Transformers via Lazy Updates
Sep 24, 2026Wanqi Yang, Shiwei LiuEfficient Transformer Inference Recurrent Transformers
ELLIS Institute Tübingen Max Planck Institute for Intelligent Systems Tübingen AI Center

Nov 3, 2026J/K move · Enter open · S save

LoopKV: Loop-Aware KV Cache Compression for Serving Looped Transformers

Tianle Zhou, Yang Zhang, Tianyu Wu, Zhuohan Wang+2

Harvard University · University of California San Diego · University of Chicago

KV-Cache Compression LLM Inference Acceleration KV-Cache Quantization

Open paper AskPDF arXiv

Looped transformers apply a shared block several times per token to improve accuracy without adding parameters. For models such as Ouro that keep a separate key-value (KV) cache per loop, cache memory limits the batch size that amortizes repeated weight reads. Uniform-bit quantization and fixed rules per loop do not allocate precision according to each layer's sensitivity in each loop. Measurements on Ouro show that quantization sensitivity varies across (loop, layer) cells, and the sum of individual cell damages approximates the overall output damage. Sharing damage also varies by cell, with later loops generally tolerating sharing better. Skipping low-attention values to save reads produces errors that compound across loops. LoopKV uses a one-time offline profile to choose 4-bit storage, 2-bit storage or reuse of the final loop's KV for each cell. The planner minimizes reads under budgets for predicted damage and storage. During decoding, each loop reads the high 2 bits of each 4-bit value and fetches the low bits only where its own attention weights require 4-bit precision. A single fused kernel in vLLM executes the plan. Serving Ouro-1.4B on one RTX PRO 6000, the memory-optimized LoopKV-M plan stores 35% less KV than uniform 4-bit quantization (KIVI-4) on the same fused kernel and CUDA graph. It sustains 1.7-2.2x KIVI-4's goodput under a latency service-level objective with a 16K-token maximum length. None of its averages on six public benchmarks falls significantly below BF16's.

Abstract

KV-Cache Quantization

Explore similar work

LoopKV: Loop-Aware KV Cache Compression for Serving Looped Transformers

ResidualQuant: KV Cache Quantization for Looped Transformers with 2-Bit Residuals

FlashLoop: Fast and Memory-Efficient Looped Transformers via Lazy Updates