cs.LGNov 3, 2026

LoopKV: Loop-Aware KV Cache Compression for Serving Looped Transformers

Authors: Tianle Zhou, Yang Zhang, Tianyu Wu, Zhuohan Wang, William Nixon, Juncheng Yang

Organizations: Harvard University · University of California San Diego · University of Chicago

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Looped transformers apply a shared block several times per token to improve accuracy without adding parameters. For models such as Ouro that keep a separate key-value (KV) cache per loop, cache memory limits the batch size that amortizes repeated weight reads. Uniform-bit quantization and fixed rules per loop do not allocate precision according to each layer's sensitivity in each loop. Measurements on Ouro show that quantization sensitivity varies across (loop, layer) cells, and the sum of individual cell damages approximates the overall output damage. Sharing damage also varies by cell, with later loops generally tolerating sharing better. Skipping low-attention values to save reads produces errors that compound across loops. LoopKV uses a one-time offline profile to choose 4-bit storage, 2-bit storage or reuse of the final loop's KV for each cell. The planner minimizes reads under budgets for predicted damage and storage. During decoding, each loop reads the high 2 bits of each 4-bit value and fetches the low bits only where its own attention weights require 4-bit precision. A single fused kernel in vLLM executes the plan. Serving Ouro-1.4B on one RTX PRO 6000, the memory-optimized LoopKV-M plan stores 35% less KV than uniform 4-bit quantization (KIVI-4) on the same fused kernel and CUDA graph. It sustains 1.7-2.2x KIVI-4's goodput under a latency service-level objective with a 16K-token maximum length. None of its averages on six public benchmarks falls significantly below BF16's.

Figures & tables

Figure 1: Overview of LoopKV. The offline phase of LoopKV, run once per model, chooses a storage option for every (loop, layer) pair, which we call a cell, from the KL damage measured by profiling each cell independently (C1). A cell may reuse the final loop’s KV through a pointer when its KL damage is low, which we show costs little accuracy (C2). In the online phase, V is stored as a high and a low 2-bit plane, so that the attention scores decide at decode time which tokens also read the low plane (C3). We then combine all three into one fused kernel that runs every cell type in a single launch. With vLLM’s whole-step CUDA graph, this cuts LoopKV-M’s host launches per decode step from 504 to 17.
Figure 2: P1: damage depends on the cell. Each point shows the damage when one cell (one layer in one loop) is stored at INT2 while every other cell stays exact. The damage on the y-axis is the share of next-token predictions that change (fp32, 16 documents). The top 19 of the 72 cells hold half of the total difference, and KL ranks the cells alike (Spearman 0.95).
Figure 3: P2: some cells can share with little loss. Each point shows the damage (the y-axis, as in Figure 2 ) when one cell reads the final loop’s KV instead of storing its own, while every other cell stays exact. Some cells (green) do less damage when sharing than when stored at INT2
Figure 4: P3: the loops compound read errors. Quest drops the low-ranked far pages that are not in the sink and window, while LoopKV-style reading keeps them and reads the values of low-attention far tokens at 2 bits. Both read 85% of the far bytes. (a) Relative damage † of Quest as kk grows, applied to the first kk loops of Ouro-1.4B, to kk of four layer slices of Ouro-1.4B that each span all loops, or to kk of four layer groups of Qwen2.5 and Qwen3. (b) Damage of LoopKV-style reading and of Quest in all four loops of Ouro-1.4B at the same 85% of bytes read. Damage is the fp32 KL to the exact model. † Damage when the kk groups are treated together, divided by the sum of their damages when page skipping is applied to each group alone.
Figure 5: LoopKV serves more requests with no significant accuracy loss on six public suites (tested on Ouro-1.4B). (a) Goodput relative to BF16 on one RTX PRO 6000 for the memory-optimized plan (LoopKV-M), which minimizes reads under a tight storage budget, the read-optimized plan (LoopKV-R), which minimizes reads with no storage budget, and comparable methods. Goodput is the highest request rate at which 95% of requests meet the SLO 2 2 2 A request meets the SLO if its first token arrives within 10 s and its mean time per output token stays below 3τ0=55.83\tau_{0}{=}55.8 ms, for AIME prompts whose contexts grow to 16K tokens (§ 5.1 ). . In the figure, LoopKV-M sustains 8.0 ×\times BF16’s goodput and 2.0 ×\times that of KIVI-4 (§ 5.4 ). (b) Over all the public suites we tested (Table 1 ), neither LoopKV-M nor LoopKV-R is significantly less accurate than BF16, whereas naive INT2 on loops 1–3 in the KIVI style and naive last-loop KV reuse lose accuracy significantly on several suites. Other comparable methods: KIVI-4 + V adds our V precision to KIVI-style INT4 on every loop, on the same fused kernel and graph. BF16, FP8 and FP8 + loop-3 sharing use vLLM’s own kernels, and KIVI-4 is uniform INT4 without V precision.
Figure 6: The read-optimized plan beats the best per-loop allocation plan of LoopKV, i.e., INT4 in every loop with loop 3 sharing loop 4’s KV, which has the lowest damage of all per-loop plans that store less than uniform INT4, on KL and throughput (42% lower KL and 5% higher throughput). The memory-optimized plan stores 14% less than this plan and decodes 6% faster. At equal bytes, the read- and memory-optimized plans have 0.36 ×\times and 0.43 ×\times the measured KL of the best per-loop plans, i.e., INT4 in loops 1–2 with INT2 in loop 3, and INT4 in loop 1, INT2 in loop 2 and sharing in loop 3.
 R16KR32KLBMATH500GSM8KL5
Ouro-1.4B
BF1671.4454.0425.2984.576.670.6
LoopKV-M72.1253.6325.1986.077.171.1
LoopKV-R72.4953.8625.3284.576.670.3
KIVI-4 + V71.3754.5725.2584.576.972.0
INT2, loops 1–365.50—24.4386.077.069.3
Table 1: On every model and suite, no LoopKV average accuracy falls significantly below BF16 (best in bold). The other methods in the table include KIVI-4 + V, i.e., uniform 4-bit quantization plus LoopKV’s V precision, naive uniform INT2 quantization of loops 1–3, and naive last-loop KV reuse. R16K and R32K denote RULER-16K and RULER-32K Hsieh et al. (2024) , LB denotes LongBench v1 Bai et al. (2024) , and L5 denotes AIME + MATH level 5 Hendrycks et al. (2021) .
Figure 7: LoopKV keeps more requests resident: in this sweep it matches or beats the other configurations at each time per token and keeps scaling after they run out of KV memory. Ouro-1.4B on one RTX PRO 6000: decode tok/s against time per output token as the batch grows, up to each configuration’s largest resident batch (numbers); INT4 configurations on the fused kernel and whole-step graph, BF16 and FP8 as vLLM serves them; time per token on a log scale; the dotted line marks the SLO’s time-per-token bound ( 3τ03\tau_{0} , § 5.1 ). KIVI-4 + our V is an ablation.
Figure 8: LoopKV’s storage saving becomes throughput where memory binds. Peak decode tok/s at the resident peak for Ouro-1.4B (a, b) and Ouro-2.6B (c, d) on an RTX PRO 6000, each GPU’s arms in one session, fused kernel and graph for INT4 arms; Ouro-1.4B within 1.1% of the separate runs quoted in the text (repeat means: § 5.3 ); 32K and a 32 GB GPU: Tables 8 , 7 and 21 ; Dory-4B: Table 20 .
Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

 planfp32 KLMQ641 (stress)RULER-style 16KMATH500 (200)GSM8K (1,319)long reasoning (194)
LoopKVread-optimized0.87+0.8+0.8−1.6-1.6———
+ V precision0.87+0.0+0.0−1.1-1.1+3.0+3.0−0.5-0.5+1.14+1.14
memory-optimized1.80−0.8-0.8−0.2-0.2+2.0+2.0+0.8+0.8—
+ V precision1.80+0.3+0.3—+1.5+1.5—−0.43-0.43
baselinesKIVI-40.46+0.9+0.9−1.7-1.7———
KIVI-221.5−13-13 to −18†-18^{\dagger}————
Appendix
Table 2: Quality before kernel fusion, including the in-house stress tests MQ641 and RULER-style: neither knapsack plan loses significantly against BF16; read-optimized + V meets the 3-point margin on MQ641, MATH500, GSM8K and long reasoning but not on RULER-style. fp32 KL / KLshare\mathrm{KL}_{\mathrm{share}} ( KLshare\mathrm{KL}_{\mathrm{share}} =1.20×10−4=1.20{\times}10^{-4} ; 16 documents); tasks: Δ\Delta (pp) vs served BF16, + / - where the paired 95% CI lies above / below zero (single runs; greedy except long reasoning). “—” = not run. † Earlier RTX PRO 4500 harness, vs its own BF16. ‡ Codec only, no sparse path. FP8 / FP8 + loop-3 sharing: fp32 KL 0.19 ×\times / 1.20 ×\times , no runs here.
armofficial RULER-16Kofficial RULER-32KLongBench v1 (16)MATH500 (200)GSM8K (1,319)MQ641 (stress)
KIVI-4 + V−0.07-0.07 ; vt −3.0-3.0+0.53+0.53−0.04-0.04 ; qasper −0.87-0.87 , gov_report −0.64-0.64+0.0+0.0+0.3+0.3+0.5+0.5
LoopKV-R+1.05++1.05^{+} ; qa_1 +7.0+7.0−0.18-0.18+0.03+0.03 ; multifieldqa_en +0.93+0.93+0.0+0.0+0.0+0.0+0.6+0.6
LoopKV-M+0.68+0.68 ; fwe +5.0+5.0 , qa_1 +8.0+8.0−0.41-0.41 ; vt −21.2-21.2 , fwe +18.3+18.3 , cwe +0.6∗+0.6^{*}−0.10-0.10+1.5+1.5+0.5+0.5+0.5+0.5
FP8 + loop-3 sharing (as served)−0.22-0.22 ; vt −6.8-6.8 , cwe +1.5+1.5+0.29+0.29 (A100-80); vt −14.2-14.2 , fwe +19.7+19.7+0.07+0.07 ; narrativeqa −0.44-0.44 , musique +1.64+1.64 ∗+2.0+2.0−1.1-1.1—
Appendix
Table 3: Quality on the fused path (fused kernel, whole-step graph, V-precision seed m0m_{0} as served): Δ\Delta (pp) vs served BF16, + / - where the paired 95% CI lies above / below zero; tasks listed if significant (unadjusted); ∗ BF16-floor task. FP8 + loop-3 sharing runs as vLLM serves it (no LoopKV kernel); its RULER-32K run is on an A100-80 against BF16 on the same GPU. RULER tasks: vt variable tracking, fwe frequent-words extraction, cwe common-words extraction, qa_1 single-hop QA; LongBench subsets by name.
suite (items)KIVI-4 + VLoopKV-RLoopKV-MKIVI-4 †KIVI-4 + loop-3 sharing †FP8 + loop-3 sharing
RULER-16K (1,300)30 / 3726 / 4454 / 7129 / 3145 / 8766 / 80
RULER-32K (1,300)37 / 4744 / 45 †84 / 8940 / 47——
LongBench v1 (3,750)499 / 480513 / 540556 / 596483 / 495579 / 570681 / 719
MATH500 (200)8 / 89 / 97 / 10—9 / 145 / 9
GSM8K (1,319)21 / 2522 / 2227 / 34—27 / 3059 / 45
long reasoning (194)—34 / 39 †33 / 38 †———
Appendix
Table 4: Answer flips against BF16: items whose score went down / up (graded suites: items whose official score changed, down / up). † Before fusion. Long reasoning: items whose sample accuracy went down / up.
armKiB / tokenMATH500 (200)GSM8K (1,319)
BF161,53689.579.7
KIVI-4 + V480 (31%)90.5, +1.0+1.079.5, −0.2-0.2
LoopKV-M310 (20%)89.5, +0.0+0.079.5, −0.2-0.2
LoopKV-R430 (28%)90.5, +1.0+1.0—
BF16 §1,536—79.8
LoopKV-R §430 (28%)—79.2, −0.5-0.5
Appendix
Table 5: Ouro-2.6B served (fused path): no quantized arm differs significantly from BF16; LoopKV-M misses the 3-point margin on MATH500 by half a point. MATH500 on one RTX PRO 6000, every arm at our 8,192-token output cap; GSM8K on A40 GPUs, same-GPU pairs. Accuracy (%) and Δ\Delta vs BF16 on the same GPU (pp, from unrounded scores); history KV stored per token. § Second set of A40s.
configurationbytesKL8K16Ksame block ×\timeshost launches
KIVI-4240/2400.46494.4249.1——
read-optimized plan (C1)215/2150.87528.9271.5—433
memory-optimized plan (C1 + C2)155/2201.80535.1280.7——
+ V precision (C3)155/1891.80561.6295.51.044 / 0.98504
+ fused kernel155/1891.80571.2300.01.019 / 1.033332
+ whole-step CUDA graph (LoopKV-M)155/1891.80575.9301.91.006 / 1.06717
Appendix
Table 6: Ablation of LoopKV’s components on one RTX PRO 6000: each step improves throughput, KV footprint, or batch-1 latency at a stated KL. Columns report stored / read history KV (KiB/token), fp32 KL / KLshare\mathrm{KL}_{\mathrm{share}} , peak decode throughput at 8K and 16K (tok/s), relative speedup over the preceding row measured back-to-back in the same job (“same block ×\times ”, at 8K peak / B=1B{=}1 ), and host launches per step. The read- and memory-optimized plans are separate knapsack solutions, and the indented rows incrementally add C3, kernel fusion, and whole-step CUDA graphs to the memory-optimized plan.
Figure 9: All served configurations at 8K context, including LoopKV’s own variants (Figure 10 shows one point per comparable method): knapsack plans without V precision, KIVI-4 + V, hand-made placements with sharing, and INT4 designs with residuals. KIVI-4 + V adds our V precision (C3) to KIVI-4 and is an ablation; the sharing points combine loop-3 sharing with BF16, FP8, KIVI-4 or hand-made INT4 / INT2 placements. The frontier with V precision runs through uniform INT4, KIVI-4 + V and LoopKV-R and LoopKV-M; every hand-made and residual design with at most LoopKV-M’s KL lies behind it. Points without +V use our unfused reader, except BF16 and FP8 (vLLM) and uniform INT2 (a separate run).
Figure 10: At 8K context, LoopKV-R and LoopKV-M (fused kernel and graph) lie on the measured frontier of served configurations between KIVI-4 (0.46 ×\times KLshare\mathrm{KL}_{\mathrm{share}} ) and KIVI-2 (21.5 ×\times KLshare\mathrm{KL}_{\mathrm{share}} ), the only faster method. Comparable methods, one point each: BF16 and FP8 as vLLM serves them, KIVI-4 and KIVI-2, and INT2 residuals for loops 1–2 against an INT4 loop 4, with loop 3 shared, in the style of ResidualQuant Kim et al. (2026) (4.38 ×\times KLshare\mathrm{KL}_{\mathrm{share}} ; our reimplementation of ResidualQuant itself has 2.01 ×\times , Table 2 ); the loop-sharing points combine loop-3 sharing with BF16 ( KLshare\mathrm{KL}_{\mathrm{share}} ), FP8 and KIVI-4. The KIVI, sharing and residual points use our unfused reader. Figure 9 adds KIVI-4 + V, also on the frontier, and our other variants. Single runs.
Figure 11: LoopKV-M sustains 8.0 ×\times BF16’s 16K SLO goodput on one RTX PRO 6000, and 2.0 ×\times that of KIVI-4 on the same kernel. Highest request rate with 95% of requests in SLO ( τ0=18.60\tau_{0}{=}18.60 ms, § 5.1 ), arrival seed 101, which every arm ran, relative to BF16’s 0.0119 req/s; on seeds 102 and 103, LoopKV-M sustains 1.72 ×\times and 2.20 ×\times KIVI-4’s rate and LoopKV-R 1.29 ×\times and 1.60 ×\times .
model, GPUcontextBF16uniform INT4LoopKV-RLoopKV-M
Ouro-1.4B, A100-808K188.4 (10)336.6 (27) 1.79 ×\times377.6 (30) 2.00 ×\times408.5 (42) 2.17 ×\times
 16K95.0 (5)161.3 (11) 1.70 ×\times189.9 (13) 2.00 ×\times209.2 (20) 2.20 ×\times
 32K22.7 (2)84.6 (6) 3.73 ×\times105.4 (8) 4.65 ×\times107.0 (11) 4.72 ×\times
Ouro-2.6B, RTX PRO 60008K81.3 (6)208.3 (16) 2.56 ×\times240.4 (18) 2.96 ×\times263.9 (23) 3.25 ×\times
 16K30.8 (3)109.5 (8) 3.56 ×\times124.1 (8) 4.03 ×\times136.8 (11) 4.44 ×\times
 32K6.2 (1)52.7 (4) 8.51 ×\times58.6 (4) 9.46 ×\times68.1 (5) 11.0 ×\times
Appendix
Table 7: Peak decode tok/s at the resident peak (batch in parentheses) and the ratio to BF16 on the same GPU, fused kernel and whole-step graph, one run per cell. Uniform INT4 is KIVI-4 without V precision; LoopKV-R and -M are the read- and memory-optimized plans with V precision; Ouro-2.6B uses its own allocator plans.
planbytes8K16K32K×\times BF16 / FP8+S, 8K
BF16768/768170.8 (8)89.0 (6)—1.00 / 0.48
FP8384/384328.8 (16)176.3 (13)—1.93 / 0.92
FP8 + sharing (FP8+S)288/384358.4 (33)180.7 (16)—2.10 / 1.00
uniform INT4240/240494.4 (34)249.1 (14)—2.89 / 1.38
uniform-INT4 + sharing180/240505.4 (40)264.2 (20)—2.96 / 1.41
+ V precision180/200521.1 (40)277.4 (20)—3.05 / 1.45
Appendix
Table 8: With the fused kernel and whole-step graph, LoopKV-R / -M decode 5.4 / 9.3% (8K), 7.8 / 13.2% (16K) and 7.4 / 15.6% (32K) more than KIVI-4 + V given the same kernel and graph (single runs; repeat means at 8K and 16K: § 5.3 ). Peak decode tok/s (batch); KiB per token, stored / read. Top two blocks: before fusion, uniform INT4 with interleaved V; bottom: fused + graph, two-plane V, one block at 8K and 16K (read-optimized + V at batch 36: 552.7) and a second block on the same GPU at 32K. BF16 peaks at batch 8 at 8K: at batch 12 stock FlashAttention-2 falls to one split per head (154.9 tok/s). At batch 1 (8K, bottom block) the plans match KIVI-4 + V: 79.1 and 78.9 vs 80.3 tok/s.
   capacity (req/s), τ0=18.60\tau_{0}{=}18.60 ms or as notedfirst failing rung, 3τ03\tau_{0} , 18.60 ms
armseedrungs (req/s)2τ02\tau_{0}3τ03\tau_{0}5τ05\tau_{0}3τ03\tau_{0} , 20.69attainmentT / P / Bpreempt.
BF161010.0119–0.0170<< 0.01190.0119≥\geq 0.01700.01430.6050 / 17 / 03
FP81010.0305–0.0476<< 0.03050.0381≥\geq 0.04760.03810.5350 / 20 / 06
FP8 + sharing1010.0305–0.0476<< 0.03050.0305≥\geq 0.04760.03810.9300 / 3 / 00
uniform INT41010.0476–0.0714<< 0.04760.04760.05950.05950.9301 / 2 / 092
 1020.0476–0.09520.04760.08330.08330.08330.8575 / 0 / 293
Appendix
Table 9: SLO ladders on one RTX PRO 6000 (Ouro-1.4B, 16K maximum length, fused path for the INT4 arms; 60 requests per cell, of which 43, 49 and 47 are scored for seeds 101–103). Rungs: lowest and highest rate tested; INT4 rungs step by 0.0119 req/s. Capacity: highest tested rate before the first rung below 95% attainment, with TTFT ≤\leq 10 s and TPOT ≤kτ0\leq k\tau_{0} ; << : the lowest rung fails; ≥\geq : no tested rung fails. First failing rung (the rung above the 3τ03\tau_{0} capacity at τ0=18.60\tau_{0}{=}18.60 ms): attainment, misses by component (TTFT only / TPOT only / both) and preemptions. Ratio rows: capacity ratios on the same trace, and the share of 10,000 request-bootstrap draws in which the 3τ03\tau_{0} ratio stays at 1.5 ×\times or more.
Figure 12: SLO attainment against arrival rate for the INT4 arms on the fused path (one RTX PRO 6000, 16K maximum length; TTFT ≤\leq 10 s and TPOT ≤3τ0\leq 3\tau_{0} at τ0=18.60\tau_{0}{=}18.60 ms; red line: the 95% rule). Each point is one cell of 60 requests, of which 43–49 are scored; Table 9 gives the scored counts and the other SLO bounds.
arm8K peak16K peakB=1B{=}1 , 8K (ms)MATH500 (%)GSM8K (%)
BF161.00 ×\times1.00 ×\times10.6883.276.6
FP82.08 ×\times2.05 ×\times———
KIVI-42.27 ×\times1.96 ×\times10.6383.277.0
KIVI-4 + V2.26 ×\times1.96 ×\times10.5585.476.5
LoopKV-R2.54 ×\times2.41 ×\times10.3584.676.8
LoopKV-M2.69 ×\times2.69 ×\times10.3384.477.0
Appendix
Table 10: On an H100 SXM (80 GB), LoopKV-M decodes 2.69 ×\times as fast as BF16 at 8K and 16K, against 2.27 ×\times and 1.96 ×\times for KIVI-4, and no INT4 arm differs significantly from BF16 on MATH500 or GSM8K. Own-peak decode throughput relative to BF16 (each arm on its own H100; a rerun of every arm on one GPU agrees within 2%), step time at B=1B{=}1 and 8K, and served accuracy on MATH500 (500 items) and GSM8K (1,319 items). The H100 kernel reads both V planes.
KV formatstoredreadtok/s 8K / 16K
BF16768768170.8 / 0 89.0
BF16 + loop-3 sharing ( KLshare\mathrm{KL}_{\mathrm{share}} )576768182.1 / 0 95.8
FP8384384328.8 / 176.3
FP8 + loop-3 sharing288384358.4 / 180.7
INT4 final + INT4 residuals180360370.3 / 192.0
KIVI-4 + loop-3 sharing180240505.4 / 264.2
Appendix
Table 11: Once quantized, throughput follows bytes read: at equal storage, own INT4 codes decode 1.36 ×\times faster than residuals at 8K. KiB per token of history (4 loops); measured peaks in vLLM on one RTX PRO 6000, except the 16K peak of INT4 residuals, run on the second GPU (§ 5.1 ). The cells of the two quantized rows that ran on this GPU, served by our plugin before fusion, lie at 98–103% of Eq. 1 ; vLLM’s own BF16 and FP8 rows run faster than Eq. 1 predicts (BF16 at 8K: 170.8 against about 146 tok/s).
planrulestored / readpredictedmeasuredserved KLtok/s 8K / 16K
read-optimizedmin reads, damage ≤\leq pred. of INT4 residuals215 / 2151.150.871.28528.9 / 271.5
memory-optimizedmin reads, stored ≤\leq 156, damage ≤\leq pred. of layer-split residuals155 / 2201.971.801.77535.1 / 280.7
dmin reads, damage ≤\leq 1.42210 / 2101.411.111.42519.4 / 276.2
cmin damage, stored ≤\leq 156156 / 3211.091.081.41401.9 / 209.1
amin damage, read ≤\leq 240237 / 2400.670.471.01479.8 / 213.0
bmin damage, read ≤\leq 312239 / 3120.490.370.98398.0 / 179.3
Appendix
Table 12: Stored / read: KiB per token (4 loops). Predicted (sum of single-cell damages plus source) and measured fp32 KL / KLshare\mathrm{KL}_{\mathrm{share}} ; served KL relative to the served BF16 sharing reference; peak tok/s in vLLM; no V precision, before fusion.
designstored / readtok/s 8K / 16Kserved KLfp32 KLMQ641
INT4 residuals180 / 360370.3 / 192.01.531.42−0.8-0.8
loop 1 INT4, loop 2 INT2156 / 336386.3 / 199.71.882.12−0.8-0.8
layer split: layers 1–12 INT4, 13–24 INT2156 / 336385.6 / 200.11.791.83−2.3-2.3
INT2 residuals132 / 312416.8 / 215.92.974.38−3.0-3.0
Appendix
Table 13: Residual designs served in vLLM before fusion (stored / read: KiB per token; served KL relative to the served BF16 sharing reference; fp32 KL / KLshare\mathrm{KL}_{\mathrm{share}} ; MQ641: Δ\Delta in pp vs BF16). The 16K peak of INT4 residuals ran on the second GPU (§ 5.1 ).
loopsτ\tauKL / KLshare\mathrm{KL}_{\mathrm{share}}Δ\Deltafull-precision V fractionread KiB
1–232.261+0.760+0.760.001 / .000216.0
1–261.768+0.267+0.267.008 / .002216.1
1–281.606+0.105+0.105.020 / .007216.3
1–2101.502+0.001+0.001.042 / .024216.8
1–2121.492−0.009-0.009.090 / .067217.9
1–2161.501−0.000-0.000.321 / .303223.5
Appendix
Table 14: V precision threshold sweep (base: KIVI-4 + loop-3 sharing, 1.51 ×\times KLshare\mathrm{KL}_{\mathrm{share}} , 1.50 ×\times with explicit attention; fp32, 16 documents; Δ\Delta against the same base with explicit attention).
hot fraction of the far history, τ=14\tau{=}14Jaccard of hot sets, τ=14\tau{=}14
loop 1 / 2 / 3 / 4.27 / .24 / .25 / .25loop 1 vs 2 / 3 / 4.44 / .40 / .39
union of 4 loops.37loops 2–3 / 2–4 / 3–4.63 / .57 / .76
shared cell 3s.253s vs 4.83
V lo-plane bytes, write-time per-token precision / per-loop read-time rule: 2.2 ×\times ( τ=10\tau{=}10 ), 1.6 ×\times ( τ=14\tau{=}14 ), 1.3 ×\times ( τ=18\tau{=}18 )
Appendix
Table 15: Loops attend to overlapping but different history tokens, so a per-loop read-time V precision reads fewer lo-plane bytes than a write-time per-token precision that all loops share and that keeps every token any loop marks hot. Ouro-1.4B, fp32, 16 documents, last 256 query positions; hot = attention weight above the τ\tau threshold; 3s = loop 3’s query on loop 4’s keys (a shared cell).
componentplain+ V precision (window seed)+ V precision (sink seed)
history reader53.6950.3850.71
seed—2.020.51
sink + window FlashAttention-27.177.217.21
GEMMs8.238.298.28
GPU busy71.3870.4269.07
Appendix
Table 16: GPU time per decode step (ms), KIVI-4 + loop-3 sharing, 8K, batch 40 (torch.profiler), before fusion: seed, history reader and sink/window attention run as separate kernels; the two V-precision columns take m0m_{0} from the window or from the 32 sink keys.
cellBF16 INT2BF16 INT8BF16 ±\pm 1 ulpfp32 INT2fp32 INT8fp32 ±\pm 1 ulpfp32 INT2 / ulp
loop 1, layer 11.63e-41.37e-41.37e-42.89e-57.5e-88.3e-8348
loop 1, layer 121.44e-41.23e-41.25e-41.78e-53.0e-83.2e-8561
loop 2, layer 121.18e-41.10e-41.08e-47.53e-64.2e-91.7e-8433
loop 3, layer 120.98e-40.98e-40.98e-48.30e-71.0e-97.2e-9115
loop 3, layer 240.92e-40.90e-40.90e-49.26e-81.2e-103.4e-927
Appendix
Table 17: KL of single-cell perturbations in BF16 and fp32 activations (16 documents). INT2 and INT8 code the cell as a residual against loop 4’s INT4 source, unlike the own codes of Figure 2 ; ±\pm 1 ulp changes the cell’s KV by one unit in the last place.
modelbytesknapsackuniform, firstuniform, last
Dory-4B−10%-10\%1.332.0029.3
Dory-4B−25%-25\%2.442.9233.4
Huginn-3.5B−10%-10\%1.041.2085.0
Huginn-3.5B−25%-25\%2.783.6298.3
Appendix
Table 18: The same knapsack has lower mean KL than uniform plans of equal bytes on Dory and Huginn; against INT2 in the first loops, the CIs separate only for Dory at −10%-10\% (Huginn: 6 short documents). fp32 KL relative to uniform INT4 on all loops; Dory 16 documents, Huginn 6. At Dory −10%-10\% the knapsack has 1.33 [1.25, 1.40] against 2.00 [1.59, 2.35] for INT2 in the first loops (95% CIs); the other rows’ CIs overlap and are not shown. Uniform plans of the same bytes put INT2 in the first or the last loops.
planstored / readKL+ V precision (read)Δ\Delta V precision
KIVI-4480 / 4800.82——
KIVI-4 + loop-3 sharing360 / 4801.86——
INT4/INT2 + sharing312 / 4326.30——
INT2 + sharing264 / 38412.70——
read-optimized (knapsack)430 / 4301.391.41 (374.1)+0.021+0.021
memory-optimized (knapsack)310 / 4303.043.04 (373.9)+0.001+0.001
Appendix
Table 19: Ouro-2.6B: at equal bytes the memory-optimized plan has 0.48 ×\times the KL of the hand-made INT4/INT2 + sharing plan (95% document-bootstrap CIs overlap; not shown), and the read-optimized plan 1.39 ×\times against KIVI-4’s 0.82 ×\times at 10% fewer reads. fp32 KL / KLshare\mathrm{KL}_{\mathrm{share}} of Ouro-2.6B, 8 documents; KiB per token (4 loops); V precision at τ=14\tau{=}14 on all loops, Δ\Delta paired.
armbytesMATH500 ( Δ\Delta )8K peak16K peak
last-loop BF161.0091.5352.0 (40)219.7 (24)
LoopKV-R0.8591.5 ( +0.0+0.0 )352.9 (40)221.6 (24)
LoopKV-M0.5491.7 ( +0.2+0.2 )362.5 (56)247.9 (48)
rec5 (fixed)0.6291.7 ( +0.2+0.2 )367.7 (56)254.8 (40)
full9 (fixed)0.31—370.3 (48)259.6 (56)
Appendix
Table 20: Dory-4B with allocator and fixed plans on one RTX PRO 4500 (32 GB), graph mode, fused GQA reader. MATH500: 484 items, greedy, 8,192-token cap, paired against last-loop BF16. Bytes: KV stored per token relative to last-loop BF16. Peaks: tok/s at the throughput peak (batch at the peak, which can lie below the largest resident batch).
contextKIVI-4KIVI-4 + VLoopKV-RLoopKV-Mgain, own peak (LoopKV-R / -M)gain, same batch (LoopKV-R / -M)
8K190.4 (10)202.9 (10)218.9 (11)242.2 (15)1.078 / 1.1941.042 / 1.036
16K100.7 (5)109.1 (5)122.3 (6)132.6 (8)1.121 / 1.2151.037 / 1.031
32K47.2 (2)50.6 (2)60.2 (3)65.5 (4)1.190 / 1.2931.042 / 1.033
Appendix
Table 21: Ouro-1.4B on a 32 GB RTX PRO 4500, 1024-token prefill chunk: peak decode tok/s (resident batch). The plans’ gain over KIVI-4 + V grows with context because they keep more requests resident. 8K: one repeat; 16K and 32K: mean of three fresh processes at the peaks.
designbytesKLresident batch ×\timestok/s ×\times
32-token groups with V precision: served
KIVI-4 + V240/2400.4611
+ loop-3 sharing (hand-made)180/2401.511.33 (1.43)1.07
LoopKV-R (knapsack)215/2150.871.12 (1.14)1.08 (1.08)
LoopKV-M (knapsack)155/2201.801.55 (1.71)1.16 (1.13)
Appendix
Table 22: At the 32-token groups we serve, the read-optimized plan has lower KL and fewer reads than KIVI-4 + loop-3 sharing, the memory-optimized plan stores less at a higher KL (1.80 ×\times against 1.51 ×\times KLshare\mathrm{KL}_{\mathrm{share}} ), and stored bytes set the resident batch. Bytes: KiB per token of history (4 loops), stored / read without V precision; KL: fp32 KL / KLshare\mathrm{KL}_{\mathrm{share}} of the stored codes, which V precision changes by at most 1.6% (§ 3.3 ). Resident batch and peak decode tok/s at 16K, relative to KIVI-4 + V (14 requests, 267.8 tok/s on the fused path), are predicted for every row with V precision: the batch scales with stored bytes from KIVI-4 + V’s 14, and the step time follows Eq. 1 with the measured reads. Parentheses: measured, from the fused-path block of Table 8 and, for the hand-made plan’s batch, from its run before fusion.
Figure 13: No codec reading less than 216 KiB matches the read-optimized plan’s KL (0.87 ×\times KLshare\mathrm{KL}_{\mathrm{share}} at 215 KiB without and 186 KiB with V precision). Ouro-1.4B, 16 documents, fp32 KL / KLshare\mathrm{KL}_{\mathrm{share}} against history bytes read per token; published recipes in our implementation: TurboQuant without QJL, KVQuant without Fisher weighting. Grey: codecs on all four loops; orange: cross-loop residual codecs (FlashLoop’s chain reads all earlier planes); purple and blue: LoopKV plans without and with V precision.
formatbits / elementKiB per tokenKLKL with Hadamard
KIVI-4, group 32 (asymmetric INT4)5.02400.460.31
INT4 symmetric, group 165.02401.250.67
NVFP4 (blocks of 16, E4M3 scale)4.52162.080.97
MXFP4 (blocks of 32, E8M0 scale)4.252046.402.46
FP8 E4M3 (reference)8.03840.19—
Appendix
Table 23: 4-bit formats on all four loops (fp32 KL / KLshare\mathrm{KL}_{\mathrm{share}} , 16 documents). Bits per element include scales and zero points; KL with Hadamard: the same format after a Hadamard rotation.
More in this topic

KV-Cache Compression

Key-value cache compression reduces the memory and bandwidth needed to retain attention states during long-context inference.

See all 279 papers

Explore similar work

CardsList

  1. LoopVQ: Sub 1-bit KV Cache Quantization for Looped Transformers

    Nov 3, 2026Minsoo Cheong, Sungjoo YooKV-Cache QuantizationLLM Inference Acceleration

    Seoul National University

  2. ResidualQuant: KV Cache Quantization for Looped Transformers with 2-Bit Residuals

    Oct 7, 2026Heejun Kim, Junyoung Lee, SangLyul Cho +3KV-Cache QuantizationEfficient Transformer Inference

    KAIST · Yonsei University · Seoul National University

  3. FlashLoop: Fast and Memory-Efficient Looped Transformers via Lazy Updates

    Sep 24, 2026Wanqi Yang, Shiwei LiuEfficient Transformer InferenceRecurrent Transformers

    ELLIS Institute Tübingen Max Planck Institute for Intelligent Systems Tübingen AI Center