Looped language models deepen computation without adding parameters, yet caching full-context key-value (KV) states across layers and rounds severely bounds serving batch size. We design a looped causal encoder-decoder (CED) that achieves recurrent depth with zero round-wise KV growth, addressing where and how to loop an asymmetric stack in which only sparse encoder layers generate global KV. We find that recurrence must be placed in the encoder to iteratively refine global KV representations rather than re-reading static decoder projections, which both drives language-modeling gains and preserves long-context scaling. Furthermore, training the encoder loop under stochastic recurrence depth induces a fixed-point equilibrium where every round losslessly shares the last round's KV cache. Compounding cross-layer and cross-round KV sharing, the resulting Looped CED matches an unlooped CED's minimal memory footprint and recovers 86-93% of a looped Transformer's gain with 1/6 of its global KV. To test serving at production scale, we loop the 552B-parameter DeepSeek-V4.1-Flash for 8 rounds on one 4xGB200 node: with per-layer and per-round KV the loop cannot fit a single 128K-token request, whereas Looped CED holds 36.8x less KV per token, fits 38 such requests (as many as the unlooped model), and serves 3.6x higher throughput on 8K-token prompts.
Figures & tables
Figure 1: Where and how to loop a Causal Encoder–Decoder. (a) Our 12-layer models; a loop repeats LR=4 layers for T rounds. Prefill runs only the layers left of the dashed line, so an encoder loop puts all T rounds on the prefill path and a decoder loop does not; a dense loop runs every layer in prefill and decode. (b) Global KV per token in units of one full-context layer ( L=12 , LR=4 , T=8 , ns=1 encoder KV source). Hatched bars keep KV for every round; solid bars keep only the last round’s KV, which stochastic-depth training makes lossless (§ 3.2 ).
Figure 2
Figure 3: Position-wise BPB across long documents. Pooled over PG19 test, FinePDFs, and GovReport (Appendix H ); 150M models trained on 6.0B tokens. (a) Absolute BPB as a function of available context length; lines fit a+bn−c . (b) Unlooped models minus the unlooped dense model. (c) Looped models minus the L2–5 dense loop. In (b) and (c), lines are least-squares fits in log2n . Markers are measurements; solid curves are looped (layers 2–5; the CED decoder loop, layers 7–10), dashed curves are unlooped. The red dashed line marks the 128-token SWA window (per-corpus breakdowns in Figure 9 ).
Table 4
Figure 4: Serving looped DeepSeek-V4.1-Flash on 4×GB200 : last-round KV vs. per-round KV. (a) Throughput vs. concurrency on 8K-input / 1K-output requests; open markers show where per-round KV fills the KV pool. (b) Peak preemption-free throughput; bold ratios are last-round CED over the baseline with per-layer and per-round KV. (c) Median time to first token on a multi-turn agent trace. Setup and further numbers are in Appendix I .
Workload / Metric ( T=8 , GB200)
Per-layer KV, per-round KV
CED, per-round KV
Looped CED (Ours)
Advantage (Ours)
Global KV per Token (KB)
60.0
8.5
1.6
36.8× smaller ( 5.2× vs. CED per-round)
Usable KV Pool per Node (M tok)
0.13
0.98
5.11
39× larger ( 5.2× vs. CED per-round)
8K In / 1K Out (tok/s/GPU)
50.9
142.5
185.1
3.6× higher ( 1.3× vs. CED per-round)
32K HBM Decode (tok/s/GPU)
—
289.3
818.5
2.8× higher decode throughput
128K Long-Context (tok/s/GPU)
OOM ( 0 req)
7 reqs max
199.2 ( 38 reqs)
38 reqs vs. OOM ( 16.5× at T=4 )
Multi-Turn Agent TTFT ( c=512 )
—
345.1s
11.6s
29.7× lower latency
Table 4: Production serving summary at T=8 on 4×GB200 (TP4) with DeepSeek-V4.1-Flash in SGLang. Comparing the baseline with per-layer and per-round KV , CED with per-round KV , and Looped CED (ours; CED, last-round KV) ; KV pools at the server settings of Figure 4 (a). All three loop L2–19 with the same MoE FLOPs per decoded token; the baseline also runs every layer on every prompt token in prefill. Full T=4 and 8×H200 results are in Appendix I .
Figure 5: Looped CED at interactive speeds and on real workloads. (a, b) Output throughput against p90 per-user decode speed on 8K-input / 1K-output requests, preemption-free points only, on (a) one 4×GB200 tray and (b) one 8×H200 node. (c) Measured GPU-seconds per completed request relative to unlooped CED on four closed-loop workloads at T=8 ( 4×GB200 ; 32, 64, 64 and 8 concurrent requests). Setup, model weights and further results are in Appendix B .
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Last-round KV when serving a prompt. The prompt is prefilled in one parallel pass, and the next 128 tokens are decoded one at a time; the plot shows their BPB with last-round KV minus with per-round KV, pooled over PG19 test, FinePDFs and GovReport, with 95% paired bootstrap intervals. Each panel compares the looped models of one size and token budget (dense loop: L2–5; L4–7 in the last panel). The fixed-round CED encoder loop (150M) is off the scale.
Figure 7: Further serving measurements. (a) As Figure 5 (a), for ShareGPT on one GB200 tray (dummy weights): output tokens/s per GPU against the p90 per-user decode speed ( 1000/p90 TPOT ), one point per concurrency, preemption-free points only. (b, c) Mooncake tool-agent trace, closed loop, T=8 , real weights (the runs of Figure 4 c): (b) prefix-cache hit rate over the second half of each run; (c) share of requests with TTFT ≤10 s.
8K in / 1K out, peak
128K decode at the KV limit
KV across layers
KV across rounds
T=4
T=8
T=4
T=8
CED (Looped CED, ours)
last round
391.8 ( c 512)
185.1 ( c 512)
347.4 (38)
199.2 (38)
CED
per round
308.4 ( c 128)
142.5 ( c 64)
186.1 (13)
69.4 (7)
per layer
last round
161.6 ( c 32)
92.4 ( c 32)
51.7 (3)
29.6 (3)
per layer (baseline)
per round
127.3 ( c 16)
50.9 ( c 8)
21.1 (1)
does not fit (0)
Cross-layer sharing, both last-round KV (CED ÷ per layer)
2.43×
2.00×
6.72×
6.73×
Appendix
Table 5: Cross-layer and cross-round sharing, measured separately ( 4×GB200 , dummy weights). Left: peak preemption-free output tokens/s per GPU on 8K/1K, with the largest preemption-free concurrency in parentheses. Right: decode tokens/s per GPU at the largest batch of 128K-token requests that fits the KV pool, with that batch in parentheses. Without a loop, CED reaches 887.8 and 780.3 and the per-layer-KV model 371.7 and 116.7. The ratios compare rows at equal T .
Prompt length (tokens)
Ratio
512
2K
4K
16K
64K
H200
CED, no loop: full replay ÷ 128-token tail
1.32
1.45
1.50
1.60
1.75
H200
Per-layer KV ÷ CED, encoder loop T=8
1.87
1.45
1.44
1.69
2.43
GB200
Per-layer KV ÷ CED, encoder loop T=8
1.65
1.48
1.41
1.40
1.48
Context length / prefill chunk (tokens)
64K / 4K
64K / 8K
64K / 16K
128K / 4K
128K / 8K
Appendix
Table 6: Prefill cost of running every layer over the prompt. Top: ratio of per-request prefill time for uncached prompts (batch 1, real weights); both models keep last-round KV. Full replay runs the CED decoder over every prompt token instead of the last 128. Bottom: ratio of new prompt tokens prefilled per second behind one cached prefix ( 4×GB200 , dummy weights, 4K new tokens per request). Values above 1 mean the CED variant is faster.
4×GB200
8×H200
Model
KV across rounds
8K
128K
tok/s/GPU at 128K
120K
524K
1M
CED, no loop
—
604
38
780.3
195
44
23
CED encoder loop, T=4
last round
604
38
347.4
—
—
—
CED encoder loop, T=8
last round
604
38
199.2
195
44
23
CED decoder loop, T=8
last round
—
—
—
195
44
23
CED encoder loop, T=4
per round
215
13
186.1
—
—
—
Appendix
Table 7: Requests that fit the KV pool at long context. 4×GB200 : one tray with the flags of Figure 4 , dummy weights; requests of L+256 tokens; the last column is decode tokens/s per GPU at the largest 128K batch that fits. The CED limit of 604 at 8K is computed from the pool (the sweep stopped at 256). 8×H200 : one node, real weights, requests that fit at full length. The H200 per-round pool also charges each round’s sliding-window rings; the GB200 one does not.
Figure 8: Fixed-point analysis. The 150M CED encoder loop (6.0B tokens), trained with a random or a fixed number of rounds and run for up to 32 rounds on FineWeb-Edu validation data in fp32. (a) Median relative change of the core state between rounds, ρr=∥hr−hr−1∥/∥hr−1∥ , over tokens, with the interquartile band. (b) BPB after r rounds minus BPB after the trained mean of 8 rounds (dashed line).
Figure 9: Gap to dense on each corpus. BPB minus the dense model with the same loop setting, by tokens of context, for each corpus (150M models, 6.0B tokens). Markers are measurements; lines are least-squares fits in log2 of the context beyond the SWA window (red dashed line). Top: no loop. Bottom: loop over layers 2–5. FineWeb-Edu counts each token’s context within its own document and is not averaged over offsets.
Layers
12
Model size (non-embedding)
90M / 150M / 340M
Hidden size d
768 / 1024 / 1536
Heads (query = KV)
6 / 8 / 12, dimension 128
FFN
4 d , ReLU 2
Value embeddings
layers 1, 3, 5, 7, 9, 11
SWA window
128
Appendix
Table 8: Hyperparameters. Shared by every 12-layer run, read from the run configurations.
Model
T=1
2
4
8
12
16
CED, encoder loop
0.9695
0.8893
0.8537
0.8465
0.8462
0.8462
CED, enc. loop, 29.6B
0.9552
0.8622
0.8206
0.8124
0.8122
0.8122
CED, decoder loop
0.9156
0.8795
0.8668
0.8648
0.8648
0.8648
Dense, loop L2–5
0.9647
0.8877
0.8527
0.8457
0.8455
0.8455
Dense, loop L4–7
0.9508
0.8874
0.8574
0.8511
0.8507
0.8507
Appendix
Table 9: Test-time rounds. BPB when evaluating with T rounds; the models were trained with 8 rounds on average. Parallel forward pass with per-round KV on the first 512 validation samples, as in Table 3 .
Tp
Td=2
4
8
16
1
0.8633
0.8251
0.8179
0.8178
2
0.8546
0.8150
0.8077
0.8075
4
0.8517
0.8121
0.8047
0.8046
8
0.8517
0.8118
0.8044
0.8042
16
0.8517
0.8118
0.8044
0.8042
Appendix
Table 10: Fewer rounds for prompt tokens. CED encoder loop after 29.6B tokens: BPB on the second half of each sequence, with the first half (the prompt) run at Tp rounds and the second half at Td rounds.
a − b
Long docs
PG19
PDFs
Gov.
FW
Gap to dense, loop L2–5
CED enc. loop
+0.1 [ − 0.2, +0.3]
− 0.1
− 0.8
+1.0
+0.5
YOCO-U loop
+3.7 [+3.4, +4.0]
+3.4
+4.2
+3.6
+2.2
Gap to dense, no loop
CED
− 0.1 [ − 0.3, +0.2]
− 0.6
− 0.7
+1.1
+0.4
YOCO-U
+2.4 [+2.1, +2.6]
+2.6
+2.5
+1.9
+1.2
Appendix
Table 11: How fast BPB gaps grow with context. The least-squares slope of BPB(a) − BPB(b) over the four context buckets beyond the SWA window (128–255, …, 1024–2047 tokens) against log2 of the context, in 10−3 BPB per doubling. Positive means that a falls further behind b as the context grows. Long docs pools PG19 test, FinePDFs and GovReport, with a 95% paired bootstrap interval; FW is the FineWeb-Edu validation set. Hidden size 1024, 6.0B tokens.
Model
1–127
128–255
256–511
512–1023
1024–2047
mean
Dense, no loop
1.2757
1.1054
1.0695
1.0420
1.0263
1.0562
Dense, loop L2–5
1.2433
1.0666
1.0287
1.0005
0.9834
1.0148
Dense, loop L4–7
1.2474
1.0741
1.0367
1.0090
0.9925
1.0232
CED, no loop
1.2718
1.0995
1.0631
1.0360
1.0200
1.0501
CED, encoder loop
1.2441
1.0671
1.0293
1.0012
0.9841
1.0155
CED, decoder loop
1.2639
1.0906
1.0547
1.0282
1.0130
1.0426
Appendix
Table 12: Long-document BPB by context bucket. PG19 test, FinePDFs and GovReport (512, 512 and 396 samples of 2048 tokens, each averaged over six offsets in its document); 150M models. One forward pass per sample with per-round KV and 8 rounds for looped models. Rows above the line trained on 6.0B tokens.
Figure 10: In-context recall. BPB on the second half of samples whose tokens 1,024–2,047 repeat tokens 0–1,023 verbatim (PG19 test and arXiv, 1024 samples), per 64-token bin; the first copied bin is left out. Each panel compares the models of one size and token budget. Solid: looped (L2–5; decoder loop L7–10; dense L4–7 in the last panel); dashed: unlooped.
Analytic Roofline Model
Measured 4×GB200 (TP4)
Measured 8×H200 (TP8)
Model Variant
Prefill a
Decode 8–32K
KV Pool
Prefill a
Decode 8–32K
Decode 120K
Prefill a
CED, no loop (reference)
1.00×
1.00×
1.00×
1.00×
1.00×
1.00×
1.00×
Stock SGLang (no plugin)
1.00×
1.00×
1.00×
1.00×
1.02×
—
0.98×
Looped CED, T=2
1.87×
1.45×
1.00×
1.54×
1.39×
—
1.61×
Looped CED, T=4
3.60×
2.34×
1.00×
2.72×
2.13×
—
2.87×
Looped CED, T=8
7.06×
4.12×
1.00×
5.12×
3.95×
3.83×
5.42×
Appendix
Table 13: Unit prefill and decode cost multipliers relative to unlooped CED ( T=1 ) across 4×GB200 (TP4) and 8×H200 (TP8). The analytic columns are a roofline model of FLOPs and bytes at GB200 peak rates, with expert parallelism over 4 GPUs and all of L20 run on every prompt token, as in SGLang. Prefill a is the per-token slope of tpre(L)=b+aL+cL2 , fitted over uncached prompts of 512–65,536 tokens at batch 1; for CED T=1 it is 18.8μ s per prompt token per GB200 tray and 60.8μ s per H200 node. Decode is the per-step time averaged over 8K and 32K contexts at batches 16–256; for CED T=1 the step at 8K context and batch 32 takes 8.6 ms. GB200 120K uses distinct 120K-token contexts (per-layer KV, T=8 : batches 1 and 8 only). Unit costs do not depend on the round policy, so each row holds for last-round and per-round KV alike.
Rounds ( T )
Per-round KV factor
Pool (last-round / per-round)
ISL = 8K
ISL = 16K
ISL = 32K
T=2
1.60×
8.11M/5.07M
≥1.12× ( 768 vs. 570 )
1.17× ( 487 vs. 288 )
1.34× ( 245 vs. 144 )
T=4
2.80×
8.11M/2.90M
≥1.42× ( 768 vs. 342 )
1.58× ( 487 vs. 174 )
1.80× ( 245 vs. 87 )
T=8
5.20×
8.11M/1.56M
≥2.05× ( 768 vs. 184 )
2.35× ( 487 vs. 93 )
2.83× ( 245 vs. 47 )
Appendix
Table 14: Pure decode throughput saturation (last-round vs. per-round KV) for the CED encoder loop on 4×GB200 (real weights; each request has its own context, whose KV is allocated without being computed, and generates 256 tokens). Each cell is the ratio of the two policies’ highest decode throughput; in parentheses, the batch at which each was measured. Last-round KV at 8K is capped at 768 requests (pool 80% full), so the 8K ratios are lower bounds.
LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.
Looped Transformers increase effective depth by re-executing shared layers, but three limitations hinder their adoption: memory grows with the loop count, compute can only be adjusted in whole-block steps, and the loop count is hard to change after training. We propose LooM (Loop-invariant Memory), which addresses each in turn. In a causal encoder-decoder (CED), LooM loops only the decoder layers after the first, which read a global memory written once per token, and each of them publishes the key-value (KV) entry for a token only on its first visit, so the cache does not grow with the loop count. These layers loop independently, providing every budget from 3 to 21 layer executions per token, and one model trained on randomly sampled budgets covers all of them without a router. With enough loops, LooM outperforms a single-pass baseline trained on the same data on language modeling, zero-shot accuracy and long-context perplexity, while using the same cache size. Our analysis further shows that the training budget distribution decides what extra depth buys. These results suggest that a CED is a natural place to loop: memory is written once, and depth becomes a budget chosen at deployment.
Lele Yang, Muxi Diao, Zhonghao Yan +7
Beijing University of Posts and Telecommunications
Depth recurrence increases effective model depth by repeatedly applying a fixed computation block. However, sharing attention parameters does not share the key-value (KV) states generated at different recurrence iterations, which causes memory growth with recurrence depth during autoregressive decoding. We study how to reduce this memory cost through architectural and cache-sharing approaches within a common depth-recurrent framework. We first study a looped hybrid that repeats a stack of Transformer layers, with recurrent mLSTM or Gated DeltaNet blocks replacing most attention layers and periodic Transformer attention anchors retained. We then restrict recurrence to a dedicated recurrent workspace in a Parcae-style architecture and test its stabilising adapter as a wrapper for the looped hybrid. Finally, we train a looped Transformer with recurrent KV replay, where only selected recurrence iterations write KV states that later iterations reuse. This trains the model to operate with the same shared KV topology used during inference. We find that looped hybrids require sufficiently frequent attention anchors, with performance depending strongly on the recurrent mixer. For cross-iteration KV sharing, recurrent KV replay allows multiple recurrence iterations to reuse the same KV states while retaining performance close to the full looped Transformer with separate KV caches for each iteration.
Looped transformers are a parameter-efficient alternative to standard transformers for language modeling. By reapplying the same layers across multiple loop steps, recent looped language models match the performance of non-looped models several times their size. While this architecture effectively reduces parameter memory, each loop step still stores its own keys and values, so the key-value (KV) cache grows linearly with the number of loop steps and becomes the main memory bottleneck. We observe that the KV cache of looped models is highly redundant along the loop axis and that the loop steps differ substantially in their sensitivity to quantization. Based on these observations, we propose LoopVQ, a KV cache vector quantization method for looped transformers that codes channels of multiple loop steps jointly, and fine-tunes the codebooks to match the output of the full-precision model. At 0.5 bits per element, LoopVQ outperforms existing KV cache vector quantization methods for standard LLMs on general, long chain-of-thought reasoning, and long-context retrieval tasks with Ouro and Huginn models. Its design is orthogonal to these methods, and combined with TaSQ, it retains much of the BF16 accuracy even at 0.25 bits per element. We also implement LoopVQ in SGLang, where it nearly halves the decode latency at a 32K-token context and raises the peak serving throughput by up to 4.3× and 6.1× on the 1.4B and 2.6B Ouro models compared to the BF16 baseline.