Looped language models deepen computation without adding parameters, yet caching full-context key-value (KV) states across layers and rounds severely bounds serving batch size. We design a looped causal encoder-decoder (CED) that achieves recurrent depth with zero round-wise KV growth, addressing where and how to loop an asymmetric stack in which only sparse encoder layers generate global KV. We find that recurrence must be placed in the encoder to iteratively refine global KV representations rather than re-reading static decoder projections, which both drives language-modeling gains and preserves long-context scaling. Furthermore, training the encoder loop under stochastic recurrence depth induces a fixed-point equilibrium where every round losslessly shares the last round's KV cache. Compounding cross-layer and cross-round KV sharing, the resulting Looped CED matches an unlooped CED's minimal memory footprint and recovers 86-93% of a looped Transformer's gain with 1/6 of its global KV. To test serving at production scale, we loop the 552B-parameter DeepSeek-V4.1-Flash for 8 rounds on one 4xGB200 node: with per-layer and per-round KV the loop cannot fit a single 128K-token request, whereas Looped CED holds 36.8x less KV per token, fits 38 such requests (as many as the unlooped model), and serves 3.6x higher throughput on 8K-token prompts.
Figures & tables
Figure 1: Where and how to loop a Causal Encoder–Decoder. (a) Our 12-layer models; a loop repeats LR=4 layers for T rounds. Prefill runs only the layers left of the dashed line, so an encoder loop puts all T rounds on the prefill path and a decoder loop does not; a dense loop runs every layer in prefill and decode. (b) Global KV per token in units of one full-context layer ( L=12 , LR=4 , T=8 , ns=1 encoder KV source). Hatched bars keep KV for every round; solid bars keep only the last round’s KV, which stochastic-depth training makes lossless (§ 3.2 ).
Figure 2
Figure 3: Position-wise BPB across long documents. Pooled over PG19 test, FinePDFs, and GovReport (Appendix H ); 150M models trained on 6.0B tokens. (a) Absolute BPB as a function of available context length; lines fit a+bn−c . (b) Unlooped models minus the unlooped dense model. (c) Looped models minus the L2–5 dense loop. In (b) and (c), lines are least-squares fits in log2n . Markers are measurements; solid curves are looped (layers 2–5; the CED decoder loop, layers 7–10), dashed curves are unlooped. The red dashed line marks the 128-token SWA window (per-corpus breakdowns in Figure 9 ).
Table 4
Figure 4: Serving looped DeepSeek-V4.1-Flash on 4×GB200 : last-round KV vs. per-round KV. (a) Throughput vs. concurrency on 8K-input / 1K-output requests; open markers show where per-round KV fills the KV pool. (b) Peak preemption-free throughput; bold ratios are last-round CED over the baseline with per-layer and per-round KV. (c) Median time to first token on a multi-turn agent trace. Setup and further numbers are in Appendix I .
Workload / Metric ( T=8 , GB200)
Per-layer KV, per-round KV
CED, per-round KV
Looped CED (Ours)
Advantage (Ours)
Global KV per Token (KB)
60.0
8.5
1.6
36.8× smaller ( 5.2× vs. CED per-round)
Usable KV Pool per Node (M tok)
0.13
0.98
5.11
39× larger ( 5.2× vs. CED per-round)
8K In / 1K Out (tok/s/GPU)
50.9
142.5
185.1
3.6× higher ( 1.3× vs. CED per-round)
32K HBM Decode (tok/s/GPU)
—
289.3
818.5
2.8× higher decode throughput
128K Long-Context (tok/s/GPU)
OOM ( 0 req)
7 reqs max
199.2 ( 38 reqs)
38 reqs vs. OOM ( 16.5× at T=4 )
Multi-Turn Agent TTFT ( c=512 )
—
345.1s
11.6s
29.7× lower latency
Table 4: Production serving summary at T=8 on 4×GB200 (TP4) with DeepSeek-V4.1-Flash in SGLang. Comparing the baseline with per-layer and per-round KV , CED with per-round KV , and Looped CED (ours; CED, last-round KV) ; KV pools at the server settings of Figure 4 (a). All three loop L2–19 with the same MoE FLOPs per decoded token; the baseline also runs every layer on every prompt token in prefill. Full T=4 and 8×H200 results are in Appendix I .
Figure 5: Looped CED at interactive speeds and on real workloads. (a, b) Output throughput against p90 per-user decode speed on 8K-input / 1K-output requests, preemption-free points only, on (a) one 4×GB200 tray and (b) one 8×H200 node. (c) Measured GPU-seconds per completed request relative to unlooped CED on four closed-loop workloads at T=8 ( 4×GB200 ; 32, 64, 64 and 8 concurrent requests). Setup, model weights and further results are in Appendix B .
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Last-round KV when serving a prompt. The prompt is prefilled in one parallel pass, and the next 128 tokens are decoded one at a time; the plot shows their BPB with last-round KV minus with per-round KV, pooled over PG19 test, FinePDFs and GovReport, with 95% paired bootstrap intervals. Each panel compares the looped models of one size and token budget (dense loop: L2–5; L4–7 in the last panel). The fixed-round CED encoder loop (150M) is off the scale.
Figure 7: Further serving measurements. (a) As Figure 5 (a), for ShareGPT on one GB200 tray (dummy weights): output tokens/s per GPU against the p90 per-user decode speed ( 1000/p90 TPOT ), one point per concurrency, preemption-free points only. (b, c) Mooncake tool-agent trace, closed loop, T=8 , real weights (the runs of Figure 4 c): (b) prefix-cache hit rate over the second half of each run; (c) share of requests with TTFT ≤10 s.
8K in / 1K out, peak
128K decode at the KV limit
KV across layers
KV across rounds
T=4
T=8
T=4
T=8
CED (Looped CED, ours)
last round
391.8 ( c 512)
185.1 ( c 512)
347.4 (38)
199.2 (38)
CED
per round
308.4 ( c 128)
142.5 ( c 64)
186.1 (13)
69.4 (7)
per layer
last round
161.6 ( c 32)
92.4 ( c 32)
51.7 (3)
29.6 (3)
per layer (baseline)
per round
127.3 ( c 16)
50.9 ( c 8)
21.1 (1)
does not fit (0)
Cross-layer sharing, both last-round KV (CED ÷ per layer)
2.43×
2.00×
6.72×
6.73×
Appendix
Table 5: Cross-layer and cross-round sharing, measured separately ( 4×GB200 , dummy weights). Left: peak preemption-free output tokens/s per GPU on 8K/1K, with the largest preemption-free concurrency in parentheses. Right: decode tokens/s per GPU at the largest batch of 128K-token requests that fits the KV pool, with that batch in parentheses. Without a loop, CED reaches 887.8 and 780.3 and the per-layer-KV model 371.7 and 116.7. The ratios compare rows at equal T .
Prompt length (tokens)
Ratio
512
2K
4K
16K
64K
H200
CED, no loop: full replay ÷ 128-token tail
1.32
1.45
1.50
1.60
1.75
H200
Per-layer KV ÷ CED, encoder loop T=8
1.87
1.45
1.44
1.69
2.43
GB200
Per-layer KV ÷ CED, encoder loop T=8
1.65
1.48
1.41
1.40
1.48
Context length / prefill chunk (tokens)
64K / 4K
64K / 8K
64K / 16K
128K / 4K
128K / 8K
Appendix
Table 6: Prefill cost of running every layer over the prompt. Top: ratio of per-request prefill time for uncached prompts (batch 1, real weights); both models keep last-round KV. Full replay runs the CED decoder over every prompt token instead of the last 128. Bottom: ratio of new prompt tokens prefilled per second behind one cached prefix ( 4×GB200 , dummy weights, 4K new tokens per request). Values above 1 mean the CED variant is faster.
4×GB200
8×H200
Model
KV across rounds
8K
128K
tok/s/GPU at 128K
120K
524K
1M
CED, no loop
—
604
38
780.3
195
44
23
CED encoder loop, T=4
last round
604
38
347.4
—
—
—
CED encoder loop, T=8
last round
604
38
199.2
195
44
23
CED decoder loop, T=8
last round
—
—
—
195
44
23
CED encoder loop, T=4
per round
215
13
186.1
—
—
—
Appendix
Table 7: Requests that fit the KV pool at long context. 4×GB200 : one tray with the flags of Figure 4 , dummy weights; requests of L+256 tokens; the last column is decode tokens/s per GPU at the largest 128K batch that fits. The CED limit of 604 at 8K is computed from the pool (the sweep stopped at 256). 8×H200 : one node, real weights, requests that fit at full length. The H200 per-round pool also charges each round’s sliding-window rings; the GB200 one does not.
Figure 8: Fixed-point analysis. The 150M CED encoder loop (6.0B tokens), trained with a random or a fixed number of rounds and run for up to 32 rounds on FineWeb-Edu validation data in fp32. (a) Median relative change of the core state between rounds, ρr=∥hr−hr−1∥/∥hr−1∥ , over tokens, with the interquartile band. (b) BPB after r rounds minus BPB after the trained mean of 8 rounds (dashed line).
Figure 9: Gap to dense on each corpus. BPB minus the dense model with the same loop setting, by tokens of context, for each corpus (150M models, 6.0B tokens). Markers are measurements; lines are least-squares fits in log2 of the context beyond the SWA window (red dashed line). Top: no loop. Bottom: loop over layers 2–5. FineWeb-Edu counts each token’s context within its own document and is not averaged over offsets.
Layers
12
Model size (non-embedding)
90M / 150M / 340M
Hidden size d
768 / 1024 / 1536
Heads (query = KV)
6 / 8 / 12, dimension 128
FFN
4 d , ReLU 2
Value embeddings
layers 1, 3, 5, 7, 9, 11
SWA window
128
Appendix
Table 8: Hyperparameters. Shared by every 12-layer run, read from the run configurations.
Model
T=1
2
4
8
12
16
CED, encoder loop
0.9695
0.8893
0.8537
0.8465
0.8462
0.8462
CED, enc. loop, 29.6B
0.9552
0.8622
0.8206
0.8124
0.8122
0.8122
CED, decoder loop
0.9156
0.8795
0.8668
0.8648
0.8648
0.8648
Dense, loop L2–5
0.9647
0.8877
0.8527
0.8457
0.8455
0.8455
Dense, loop L4–7
0.9508
0.8874
0.8574
0.8511
0.8507
0.8507
Appendix
Table 9: Test-time rounds. BPB when evaluating with T rounds; the models were trained with 8 rounds on average. Parallel forward pass with per-round KV on the first 512 validation samples, as in Table 3 .
Tp
Td=2
4
8
16
1
0.8633
0.8251
0.8179
0.8178
2
0.8546
0.8150
0.8077
0.8075
4
0.8517
0.8121
0.8047
0.8046
8
0.8517
0.8118
0.8044
0.8042
16
0.8517
0.8118
0.8044
0.8042
Appendix
Table 10: Fewer rounds for prompt tokens. CED encoder loop after 29.6B tokens: BPB on the second half of each sequence, with the first half (the prompt) run at Tp rounds and the second half at Td rounds.
a − b
Long docs
PG19
PDFs
Gov.
FW
Gap to dense, loop L2–5
CED enc. loop
+0.1 [ − 0.2, +0.3]
− 0.1
− 0.8
+1.0
+0.5
YOCO-U loop
+3.7 [+3.4, +4.0]
+3.4
+4.2
+3.6
+2.2
Gap to dense, no loop
CED
− 0.1 [ − 0.3, +0.2]
− 0.6
− 0.7
+1.1
+0.4
YOCO-U
+2.4 [+2.1, +2.6]
+2.6
+2.5
+1.9
+1.2
Appendix
Table 11: How fast BPB gaps grow with context. The least-squares slope of BPB(a) − BPB(b) over the four context buckets beyond the SWA window (128–255, …, 1024–2047 tokens) against log2 of the context, in 10−3 BPB per doubling. Positive means that a falls further behind b as the context grows. Long docs pools PG19 test, FinePDFs and GovReport, with a 95% paired bootstrap interval; FW is the FineWeb-Edu validation set. Hidden size 1024, 6.0B tokens.
Model
1–127
128–255
256–511
512–1023
1024–2047
mean
Dense, no loop
1.2757
1.1054
1.0695
1.0420
1.0263
1.0562
Dense, loop L2–5
1.2433
1.0666
1.0287
1.0005
0.9834
1.0148
Dense, loop L4–7
1.2474
1.0741
1.0367
1.0090
0.9925
1.0232
CED, no loop
1.2718
1.0995
1.0631
1.0360
1.0200
1.0501
CED, encoder loop
1.2441
1.0671
1.0293
1.0012
0.9841
1.0155
CED, decoder loop
1.2639
1.0906
1.0547
1.0282
1.0130
1.0426
Appendix
Table 12: Long-document BPB by context bucket. PG19 test, FinePDFs and GovReport (512, 512 and 396 samples of 2048 tokens, each averaged over six offsets in its document); 150M models. One forward pass per sample with per-round KV and 8 rounds for looped models. Rows above the line trained on 6.0B tokens.
Figure 10: In-context recall. BPB on the second half of samples whose tokens 1,024–2,047 repeat tokens 0–1,023 verbatim (PG19 test and arXiv, 1024 samples), per 64-token bin; the first copied bin is left out. Each panel compares the models of one size and token budget. Solid: looped (L2–5; decoder loop L7–10; dense L4–7 in the last panel); dashed: unlooped.
Analytic Roofline Model
Measured 4×GB200 (TP4)
Measured 8×H200 (TP8)
Model Variant
Prefill a
Decode 8–32K
KV Pool
Prefill a
Decode 8–32K
Decode 120K
Prefill a
CED, no loop (reference)
1.00×
1.00×
1.00×
1.00×
1.00×
1.00×
1.00×
Stock SGLang (no plugin)
1.00×
1.00×
1.00×
1.00×
1.02×
—
0.98×
Looped CED, T=2
1.87×
1.45×
1.00×
1.54×
1.39×
—
1.61×
Looped CED, T=4
3.60×
2.34×
1.00×
2.72×
2.13×
—
2.87×
Looped CED, T=8
7.06×
4.12×
1.00×
5.12×
3.95×
3.83×
5.42×
Appendix
Table 13: Unit prefill and decode cost multipliers relative to unlooped CED ( T=1 ) across 4×GB200 (TP4) and 8×H200 (TP8). The analytic columns are a roofline model of FLOPs and bytes at GB200 peak rates, with expert parallelism over 4 GPUs and all of L20 run on every prompt token, as in SGLang. Prefill a is the per-token slope of tpre(L)=b+aL+cL2 , fitted over uncached prompts of 512–65,536 tokens at batch 1; for CED T=1 it is 18.8μ s per prompt token per GB200 tray and 60.8μ s per H200 node. Decode is the per-step time averaged over 8K and 32K contexts at batches 16–256; for CED T=1 the step at 8K context and batch 32 takes 8.6 ms. GB200 120K uses distinct 120K-token contexts (per-layer KV, T=8 : batches 1 and 8 only). Unit costs do not depend on the round policy, so each row holds for last-round and per-round KV alike.
Rounds ( T )
Per-round KV factor
Pool (last-round / per-round)
ISL = 8K
ISL = 16K
ISL = 32K
T=2
1.60×
8.11M/5.07M
≥1.12× ( 768 vs. 570 )
1.17× ( 487 vs. 288 )
1.34× ( 245 vs. 144 )
T=4
2.80×
8.11M/2.90M
≥1.42× ( 768 vs. 342 )
1.58× ( 487 vs. 174 )
1.80× ( 245 vs. 87 )
T=8
5.20×
8.11M/1.56M
≥2.05× ( 768 vs. 184 )
2.35× ( 487 vs. 93 )
2.83× ( 245 vs. 47 )
Appendix
Table 14: Pure decode throughput saturation (last-round vs. per-round KV) for the CED encoder loop on 4×GB200 (real weights; each request has its own context, whose KV is allocated without being computed, and generates 256 tokens). Each cell is the ratio of the two policies’ highest decode throughput; in parentheses, the batch at which each was measured. Last-round KV at 8K is capped at 768 requests (pool 80% full), so the 8K ratios are lower bounds.
LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.