Looped language models deepen reasoning at a compact parameter count by iterating a shared layer stack T times per token, turning recurrence depth directly into sequential decoding latency. Conventional early-exit decoding halts after initial passes without cache completion, truncating cached key-value states of past tokens to shallow depth. We study the opposite strategy, late entry, where new tokens skip initial passes and enter directly into the final K < T passes over a deep context cache. Once lightweight fine-tuning resolves hidden-state norm mismatch at the entry boundary, deep cached context substitutes for skipped token-local passes, allowing late entry to outperform matched early exit across model families and pass budgets. To turn this asymmetry into wall-clock speedups without stalling to build deep caches, candidate blocks verify through only the final K passes on the critical path while previously committed tokens complete passes 1 -> T alongside subsequent steps at zero extra serial calls. Projecting these finished T-pass wake states inside the existing output GEMM (Wake verification, governed by logit-margin tolerance tau in [0, inf]) unifies greedy-exact T-pass decoding (tau = 0, matching K = T block decoding within bf16 reduction order), margin-gated verification (tau > 0), and non-rollback lossy late entry (tau = inf) inside a single static CUDA graph. On Ouro-2.6B (T = 4), K = 2 Wake reaches 5.2x greedy-exact speedup at tau = 0 (+26% over K = 4 block decoding on GSM8K) and 6.8x at tau = inf (+16.9 points over trained early exit at matched speed). Scaling test-time recurrence across T in {8, 16, 32, 64} on Huginn-3.5B cuts K = T/4 rollbacks 21.5x with depth, driving greedy-exact (tau = 0) speedups to 6.1x (T = 32) and 8.4x (T = 64, +153% over K = 64 block) and tau = inf speedups to 10.6 to 12.5x.
Figures & tables
Figure 1: Early exit vs. late entry in looped language models. (a) Full depth ( ): runs all T=4 passes. (b) Early exit ( ): halts after pass 1 , truncating context KV caches to shallow depth. (c) Late entry ( ): skips passes 1 to 3 on the new token ( clips ) and enters pass 4 over a full-depth context cache completed in parallel (Section 3 ). (d) Unified Wake and late-entry frontier (GSM8K, B=8 , single H100): reusing completed T -pass wake states lifts throughput beyond the K=T block baseline with greedy-exact fidelity at τ=0 ( 5.2 × , +26% on Ouro-2.6B; 6.1 to 8.4 × , +82% to +153% on Huginn-3.5B across T=32→64 ) and reaches 6.8 × and 10.6 × ( 12.5 × on MBPP) at τ=∞ .
Input state supplied to pass 4 ( K=1 over four-pass context)
Released Ouro-1.4B
Released Ouro-2.6B
Raw token embedding (unadapted late entry)
.552
.751
Closed-form linear projection of the token embedding
.816
.919
True pass-3 hidden state (diagnostic oracle ceiling)
.991
.992
Reference: 1-pass early exit (pass 1 only for all tokens)
.938
.949
Table 1: Diagnosing raw late-entry failure on off-the-shelf purely looped checkpoints. Each row reports GSM8K teacher-forced next-token agreement with full-depth ( T=4 ) decoding across 78,693 token positions on Ouro-1.4B and 70,101 positions on Ouro-2.6B. In the top three rows, the current token executes only pass 4 ( K=1 ) over a four-pass context cache; the bottom row runs pass 1 only for both token and context. A closed-form linear map recovers 60% to 70% of the gap to the pass-3 oracle without changing a single model weight.
Figure 2: Why a single adapted late pass suffices: deep context KV substitutes for token-local passes on fine-tuned Ouro-2.6B. (a) Mechanism: under shallow early exit ( K=2,D=2 , ), token xi executes two passes but attends only to unrefined two-pass prefix caches; under adapted late entry ( K=1,D=4 , ), multi-pass contextual reasoning is already stored in the four-pass prefix cache, allowing a single aligned pass 4 to query deep context states directly from e(xi) . (b) Teacher-forced decoupling: varying current-token passes K (executing passes D−K+1,…,D ) and context depth D∈{2,4} on GSM8K ( K=D=4 is 1.000 by definition) reveals that one pass over deep context ( .962 ) beats two passes over shallow context ( .954 ). (c) Free decoding: in single-token greedy generation at K=2 , completing four-pass context caches lifts task accuracy by +10.4 points on GSM8K and +12.0 points on IFEval.
Figure 3: Pass-coherent (K,V) addressing and confidence localization on Ouro-2.6B ( fp32 probes across 30,736 GSM8K positions). (a) Cross-pass (K,V) interventions: swapping keys and values across passes collapses agreement to .000 (Pass-1 K + Pass-4 V ) or .296 (Pass-4 K + Pass-1 V ), whereas Stage-2 alignment reaches linear CKA .961 with true pass-4 states and .962 ( K=1 ) to .984 ( K=2 ) agreement over coherent deep (K,V) caches. (b) Confidence-binned agreement: adapted late entry reaches 1.000 ( K=2 ) and .998 ( K=1 ) on the 68% of confident tokens ( pmax≥.95 ), isolating residual discrepancies to the 6% uncertain tail ( pmax<.50 ) targeted by Wake verification.
Ouro-2.6B ( 48 layers, T=4 )
Ouro-1.4B ( 24 layers, T=4 )
GSM8K
IFEval
GSM8K
IFEval
Passes / token
First K
Last K
First K
Last K
First K
Last K
First K
Last K
K=1 ( vs. )
.898
.962
.757
.887
.842
.948
.743
.884
K=2 ( vs. )
.954
.984
.866
.943
.922
.976
.839
.941
K=3 ( vs. )
.982
.992
.949
.969
.967
.990
.938
.968
Table 2: Same-checkpoint comparison of First K versus Last K passes across scales and tasks. Entries report teacher-forced next-token agreement with full-depth ( T=4 ) decoding on identical Stage-2 checkpoints ( 70,101 GSM8K positions on Ouro-2.6B; 28,939 GSM8K and 61,723 IFEval positions on Ouro-1.4B). “First K ” ( ) runs passes 1,…,K over a depth- K context; “Last K ” ( ) runs passes T−K+1,…,T over a four-pass ( D=4 ) context.
Figure 4: Inside a single late-entry block step ( T=4,K=2 , block size B=2 ). The last committed anchor token (“ 3 ”) and B=2 candidate proposals (“ red ”, “ apples ”) each append a branch of B=2 mask placeholders, forming (1+B)2=9 parallel rows that enter directly at pass 3 . Prefix verification accepts “ red ” ( ✓ ), replaces the rejected proposal with bonus token “ balls ” ( × ), and reuses the placeholder predictions behind “ red ” (“ and ”, “ 2 ”) as draft proposals to be verified from anchor “ balls ” in the next step.
Figure 5: Multi-step pipeline of late-entry decoding ( T=4,K=2 ). Each step executes K=2 serial forward calls. Tentatively accepted token group A ( 1+c≤1+B tokens) runs late-entry passes 3 and 4 in Step 1 (writing provisional KV states P3∗,P4∗ ), and then recomputes true passes 1→4 as parallel batch rows inside the K=2 calls of Step 2 (passes 1,2 ) and Step 3 (passes 3,4 ), adding zero extra serial forward calls .
Figure 6: Worked 3-step walkthrough of Wake verification and rollback ( T=4,K=2 ). Step 1: over permanently verified prefix "2 + 3" ( <u ), passes 3 and 4 tentatively accept Group A =["=","6"] at [u,v) , slipping on position u="=" ( y^u="6" instead of "5" ). Step 2: the frontier tentatively emits ["."," The"] while Group A computes exact passes 1 and 2 inside the same forward calls. Step 3: Group A finishes passes 3 and 4 and projects inside the same single lm_head GEMM ; detecting yu∗="5"="6" ( Δz=2.4 ), τ=0 truncates the tentative tail beyond u , retains exact KV states on "2 + 3 =" ( ≤u ), commits "5" at u+1 , and resumes.
Figure 7: Accuracy-throughput frontier of late-entry decoding on Ouro-2.6B across the full GSM8K ( n=1,319 ) and IFEval ( n=541 ) test sets for pass budgets K∈{2,3,4} and block sizes B∈{4,8} ( K=1 appears in Figure 8 and Appendix D ). LoopSpec marks its fastest released lossless ( T=4 ) reference point on GSM8K.
Full depth
K=4 (full)
K=3 (late entry)
K=2 (late entry)
Benchmark
n
Acc.
Gap
Speedup
Gap
Speedup (vs. K=4 )
Gap
Speedup (vs. K=4 )
GSM8K
1,319
.873
− 0.6
4.05 ×
− 2.2 †
5.09 × (+26%)
− 3.4 †
6.73 × (+66%)
IFEval
541
.732
+0.7
2.16 ×
+0.2
2.82 × (+31%)
− 3.3 †
3.78 × (+75%)
MATH500
500
.532
− 0.6
4.06 ×
+0.4
4.99 × (+23%)
− 4.0 †
6.49 × (+60%)
MBPP
500
.616
+1.0
3.65 ×
+0.2
4.48 × (+23%)
− 1.4
5.80 × (+59%)
BBH
6,511
.577
+ 0.0
3.30 ×
− 1.3 †
4.30 × (+30%)
− 2.8 †
6.16 × (+87%)
Table 3: Full-dataset accuracy and throughput on Ouro-2.6B ( T=4 ) across six benchmarks at block size B=8 ( 21,403 total items, single H100 GPU). “Full depth” is sequential T=4 accuracy. “Gap” is accuracy minus full-depth sequential accuracy in percentage points ( † marks paired sign-test p<.05 ). “Speedup” divides decode tokens/s by same-job full-depth sequential tokens/s, while (+X%) isolates the incremental throughput from late entry over K=4 block decoding.
Early exit, trained
Late entry, trained
Passes / token
Acc.
tok/s
Acc.
tok/s
Gap (pts)
Sign-test p
GSM8K (1,319 test problems)
K=1 ( vs. )
.330
657
.629
544
+29.9 †
4.4×10−77
K=2 ( vs. )
.669
431
.839
427
+16.9 †
2.7×10−35
K=3 ( vs. )
.785
314
.851
323
+6.5 †
8.3×10−8
K=4 (full depth)
.822
246
.867
257
+4.5 †
2.2×10−5
Table 4: End-to-end comparison of trained early exit and late entry on Ouro-2.6B under matched Stage-2 fine-tuning, decoded through the identical B=8 block engine on full test sets ( † marks paired sign-test p<.05 ; gaps computed prior to rounding). At K<4 , early exit runs passes 1,…,K over a depth- K cache, whereas late entry runs passes T−K+1,…,T with pipelined full-depth cache completion.
Figure 8: Accuracy versus decode throughput for trained late entry and trained early exit on Ouro-2.6B at B=8 (plotting Table 4 ). Vertical calipers mark matched-throughput operating points ( +16.9 points at ∼430 tokens/s on GSM8K; +4.3 and +1.8 points on IFEval).
Lossless T=4 verification
Lossy late entry (ours)
Benchmark
Full depth (tok/s)
LoopSpec
LoopAhead
Ours K=4
K=3
K=2
GSM8K
67.2 (.873)
2.84 ×
2.94 ×
3.82 × .867
4.80 × .851
6.35 × .839
IFEval
64.7 (.732)
2.34 ×
2.67 ×
2.12 × .739
2.75 × .734
3.68 × .699
MATH500
62.7 (.532)
2.85 ×
2.59 ×
4.11 × .526
5.07 × .536
6.63 × .492
MBPP
65.1 (.616)
2.86 ×
2.90 ×
3.64 × .626
4.51 × .618
5.95 × .602
BBH
65.2 (.577)
2.62 ×
3.25 ×
3.29 × .577
4.35 × .564
6.33 × .549
Table 5: Full-depth ( T=4 ) speculative decoders versus non-rollback ( τ=∞ ) late entry on Ouro-2.6B across all six benchmarks ( 21,403 total items, single H100 GPU). Speedups divide each method’s decode tokens/s by the fastest measured full-depth sequential decoder on that benchmark. LoopSpec, LoopAhead, and our K=4 block baseline run full-depth T=4 verification; K=3 and K=2 ( τ=∞ ) trade bounded accuracy shifts (small numbers report full-set accuracy, with full-depth sequential accuracy beside each denominator) for higher throughput, while Section 4.5 activates τ=0 Wake to surpass K=4 with greedy-exact fidelity.
Figure 9: Generalization to 32-pass middle-recurrent Huginn-3.5B ( T=32 , matched 7,406 -step Stage-2 checkpoints). (a) Teacher-forced probe: across all pass budgets K<32 on GSM8K, late entry over a 32 -pass context outperforms trained early exit, with 4 late passes ( .954 ) exceeding 8 trained early-exit passes ( .944 ). (b) End-to-end GSM8K accuracy-throughput frontier ( B=8,n=1,319 ): inside our unified CUDA-graphed paged FlashAttention-3 engine, late entry outperforms trained early exit over a broad range of practical operating points, reaching 5.84 × ( 126.8 tokens/s, .405 ) at K=16 , 7.27 × ( 157.8 tokens/s, .395 ) at K=12 , 9.13 × ( 198.1 tokens/s, .354 ) at four-stage K=8 , and 9.93 × ( 215.5 tokens/s, .369 ) under a two-stage partial-cache K=8 variant ( +9.7 points over trained early exit at ∼220 tokens/s).
Figure 10: Unified Wake verification spectrum ( τ∈[0,∞] ) and drafter factorization on GSM8K ( n=1,319,B=8 ). (a) Ouro-2.6B ( T=4 ): τ=0 Exact Wake at K=2 lifts throughput +26% beyond the K=4 block baseline ( 4.13×→5.20× ) with identical .867 accuracy and 0.007% committed-token discrepancy, while margin gating ( τ∈[0.15,0.30] ) reaches 5.52 to 5.77 × ( +34% to +40% ) at .877 to .864 accuracy before connecting smoothly to τ=∞ ( 6.82× ). (b) Huginn-3.5B ( T=32 ): τ=0 Exact Wake eliminates four-stage K=8 lag completely, reaching 5.21 × ( +56% ) at K=16 , 6.09 × ( +82% ) at K=8 , and 7.15 × ( +114% ) at τ=0.25 with zero accuracy drop.
Figure 11: Intra-model recurrence depth scaling on Huginn-3.5B across T∈{8,16,32,64} ( 36→260 effective Transformer layers/token, identical checkpoint and engine, full GSM8K n=1,319 ). (a) Decode speedup vs. T : while full-depth K=T block speedup plateaus near 3.3× , τ=0 greedy-exact Wake scales from 1.96× ( T=8 ) to 8.42 × ( T=64 , +153% over K=64 block) and τ=∞ late entry reaches 10.57 × . (b) Wake rollback collapse: τ=0 rollbacks per 100 emitted tokens fall 21.5 × across T=8→64 at both K=T/4 ( 12.5→0.58 ) and K=T/2 ( 5.6→0.26 ).
Figure 12: Physical H100 Roofline and zero-shot multi-request throughput law on Ouro-2.6B ( T=4 ). (a) Single-pass Roofline: at batch size C=1 , appending wake catch-up rows ( 81→99 rows) sits deep inside the HBM weight-streaming plateau well below the hardware ridge point Mridge(1)≈348 rows/pass. (b) Zero-shot concurrency prediction: calibrating Equation ( 3 ) strictly on full-depth K=4 modes predicts unseen K∈{2,3} continuous-batching throughput across C∈[1,32] with 8.3% MAPE and selects the optimal block size B∗(C)∈{4,8} on 11/12 (91.7%) operating points.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Training configuration
Full-depth acc. (GSM8K / IFEval)
Late-entry probe
Late-entry decode ( K=2 )
None (released base model)
.729 / .442
.751
n/a (no drafters)
Stage 1 only (full-depth SFT)
.880 / .728
.699
n/a (no drafters)
Stage 1 + Stage 2, late entry off
.820 / .708
.738
.636 / .529
Stage 1 + Stage 2 (our full recipe)
.873 / .732
.962
.839 / .699
Appendix
Table 9: Training-stage ablation on Ouro-2.6B across full GSM8K ( n=1,319 ) and strict IFEval ( n=541 ). The probe column reports teacher-forced agreement at K=1,D=4 ; the rightmost column reports end-to-end block decoding accuracy at K=2,B=8 .
Input state supplied to pass 4 ( K=1 over four-pass context)
Released Ouro-1.4B
Released Ouro-2.6B
Raw token embedding (unadapted late entry)
.552
.751
Normalized and magnitude-rescaled embedding
.245
.218
Closed-form linear projection of the embedding
.816
.919
Preceding token’s deep hidden state alone
.030
.039
True pass-3 hidden state (diagnostic oracle ceiling)
.991
.992
Reference: 1-pass early exit (pass 1 only)
.938
.949
Appendix
Table 10: Extended pass-4 entry-state diagnostics on off-the-shelf Ouro-1.4B and Ouro-2.6B (GSM8K teacher-forced agreement with full depth; top five rows run pass 4 only over a four-pass context cache).
GSM8K ( n=1,319 )
IFEval ( n=541 )
Setting
B
tok/s
Speedup
Acc.
tok/s
Speedup
Acc.
Full-depth sequential
n/a
63.4
1.00 ×
.873
63.1
1.00 ×
.732
K=4 ( )
4
210.5
3.32 ×
.870
137.4
2.18 ×
.719
8
256.6
4.05 ×
.867
136.7
2.16 ×
.739
K=3 ( )
4
269.7
4.26 ×
.862
175.0
2.77 ×
.728
8
322.5
5.09 ×
.851
178.1
2.82 ×
.734
Appendix
Table 11: Complete (K,B) accuracy-throughput grid on full GSM8K and IFEval test sets (batch size 1 , greedy decoding, single H100 GPU).
GSM8K
IFEval
Verification rule
Speedup
Acc.
Speedup
Acc.
Strict argmax , K=4
4.06 ×
.867
2.17 ×
.739
Relaxed θ=.40 , K=4
4.14 ×
.867
2.19 ×
.726
Relaxed θ=.30 , K=4
4.30 ×
.861
2.26 ×
.730
Relaxed θ=.20 , K=4
4.40 ×
.841
2.36 ×
.719
Relaxed θ=.10 , K=4
4.49 ×
.834
2.56 ×
.734
Appendix
Table 12: Relaxed probability-threshold verification at full depth ( K=4,B=8 ) versus strict greedy ( argmax ) late entry. Accepting proposals whose full-depth probability exceeds θ yields smaller speedups and lower accuracy than reducing K under strict argmax verification.
Full-depth floor
K=4
K=3
K=2
Benchmark
diff
margin
diff
margin
diff
margin
diff
margin
GSM8K
0.27%
.04 (.10)
0.01%
.06 (.11)
1.12%
.11 (.48)
2.08%
.22 (.87)
IFEval
0.64%
.03 (.09)
0.41%
.03 (.09)
3.17%
.13 (.50)
5.86%
.23 (.77)
MATH500
0.25%
.05 (.11)
0.09%
.05 (.12)
1.16%
.13 (.53)
2.16%
.23 (1.00)
MBPP
0.33%
.04 (.10)
0.02%
.07 (.30)
1.04%
.10 (.39)
1.87%
.18 (.71)
BBH
0.27%
.04 (.11)
0.06%
.04 (.12)
1.33%
.15 (.77)
2.43%
.33 (1.41)
Appendix
Table 13: Committed-token fidelity audit at block size B=8 . “diff” is the percentage of committed positions whose token differs from a teacher-forced full-depth argmax replay; “margin” reports the median (and 90th percentile) top-2 logit margin at differing positions.
GSM8K
IFEval
Cache-completion policy
Decoding mode
K=2
K=3
K=2
K=3
Synchronous (repair before next step)
Sequential
.852
.864
.684
.717
Pipelined (lagged across steps, ours)
Sequential
.859
.872
.695
.723
Pipelined (lagged across steps, ours)
Block ( B=8 )
.839
.851
.699
.734
Never (keep provisional cache)
Block ( B=8 )
.798
.848
.688
.721
Appendix
Table 14: Cache-completion policy ablation on full GSM8K ( n=1,319 ) and IFEval ( n=541 ). Full-depth sequential references are .873 and .732 .
Figure 13: Position-resolved deviation audit at B=8 and K∈{2,3} across 400 GSM8K and 200 IFEval examples. Left subpanels group bonus tokens by accepted count c ( c=0 means all B=8 proposals were rejected); right subpanels show deviation rates across accepted in-block positions 1→8 .
Early exit, trained
Late entry, trained
Core passes / token
Acc.
tok/s
Acc.
tok/s
Gap (pts)
Sign-test p
GSM8K (1,319 test problems; full-depth sequential: .393 at 21.7 tok/s)
K=8 ( 4 -stg full)
.271
223
.354
198
+8.3 †
3.1×10−10
K=8 ( 2 -stg partial)
.271
223
.369
215
+9.7 †
3.4×10−14
K=12
.342
174
.395
158
+5.3 †
7.5×10−5
K=16
.372
140
.405
127
+3.3 †
.015
Appendix
Table 16: Trained early exit versus late entry on Huginn-3.5B ( T=32 ) under matched Stage-2 fine-tuning inside the unified CUDA-graphed paged FlashAttention-3 engine at block size B=8 across full GSM8K ( n=1,319 ) and strict IFEval ( n=541 ), following Table 4 ( † marks paired sign-test p<.05 ; gaps computed prior to rounding; “ K=8 ( 2 -stg partial)” completes passes 17→32 and is compared against 8 -pass early exit at matched ∼220 tokens/s).
Figure 14: Steady-state concurrency scaling on Ouro-2.6B ( T=4 , GSM8K, single H100 GPU) across C∈{1,2,4,8,16,32} concurrent requests. (a) Speedup over continuous-batching AR at the same C : fixed large blocks ( B=8 ) decay steeply as C×99 rows saturate compute, whereas selecting smaller discrete block profiles with load ( B∈{8,4,2} , shaded K=2 profile envelope) sustains 2.31 × at C=32 ( K=2 ) and 1.71 × ( K=3 ). (b) System decode throughput vs. per-request speed ( C=1 at bottom-right to C=32 at top-left along each curve).