Late Entry in Looped Language Models
Organizations: Nanyang Technological University. · Singapore University of Technology and Design. · The Chinese University of Hong Kong.
Abstract
Looped language models deepen reasoning at a compact parameter count by iterating a shared layer stack T times per token, turning recurrence depth directly into sequential decoding latency. Conventional early-exit decoding halts after initial passes without cache completion, truncating cached key-value states of past tokens to shallow depth. We study the opposite strategy, late entry, where new tokens skip initial passes and enter directly into the final K < T passes over a deep context cache. Once lightweight fine-tuning resolves hidden-state norm mismatch at the entry boundary, deep cached context substitutes for skipped token-local passes, allowing late entry to outperform matched early exit across model families and pass budgets. To turn this asymmetry into wall-clock speedups without stalling to build deep caches, candidate blocks verify through only the final K passes on the critical path while previously committed tokens complete passes 1 -> T alongside subsequent steps at zero extra serial calls. Projecting these finished T-pass wake states inside the existing output GEMM (Wake verification, governed by logit-margin tolerance tau in [0, inf]) unifies greedy-exact T-pass decoding (tau = 0, matching K = T block decoding within bf16 reduction order), margin-gated verification (tau > 0), and non-rollback lossy late entry (tau = inf) inside a single static CUDA graph. On Ouro-2.6B (T = 4), K = 2 Wake reaches 5.2x greedy-exact speedup at tau = 0 (+26% over K = 4 block decoding on GSM8K) and 6.8x at tau = inf (+16.9 points over trained early exit at matched speed). Scaling test-time recurrence across T in {8, 16, 32, 64} on Huginn-3.5B cuts K = T/4 rollbacks 21.5x with depth, driving greedy-exact (tau = 0) speedups to 6.1x (T = 32) and 8.4x (T = 64, +153% over K = 64 block) and tau = inf speedups to 10.6 to 12.5x.
Figures & tables
| Input state supplied to pass 4 ( over four-pass context) | Released Ouro-1.4B | Released Ouro-2.6B |
|---|---|---|
| Raw token embedding (unadapted late entry) | .552 | .751 |
| Closed-form linear projection of the token embedding | .816 | .919 |
| True pass-3 hidden state (diagnostic oracle ceiling) | .991 | .992 |
| Reference: 1-pass early exit (pass 1 only for all tokens) | .938 | .949 |
| Ouro-2.6B ( layers, ) | Ouro-1.4B ( layers, ) | |||||||
| GSM8K | IFEval | GSM8K | IFEval | |||||
| Passes / token | First | Last | First | Last | First | Last | First | Last |
| ( vs. ) | .898 | .962 | .757 | .887 | .842 | .948 | .743 | .884 |
| ( vs. ) | .954 | .984 | .866 | .943 | .922 | .976 | .839 | .941 |
| ( vs. ) | .982 | .992 | .949 | .969 | .967 | .990 | .938 | .968 |
| Full depth | (full) | (late entry) | (late entry) | |||||
|---|---|---|---|---|---|---|---|---|
| Benchmark | Acc. | Gap | Speedup | Gap | Speedup (vs. ) | Gap | Speedup (vs. ) | |
| GSM8K | 1,319 | .873 | 0.6 | 4.05 | 2.2 † | 5.09 (+26%) | 3.4 † | 6.73 (+66%) |
| IFEval | 541 | .732 | +0.7 | 2.16 | +0.2 | 2.82 (+31%) | 3.3 † | 3.78 (+75%) |
| MATH500 | 500 | .532 | 0.6 | 4.06 | +0.4 | 4.99 (+23%) | 4.0 † | 6.49 (+60%) |
| MBPP | 500 | .616 | +1.0 | 3.65 | +0.2 | 4.48 (+23%) | 1.4 | 5.80 (+59%) |
| BBH | 6,511 | .577 | + 0.0 | 3.30 | 1.3 † | 4.30 (+30%) | 2.8 † | 6.16 (+87%) |
| Early exit, trained | Late entry, trained | |||||
| Passes / token | Acc. | tok/s | Acc. | tok/s | Gap (pts) | Sign-test |
| GSM8K (1,319 test problems) | ||||||
| ( vs. ) | .330 | 657 | .629 | 544 | +29.9 † | |
| ( vs. ) | .669 | 431 | .839 | 427 | +16.9 † | |
| ( vs. ) | .785 | 314 | .851 | 323 | +6.5 † | |
| (full depth) | .822 | 246 | .867 | 257 | +4.5 † | |
| Lossless verification | Lossy late entry (ours) | |||||
|---|---|---|---|---|---|---|
| Benchmark | Full depth (tok/s) | LoopSpec | LoopAhead | Ours | ||
| GSM8K | 67.2 (.873) | 2.84 | 2.94 | 3.82 .867 | 4.80 .851 | 6.35 .839 |
| IFEval | 64.7 (.732) | 2.34 | 2.67 | 2.12 .739 | 2.75 .734 | 3.68 .699 |
| MATH500 | 62.7 (.532) | 2.85 | 2.59 | 4.11 .526 | 5.07 .536 | 6.63 .492 |
| MBPP | 65.1 (.616) | 2.86 | 2.90 | 3.64 .626 | 4.51 .618 | 5.95 .602 |
| BBH | 65.2 (.577) | 2.62 | 3.25 | 3.29 .577 | 4.35 .564 | 6.33 .549 |
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Training configuration | Full-depth acc. (GSM8K / IFEval) | Late-entry probe | Late-entry decode ( ) |
|---|---|---|---|
| None (released base model) | .729 / .442 | .751 | n/a (no drafters) |
| Stage 1 only (full-depth SFT) | .880 / .728 | .699 | n/a (no drafters) |
| Stage 1 + Stage 2, late entry off | .820 / .708 | .738 | .636 / .529 |
| Stage 1 + Stage 2 (our full recipe) | .873 / .732 | .962 | .839 / .699 |
| Input state supplied to pass 4 ( over four-pass context) | Released Ouro-1.4B | Released Ouro-2.6B |
| Raw token embedding (unadapted late entry) | .552 | .751 |
| Normalized and magnitude-rescaled embedding | .245 | .218 |
| Closed-form linear projection of the embedding | .816 | .919 |
| Preceding token’s deep hidden state alone | .030 | .039 |
| True pass-3 hidden state (diagnostic oracle ceiling) | .991 | .992 |
| Reference: 1-pass early exit (pass 1 only) | .938 | .949 |
| GSM8K ( ) | IFEval ( ) | ||||||
|---|---|---|---|---|---|---|---|
| Setting | tok/s | Speedup | Acc. | tok/s | Speedup | Acc. | |
| Full-depth sequential | n/a | 63.4 | 1.00 | .873 | 63.1 | 1.00 | .732 |
| ( ) | 4 | 210.5 | 3.32 | .870 | 137.4 | 2.18 | .719 |
| 8 | 256.6 | 4.05 | .867 | 136.7 | 2.16 | .739 | |
| ( ) | 4 | 269.7 | 4.26 | .862 | 175.0 | 2.77 | .728 |
| 8 | 322.5 | 5.09 | .851 | 178.1 | 2.82 | .734 | |
| GSM8K | IFEval | |||
| Verification rule | Speedup | Acc. | Speedup | Acc. |
| Strict , | 4.06 | .867 | 2.17 | .739 |
| Relaxed , | 4.14 | .867 | 2.19 | .726 |
| Relaxed , | 4.30 | .861 | 2.26 | .730 |
| Relaxed , | 4.40 | .841 | 2.36 | .719 |
| Relaxed , | 4.49 | .834 | 2.56 | .734 |
| Full-depth floor | ||||||||
|---|---|---|---|---|---|---|---|---|
| Benchmark | diff | margin | diff | margin | diff | margin | diff | margin |
| GSM8K | 0.27% | .04 (.10) | 0.01% | .06 (.11) | 1.12% | .11 (.48) | 2.08% | .22 (.87) |
| IFEval | 0.64% | .03 (.09) | 0.41% | .03 (.09) | 3.17% | .13 (.50) | 5.86% | .23 (.77) |
| MATH500 | 0.25% | .05 (.11) | 0.09% | .05 (.12) | 1.16% | .13 (.53) | 2.16% | .23 (1.00) |
| MBPP | 0.33% | .04 (.10) | 0.02% | .07 (.30) | 1.04% | .10 (.39) | 1.87% | .18 (.71) |
| BBH | 0.27% | .04 (.11) | 0.06% | .04 (.12) | 1.33% | .15 (.77) | 2.43% | .33 (1.41) |
| GSM8K | IFEval | ||||
|---|---|---|---|---|---|
| Cache-completion policy | Decoding mode | ||||
| Synchronous (repair before next step) | Sequential | .852 | .864 | .684 | .717 |
| Pipelined (lagged across steps, ours) | Sequential | .859 | .872 | .695 | .723 |
| Pipelined (lagged across steps, ours) | Block ( ) | .839 | .851 | .699 | .734 |
| Never (keep provisional cache) | Block ( ) | .798 | .848 | .688 | .721 |
| Early exit, trained | Late entry, trained | |||||
| Core passes / token | Acc. | tok/s | Acc. | tok/s | Gap (pts) | Sign-test |
| GSM8K (1,319 test problems; full-depth sequential: .393 at 21.7 tok/s) | ||||||
| ( -stg full) | .271 | 223 | .354 | 198 | +8.3 † | |
| ( -stg partial) | .271 | 223 | .369 | 215 | +9.7 † | |
| .342 | 174 | .395 | 158 | +5.3 † | ||
| .372 | 140 | .405 | 127 | +3.3 † | ||
Efficient Language Model Inference
Efficient language model inference reduces the compute, memory, latency, and energy needed to generate useful outputs.
See all 354 papers