LEV: Thinking Fast and Deep with Looped Decision Models
Organizations: Lehigh University · University of British Columbia · Vector Institute
Abstract
Decision models answer typed questions about an input with one probability per option, computed in a single forward pass without generating text. Like the fast System 1 of dual-process theory, they answer at once, but their depth is fixed at training time, so they spend the same computation on easy and hard questions. Models that reason in generated text can think longer, but they pay for every extra step in decoded tokens. We present LEV, a decision model that thinks deeper by rerunning its layers in a loop instead of generating tokens. LEV builds on a language model pretrained as a loop, reads the options after every loop step and trains all steps jointly, so one checkpoint can answer at any depth. On three looped backbones, accuracy rises with depth on the same weights, mostly on hard questions, and a 2.6B LEV goes from 63.2% at depth one to 76.2% at depth four on JevBench. On-demand depth, set per question by a small stopper with a threshold fitted on unlabelled requests, saves 15% to 24% of the computation at a cost of under one accuracy point, level with or slightly above a fixed mix of depths. On a locked test set, the 1.4B and 3B models are 5.4 to 8.8 points more accurate than dense (non-looped) models and cascades of the same latency below 50 ms.
Figures & tables
| System | Depth | Latency | Tuning | Held-out | Benchmark | ||||
|---|---|---|---|---|---|---|---|---|---|
| (ms) | Kev-Seen | Kev-Skills | Kev-Transfer | Kev-Breadth | LevBench | Avg. | JevBench | ||
| LEV (one checkpoint per model, read at every depth) | |||||||||
| 1 | 0 15 | 78.7 | 47.6 | 50.9 | 57.0 | 44.5 | 50.8 | 58.0 | |
| 2 | 0 30 | 83.7 | 58.2 | 60.4 | 63.4 | 53.0 | 58.2 | 65.4 | |
| 3 | 0 44 | 85.7 | 62.5 | 65.9 | 66.4 | 56.4 | 61.4 | 68.0 | |
| LEV-1.4B [-1pt] Ouro-1.4B | 4 | 0 58 | 85.9 | 63.7 | 66.4 | 66.0 | 58.1 | 62.1 | 68.0 |
| Model | Deepest | On-demand | Cost | vs. deepest | vs. frontier |
|---|---|---|---|---|---|
| Development | |||||
| LEV-1.4B | 61.3 | 61.0 | 0.78 | ||
| LEV-2.6B | 65.3 | 64.6 | 0.76 | ||
| LEV-3B | 62.1 | 61.3 | 0.85 | ||
| Locked test | |||||
| LEV-1.4B | 60.1 | 59.3 | 0.80 | ||
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
| Variant | Seeds | Kev-Seen | Kev-Transfer | Kev-Skills |
|---|---|---|---|---|
| Baseline accuracy | 3 | 81.1 | 56.4 | 34.1 |
| Default loop | 3 | |||
| 3 passes | 1 | |||
| 16 layers | 1 | |||
| Gate lr | 1 | |||
| Gate lr | 1 |
| Setting | Value |
|---|---|
| Adapter and head | |
| LoRA rank / / dropout | 16 / 32 / 0.05 |
| LoRA targets | attention + MLP |
| Pointer head dimension | 256 |
| Optimisation | |
| Optimiser | AdamW, weight decay 0.01 |
| Group | Panel | Suite, partition | Questions | Content |
| Tuning | Kev-Seen | decision-v7 , dev. | 1,464 | Ten public classification and inference datasets, generated policies and rules |
| Kev-Transfer | transfer-v9 , dev. | 1,264 | New sources such as MMLU-Pro and PAWS, held-out rule families, unknowable items | |
| Kev-Skills | hard-v1 , dev. | 1,083 | Seven programmatically labelled skill families, held-out templates | |
| Held-out | LevBench | ours, dev. | 4,370 | Tiers id , near , deep , and far by distance from the training data |
| Kev-Breadth | breadth-v1 , dev. | 3,023 | 14 datasets from five application areas, never trained on | |
| Benchmark | JevBench | public items | 231 | 48 easy, 72 standard, and 111 hard items |
| Part of the pool | Train-60k | Train-120k |
|---|---|---|
| Generated reasoning families | 10,222 | 31,621 |
| Six further public datasets | 7,028 | 21,741 |
| Public sources of Kev-Seen | 5,111 | 15,810 |
| New seeds, skill generators | 3,833 | 11,858 |
| ARC, OpenBookQA, CommonsenseQA | 2,466 | 7,630 |
| All | 28,660 | 88,660 |
| Kev-0.8B | Kev-4B | |||
|---|---|---|---|---|
| Ours | Harness | Ours | Harness | |
| Acc. (%) | 63.6 | 63.6 | 75.8 | 75.8 |
| Hard acc. (%) | 36.9 | 36.0 | 54.1 | 54.1 |
| Hard ECE | 0.172 | 0.181 | 0.112 | 0.112 |
| System | Params | Depth | Latency (ms) | Easy | Standard | Hard | All | Intelligence |
| LEV, one checkpoint per model read at every depth | ||||||||
| 1 | 15 | 100.0 | 70.8 | 31.5 | 58.0 | 41.7 | ||
| 2 | 30 | 100.0 | 80.6 | 40.5 | 65.4 | 51.6 | ||
| 3 | 44 | 100.0 | 81.9 | 45.0 | 68.0 | 55.2 | ||
| LEV-1.4B [-1pt] Ouro-1.4B | 1.4B | 4 | 58 | 97.9 | 84.7 | 44.1 | 68.0 | 55.7 |
| 1 | 51 | 97.9 | 76.4 | 39.6 | 63.2 | 48.1 | ||
| Model | Panel | Deepest | On-demand | Cost | vs. deepest | vs. frontier | ||
|---|---|---|---|---|---|---|---|---|
| LEV-1.4B | LevBench | 58.1 | 57.5 | 0.80 | ||||
| Kev-Breadth | 66.0 | 66.1 | 0.75 | |||||
| held-out, pooled | 61.3 | 61.0 | 0.78 | |||||
| JevBench | 68.0 | 67.1 | 0.59 | |||||
| LEV-2.6B | Kev-Seen | 86.1 | 85.9 | 0.44 | ||||
| Kev-Transfer | 72.5 | 71.5 | 0.71 | |||||
| Model | Panel (test) | Deepest | On-demand | Cost | vs. deepest | vs. frontier | ||
|---|---|---|---|---|---|---|---|---|
| LEV-1.4B | LevBench | 58.4 | 57.5 | 0.81 | ||||
| Kev-Breadth | 64.9 | 64.7 | 0.75 | |||||
| held-out, pooled | 60.1 | 59.3 | 0.80 | |||||
| Kev-Seen | 85.3 | 84.0 | 0.50 | |||||
| Kev-Transfer | 70.6 | 68.4 | 0.75 | |||||
| Kev-Skills | 70.5 | 69.5 | 0.86 | |||||
| Model | Panel | Deepest | On-demand | Cost | vs. deepest | vs. frontier | ||
|---|---|---|---|---|---|---|---|---|
| LEV-1.4B | documents-v1 | 87.1 | 87.0 | 0.57 | ||||
| devtools-v1 | 67.0 | 66.4 | 0.67 | |||||
| MMLU-Pro | 38.6 | 37.6 | 0.88 | |||||
| LEV-2.6B | documents-v1 | 88.7 | 87.9 | 0.43 | ||||
| devtools-v1 | 69.2 | 68.1 | 0.74 | |||||
| MMLU-Pro | 48.7 | 47.9 | 0.83 | |||||
| Tier | Depth 1 / 4 | On-demand | Cost | vs. frontier |
|---|---|---|---|---|
| id | 62.6 / 75.8 | 75.3 | 0.66 | |
| near | 42.0 / 71.8 | 69.8 | 0.79 | |
| deep | 34.5 / 45.4 | 44.9 | 0.87 | |
| far | 41.5 / 57.5 | 56.8 | 0.75 |
| Stopped at depth 1 | AUROC | |||
|---|---|---|---|---|
| Panel | Top prob. | Acc. | Right/wrong | Fixed/not |
| Calibration | 95.3 | 94.8 | 0.82 | 0.50 |
| Kev-Seen | 95.5 | 94.7 | 0.83 | 0.46 |
| Kev-Transfer | 93.3 | 78.9 | 0.72 | 0.60 |
| Kev-Skills | 88.8 | 81.6 | 0.67 | 0.57 |
| LevBench | 93.0 | 88.2 | 0.73 | 0.61 |
| AUROC | ECE, Kev-Breadth | |||
| Training set | LevBench | Kev-Breadth | Depth 1 | Deepest |
| LEV-1.4B | ||||
| Train-31k | 0.59 | 0.53 | 0.11 | 0.08 |
| Train-60k | 0.58 | 0.49 | 0.11 | 0.10 |
| Train-120k | 0.59 | 0.50 | 0.12 | 0.12 |
| full | 0.56 | 0.49 | 0.12 | 0.09 |
| LEV-1.4B | LEV-2.6B | LEV-3B | |||||||
| Rule | Acc. | Cost | vs. frontier | Acc. | Cost | vs. frontier | Acc. | Cost | vs. frontier |
| Rules from the early-exit literature | |||||||||
| top probability | 60.1 | 0.79 | 64.2 | 0.81 | 60.9 | 0.81 | |||
| margin | 59.6 | 0.75 | 63.8 | 0.78 | 60.3 | 0.77 | |||
| entropy | 59.8 | 0.80 | 62.8 | 0.76 | 58.2 | 0.66 | |||
| patience, | 58.7 | 0.62 | 62.9 | 0.63 | – | – | – | ||
| Test | Development | ||||
| Model | Compared with | LevBench | Kev-Breadth | LevBench | Kev-Breadth |
| LEV-1.4B | Qwen3-1.7B Qwen3.5-4B | ||||
| best dense option | |||||
| LEV-2.6B | Qwen3.5-2B Qwen3.5-4B | ||||
| Qwen3-1.7B Qwen3.5-4B | |||||
| Kev-0.8B Kev-4B | |||||
| LevBench | Kev-Breadth | JevBench | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Model | Training set | Deepest | On-demand | Cost | vs. frontier | Deepest | On-demand | Cost | vs. frontier | Deepest |
| LEV-1.4B | Train-31k | 58.1 | 57.5 | 0.80 | 66.0 | 66.1 | 0.75 | 68.0 | ||
| Train-60k | 70.8 | 70.0 | 0.67 | 66.7 | 65.9 | 0.75 | 69.7 | |||
| Train-120k | 73.0 | 71.9 | 0.61 | 65.2 | 64.8 | 0.75 | 68.0 | |||
| full | 57.9 | 56.8 | 0.73 | 63.0 | 63.0 | 0.74 | 68.8 | |||
| LEV-2.6B | Train-31k | 63.6 | 62.8 | 0.76 | 67.7 | 67.3 | 0.76 | 76.2 | ||
| Training set | Depth 1 | Deepest | Cost | vs. frontier |
|---|---|---|---|---|
| LEV-1.4B | ||||
| Train-31k | 49.6 | 61.3 | 0.78 | |
| Train-60k | 60.1 | 69.1 | 0.70 | |
| Train-120k | 61.3 | 69.8 | 0.67 | |
| full | 50.7 | 60.0 | 0.73 | |
| LEV-2.6B | ||||
| LevBench | Kev-Breadth | |||||
|---|---|---|---|---|---|---|
| Model | 31k | 60k | 120k | 31k | 60k | 120k |
| LEV-2.6B | 63.6 | 73.8 | 77.0 | 67.7 | 67.3 | 67.3 |
| Qwen3.5-2B | 53.1 | 66.3 | 69.9 | 64.7 | 64.3 | 65.1 |
| LEV-3B | 59.4 | 71.2 | 73.6 | 66.1 | 66.5 | 66.5 |
| Qwen3.5-4B | 62.1 | 71.8 | 74.2 | 67.4 | 69.4 | 69.4 |
| LevBench | Kev-Breadth | |||||
|---|---|---|---|---|---|---|
| Depth | Joint | Single | Joint | Single | ||
| 1 | 46.9 | 48.2 | 55.7 | 56.9 | ||
| 2 | 58.6 | 58.3 | 65.3 | 64.7 | ||
| 4 | 63.6 | 63.2 | 67.7 | 67.9 | ||
| Recipe | Depth 1 | Deepest | On-demand | Cost | vs. frontier |
|---|---|---|---|---|---|
| LEV-2.6B | |||||
| default | 50.5 | 65.3 | 64.6 | 0.76 | |
| Brier term | 51.2 | 65.9 | 65.2 | 0.78 | |
| 48.6 | 65.8 | 65.0 | 0.78 | ||
| last step only | 38.4 | 65.1 | 64.4 | 0.82 | |
| six steps | 50.0 | 66.2 | 65.7 | 0.67 | |
Efficient Language Model Inference
Efficient language model inference reduces the compute, memory, latency, and energy needed to generate useful outputs.
See all 318 papers