How Many Candidates Does a One-Step World Model Need? A Random-Codebook Account of Nearest-of-m Training on Frozen DINO Tokens
Organizations: School of Information and Communication Engineering, Chungbuk National University, Cheongju, Republic of Korea · School of Computer Science, Chungbuk National University, Cheongju, Republic of Korea
Abstract
One-step stochastic world models avoid the mode averaging of deterministic feature predictors without the tens of network evaluations per step that diffusion and flow matching require. The simplest such objective is nearest-of-m matching (conditional implicit maximum likelihood estimation, IMLE): draw m candidate futures and pull only the one closest to the observed outcome. Recent work conjectures that a small m suffices for feature-space world models and that a strictly proper energy score should replace the hard minimum. We test both claims with one-step world models on frozen DINOv2/v3 patch tokens, using benchmarks whose futures can be enumerated. Under mode-like uncertainty, nearest-of-m with m=16 covers the enumerated futures better than 50-step flow matching (at about 1/30 of its latency), one-step flow generators and an energy-score generator at matched parameter counts, and its rollouts stay on the token manifold where flow rollouts drift. One quantity governs the choice of m: the nearest-of-m objective is the expected distortion of a random codebook of m draws from the generator. This view yields three predictions, recorded before the corresponding runs: (i) memorization of the realized outcome in contexts that never repeat, whose strength is governed by the codebook's miss distortion I(m) rather than its miss probability, and which an energy-score generator does not escape by raising m; (ii) a flattening of mode probabilities with m, matching a known optimum within 0.01; and (iii) under-dispersion on continuous high-dimensional outcomes, which bounds the method on real robot video. Per-step matching on self-rollouts cuts long-horizon error by a third on a stochastic Push-T variant; an ablation separates its manifold and coverage benefits. Planning success does not improve despite accurate risk estimates, a negative result we report with its diagnosis.
Figures & tables
| Name | Source | Stochasticity | Repeat | Actions | Model |
|---|---|---|---|---|---|
| A1 Bouncing Shapes | FlowWM generator | normal bounce vs. reversal per collision | yes | none | 8M (joint 27–29M) |
| A2-div / A2-fix bounce | ours | directions at a wall; asymmetric | no / partly | none | 8M |
| Push (branch risk) | ours | (2 modes), ice (3), absorbing pit | no | 2-D | 8M |
| B stochastic Push-T | DINO-WM data, re-simulated | hidden center of gravity, 4 modes | no | 10-D | 21M |
| BAIR robot pushing | Ebert et al. (2017) | continuous, high-dimensional | no | 4-D | 21M |
| Method | NFE | Coverage | Precision | kNN recall | Det. err. (px) | Memo. index | ms/step | Train (h) |
|---|---|---|---|---|---|---|---|---|
| Regression | 1 | 0.00 | 4.4 | 0.3 | ||||
| FM-50 (FlowWM recipe) | 50 | 107 | 0.4 | |||||
| FM-2 (same model) | 2 | 6.5 | 0.4 | |||||
| iMF ( steps) | 1 | 3.4 | 3.2 | |||||
| Shortcut ( steps) | 1 | 4.3 | 0.7 | |||||
| Energy score, | 1 | 3.1 | 1.2 |
| Autoregressive | Joint 16 16 | ||||
|---|---|---|---|---|---|
| Method | NFE | probe | heatmap | probe | heatmap |
| Regression | 1 | ||||
| FM-50 | 50 | ||||
| FM-2 | 2 | ||||
| IMLE-WM, | 1 | ||||
| IMLE-WM, | 1 | ||||
| Method | NFE | precision | recall | precision | recall |
|---|---|---|---|---|---|
| GT spread (reference) | 30.8 | 48.3 | |||
| Regression | 1 | ||||
| FM-50 | 50 | ||||
| FM-2 | 2 | ||||
| iMF | 1 | ||||
| Stochastic Push-T | prec. / recall | prec. / recall | Train (h) |
|---|---|---|---|
| One-step matching | 25.7 / 25.4 | 41.6 / 43.7 | 1.3 |
| Per-step, feed matched | 18.8 / 18.4 | 24.6 / 29.3 | 5.3 |
| Paired change | / | / | |
| One-step matching, 12k steps | 21.8 / 22.1 | 34.1 / 37.8 | 2.2 |
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
| Chance | Predicted [interval] | Observed | |||
| 2 | 2 | 0.500 | 0.50 | 0.77 [0.74, 0.81] | 0.956 (miss) |
| 2 | 4 | 0.125 | 0.50 | 0.57 [0.54, 0.61] | 0.568 |
| 2 | 8 | 0.008 | 0.50 | 0.51 [0.50, 0.54] | 0.528 |
| 2 | 16 | 0.50 | 0.51 [0.50, 0.54] | 0.522 | |
| 16 | 4 | 0.234 | 0.0625 | 0.31 [0.25, 0.37] | 0.221 (miss) |
| 16 | 8 | 0.074 | 0.0625 | 0.15 [0.09, 0.21] | 0.144 |
| Randomization | Deviation (px) | Decision |
|---|---|---|
| Block mass – | 0.0 / 0.0 | no effect (quasi-static) |
| Contact friction – | 0.0 / 0.0 | no effect |
| Damping – | 25 / 346 | unrealistic sliding |
| Center of gravity, radius px, 4 positions + 3 px jitter | 28 / 138 | adopted |
| Center of gravity, radius px | 56 / 195 | too large |
| Variant | Coverage | Precision | kNN recall | Det. err. (px) | Memo. (train / val) | Train (h) |
|---|---|---|---|---|---|---|
| IMLE-WM , full noise injection, wide head | 0.934 [0.917, 0.949] | 0.83 | 0.94 | 2.31 | 0.29 / 0.24 | 1.93 |
| IMLE-WM , late fusion, wide head (adopted) | 0.925 [0.907, 0.942] | 0.83 | 0.95 | 1.56 | 0.29 / 0.27 | 1.05 |
| IMLE-WM , full injection, narrow head (6.6M) | 0.930 [0.909, 0.948] | 0.84 | 0.94 | 2.02 | 0.30 / 0.25 | 1.71 |
| IMLE-WM , pilot recipe (three spatial stages) | 0.91 [0.89, 0.93] | 0.81 | 0.94 | 2.67 | 0.30 / 0.22 | 3.2 |
| FM, default recipe, 20 Euler steps | 0.58 [0.55, 0.61] | 0.38 | 0.68 | 4.64 | – | 0.6 |
| FM, FlowWM recipe, 50 steps (adopted) | 0.79 [0.77, 0.82] | 0.51 | 0.91 | 2.27 | 0.29 / 0.24 | 0.4 |
| Joint 16 16 | Autoregressive | |||||
|---|---|---|---|---|---|---|
| Method | probe | heatmap | heatmap, noise 0.5 | probe | heatmap | heatmap, noise 0.5 |
| Regression | 5.14 | 22.48 | 9.67 | 8.09 | 8.41 | 8.53 |
| FM-50 | 3.86 | 12.59 | 15.00 | 12.28 | 14.11 | 18.38 |
| FM-2 | 3.83 | 14.80 | 16.43 | 9.95 | 12.49 | 16.85 |
| Energy score, | – | – | – | 8.94 | 33.77 | 11.65 |
| IMLE-WM, | 3.30 | 11.73 | 5.92 | 6.61 | 6.83 | 7.04 |
| 2 | 4 | 8 | 16 | 4 | 8 | 16 | 32 | 4 | 8 | 16 | 32 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Memo. index | 0.96 | 0.57 | 0.53 | 0.52 | 0.50 | 0.31 | 0.27 | 0.26 | 0.31 | 0.20 | 0.15 | 0.14 |
| Coverage | 0.66 | 0.92 | 0.97 | 0.99 | 0.65 | 0.89 | 0.92 | 0.90 | 0.41 | 0.58 | 0.64 | 0.62 |
| Precision | 0.78 | 0.85 | 0.85 | 0.68 | 0.69 | 0.76 | 0.77 | 0.72 | 0.54 | 0.62 | 0.64 | 0.60 |
| Det. err. (px) | 0.99 | 1.47 | 2.64 | 4.01 | 1.23 | 1.70 | 2.91 | 3.46 | 1.20 | 1.46 | 1.64 | 2.78 |
| Variant | Coverage | Precision | kNN recall | Memo. (train / held-out) | Det. err. (px) | Train (h) |
|---|---|---|---|---|---|---|
| 0.46 | 0.68 | 0.65 | 0.75 / 0.25 | 1.14 | 0.5 | |
| + RS-IMLE rejection ( ) | 0.46 | 0.69 | 0.65 | 0.74 / 0.25 | 1.17 | 0.5 |
| + context noise | 0.74 | 0.61 | 0.86 | 0.27 / 0.24 | 1.92 | 0.5 |
| + context noise | 0.88 | 0.77 | 0.92 | 0.32 / 0.22 | 1.65 | 0.7 |
| 0.93 | 0.82 | 0.94 | 0.29 / 0.25 | 1.55 | 1.1 |
| Method | Frequent-mode share | Theory | Share FM-50 (measured / theory) | Coverage | Memo. index |
|---|---|---|---|---|---|
| IMLE-WM, | 0.823 | 0.787 | / | 0.75 | 0.93 |
| IMLE-WM, | 0.651 | 0.607 | / | 0.96 | 0.68 |
| IMLE-WM, | 0.573 | 0.522 | / | 0.99 | 0.56 |
| FM-50 | 0.832 | 0.787 | – | 0.95 | 0.73 |
| FM-2 | 0.915 | – | / – | 0.80 | 0.78 |
| Energy score, | 0.886 | 0.787 | / | 0.58 | 0.91 |
| Method | NFE | best-of-16 ( ) | mean | manifold | spread | FD | recall | ||
|---|---|---|---|---|---|---|---|---|---|
| Regression | 1 | 0.268 | 0.481 | 0.556 | 0.556 | 1.27 | 0 | 1.44 / 2.65 | 0.21 / 0.08 |
| FM-50 | 50 | 0.338 | 0.681 | 0.843 | 0.885 | 1.78 | 0.94 | 0.53 / 1.38 | 0.73 / 0.84 |
| FM-2 | 2 | 0.302 | 0.590 | 0.734 | 0.756 | 1.45 | 0.72 | 1.42 / 2.75 | 0.26 / 0.08 |
| Shortcut | 1 | 0.401 | 0.902 | 1.173 | 1.197 | 1.71 | 1.48 | 0.95 / 1.77 | 0.42 / 0.26 |
| IMLE-WM, | 1 | 0.260 | 0.460 | 0.532 | 0.559 | 1.28 | 0.18 | 1.35 / 2.35 | 0.24 / 0.12 |
| IMLE-WM, | 1 | 0.258 | 0.456 | 0.527 | 0.576 | 1.30 | 0.29 | 1.22 / 2.09 | 0.28 / 0.15 |
| A2-div, | A1 Bouncing Shapes, | B Push-T, | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Method | manifold | best-of-16 | spread | manifold | best-of-16 | spread | manifold | best-of-16 | spread |
| Regression | 1.80 | 32.2 | 0 | 2.04 | 34.0 | 0 | 3.21 | 102.2 | 0 |
| FM-50 | 6.16 | 9.8 | 25.7 | 2.95 | 28.0 | 25.1 | 4.60 | 67.4 | 36.4 |
| FM-2 | 6.36 | 10.6 | 26.8 | 3.03 | 23.4 | 26.1 | 4.37 | 69.2 | 29.7 |
| iMF | 8.04 | 10.1 | 29.0 | 3.57 | 33.4 | – | 5.81 | 84.1 | 57.0 |
| IMLE-WM, | 1.74 | 10.5 | 23.9 | 1.98 | 19.0 | – | 3.20 | 88.7 | 9.2 |
| Comparison | Metric | Difference | Start-state CI | Hierarchical CI |
|---|---|---|---|---|
| IMLE-WM 16 vs. FM-50 | recall | |||
| precision | ||||
| recall | ||||
| precision | ||||
| IMLE-WM 16 vs. FM-2 | recall | |||
| precision |
| Setting | Method | Coverage | Precision | kNN recall | Memo. index | Det. err. (px) | Manifold |
| A2-div | Energy score, (3 seeds) | 0.36 | 0.61 | 0.77 | 0.87 | 1.06 | 1.19 |
| Energy score, | 0.28 | 0.61 | 0.60 | 0.97 | 1.06 | – | |
| Per-token energy score, | 0.22 | 0.55 | 0.44 | 0.97 | 1.33 | 1.12 | |
| IMLE-WM, (3 seeds) | 0.46 | 0.68 | 0.65 | 0.75 | 1.14 | 0.84 | |
| IMLE-WM, (3 seeds) | 0.93 | 0.82 | 0.94 | 0.29 | 1.55 | 0.86 | |
| A2-fix | Energy score, | 0.95 | 0.81 | 1.00 | 0.27 | 0.20 |
| Fed-back state | Manifold | Best-of-16 | Spread | F1 |
|---|---|---|---|---|
| One-step matching | 2.17 | 23.2 | 17.0 | 8.74 |
| Per-step, feed matched | 1.92 | 17.7 | 26.3 | 8.51 |
| Per-step, feed random candidate | 1.81 | 22.9 | 12.4 | 8.77 |
| Per-step, feed true frame | 2.14 | 23.6 | 16.6 | 8.62 |
| Stochastic Push-T | Success | s / step |
|---|---|---|
| Regression | % | 4.3 |
| FM-2 | % | 8.5 |
| FM-50 | 30% | 200 |
| IMLE-WM, | % | 4.0 |
World Model Learning
World model learning builds predictive models of how environments change under actions to support planning, simulation, and decision-making.
See all 456 papers