World models simulate the consequences of action candidates, but good planning need not preserve every physical distinction required for accurate prediction. We formalize this gap through a hierarchy of mechanism, response, and decision sufficiency. Given a candidate set, the planning query determines which physical variations matter and how precisely they must be preserved: coarse decisions can discard much of the information needed for prediction, whereas fine decisions may require nearly the same resolution. In practice, planners often adaptively search to construct candidates, and information unnecessary for final selection may still be needed to discover good candidates. What a world model must preserve therefore depends on the query, the candidate set, and the planner. We study these effects in a collision system, nonlinear dynamics, and robotic planning. These varying requirements raise a design question: where should query information enter the planning system? A model that jointly generates actions and outcomes conditioned on the query achieves lower regret than an action-conditioned world model on seen objectives, but this advantage largely disappears when generalizing to unseen objectives. Motivated by this, we propose a modular design in which the query determines where to look and an action-conditioned model predicts what will happen, allowing the same predictions to be reused across objectives.
Figures & tables
Figure 1: Overview of experimental settings.
Figure 2: Query-dependent response–decision gaps. (a) Analytic cover complexity at matched tolerances Δ=2ϵ : the response–decision gap grows with precision for the threshold queries ( q1,q2 ) but remains bounded for tracking ( q3 ). (b) Learned rate–regret frontiers. Decision supervision requires fewer coded bits for the threshold queries, while no systematic saving appears for tracking.
Figure 3: Resolution and horizon shape information needs. (a,b) Median required coded rate at normalized regret δ=0.02 as candidate resolution and planning horizon vary. Shaded regions span the training seeds. (c,d) Example height trajectories and corresponding state snapshots for the true dynamics, learned model, and mean-mechanism baseline.
Figure 4: Candidate sets and planning stages shape information requirements. (a) Broader candidate sets amplify the gap between one-bit messages and full predictions. (b) One-bit messages preserve final selection but substantially degrade search. (c) Under a matched communication budget, richer feedback is more useful earlier in CEM search.
Figure 5: Supervision shapes planning and closed-loop behavior. (a) Push-T planning regret and mean closed-loop final cost (averaged over 128 episodes and 3 seeds). Full is uncompressed, others use 1-bit representations. (b) Representative rollouts under a common target query.
Figure 6: Query-guided proposals improve search efficiency. (a,b) Candidate outcomes from 32 proposals for the same scene and target. Rings mark the candidates selected by the learned planner; stars mark the lowest-cost candidates in each set for evaluation only. (c) Mean success across 512 new scenes and four target directions as the candidate budget varies. (d) Recorded cube positions along the selected plans in the illustrated scene.
Figure 7: Query placement across planning interfaces. ACWM is query-independent in both proposal and prediction; Joint conditions both on the query; and Modular guides proposals by the query while keeping physical prediction action-conditioned.
Figure 8: Query placement shapes performance across objectives. (a) Regret contrasts and decomposition on seen and held-out compositions. (b) Seen and held-out regret across the three interfaces. (c) Qualitative executions across objective compositions, including one held out from training.
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Planner operation
Required cost information
Why it is sufficient
Final selection
Minimizing candidate index
Determines the action selected from the current candidate set.
CEM update
Elite subset (top- k lowest-cost candidates)
The actions in the elite set determine the refitted sampling distribution.
MPPI update
Normalized weights wi∝exp(−Ji/λ)
The update averages candidate actions using these weights. Adding the same constant to every candidate cost leaves all normalized weights unchanged, so the absolute cost offset is unnecessary.
Appendix
Table 1: Information sufficient for representative planner operations, with candidate actions, planner state, and random draws fixed.
Component
Architecture and information access
Physical encoder
Physical input and public embeddings, followed by three width-256 SiLU/LayerNorm layers with separate symbol and length outputs.
Public embeddings
Query and known-condition embeddings, each with 128 dimensions. Fixed auxiliary features contain no world information.
Message decoder
Active codewords concatenated with zeros for inactive positions and a 16-dimensional length embedding, then projected to 128 dimensions.
Candidate head
Three-layer [message,public,action]→256→256→1 SiLU network, with a three-layer width-128 action embedding.
Entropy model
Two causal Transformer blocks with width 128, four heads, feed-forward width 512, and no dropout. The model predicts message length followed by active symbols conditioned on preceding symbols and public inputs.
Appendix
Table 2: Shared coded-rate architecture.
Stage
Maximum steps
Check interval
Checkpoint rule
Task-only training
8,000
500
Lowest native validation loss.
Each compression stage
4,000
500
Lowest refitted validation symbol NLL among checkpoints satisfying the native-loss tolerance.
Entropy refit
3,000
250
Lowest validation symbol NLL, with patience three.
Common scorer
8,000
500
Lowest validation decision loss, with patience five after 3,000 steps.
Appendix
Table 3: Training budgets and checkpoint selection for the Collision comparison.
q1 : velocity threshold
q2 : impulse threshold
q3 : tracking
δ≈
Rresp
Rdec
Sq
Rresp
Rdec
Sq
Rresp
Rdec
Sq
0.1
1.30
0.92
+0.38
1.21
0.93
+0.27
2.59
2.66
−0.08
0.01
1.98
1.36
+0.62
1.66
1.09
+0.58
5.53
6.55
−1.02
0.001
3.32
1.76
+1.56
2.74
1.13
+1.61
9.58†
9.97†
−0.32†
Appendix
Table 4: Required coded rates and paired response-minus-decision savings (bits per context). Each method uses five training seeds; † marks partial frontier support.
Training seed
Train end / best ckpt. (k steps)
RMSE (1 / 2 / 4 s)
Height MAE
K=8 acc. / regret
K=64 acc. / regret
0
30 / 30
0.019 / 0.040 / 0.188
0.0082
0.908 / 0.0003
0.574 / 0.0007
1
13 / 8
0.055 / 0.116 / 0.363
0.0206
0.798 / 0.0019
0.398 / 0.0025
2
11 / 6
0.072 / 0.125 / 0.465
0.0314
0.727 / 0.0032
0.355 / 0.0042
3
30 / 28
0.020 / 0.040 / 0.231
0.0084
0.912 / 0.0003
0.575 / 0.0007
4
11 / 6
0.059 / 0.114 / 0.407
0.0252
0.772 / 0.0026
0.369 / 0.0038
Mean-mechanism baseline
—
0.436 / 0.732 / 1.082
0.0845
0.576 / 0.0111
0.268 / 0.0141
Appendix
Table 5: Temporal-model training and autoregressive-rollout fidelity. RMSE is computed on standardized states, height MAE at T=4 s, and planning columns report winner accuracy / normalized regret.
Input
δ
Resolution at T=4 s: R ( K=2/8/64 )
Horizon at K=8 : R ( T=1/2/4 s)
Learned
0.05
0.000 / 0.048 / 0.059
0.022 / 0.029 / 0.048
Learned
0.02
0.016 / 0.080 / 0.782
0.039 / 0.046 / 0.080
Known
0.05
0.000 / 0.024 / 0.028
0.026 / 0.038 / 0.024
Known
0.02
0.059 / 0.055 / 0.656
0.047 / 0.060 / 0.061
History
0.05
0.000 / 0.034 / 0.025
0.029 / 0.030 / 0.034
History
0.02
0.022 / 0.093 / 0.929
0.052 / 0.105 / 0.093
Appendix
Table 6: Required coded rates as candidate resolution and planning horizon vary. Values are median required rates over matched supported seeds.
Figure 9: Autoregressive rollout fidelity. State RMSE over the rollout horizon, maximum-height MAE, and planning regret for individual training seeds (thin orange), their medians (thick orange), and the mean-mechanism baseline (blue).
Figure 10: Example autoregressive rollouts. Configuration snapshots and height trajectories for three predeclared test cases. We compare the true dynamics (black), learned model (orange dashed), and mean-mechanism baseline (blue). Faint trails show the preceding 0.6 s of motion.
Stage
Steps
Batch
Checkpoint rule
Encoder and native head
6,000
64
Lowest native validation loss after a 300-step warmup.
Common scorer
4,000
128
Lowest validation decision loss after a 200-step warmup.
Appendix
Table 7: Training budgets for Push-T message representations and their common scorer. Batch sizes count candidate sets.
Figure 11: Selection difficulty depends on the supplied candidate set. (a) Full predictions, Decision (1 bit), and Cost (4 bit) evaluated on identical candidate sets grouped by breadth. (b) Selection regret for Cost messages as message capacity increases.
Component
Training objective
Steps
Physical predictor
Terminal-feature prediction and physical-trajectory prediction
12,000
Query-aware / query-blind proposals
Cost-weighted mixture negative log likelihood
8,000
Appendix
Table 8: Training objectives and budgets for the PushCube proposal experiment.
Figure 12: Candidate quality and selection quality. (a) Realized candidate directions. (b) Costs of the learned-selected candidate (solid) and the best candidate in each set (dashed) across candidate budgets. (c) An example selection failure at N=128 . Gray denotes query-blind proposals and green denotes query-aware proposals.
Stage
Steps
Batch
Training setup
Encoder and native head
6,000
64 sets
End-to-end representation training
Common readout
4,000
64 sets
Frozen discrete codes
Appendix
Table 9: Training budgets for Push-T supervision experiments.
1 bit
8 bit
Value
Planner
Hybrid
Value
Hybrid
Regret
Jaccard
Regret
Jaccard
Regret
Jaccard
Regret
Jaccard
Regret
Jaccard
0.448
0.082
0.353
0.124
0.272
0.136
0.229
0.161
0.306
0.142
Appendix
Table 10: Push-T confirmation and capacity control. Lower regret and higher Jaccard are better.
Figure 13: Controls for supervision and planner settings. (a,b) Development elite agreement and planning regret. (c) Effect of additional training data. (d) Hybrid–Value regret across CEM candidate counts and elite fractions, where negative values favor Hybrid.
Figure 14: Example closed-loop outcomes. Recorded trajectories showing (a) a one-bit Hybrid advantage and (b) an eight-bit Value advantage. The green outline marks the scoring target, the gray shape the block, and blue the pusher.
Interface
Proposal
Physical predictor
Predictive exposure
ACWM
Query-blind
Action-conditioned, no query
Broad actions
Joint
Query-aware
Query-conditioned
Query-weighted actions
Modular
Query-aware
Action-conditioned, no query
Broad actions
Appendix
Table 11: Query placement and predictive training exposure for the three primary interfaces.
Comparison
Total regret
Candidate set
Selection
Modular − ACWM, held-out
−0.011
−0.024
+0.013
Modular − Joint, seen
−0.014
–
–
Appendix
Table 12: Regret decomposition and objective-shift controls.
ACWM
Joint
Modular
Joint+action
Params. (M)
Seen
Held-out
Params. (M)
Seen
Held-out
Params. (M)
Seen
Held-out
Params. (M)
Seen
Held-out
3.80
0.250
0.164
4.36
0.223
0.160
4.80
0.209
0.153
7.54
0.207
0.148
Appendix
Table 13: Interface size and planning regret. Regret is averaged within the seen and held-out objective groups.
Figure 15: Three interfaces under different objective compositions. Each column after Start shows a separately reset rollout from the same initial physical state. The final composition was held out from training.
Visual world models enable goal-directed control through decision-time action search, but their deployment efficiency is often limited by conservatively large planning budgets. We show that competitive task performance can be achieved without agreement with the Full-budget action, that sufficient budgets vary across model--task pairs, and that iterative planners repeatedly encode solve-invariant context. To address these inefficiencies, we propose {SufficientPlan}, a simple deployment framework that requires no modification to pretrained world models or planner updates. Its {Paired Sequential Budget Certification (PSBC)} component uses paired closed-loop evidence to search for and certify a reduced model--task-specific budget within a predefined Full-performance tolerance. Its {Static-Context Reuse (SCR)} component caches observation and goal representations across search iterations while preserving candidate-dependent planning and selected actions. Experiments across multiple world-model backbones and visual-control tasks show that SufficientPlan substantially reduces search budgets and planning latency while maintaining competitive control performance.
Changbai Li, Sirui Li, Yichen Yang +4
Beihang University · Chengdu University of Technology · Nanyang Technological University
A world model matters to an agent only through the state it constructs. That state must preserve some information, discard other information, and support some future function: prediction, control, planning, memory, grounding, or counterfactual reasoning. This paper treats world-model research as latent state design under sufficiency constraints. We propose a functional taxonomy that groups methods by what their latent state is for, rather than by architecture or application domain: predictive embedding, recurrent belief state, object/causal structure, latent action interface, grounded planning interface, and memory substrate. These roles expose distinctions that architecture-based groupings hide, including the gap between predictive sufficiency and control sufficiency, and the gap between passive video prediction and counterfactual action modeling. The taxonomy supports an evaluation framework that judges a model by the sufficiency constraint its latent state was built to satisfy. We compare methods along seven axes: representation, prediction, planning, controllability, causal/counterfactual support, memory, and uncertainty. We use the resulting matrix as a diagnostic for what a latent state preserves, discards, and enables. The conclusion that follows is that an actionable world model is the one whose state construction matches the task, not the one that preserves the most information.
Different research lines use the term world model in different ways, yet they share a common aim: to capture how the world evolves under action in a form that supports perception, simulation, and planning. Two prominent realizations are neural predictors that learn dynamics in continuous vector spaces, and hand-built physics engines that expose explicit state and physical laws. Neural predictors scale from data but leave the form of the dynamics implicit; physics engines are inspectable and editable but difficult to construct at scale. We introduce VisualPatchWorld (VPW), which represents world dynamics as code. VPW first selects a qualitative dynamical form with short active probes, then fits that form's free parameters from recorded state-action traces by minimizing multi-step prediction error. The resulting programs can be rolled forward like a simulator, inspected in source form, and used inside model-predictive control; image-derived scene graphs can supply the live state at replan time. Across comparisons with prior code-based world models, VPW attains 69.0% mean planning success and exceeds the strongest code baseline by 23.5 points. The largest gains arise when choosing the correct qualitative dynamics is essential. Under the same planner, the induced models approach ground-truth engine success on navigation and grasp-rich control; a residual gap remains for contact-rich pushing, and checking a shortlist of promising plans in the engine closes most of that gap. These results establish a practical route toward automatically constructed code world models that are useful for planning. Code is available at https://github.com/HKBU-KnowComp/VisualPatchWorld/.