Sampling-based planners such as MPPI and CEM query a learned world model thousands of times per control step, so low-bit world models are an obvious way to make model-based control cheaper on a robot. With round-to-nearest activation quantization and no calibration, we find that planning can collapse, by up to 100 points of success, while the world model's latent one-step prediction error stays no larger than that of a stochastic rounding scheme that plans well. At the same bit width, what decides the outcome is not the size of the rounding error but whether it persists across planner iterations and is shared across candidates. Round-to-nearest leaves an error that persists, and an iterating planner drifts toward the candidates that this error favors; in TD-MPC2, holding the same dither fixed within a step or an episode brings the drift back. Dither shared by all candidates and redrawn at every iteration removes most of the damage at the same bit width and without training: at least half of it in all 19 affected Meta-World cells (task × bit width) of TD-MPC2 (median 95%), in 4 of 5 DINO-WM cells, and in all 8 affected tasks of a 317M-parameter multi-task TD-MPC2. Captured as CUDA graphs on an Ampere GPU, a W8A4 integer TD-MPC2 world model runs its planning rollouts 1.9-3.3× and the whole planning step 1.3-1.5× faster than FP32, though not faster than BF16 with pre-converted weights; shared dither adds at most 2.3% latency. In TD-MPC2, block-scaled NVFP4 removes most of the damage and MXFP4 part of it, which ties the failure to coarse grids with a shared scale. Hypotheses, thresholds and analysis code were fixed before the data that test them, and the hypotheses that failed are reported.
Figures & tables
Hypothesis and test
Outcome
H1
det ’s latent one-step error ≤ sh-iter ’s in some affected cell of each family
Spearman(persistent share of det ’s error, red of sh-iter ) ≥0.5 over 12 tasks
holds: 0.83
H2a ′†
det drift <0 and drift( sh-iter )/drift( det ) <0.5
holds: 4/4 tasks (ratio ≤0.24 )
H2c
Spearman(candidate count, ∣OEIdet∣ and Ddet ) ≥0.8 , elites fixed
holds: 3/4 tasks
H4a
Spearman(table size of fixed dither, its damage) ≤−0.8
holds: 5/6 tasks (three exactly at −0.8 )
TABLE I: Hypotheses and outcomes. Each test was fixed before its data; rows marked † were added after the first stage and tested on new seeds; ‡ marks the extension experiments run after the main study. Mechanism tasks: mw-push, mw-pick-place, cheetah-run, PushT.
Fig. 1: Every affected cell. (a) Damage under sh-iter vs. det (success points on Meta-World and DINO-WM, relative return on DMC); below the dashed line sh-iter removes at least half of the damage. (b) det ’s extra damage against the ratio of the two latent one-step errors: most cells sit in the shaded region, where det ’s error is the smaller one.
Fig. 2: Mechanism. (a) Favorable drift of the population-mean error kept by each variant, relative to det (TD-MPC2 at its primary bit width; PushT at 4 bits, where sh-step and sh-ep coincide as sh-plan); bars show 95% bootstrap intervals. (b) Damage vs. MPPI iterations (TD-MPC2, 90 paired episodes per point, 95% intervals). Fresh dither needs iterations to help; det does not improve with them.
Fig. 3: Speed-up over FP32 on an NVIDIA A40, as CUDA graphs. (a) W8A4 world-model rollouts of one MPPI step ( N candidates, 6 iterations, horizon 3). (b) Whole planning step, W8A4 and BF16 with pre-converted weights: median over N , bars show the range. (a) and (b) are two runs on different A40s.
Planning with a learned latent world model is a promising route to control from raw pixels, but a strong world model alone is not enough. We show this experimentally: even with a perfect world model (operationalized by replacing the learned forward predictor with an idealized rollout of the true environment dynamics), a finite-budget sample-based planner still fails on some tasks, indicating that the bottleneck can lie in search rather than in world-model accuracy. Motivated by this gap, we propose IMWM (Intuition Model + World Model), which pairs the world model with an intuition model trained from demonstrations to recognize promising actions. The two models collaborate through three lightweight components: (i) Retrieval Initialization, which initializes the planner's action proposal from a retrieved demonstration; (ii) Hybrid Cost, which combines the intuition score with the world-model rollout cost; and (iii) a Reliability Gate, which adjusts how much the planner trusts intuition in each setting. Across four pixel-based goal-reaching tasks (Two-Room, Reacher, Push-T, and OGBench-Cube), IMWM has higher mean success than the world-model-only planner on all four, with the largest gains on Two-Room (99.2%, +11.5 percentage points) and OGBench-Cube (94.7%, +28.5 percentage points).
Baoqi Gao, Ruize Han, Miao Wang +1
Beihang University · Shenzhen University of Advanced Technology
Neural world models coupled with model predictive control (MPC) replan at every environment step to bound accumulated prediction error, but this incurs substantial computational overhead. Reusing a cached plan reduces this overhead, yet its effectiveness depends on how prediction mismatch propagates through the local dynamics. We analyze this trade-off with a perturbation-based dynamic-regret framework and show that stale-plan penalties scale with the reuse tolerance, the accumulated mismatch since the last replanning step, and the local dynamics sensitivity. Based on this structure, we propose AdaReP, a training-free wrapper that adapts the replanning tolerance online using the current deviation from the cached rollout and a local sensitivity estimate, without modifying the learned world model or planner. Across image-space planning, latent-space control, and real-world robotic manipulation, AdaReP substantially reduces planner-side computation while maintaining comparable task performance, including over 80% fewer queries on a 50-trial physical robot study.
Yutian Cheng, Xiaojian Ma, Xianhao Wang +6
Shanghai Jiao Tong University · Beijing Institute for General Artificial Intelligence (BIGAI) · University of Science and Technology of China
Visual world models predict the consequences of actions from images, but planning with them is typically done by optimizers such as CEM and MPPI. However, these optimizers stall on long-horizon tasks where progress requires motion the objective does not reward, such as moving around a block before pushing it. This paper revisits kinodynamic sampling-based motion planning for world models, using the learned model as the propagator of a tree search in latent space. Our key insight is that the sampling strategies devised for high-dimensional search in configuration space apply directly to the latent space of a world model, where they rapidly expand through intermediate states that myopic optimizers never sample. We present two exploration strategies for the high-dimensional latent space, nearest-neighbor-guided RRT and projection-based KPIECE, with GPU-native batched expansion for efficient device utilization. We evaluate with an off-the-shelf DINO-WM on four simulated domains and a real-world block-pushing task. Our approaches match or exceed the success rate of CEM and MPPI at up to 13.8x lower planning time in simulation, and improve the success rate by 20% on the real robot.