World-model planners evaluate candidate trajectories by predicting their future consequences, implicitly relying on the plan-to-motion relationship learned during training. At deployment, a different downstream controller may execute the same planned trajectory differently, creating a planning-execution gap between the frozen model's imagined future and actual execution. A natural remedy is to replace the planned trajectory with its predicted or observed execution. However, we find that better execution fidelity does not necessarily yield better decisions: directly substituting predicted execution trajectories yields only a 0.048 PDMS improvement across 10 unseen controller configurations, far smaller than the gain from our proposed adaptation. This suggests a semantic mismatch between a planned trajectory, which serves as a reference for downstream control, and an executed trajectory, which already describes the resulting post-controller motion. We introduce ExecutionBridge, which infers controller-specific execution characteristics from planned-executed trajectory pairs and uses them to adapt the native ego latent rollout while keeping the original planned trajectory unchanged. The planner, world model, and reward head remain frozen. On NAVSIM, ExecutionBridge consistently improves planning across unseen controller configurations and multiple frozen world-model planners, with substantially larger gains in challenging driving scenes where execution mismatch more strongly affects candidate evaluation.
Figures & tables
Fig. 1: Evolution of end-to-end planning paradigms. Model-free planners predict or score trajectories directly from the current scene, without explicitly modeling their future consequences. Model-based planners additionally use a learned world model to imagine candidate-conditioned futures for trajectory evaluation. Conventional model-based planning typically assumes that candidate trajectories are executed according to the plan-to-motion relationship learned during training. Execution-adaptive model-based planning further incorporates controller-specific information, allowing candidate futures to be evaluated under deployment-specific execution.
Fig. 2: Planning–execution gap and motivation for ExecutionBridge. (a) The same planned trajectory can lead to different executions at deployment from those implicitly assumed by the frozen world model, causing its imagined future to deviate from the future actually encountered. (b) Directly substituting the planned trajectory with its executed counterpart improves execution fidelity, but introduces a plan–execution semantic mismatch : the executed trajectory can still be interpreted through the model’s implicit plan-to-motion mapping, and therefore does not necessarily improve candidate ranking. (c) ExecutionBridge instead retains the original planned trajectory as the conditioning input and uses controller-specific execution evidence to adapt the imagined future used for candidate evaluation.
Fig. 3: Architecture of ExecutionBridge. (a) Controller Encoder. Given a support set of planned–executed trajectory pairs Sc , the Controller Encoder summarizes these observations into a compact controller representation ec that captures persistent controller-specific execution characteristics. An auxiliary execution decoder predicts the execution of a held-out query trajectory for training supervision and is discarded at deployment. (b) Controller-Conditioned Latent Bridge. For each candidate, the frozen world model first produces its native rollout from the original planned trajectory. Conditioned on ec , the Latent Bridge predicts a gated correction to the ego latent, which is bounded by a trust-region projection before being injected into the rollout. The resulting controller-conditioned rollout is evaluated by the unchanged reward model for candidate scoring.
Fig. 4: Representative planning results under the standard NAVSIM setting. Compared with frozen WoTE, ExecutionBridge yields trajectory selections that are better aligned with controller-dependent execution in representative collision and drivable-area failure cases.
Method
Input
#Traj.
Traj. Eval.
NC ↑
DAC ↑
EP ↑
TTC ↑
PDMS ↑
Human
–
–
–
100.0
100.0
87.5
100.0
94.8
Constant Velocity
–
1
×
69.9
58.8
49.3
49.3
21.6
Ego Status MLP [ 29 ]
–
1
×
93.0
77.3
62.8
83.6
65.6
VADv2 [ 6 ]
C
8192
Rule-based
97.9
91.7
77.6
92.9
83.0
UniAD [ 4 ]
C
1
Rule-based
97.8
91.9
78.8
92.9
83.4
LTF [ 30 ]
C
1
×
97.4
92.8
79.0
92.4
83.8
TABLE I: Planning performance across frozen world-model planners under the standard NAVSIM setting. For each backbone, ExecutionBridge is applied while keeping its native planning components frozen.
Full set
Low-score subset
Method
PDMS ↑
Δ
PDMS ↑
Δ
Frozen WoTE
87.976
–
32.038
–
Predicted execution
88.024
+0.048
36.528
+4.490
ExecutionBridge
88.589
+0.613
46.128
+14.090
TABLE II: Planning under controller-induced execution shift. Results are averaged over ten controller configurations.
Fig. 5: Planning under unseen controller shifts. Performance of the frozen planner and ExecutionBridge across ten unseen deployment controller configurations. (a) ExecutionBridge consistently improves PDMS across controller styles (mean: 87.98→88.59 , +0.61 ). (b) It also improves TTC (mean: 94.53→94.86 , +0.33 ). (c) The improvement is substantially larger on the low-score subset, where PDMS increases from 32.04 to 46.13 ( +14.09 ). These results indicate that controller-conditioned adaptation is particularly beneficial in challenging driving scenes where candidate evaluation is more sensitive to execution differences.
Fig. 6: Adaptation under representative controller styles. We show four representative controller configurations, including aggressive, conservative, steering-lag, and steering-gain behaviors. Controller-conditioned candidate evaluation consistently improves both PDM and Time to Collision (TTC) across these diverse controller styles. Examples on the right illustrate safety-critical cases in which accounting for controller-specific execution changes the selected trajectory and avoids failures under frozen evaluation.
Fig. 7: Execution fidelity does not imply decision fidelity. (a) Predicted and exact controller execution substantially reduce geometric execution error. (b–c) Nevertheless, candidate-ranking fidelity changes only marginally on the frozen-defined difficult-scene subset. (d) Direct execution substitution improves selected-candidate quality, but ExecutionBridge performs best by preserving the original plan and adapting its imagined future. Pairwise-oracle PDMS in (d) is a candidate-level diagnostic distinct from final navtest PDMS.
Method
Adapt.
ec
r^c,k
PDMS ↑
Low-score ↑
Frozen WoTE
–
–
–
87.976
32.038
Decision-only
ξ
–
–
88.071
35.214
Score calibration
s
✓
–
88.184
40.327
Residual-conditioned
ξ
✓
✓
88.312
45.550
ExecutionBridge
ξ
✓
–
88.589
46.128
TABLE III: Ablation of adaptation target and controller information. “Adapt.” denotes adaptation of the ego latent ξ or candidate score s ; ec is the controller representation and r^c,k the candidate-specific execution residual.
N
Exec. ADE ↓
Full PDMS ↑
Low-score PDMS ↑
64
1.521
88.445
45.901
128
1.522
88.441
45.955
256
1.521
88.441
45.954
1024
1.521
88.443
45.954
TABLE IV: Sensitivity to support-set size. Results over five held-out controller styles show stable execution and planning performance across N=64 – 1024 .
World-model controllers rely on action-conditioned dynamics for prediction and planning, yet real control systems often execute commands asynchronously due to communication delay, packet loss, reordering, and actuator buffering. We study how asynchronous execution changes the action semantics assumed within world-model controllers, rather than treating it only as an external control disturbance. Through controlled interventions, we identify two architecture-dependent failure modes: planning-based controllers such as TD-MPC2 suffer from a future-action timeline mismatch between imagined and executed action sequences, while recurrent world models such as DreamerV3 can attribute observed transitions to commands that were not actually applied. Our analysis shows that TD-MPC2 requires the correct future action sequence during latent dynamics rollout, whereas DreamerV3 requires timely attribution of each transition to the action that generated it. Based on these findings, we introduce two lightweight execution-consistent interfaces, Future-Sequence for TD-MPC2 and Applied-Action Feedback for DreamerV3, that correct these mismatches without modifying the pretrained world models. Experiments across delays, packet loss, reordering, multiple control domains, measured network traces, and a process-separated asynchronous stack consistently support both diagnoses and the corresponding architecture-specific corrections.
Shengtao Wen, Xiang Chen, Yu Tian +3
Nanjing University of Aeronautics and Astronautics, Nanjing, China · Tsinghua University, Beijing, China · Nanjing University, Nanjing, China
Latent world models are typically pre-trained offline and kept frozen at test time, so planning with them can degrade under distribution shift. Test-time adaptation (TTA) can mitigate this degradation by updating the model during deployment. Unlike pre-training on fixed data, TTA for planning is closed-loop: the planner uses model rollouts to select actions, and the resulting transitions drive the next update. This raises the question: which objective should guide TTA for planning? Under a prediction-only objective, the model is fit only to the outcomes of its own actions, so it can remain wrong about the actions that would reach the goal, and low prediction error need not translate into better planning. Inspired by reward-biased maximum likelihood from adaptive control, we introduce Optimistic Test-Time Adaptation (OpTTA). Its objective has two terms, the prediction loss on observed transitions and the planner's goal-reaching cost on its candidate rollouts, both minimized over the model parameters. The planning term biases adaptation toward models that are optimistic about reaching the goal, without any additional environment interaction. Across visual and environment shifts on manipulation, reaching and navigation tasks, OpTTA raises final planning success over the frozen model by 13.5 percentage points on average, compared with 6.9 for the prediction-only objective (AdaJEPA), when only the predictor is adapted. When the encoder is adapted as well, the gains are 13.5 and 7.5 points. These experiments span three JEPA world models with different encoders and training objectives and, for temporal straightening, both CEM and gradient-based planners. Our results suggest that test-time adaptation of world models for planning benefits from an optimistic, task-aware term in addition to the prediction loss.
Akshay Mete, Dileep Kalathil, P. R. Kumar
ECE Department, Texas A&M University, College Station, Texas, USA.
Neural world models coupled with model predictive control (MPC) replan at every environment step to bound accumulated prediction error, but this incurs substantial computational overhead. Reusing a cached plan reduces this overhead, yet its effectiveness depends on how prediction mismatch propagates through the local dynamics. We analyze this trade-off with a perturbation-based dynamic-regret framework and show that stale-plan penalties scale with the reuse tolerance, the accumulated mismatch since the last replanning step, and the local dynamics sensitivity. Based on this structure, we propose AdaReP, a training-free wrapper that adapts the replanning tolerance online using the current deviation from the cached rollout and a local sensitivity estimate, without modifying the learned world model or planner. Across image-space planning, latent-space control, and real-world robotic manipulation, AdaReP substantially reduces planner-side computation while maintaining comparable task performance, including over 80% fewer queries on a 50-trial physical robot study.
Yutian Cheng, Xiaojian Ma, Xianhao Wang +6
Shanghai Jiao Tong University · Beijing Institute for General Artificial Intelligence (BIGAI) · University of Science and Technology of China