GuidedWAM: Training-Free Energy Guidance for World Action Models in Autonomous Driving
Authors: Xiyan Su, Finn Rasmus Schäfer, Hojun Lim, Frank Diermeyer, Johannes Betz, Markus Lienkamp
Organizations: Institute of Automotive Technology (FTM), Technical University of Munich, Germany · Munich Institute of Robotics and Machine Intelligence (MIRMI), Technical University of Munich, Germany · Professorship of Autonomous Vehicle Systems (AVS), Technical University of Munich, Germany
World action models (WAMs) for autonomous driving couple the driving policy and the visual world model in a single generative network whose objectives are fixed post-training. Conditioning on a new driving objective requires retraining. We study energy guidance as a general mechanism and present GuidedWAM, a model-agnostic framework for injecting driving objectives into frozen WAMs at sampling time. Our framework constructs each objective as differentiable energies on the sampler's intermediate clean-action estimate, which needs only forward access to the frozen model, and is stabilized by sampling-time guidance scheduling, per-energy normalization, and velocity-bounded correction. On the NAVSIM v2 hard-split, guidance improves all three frozen, publicly released WAMs by 1.8 to 3.6 EPDMS, averaged over three sampler seeds, and adds less than 10% to the sampling time per step. These results position energy guidance as a practical post-training control channel for WAMs.
Figures & tables
Method
Venue
Stage
NC ↑
DAC ↑
DDC ↑
TLC ↑
EP ↑
TTC ↑
LK ↑
HC ↑
EC ↑
EPDMS ↑
Diffusion and flow planners
TransFuser [ 4 ]
TPAMI’23
S1
96.2
79.5
99.1
99.5
84.1
95.1
94.2
97.5
79.1
25.1 ± 0.0
S2
77.7
70.2
84.2
98.0
85.1
75.6
45.4
95.7
75.9
DiffusionDrive [ 22 ]
CVPR’25
S1
96.8
86.0
98.8
99.3
84.0
95.8
96.7
97.6
79.6
27.5 ± 0.0
S2
80.1
72.8
84.4
98.4
85.9
76.6
46.4
96.3
72.8
GuideFlow [ 24 ]
CVPR’26
S1
96.6
80.5
96.3
99.3
82.3
94.9
91.5
97.7
67.8
27.1 ± 0.0
Table 1: NAVSIM v2 hard-split. Per-metric scores of stage one (S1, real scenes) and stage two (S2, synthetic scenes) and the combined EPDMS, in points. EG : energy guidance with a single camera-based drivable-area energy and no privileged information at inference, mean and standard deviation over three seeds. Stock rows are single runs. Baseline rows are the numbers published in the respective papers, except TransFuser from the NAVSIM v2 leaderboard, DiffusionDrive as reproduced in SimScale, and ReCogDrive and SGDrive as reproduced in the SimWAM paper. GuideFlow is its no-scorer setting. The stock DriveLaW, DriveWAM and SimWAM rows are our own runs of the released checkpoints.
Figure 1: Energy guidance against post-hoc projection on one NAVSIM v2 hard-split scene where both recover the DAC. Left : front camera with the three plans projected into the image. Right : bird’s-eye view (BEV) with the drivable energy field. The stock plan (grey) leaves the drivable area. The post-hoc projection (blue) follows it until the plan violates the constraint, causing sudden lateral change. Guidance (orange) corrects along the whole plan and reaches the same score with a smooth trajectory.
Model
EPDMS ↑
Δ
Off-road ↓
EC ↑
DriveLaW
stock
30.0
26.0
57.9
guidance
34.8
+4.8
22.7
58.0
projection
37.7
+7.7
16.6
51.2
DriveWAM
stock
35.9
20.9
73.0
guidance
39.2
+3.3
18.5
71.4
projection
40.3
+4.3
15.2
66.0
Table 2: Guidance vs post-hoc gradient projection with the same constraint. Both use the privileged drivable area with a margin of 0.5 m. Off-road is the fraction of synthetic scenes (stage 2) failing the DAC. EC is the stage-two two-frame comfort.
single energy EPDMS ↑
composition EPDMS ↑
Model
stock
LK
DA
DD
LK+DA
DA+DD
LK+DA+DD
DriveLaW
30.0
31.1
34.0
30.6
33.3
34.8
33.0
DriveWAM
35.9
37.5
39.2
36.4
39.5
40.0
39.6
SimWAM
37.8
37.9
39.2
37.8
38.2
39.6
38.9
Table 3: Composition of guidance energies on the privileged information. Lane keeping (LK): centerline attraction; drivable area (DA): drivable-area violation on the vehicle corners; driving direction (DD): vehicle center direction violation. The compositions use the best relative weights.
Figure 2: Guidance strength on single-camera-only drivable-area energy. EPDMS against the strength λ on a log axis. The dashed lines show the stock models. The open marker is the operating point reported in Table 1 .
Model
full-split best
cross-fitted
95% interval
DriveLaW
34.1
33.5
[+1.9,+6.3]
DriveWAM
38.2
37.7
[+0.5,+4.2]
SimWAM
40.0
40.0
[+0.8,+3.8]
Table 4: Cross-fitted selection of strength and margin over two halves of the 450 real frames of stage one, next to the full-split optimum. The last column is the full-split gain of the selected arm with a paired bootstrap interval, 10,000 draws. EPDMS in points.
Figure 3: Window shape against guidance strength on 40 DriveWAM DAC failure scenes under the stock sampler. Top : failure rate, the percentage of scenes whose final plan jitters more than three times the stock sampler. Bottom : ADE/FDE against the stock plan. The operating strength of DriveWAM is λ=4 .
Model
Form
EPDM ↑
guidance / step ↓
forward / step ↓
memory ↓
DriveLaW
Jacobian-free
34.1
0.1 ms
48 ms
8.2 GB
Jacobian
29.9
50.4 ms
42.4 ms
9.8 GB
DriveWAM
Jacobian-free
37.7
1.9 ms
36 ms
13.7 GB
Jacobian
36.3
72.5 ms
35.9 ms
14.3 GB
SimWAM
Jacobian-free
40.0
1.4 ms
16.3 ms
12.7 GB
Jacobian
40.0
14.3 ms
16.2 ms
12.7 GB
Table 5: Jacobian-free guidance against the full Jacobian form of Eq. ( 5 ), at the same strength and window. Times are per sampler step on one NVIDIA RTX 4090 GPU. Memory is the peak during sampling.
In autonomous driving, World-Action Models (WAMs) have improved end-to-end planning by transferring video dynamics priors to action prediction, but many still couple planning with future-video generation at inference, incurring substantial computational overhead. We present SimWAM, a simple yet effective WAM that leverages future-video prediction solely as a training-time supervision signal. It co-trains a pretrained video expert and a lightweight action expert with joint flow matching. An isolated attention mask keeps action prediction independent of future frames, allowing trajectory prediction without future-frame generation at inference. This design supports multiple pretrained video backbones and independent action-expert scaling within a shared attention interface, while preserving the joint learning objective. Moreover, we apply reinforcement learning to optimize a compositional driving reward beyond trajectory imitation. Experiments show that SimWAM achieves 91.9 PDMS on NAVSIM with a favorable trade-off between accuracy and latency among world-model-based planners, while transferring zero-shot to nuScenes. It also achieves competitive planning accuracy on WOD-E2E and PhysicalAI-Autonomous-Vehicles. These results position SimWAM as a plain yet solid baseline for efficient autonomous driving. The code and model weights are available at https://github.com/H-EmbodVis/SimWAM/.
Zongchuang Zhao, Xin Zhou, Tianyang Xu +6
Huazhong University of Science & Technology · Dongfeng Research & Development Institute
Direct-policy World Action Models (WAM) have demonstrated the benefits of decoupling action generation from explicit future visual prediction for efficient inference in autonomous driving and embodied control. However, they still require tens of denoising steps to achieve competitive action quality, incurring substantial latency and hindering real-time deployment. Simply reducing the number of denoising steps will collapse the nonlinear multi-step denoising trajectory into coarse updates, substantially degrading action quality. To reduce the denoising steps while enabling real-time action generation, we present an effective world action model with one-step action generation, namely OSWAM, for autonomous driving and embodied control. We argue that it is the absence of structured action priors in random noise initialization that forces direct-policy WAMs to recover coherent action sequences through iterative denoising. Hence, we construct imagination tokens to initialize the diffusion process, providing physical priors and action-conditioned representations. We further introduce privileged information distillation with future conditioning to enhance future-aware action generation. As a result, OSWAM can efficiently and effectively generate high quality actions in just one diffusion step. Our experiments demonstrate that OSWAM achieves comparable or even better results in both autonomous driving and embodied control than direct-policy WAM and Joint-WAM that require tens of steps. It obtains approximately 3× speedup over direct-policy WAM baselines and supports real-time compiled inference with latency as low as 40ms on an NVIDIA RTX 5090 GPU. The project page is available at https://gwenzhang.github.io/OSWAM/
We introduce Latent-WAM++, a representation-centered World Action Model (WAM) for end-to-end autonomous driving. Existing WAMs are primarily trained to predict future states or final actions, leaving the intermediate world and action representations that connect prediction to planning only indirectly constrained. Latent-WAM++ matches the supervision of each representation to its role in planning. World Representation Construction combines scene-ego future prediction, SIGReg-based distribution stabilization, and frozen geometric priors to learn a temporally predictive, non-degenerate, and geometry-aware world representation without an EMA target encoder. Trajectory MeanFlow Planner directly predicts multimodal trajectory proposals while being optimized with an Improved MeanFlow velocity-space objective, enabling one-step sampling. Visual-Action Representation Alignment further aligns the resulting pre-output action features with a stop-gradient target derived from current visual semantics. Latent-WAM++ achieves state-of-the-art performance on NAVSIM v2 and under zero-shot evaluation on HUGSIM, reaching 90.1 EPDMS on NAVSIM v2 and 47.0 average RC and 30.1 average HD-Score on HUGSIM. Training the planner end to end costs 190 GPU-hours for Latent-WAM++ versus 1,600 for Latent-WAM, a reduction of 88.1%.
Yichen Zhang, Yupeng Zheng, Linbo Wang +10
Institute of Automation, Chinese Academy of Sciences · Chongqing Changan Technology Co., Ltd.