cs.CVJul 12, 2026

Is Energy Guidance All You Need? Training-Free Norm Injection for Driving World Models

Authors: Xiyan SuFrank DiermeyerMarkus Lienkamp

Organizations: Institute of Automotive Technology, Technical University of Munich, Munich Germany

Abstract

Driving world models built on large video-diffusion backbones generate realistic scenes but are hard to control: enforcing a traffic norm typically means retraining the backbone or conditioning it on hand-built layouts. We ask whether controllability requires training at all. Our experiment shows that a rectified-flow driving world model, which jointly generates future video and a planned ego trajectory, can have its planned trajectory steered entirely at sampling time by differentiable energy functions that encode driving norms, without knowledge-specific retraining of the diffusion backbone. Concretely, we demonstrate that a world model built on Open-Sora 2.0 MM-DiT backbone can be steered to brake at a counterfactual target by injecting energy guidance at sampling time. However, we find that the generated video does not yet follow the steered trajectory through the backbone's joint self-attention and identify the cross-stream coupling as a crucial requirement for end-to-end-controllable rollouts.

Explore similar work

CardsList
  1. Orbis 2: A Hierarchical World Model for Driving

    Jul 17, 2026Sudhanshu Mittal, Arian Mousakhan, Silvio Galesso +4World ModelsSpatial Reasoning