Real-Time Generative Policy via Langevin-Guided Flow Matching for Autonomous Driving

Tianze Zhu; Yinuo Wang; Wenjun Zou; Tianyi Zhang; Likun Wang; Letian Tao; Feihong Zhang; Yao Lyu

cs.LGMar 3, 2026

Real-Time Generative Policy via Langevin-Guided Flow Matching for Autonomous Driving

Authors: Tianze Zhu, Yinuo Wang, Wenjun Zou, Tianyi Zhang, Likun Wang, Letian Tao, Feihong Zhang, Yao Lyu,

Organizations: School of Vehicle and Mobility, Tsinghua University, Beijing, China · College of Artificial Intelligence, Tsinghua University, Beijing, China

Abstract

Reinforcement learning (RL) is a fundamental methodology in autonomous driving systems, where generative policies exhibit considerable potential by leveraging their ability to model complex distributions to enhance exploration. However, their inherent high inference latency severely impedes their deployment in real-time decision-making and control. To address this issue, we propose diffusion actor-critic with entropy regulator via flow matching (DACER-F) by introducing flow matching into online RL, enabling the generation of competitive actions in a single inference step. By leveraging Langevin dynamics and gradients of the Q-function, DACER-F dynamically optimizes actions from experience replay toward a target distribution that balances high Q-value information with exploratory behavior. The flow policy is then trained to efficiently learn a mapping from a simple prior distribution to this dynamic target. In complex multi-lane and intersection simulations, DACER-F outperforms baselines diffusion actor-critic with entropy regulator (DACER) and distributional soft actor-critic (DSAC), while maintaining an ultra-low inference latency. DACER-F further demonstrates its scalability on standard RL benchmark DeepMind Control Suite (DMC), achieving a score of 775.8 in the humanoid-stand task and surpassing prior methods. Collectively, these results establish DACER-F as a high-performance and computationally efficient RL algorithm.

Figures & tables

Symbol	Value	Symbol	Value
$\Delta a_{x}$	[-0.4, 0.25]	$\Delta\delta$	[-0.025, 0.025]
$a_{x}$	[-0.8, 0.8]	$\delta$	[-0.571, 0.571]
$\rho_{\text{lat}}$	1.5	$\rho_{\text{long}}$	3.0
$\rho_{\phi}$	0.3	$\rho_{\text{acc}}$	1.0
$\rho_{\text{steer}}$	0.1	$\rho_{\text{yaw}}$	0.7
$\rho_{\text{jerk}}$	0.5	$\rho_{\Delta\text{steer}}$	0.005

TABLE I: Hyper-parameters of reward and cost.

Symbol	Value
Actor learning rate $\beta_{F}$	$1\times 10^{-4}$
Two Critic learning rate $\beta_{Q}$	$1\times 10^{-4}$
Sample batch size	200
Replay batch size	512
Optimizer	RAD
Discount factor	0.99

TABLE II: Training hyper-parameters.

Fig. 1: Total average reward comparison during training.

Fig. 2: Arrival rate comparison during training.

Fig. 3: Collision rate comparison during training.

Panel of Figure 6 — Fig. 4: Visualization of an autonomous lane change maneuver executed by the DACER-F agent (red vehicle) in the multilane scenario. The sequence (a-e) shows the agent smoothly merging into the left lane to overtake a slower vehicle.

Panel of Figure 7 — Fig. 5: Visualization of a complex left-turn maneuver at the crossroad. The DACER-F agent (red vehicle) demonstrates social awareness by waiting for oncoming traffic (a), identifying a safe gap (b), and then proceeding to cross the intersection (c-e).

Panel of Figure 8 — Fig. 6: Computational performance analysis. (a) The average iteration time reflects the training speed. (b) The inference time measures the execution latency of the policy.

Panel of Figure 9 — Fig. 7: Evaluating curves on benchmarks with 5 random seeds. The solid lines represent the mean, while the shaded regions indicate the confidence interval over five runs.

Environment	DACER-F	DIPO	DSAC	QSM	DACER	SAC	SDAC	QVPO
Humanoid-stand	775.8 $\pm$ 12.7	8.7 $\pm$ 0.0	4.4 $\pm$ 0.4	8.1 $\pm$ 0.1	8.1 $\pm$ 0.4	6.9 $\pm$ 0.1	7.7 $\pm$ 0.0	7.6 $\pm$ 0.4
Humanoid-walk	226.5 $\pm$ 129.2	2.5 $\pm$ 0.1	1.1 $\pm$ 0.2	1.9 $\pm$ 0.0	1.9 $\pm$ 0.8	2.3 $\pm$ 0.0	1.8 $\pm$ 0.0	1.6 $\pm$ 0.1
Dog-trot	585.4 $\pm$ 28.8	449.9 $\pm$ 108.3	7.7 $\pm$ 3.2	187.5 $\pm$ 48.1	9.2 $\pm$ 1.4	10.0 $\pm$ 1.5	40.4 $\pm$ 34.2	27.3 $\pm$ 26.4
Dog-stand	879.2 $\pm$ 27.9	810.0 $\pm$ 27.5	25.1 $\pm$ 3.4	787.7 $\pm$ 28.2	25.6 $\pm$ 6.9	325.9 $\pm$ 253.8	639.0 $\pm$ 60.6	71.5 $\pm$ 61.3
Dog-walk	788.7 $\pm$ 37.4	707.4 $\pm$ 105.4	10.5 $\pm$ 2.0	458.3 $\pm$ 13.6	9.0 $\pm$ 0.7	162.4 $\pm$ 215.8	179.2 $\pm$ 138.1	31.1 $\pm$ 23.8
Dog-run	232.6 $\pm$ 27.3	167.3 $\pm$ 18.1	6.7 $\pm$ 1.4	157.0 $\pm$ 19.5	6.5 $\pm$ 0.5	71.5 $\pm$ 48.6	75.5 $\pm$ 48.4	6.2 $\pm$ 0.7

TABLE III: Average Return on DMControl benchmarks with 5 random seeds. Mean

\pm

Std. Bold = best; Higher is better.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,717 papers

DreamerAD: Efficient Reinforcement Learning via Latent World Model for Autonomous Driving
Mar 25, 2026Pengxuan Yang, Yupeng Zheng, Deheng Qian +11RL for Autonomous Driving Latent World Models
Institute of Automation, CAS · Chongqing Chang’an Technology Co., Ltd · School of Advanced Interdisciplinary Sciences, UCAS +1
RAD-2: Scaling Reinforcement Learning in a Generator-Discriminator Framework
Apr 16, 2026Hao Gao, Shaoyu Chen, Yifan Zhu +4Diffusion-Based Planning RL for Diffusion Models
Huazhong University of Science & Technology · Horizon Robotics
Vault: One-Step Latent Generation with Positive-Anchored Rewards for Autonomous Driving
Jun 4, 2026Yining Xing, Zehong Ke, Zhiyuan Liu +1End-to-End Autonomous Driving One-Step Generative Modeling

Mar 25, 2026J/K move · Enter open · S save

DreamerAD: Efficient Reinforcement Learning via Latent World Model for Autonomous Driving

Pengxuan Yang, Yupeng Zheng, Deheng Qian, Zebin Xing+10

Institute of Automation, CAS · Chongqing Chang’an Technology Co., Ltd · School of Advanced Interdisciplinary Sciences, UCAS+1

RL for Autonomous Driving Latent World Models Model-Based RL Diffusion Model Acceleration

Open paper AskPDF arXiv

We introduce DreamerAD, the first latent world model framework that enables efficient reinforcement learning for autonomous driving by compressing diffusion sampling from 100 steps to 1 - achieving 80x speedup while maintaining visual interpretability. Training RL policies on real-world driving data incurs prohibitive costs and safety risks. While existing pixel-level diffusion world models enable safe imagination-based training, they suffer from multi-step diffusion inference latency (2s/frame) that prevents high-frequency RL interaction. Our approach leverages denoised latent features from video generation models through three key mechanisms: (1) shortcut forcing that reduces sampling complexity via recursive multi-resolution step compression, (2) an autoregressive dense reward model operating directly on latent representations for fine-grained credit assignment, and (3) Gaussian vocabulary sampling for GRPO that constrains exploration to physically plausible trajectories. DreamerAD achieves 87.7 EPDMS on NavSim v2, establishing state-of-the-art performance and demonstrating that latent-space RL is effective for autonomous driving.

Abstract

Reinforcement Learning

Explore similar work

DreamerAD: Efficient Reinforcement Learning via Latent World Model for Autonomous Driving

RAD-2: Scaling Reinforcement Learning in a Generator-Discriminator Framework

Vault: One-Step Latent Generation with Positive-Anchored Rewards for Autonomous Driving