Efficient Multi-Modal Planning with Reward-Guided Preference Optimization for Autonomous Driving

Chenglin Chen; Lujia Wang; Xinhu Zheng; Jun Ma; Haoang Li

cs.ROSep 30, 2026

Efficient Multi-Modal Planning with Reward-Guided Preference Optimization for Autonomous Driving

Authors: Chenglin Chen, Lujia Wang, Xinhu Zheng, Jun Ma, Haoang Li

Abstract

Safe and efficient trajectory planning is essential in autonomous driving. However, existing end-to-end approaches often fall short in both computational efficiency and safety guarantees. Methods based on imitation learning suffer from causal confusion, while rule-based scoring approaches often incur heavy computational overhead and suffer from objective misalignment. Additionally, preference-based methods rely on strict pairwise annotations, limiting data utilization. To overcome these limitations, we propose EMPlan, an efficient multi-modal trajectory planning method powered by reward-guided fine-tuning. We design a hybrid architecture that combines sparse anchors with an offset refinement module for efficient multi-modal trajectory prediction. Sparse anchors provide coarse trajectory candidates with low latency, which are subsequently refined by the offset module for higher prediction accuracy. To enhance safety without incurring additional inference costs, we adopt a two-stage training paradigm consisting of pretraining and reward-guided fine-tuning. During fine-tuning, we leverage rule-based reward signals and unpaired preference supervision to refine the pretrained policy toward safer trajectory selection. We evaluate EMPlan on the non-reactive NAVSIM benchmark, where it strikes a favorable balance between planning accuracy and efficiency, demonstrating superior performance under real-time constraints.

Figures & tables

Figure 1: Overview of our EMPlan, built upon a hybrid architecture and powered by reward-guided fine-tuning. (a) The hybrid architecture combines sparse anchors with an offset module for efficient multi-modal trajectory prediction. Reward-guided fine-tuning further aligns trajectory selection with safety preferences. (b) Our model achieves better performance in efficiency and accuracy compared with representative methods [ 2 , 3 , 11 , 12 ] . (c) Compared to score-based methods (i.e., WoTE [ 12 ] ), our model assigns lower scores to human-like but unsafe trajectories (red) and higher scores to safe ones (green).

Figure 2: Overview of the network architecture. Given multi-view images, the perception module extracts BEV representations, which are fed into a trajectory decoder to predict trajectory offsets and reward scores based on ego states and sparse anchors. The offset head refines anchor trajectories, and the trajectory with the highest aggregated reward is chosen as the final output. During training, the model is first pretrained with imitation and rule-based supervision and subsequently optimized via reward-guided fine-tuning, which integrates reward loss and preference loss to align trajectory selection with safety preferences.

Figure 3: Illustration of our two-stage training pipeline. In Stage I, $\pi_{\theta}$ is pretrained using expert demonstrations and rule-based scores to learn human-like driving priors. In Stage II, rule-based scores are combined to compute a trajectory-level reward loss across all candidates. In addition, unpaired preference data are used to compute preference loss (see Eq. ( 10 )), enabling the model to learn safer and more reliable trajectory selection.

Figure 4: Illustration of our construction of reward signal and preference data. The policy model pretrained in Stage I first generates candidate trajectories, which are then evaluated by an offline evaluator to obtain reward scores. The resulting rewards are used both for dense reward supervision and for constructing unpaired preference data by selecting $\mathcal{D}^{+}$ , $\mathcal{D}^{-}$ , $\mathcal{D}^{\textnormal{ref}}$ .

Method	Input	Supervision	Anchor	NC $\uparrow$	DAC $\uparrow$	EP $\uparrow$	TTC $\uparrow$	C $\uparrow$	PDMS $\uparrow$ (Overall)
Human	-	-	-	100.0	100.0	87.5	100.0	99.9	94.8
PDM-closed	Perception GT	-	-	94.6	99.8	89.9	86.9	99.9	89.1
Transfuser [ 2 ]	C & L	Human	-	97.7	92.8	79.2	92.8	100.0	84.0
UniAD [ 3 ]	C	Human	-	97.8	91.9	78.8	92.9	100.0	83.4
VADv2 [ 4 ]	C	Human	-	97.9	91.7	77.6	92.9	100.0	83.0
PARA-Drive [ 29 ]	C	Human	-	97.9	92.4	79.3	93.0	99.8	84.0

TABLE I: Performance comparison with representative methods on the navtest split. C: camera. L: LiDAR.

Figure 5: Qualitative comparison of predicted trajectories across challenging scenarios. We present qualitative comparisons with TransFuser [ 2 ] , Hydra-MDP [ 11 ] , and the SOTA WoTE [ 12 ] . In (a), both TransFuser and Hydra-MDP fail to respond properly to the red traffic signal, causing rear-end collisions with the leading vehicle. In addition, all baselines exhibit lane deviations in (b). WoTE tends to be overly conservative, resulting in stagnation behaviors in (c). In (d), TransFuser and Hydra-MDP proceed straight toward the construction obstruction, while WoTE avoids collision but fails to commit to the required right turn. In contrast, our EMPlan successfully executes all maneuvers.

Method	PDMS $\uparrow$	Latency $\downarrow$
Hydra-MDP [ 11 ]	86.5	104.7 ms
WoTE [ 12 ]	88.2	28.3 ms
EMPlan (Ours)	89.6	21.4 ms

TABLE II: Efficiency comparison between anchor-based methods. All models were tested on an NVIDIA RTX A6000.

Table 8

Figure 6: Visualization of reward distributions over candidate trajectories, colored by reward value. Unsafe trajectories (e.g., collisions or off-road driving, red circles) are consistently assigned low rewards, while feasible trajectories near the ground truth (yellow circles) receive higher rewards, demonstrating that the reward-guided policy effectively suppresses unsafe behaviors while preserving multimodal diversity.

Explore similar work

Jun 4, 2026cs.RO

Vault: One-Step Latent Generation with Positive-Anchored Rewards for Autonomous Driving

End-to-end autonomous driving must balance multimodal maneuver generation against real-time inference constraints. Diffusion planners capture diverse behaviors, but their iterative denoising incurs considerable inference cost in safety-critical deployment, while one-step alternatives still imitate only the single expert demonstration, so neither family certifies both the diversity and the quality of its outputs. To address this, we propose Vault, a framework that couples one-step latent generation with sample-based reinforcement learning guidance. Built on a V-JEPA 2.1 visual representation pretrained on the full NAVSIM training imagery, Vault generates trajectory candidates in a single forward pass by drifting in a VAE latent space. During training, a reward-gated positive pool collects the maximum-scoring trajectories that the model itself discovers under the official evaluator and anchors the drift targets to these certified samples. This closes a reinforcement-learning loop purely through sample-based target construction, without policy gradients, learned reward models, or extra interaction: the generator imitates the frontier of its own certified successes, so quality and multimodality improve together under a single fixed sampling configuration. A learned scorer that predicts the official score and its sub-metrics then selects the executed trajectory, and all training-only modules are discarded at inference. On the NAVSIM benchmarks, Vault achieves state-of-the-art performance at real-time inference cost, reaching 94.6 PDMS on v1 and 91.2 EPDMS on v2.

Yining Xing, Zehong Ke, Zhiyuan Liu +1

Causality-Aware End-To-End Autonomous Driving Autonomous Driving Autonomous Navigation Classical Planning Cognitive Science

Ask PDF

Apr 28, 2026cs.RO

ProDrive: Proactive Planning for Autonomous Driving via Ego-Environment Co-Evolution

End-to-end autonomous driving planners typically generate trajectories from current observations alone. However, real-world driving is highly dynamic, and such reactive planning cannot anticipate future scene evolution, often leading to myopic decisions and safety-critical failures. We propose ProDrive, a world-model-based proactive planning framework that enables ego-environment co-evolution for autonomous driving. ProDrive jointly trains a query-centric trajectory planner and a bird's-eye-view (BEV) world model end-to-end: the planner generates diverse candidate trajectories and planning-aware ego tokens, while the world model predicts future scene evolution conditioned on them. By injecting planner features into the world model and evaluating all candidates in parallel, ProDrive preserves end-to-end gradient flow and allows future outcome assessment to directly shape planning. This bidirectional coupling enables proactive planning beyond current-observation-driven decision-making. Experiments on NAVSIM v1 show that ProDrive outperforms strong baselines in both safety and planning efficiency, while ablations validate the effectiveness of the proposed ego-environment coupling design.

Chuyao Fu, Shengzhe Gan, Zhuoli Ouyang +5

1Southern University of Science and Technology · 2Hong Kong University of Science and Technology

Causality-Aware End-To-End Autonomous Driving Autonomous Driving World Model Planning Human Driving Planning Co-Evolution

Ask PDF

Jun 10, 2026cs.RO

ConsistencyPlanner: Real-time Planning with Fast-Sampling Consistency Models

Closed-loop planning in complex, real-world driving scenarios presents a critical challenge for autonomous driving systems. While traditional rule-based methods are interpretable, their predefined heuristics lack the adaptability for dynamic traffic environments. Learning-based approaches have shown considerable promise. Conversely, learning-based approaches, despite their promise, struggle to balance the modeling diverse and multimodal driving behaviors and real-time planning, often leading to indecisive or unsafe actions. To address this limitation, we propose Consistency Planner, a real-time planning framework with fast-sampling consistency models. Our approach is built upon two key technical contributions. Efficient Multimodal Sampling: We employ fast-sampling consistency models to generate a diverse set of plausible future trajectories. This enables efficient, real-time exploration of multimodal actions, overcoming the computational bottlenecks of previous iterative generative methods. Heterogeneous Feature Fusion: We introduce an attention-enhanced decoder that dynamically integrates heterogeneous input features (including scene feature and action token) into a cohesive representation for robust planning. Extensive evaluation in the Waymax simulator demonstrates superior performance in safety metrics compared to existing methods, with particularly strong results in challenging dynamic scenarios.

Qichao Zhang, Xing Fang, Jiaqi Fang +4

State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · Guangzhou Zaofu Intelligent Technology Co., Ltd. · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China

Planning

Ask PDF