CarPLAN: Context-Adaptive and Robust Planning with Dynamic Scene Awareness for Autonomous Driving
Authors: Junyong Yun, Jungho Kim, ByungHyun Lee, Dongyoung Lee, Sehwan Choi, Seunghyeop Nam, Kichun Jo, Jun Won Choi
Organizations: Department of Artificial Intelligence, Hanyang University, 04763, Republic of Korea. · Interdisciplinary Program in Artificial Intelligence, Seoul National University, 08826, Seoul, Republic of Korea · Department of Electrical and Computer Engineering, Seoul National University, 08826, Seoul, Republic of Korea. · Department of Automotive Engineering, Hanyang University, 04763, Republic of Korea.
Imitation learning (IL) is widely used for motion planning in autonomous driving due to its data efficiency and access to real-world driving data. For safe and robust real-world driving, IL-based planning requires capturing the complex driving contexts inherent in real-world data and enabling context-adaptive decision-making, rather than relying solely on expert trajectory imitation. In this paper, we propose CarPLAN, a novel IL-based motion planning framework that explicitly enhances driving context understanding and enables adaptive planning across diverse traffic scenarios. Our contributions are twofold: We introduce Displacement-Aware Predictive Encoding (DPE) to improve the model's spatial awareness by predicting future displacement vectors between the Autonomous Vehicle (AV) and surrounding scene elements. This allows the planner to account for relational spacing when generating trajectories. In addition to the standard imitation loss, we incorporate an augmented loss term that captures displacement prediction errors, ensuring planning decisions consider relative distances from other agents. To improve the model's ability to handle diverse driving contexts, we propose Context-Adaptive Multi-Expert Decoder (CMD), which leverages the Mixture of Experts (MoE) framework. CMD dynamically selects the most suitable expert decoders based on scene structure at each Transformer layer, enabling adaptive and context-aware planning in dynamic environments. We evaluate CarPLAN on the nuPlan benchmark and demonstrate state-of-the-art performance across all closed-loop simulation metrics. In particular, CarPLAN exhibits robust performance on challenging scenarios such as Test14-Hard, validating its effectiveness in complex driving conditions. Additional experiments on the Waymax benchmark further demonstrate its generalization capability across different benchmark settings.
Figures & tables
Fig. 1: Challenges in imitation learning. Minimizing imitation loss alone does not guarantee safe planning. This example demonstrates that a lower L1 loss could result in lane departures or collisions.
Fig. 2: Overall structure of CarPLAN . CarPLAN comprises two main networks: DPE and CMD. DPE is trained to predict the displacements or velocity between the AV and surrounding scene elements at each future timestep, generating Displacement-Aware Features. CMD utilizes multiple experts, dynamically selected by the Scene-Aware Router, to generate the AV’s future trajectory.
Type
Planner
Venue
Val14
Test14-Hard
Test14-Random
OLS
CLS-NR
CLS-R
OLS
CLS-NR
CLS-R
OLS
CLS-NR
CLS-R
Learning
GameFormer [ 15 ]
ICCV’23
70.3
13.3
–
61.7
7.1
6.7
68.4
11.4
9.3
PDM-Open [ 2 ]
CoRL’23
85.9
50.2
54.8
79.1
33.5
35.8
84.1
52.8
57.2
UrbanDriver [ 51 ]
PMLR’22
84.2
68.6
64.8
77.2
50.4
50.0
81.2
51.8
67.2
PlanTF [ 34 ]
ICRA’24
88.7
85.3
77.1
84.9
69.7
61.6
87.8
85.6
79.6
PLUTO [ 18 ]
arXiv’24
89.3
88.9
80.0
81.4
74.0
59.7
87.8
89.9
78.6
TABLE I: Closed-loop simulation results on Val14, Test14-Hard, and Test14-Random. The “–” symbol means the metric is unknown.
Type
Planner
Val14
Test14-Hard
Test14-Random
NR
R
NR
R
NR
R
Expert
Log-replay
93.5
75.9
86.0
65.8
94.0
75.9
Rule
IDM [ 1 ]
75.6
77.3
56.2
62.3
70.4
74.4
PDM-Closed [ 2 ]
92.8
92.1
66.0
76.1
90.2
91.2
Learning
GameFormer [ 15 ]
82.9
83.8
68.7
67.1
83.9
82.1
PDM-Hybrid [ 2 ]
92.8
92.1
66.0
76.1
90.2
91.2
TABLE II: Performance on the nuPlan benchmark with post-processing.
DPE
CMD
NR ↑
Colli ↑
DAC ↑
SSE
SAR
75.5
88.5
96.3
✓
77.0
90.3
97.6
✓
76.5
89.0
96.8
✓
✓
77.4
90.6
98.1
✓
✓
✓
78.6
92.1
98.5
TABLE III: Ablation study of CarPLAN. SSE: Scene-Specific Experts, SAR: Scene-Aware Router.
Displacement Target
NR ↑
Colli ↑
DAC ↑
No Target
77.4
90.6
98.1
Agent Only
78.1
91.6
98.0
Map Only
77.9
91.2
98.3
Agent + Map
78.6
92.1
98.5
TABLE IV: Ablation study for DPE target.
Expert
NR ↑
Colli ↑
DAC ↑
Routed
TopK
Shared
16
2
-
77.3
90.5
98.0
16
4
-
77.5
90.7
98.1
16
2
✓
78.6
92.1
98.5
16
-
✓
77.9
91.2
98.3
16
1
✓
77.2
90.5
97.7
TABLE V: Ablation study for CMD.
Method
CLS-NR ↑
FPS ↑
FLOPs (G) ↓
Baseline
75.5
17.74
0.94
Baseline + DPE
77.0
17.74
0.94
Baseline + CMD
77.4
14.94
1.08
CarPLAN
78.6
14.94
1.08
TABLE VI: Inference Efficiency Analysis
Fig. 3: Visualization of expert selection scores across layers. The dark red vehicle represents the AV, while the yellow and red lines indicate the ground truth (GT) and the predicted future trajectory with the highest score, respectively. The softmax scores are displayed for layers 2, 3, and 4. (a) In two distinct straight-driving scenarios, differences in the distribution of surrounding agents lead to distinct expert selections. (b) In similar driving scenarios, expert selection remains mostly consistent.
Method
AR ↑
OR ↓
CR ↓
PR ↑
Baseline
87.52
1.77
2.61
95.34
Baseline + DPE
88.21
1.52
2.47
95.89
Baseline + CMD
88.56
1.60
2.51
96.32
CarPLAN
89.87
1.51
2.43
97.01
TABLE VII: Ablation study on the Waymax benchmark.
Fig. 4: Qualitative results on closed-loop simulations in nuPlan benchmark. The yellow trajectory represents the recorded actual trajectory of the vehicle, while the red trajectory indicates the model’s predicted trajectory with the highest probability at each timestep. The dark red vehicle represents the AV, and black vehicles or pedestrians that turn red signify a collision occurrence. Red dashed boxes highlight critical events observed during the simulation.
Fig. 5: Qualitative results on closed-loop simulations in Waymax benchmark. The blue dots represent the trajectory executed during the closed-loop simulation. The blue bounding box denotes the AV, while gray bounding boxes indicate surrounding vehicles and boxes that turn red indicate a collision event during simulation. Green circles correspond to traffic lights in the green phase. Dark gray dots illustrate the road layout, and light gray dots represent centerlines. Red dashed boxes highlight critical events observed during the simulation.
Learning motion planning policies that remain reliable under closed-loop execution is a central challenge in Autonomous Driving (AD). Imitation Learning (IL) scales well to large AD datasets but suffers from distribution shift, causal confusion, and compounding errors under closed-loop execution. In contrast, Reinforcement Learning (RL) typically requires handcrafted rule-based rewards and massive simulation. We study a simple closed-loop IL framework that uses RL purely as an optimization tool, guided by a trajectory-distance reward derived from logged expert data. We refer to this training strategy as Expert-Guided RL (EGRL). Through a controlled empirical study we find evidence that the training procedure, rather than model size and complexity, accounts for a substantial part of the performance gains. We then introduce DriveMLP, a lightweight MLP policy trained with EGRL, as a simple instantiation of these findings. It achieves a score of 86.51 on the nuPlan Val14 Reactive benchmark, a +3.2 point improvement over the previous best IL planner (Flow Planner), with >15x faster inference. Code is available at: https://github.com/alessandro-canevaro/DriveMLP .
Alessandro Canevaro, Julian Schmidt, Hang Yu +4
Mercedes-Benz AG, Research & Development, Sindelfingen, Germany · University of Tübingen, Tübingen, Germany · Karlsruhe Institute of Technology, ITIV, Karlsruhe, Germany +2
Autonomous driving planning is a key component of IoT-enabled intelligent transportation systems, requiring vehicles to generate safe, efficient, and executable trajectories in complex urban environments from multi-source contextual information. While imitation learning (IL) has shown promise on large-scale datasets, IL-based planners still suffer from limited coverage of complex long-tail interactions, weak consistency with downstream constrained refinement, and insufficient use of high level scene semantics under real time constraints. To address these issues, this paper proposes a large language model (LLM) enhanced differentiable trajectory planning framework for IoT-enabled autonomous driving. Specifically, we introduce a surrounding agent centric data augmentation strategy to reorganize sur rounding agent trajectories as additional planning supervision, thereby improving the training distribution without collecting additional raw data. We further design a complexity-aware asyn chronous LLM-based semantic enhancement module to extract scene-related high-level semantic features with controlled online overhead. In addition, a differentiable optimization module is incorporated to refine generated trajectories with explicit residual penalties while backpropagating optimization gradients to the upstream planner. Experiments show that the proposed method achieves the best overall scores of 83.63 and 78.29 on the nuPlan closed-loop nonreactive and reactive Hard20 benchmarks, respectively, and CARLA-ROS tests further verify its online deployment and real time closed-loop execution capability.
Shihao Zhang, Jing Yang, Ziyu Song +7
State Key Laboratory of Automotive Simulation and Control, Jilin University, Changchun 130000, China · Center of Research for Cyber Security and Network (CSNET), Faculty of Computer Science and Information Technology, Universiti Malaya, 50603 Kuala Lumpur, Malaysia · Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg, Luxembourg +3
End-to-end autonomous driving planners typically generate trajectories from current observations alone. However, real-world driving is highly dynamic, and such reactive planning cannot anticipate future scene evolution, often leading to myopic decisions and safety-critical failures. We propose ProDrive, a world-model-based proactive planning framework that enables ego-environment co-evolution for autonomous driving. ProDrive jointly trains a query-centric trajectory planner and a bird's-eye-view (BEV) world model end-to-end: the planner generates diverse candidate trajectories and planning-aware ego tokens, while the world model predicts future scene evolution conditioned on them. By injecting planner features into the world model and evaluating all candidates in parallel, ProDrive preserves end-to-end gradient flow and allows future outcome assessment to directly shape planning. This bidirectional coupling enables proactive planning beyond current-observation-driven decision-making. Experiments on NAVSIM v1 show that ProDrive outperforms strong baselines in both safety and planning efficiency, while ablations validate the effectiveness of the proposed ego-environment coupling design.
Chuyao Fu, Shengzhe Gan, Zhuoli Ouyang +5
1Southern University of Science and Technology · 2Hong Kong University of Science and Technology