We present LAD, a real-time language--action planner with an interruptible architecture that produces a motion plan in a single forward pass (~20 Hz) or generates textual reasoning alongside a motion plan (~10 Hz). LAD is fast enough for real-time closed-loop deployment, achieving ~3x lower latency than prior driving language models while setting a new learning-based state of the art on nuPlan Test14-Hard and InterPlan. We also introduce RAD, a rule-based planner designed to address structural limitations of PDM-Closed. RAD achieves state-of-the-art performance among rule-based planners on nuPlan Test14-Hard and InterPlan. Finally, we show that combining RAD and LAD enables hybrid planning that captures the strengths of both approaches. This hybrid system demonstrates that rules and learning provide complementary capabilities: rules support reliable maneuvering, while language enables adaptive and explainable decision-making.
Figures & tables
Figure 1: Autonomous driving requires rule-following and semantic understanding. (Top row) A left turn at a pickup/dropoff zone: the ego vehicle (red, with planned trajectory) must navigate around vehicles blocking the lane. (Bottom row) A right turn through a crowded intersection: dense traffic from multiple directions, pedestrian crossings, and ambiguous right-of-way require reasoning beyond simple trajectory optimization. Text overlays show LAD ’s real-time situational understanding. The key insight is many scenarios labeled “hard” require only better lane-changing which our rule-based planner RAD can handle. Semantic difficulty demands language-grounded reasoning, e.g. negotiation of ambiguous traffic which LAD handles by generating both motion plans and interpretable explanations at ∼ 10Hz, enabling real-time deployment.
Figure 2: LAD Architecture. We encode scene context, adapt it into the language model’s manifold, and insert it as pseudo-tokens within the prompt. The decoder produces natural-language reasoning and a motion plan from the hidden state at <|plan|> .
Figure 3: Interruptible Anytime Inference. LAD produces a valid plan when interrupted, generating reasoning tokens if budget permits.
Figure 4: PDM-Closed’s [ 10 ] static proposal paths become blocked by obstacles with no recovery. RAD topologically replans at every timestep and augments the route with adjacent-lane centerlines.
Figure 5: RAD combines goal-directed optimization with trajectory proposal augmentation, adding feasible alternative trajectories and favoring trajectories that make progress toward the goal.
Figure 6: Scenario Understanding and Reasoning. (a) Cross-modal grounding : LAD can interpret pedestrians, lane geometry, and turn-safety from scene context. (b) Relational reasoning : LAD can infer spatial relations between agents and anticipates conflicts. (c) Conditional reasoning : LAD can evaluate counterfactuals, apply traffic laws, and identify when rules may be safely relaxed.
Type
Planner
Test14-Hard
InterPlan
Expert
Log Replay
85.96
–
Rule
IDM [ 42 ]
62.26
31
PDM-Closed [ 10 ]
75.19
42
RAD (Ours)
80.53
72
Learned
PlanTF [ 9 ]
61.61
32
PLUTO [ 8 ]
59.74
–
Table 1: Test14-Hard (Reactive) and InterPlan results. RAD rivals hybrid methods without learned components while LAD sets a new state-of-the-art among learned planners while offering textual reasoning ability.
Component
Test14-Hard (R)
PlanTF
61.61
+ 128 Objects
59.73
+ Static Objects
68.49
+ Plan Token
68.90
+ LLM/DrivingQA
69.75
+ PlanningQA ( LAD )
70.77
Table 2: LAD component ablations on Test14-Hard (Reactive). Each component contributes incrementally, with language supervision through DrivingQA and PlanningQA providing complementary gains on top of input and architectural improvements. Textual reasoning acts as a useful inductive bias for trajectory prediction.
Model
Reasoning
Latency (ms)
Hardware
DriveVLM [ 41 ]
Yes
410
Orin X–2
DriveGPT4-V2-8B [ 44 ]
No
2500
—
DriveGPT4-V2-1.5B [ 44 ]
No
345
—
DriveGPT4-V2-0.5B [ 44 ]
No
124
—
PlanTF [ 9 ]
No
12
A6000
DiffusionPlanner [ 50 ]
No
50
A6000
Table 3: LAD achieves real-time language-based planning. Without reasoning, LAD runs at 43 ms ( ∼ 20 Hz), comparable to recent closed-loop planners. With reasoning enabled (10 tokens), LAD operates at ∼ 10 Hz, demonstrating that language-based planning need not sacrifice latency for textual reasoning.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: LAD demonstrates strong spatio-temporal understanding when navigating a complex left-turn. In each case, it correctly identifies nearby pedestrians, crosswalks, and turning geometry, and produces a safe, smooth trajectory while articulating its high-level intent (e.g., “turning left,” “near pedestrian on crosswalk”). LAD ’s textual explanations remain consistent with the scene layout and the executed motion plan.
Figure 8: LAD handles diverse road configurations by accurately understanding lane topology, control rules, and surrounding agents. Its generated reasoning reflects this situational awareness (e.g., “on stopline stop sign,” “crossing an intersection”, “following lane without lead”), and its trajectory choices align with these high-level descriptions.
Figure 9: LAD produces safe and consistent trajectories through this complex roundabout while simultaneously articulating its high-level intent (e.g., “ego vehicle is traversing pickup-dropoff and is turning left”). In contrast, PlanTF [ 9 ] frequently hesitates or commits to suboptimal maneuvers. LAD ’s textual reasoning aligns with its chosen motion plan, providing interpretable justification for each action.
Figure 10: LAD correctly infers from the scene that the ego lane is blocked by static objects. It then selects the farther unblocked lane and executes a safe maneuver through the intersection, identifying relevant dynamic objects (“long vehicle”). In contrast, PlanTF [ 9 ] continues to follow the blocked lane or hesitates, failing to account for the static obstacles.
PDM-Closed
+Replan
+Vocab Aug.
+Adj. Centerlines
92.75
93.92
94.40
94.51
Appendix
Table 4: RAD Ablation on nuPlan Test14 subset
RAD
+Goal Opt.
+Rule Relaxation
84.31
90.22
92.99
Appendix
Table 5: RAD Ablation on InterPlan subset
Type
Planner
Val14 (R)
Expert
Log Replay
93.68
Rule
IDM [ 42 ]
79.31
PDM-Closed [ 10 ]
92.12
RAD (Ours)
92.31
Learned
PlanTF [ 9 ]
77.07
PLUTO [ 8 ]
80.01
Appendix
Table 6: Val14 (Reactive) results. RAD marginally improves over PDM-Closed, confirming that dynamic replanning and lane-change capability do not degrade nominal driving quality. LAD outperforms PlanTF among learned planners, demonstrating that language supervision preserves normal driving performance.
Planner
Controller
Val14 (R)
LAD
LQR
78.40
LAD
iLQR
82.47
Appendix
Table 7: Impact of Downstream Controller. Replacing the default LQR controller with iLQR improves LAD ’s score by 4.07, consistent with the hidden imitation gap arising due to not accounting for downstream controller [ 9 ] . Thus, controller-awareness is an important consideration for trajectory-based planners.
Val14
Test14-Hard
Method
Score
Human Gap
Score
Human Gap
PDM-Closed
93.20
0.80
75.19
10.77
RAD
92.31
1.69
80.53
5.43
Human (Expert)
94.00
–
85.96
–
Appendix
Table 8: Human-gap analysis across splits. Both RAD and PDM-Closed achieve near-human performance on Val14, but RAD maintains a significantly smaller gap on Test14-Hard, suggesting that RAD ’s lane-change and replanning capabilities generalize to long-tail scenarios without overfitting.
Figure 11: DrivingQA generation prompt. Template used to synthesize trajectory-grounded QA pairs from driving logs, for LAD ’s scalable language supervision pipeline.
Natural-language instructions promise controllable end-to-end driving, but their benefit can be hidden when planners already receive reliable high-level commands. We propose NudgeVAD, a frozen-planner residual framework that uses language as a calibrated nudge to a VAD trajectory. With identity-initialized FiLM and a zero-initialized residual head, NudgeVAD is equivalent to the frozen planner at initialization, so learned deviations arise only from language-conditioned residuals. We evaluate NudgeVAD along a command-reliability axis. With reliable commands, language improves the initial planner but becomes nearly redundant once compared against VAD-FT (UNCOND), a compute-matched VAD model fine-tuned without language. With random commands, however, language becomes essential: detaching text degrades ADE6s to 3.166 m, while NudgeVAD with text recovers 2.806 m and outperforms VAD-FT (UNCOND) by 0.312 m. These results show that language is not universally additive; it is most valuable when the categorical command channel is unreliable.
Vision-language models (VLMs) have emerged as a promising direction for end-to-end autonomous driving (AD) by jointly modeling visual observations, driving context, and language-based reasoning. However, existing VLM-based systems face a trade-off between high-level reasoning and motion planning: large models offer strong semantic understanding but are costly to adapt for precise control, whereas small VLM models can be fine-tuned efficiently but often exhibit weaker reasoning. We propose NaviDriveVLM, a decoupled framework that separates reasoning from action generation using a large-scale Navigator and a lightweight trainable Driver. This design preserves reasoning ability, reduces training cost, and provides an explicit interpretable intermediate representation for downstream planning. Experiments on the nuScenes benchmark show that NaviDriveVLM outperforms large VLM baselines in end-to-end motion planning.
Ximeng Tao, Pardis Taghavi, Dimitar Filev +2
J. Mike Walker ’66 Department of Mechanical Engineering, Texas A&M University, College Station, TX 77843, USA · The Department of Engineering Technology and Industrial Distribution Texas A&M University, College Station, TX 77843, USA
Recent advancements in Language Models (LMs) have demonstrated strong semantic reasoning capabilities, enabling their application in high-level decision-making for autonomous driving (AD). However, LMs operate over discrete token spaces and lack the ability to generate continuous, physically feasible trajectories required for motion planning. Meanwhile, diffusion models have proven effective at generating reliable and dynamically consistent trajectories, but often lack semantic interpretability and alignment with scene-level understanding. To address these limitations, we propose KnowDiffuser, a knowledge-guided motion planning framework that tightly integrates the semantic understanding of language models with the generative power of diffusion models. The framework employs a language model to infer context-aware meta-actions from structured scene representations, which are then mapped to prior trajectories that anchor the subsequent denoising process. A two-stage truncated denoising mechanism refines these trajectories efficiently, preserving both semantic alignment and physical feasibility. Experiments on the nuPlan benchmark demonstrate that KnowDiffuser significantly outperforms existing planners in both open-loop and closed-loop evaluations, establishing a robust and interpretable framework that effectively bridges the semantic-to-physical gap in AD systems.
Fan Ding, Xuewen Luo, Fengze Yang +4
Monash University Malaysia. · University of Utah, USA.