cs.ROMar 30, 2026
SaveRAD-LAD: Rule and Language Grounded Autonomous Driving in Real-Time
Organizations: Carnegie Mellon University · NEC Labs America · UC San Diego
Abstract
We present LAD, a real-time language--action planner with an interruptible architecture that produces a motion plan in a single forward pass (~20 Hz) or generates textual reasoning alongside a motion plan (~10 Hz). LAD is fast enough for real-time closed-loop deployment, achieving ~3x lower latency than prior driving language models while setting a new learning-based state of the art on nuPlan Test14-Hard and InterPlan. We also introduce RAD, a rule-based planner designed to address structural limitations of PDM-Closed. RAD achieves state-of-the-art performance among rule-based planners on nuPlan Test14-Hard and InterPlan. Finally, we show that combining RAD and LAD enables hybrid planning that captures the strengths of both approaches. This hybrid system demonstrates that rules and learning provide complementary capabilities: rules support reliable maneuvering, while language enables adaptive and explainable decision-making.
Figures & tables
Figure 1: Autonomous driving requires rule-following and semantic understanding. (Top row) A left turn at a pickup/dropoff zone: the ego vehicle (red, with planned trajectory) must navigate around vehicles blocking the lane. (Bottom row) A right turn through a crowded intersection: dense traffic from multiple directions, pedestrian crossings, and ambiguous right-of-way require reasoning beyond simple trajectory optimization. Text overlays show LAD ’s real-time situational understanding. The key insight is many scenarios labeled “hard” require only better lane-changing which our rule-based planner RAD can handle. Semantic difficulty demands language-grounded reasoning, e.g. negotiation of ambiguous traffic which LAD handles by generating both motion plans and interpretable explanations at 10Hz, enabling real-time deployment.
Figure 2: LAD Architecture. We encode scene context, adapt it into the language model’s manifold, and insert it as pseudo-tokens within the prompt. The decoder produces natural-language reasoning and a motion plan from the hidden state at <|plan|> .
Figure 3: Interruptible Anytime Inference. LAD produces a valid plan when interrupted, generating reasoning tokens if budget permits.
Figure 4: PDM-Closed’s [ 10 ] static proposal paths become blocked by obstacles with no recovery. RAD topologically replans at every timestep and augments the route with adjacent-lane centerlines.
Figure 5: RAD combines goal-directed optimization with trajectory proposal augmentation, adding feasible alternative trajectories and favoring trajectories that make progress toward the goal.
Figure 6: Scenario Understanding and Reasoning. (a) Cross-modal grounding : LAD can interpret pedestrians, lane geometry, and turn-safety from scene context. (b) Relational reasoning : LAD can infer spatial relations between agents and anticipates conflicts. (c) Conditional reasoning : LAD can evaluate counterfactuals, apply traffic laws, and identify when rules may be safely relaxed.
| Type | Planner | Test14-Hard | InterPlan |
| Expert | Log Replay | 85.96 | – |
| Rule | IDM [ 42 ] | 62.26 | 31 |
| PDM-Closed [ 10 ] | 75.19 | 42 | |
| RAD (Ours) | 80.53 | 72 | |
| Learned | PlanTF [ 9 ] | 61.61 | 32 |
| PLUTO [ 8 ] | 59.74 | – |
Table 1: Test14-Hard (Reactive) and InterPlan results. RAD rivals hybrid methods without learned components while LAD sets a new state-of-the-art among learned planners while offering textual reasoning ability.
| Component | Test14-Hard (R) |
| PlanTF | 61.61 |
| + 128 Objects | 59.73 |
| + Static Objects | 68.49 |
| + Plan Token | 68.90 |
| + LLM/DrivingQA | 69.75 |
| + PlanningQA ( LAD ) | 70.77 |
Table 2: LAD component ablations on Test14-Hard (Reactive). Each component contributes incrementally, with language supervision through DrivingQA and PlanningQA providing complementary gains on top of input and architectural improvements. Textual reasoning acts as a useful inductive bias for trajectory prediction.
| Model | Reasoning | Latency (ms) | Hardware |
| DriveVLM [ 41 ] | Yes | 410 | Orin X–2 |
| DriveGPT4-V2-8B [ 44 ] | No | 2500 | — |
| DriveGPT4-V2-1.5B [ 44 ] | No | 345 | — |
| DriveGPT4-V2-0.5B [ 44 ] | No | 124 | — |
| PlanTF [ 9 ] | No | 12 | A6000 |
| DiffusionPlanner [ 50 ] | No | 50 | A6000 |
Table 3: LAD achieves real-time language-based planning. Without reasoning, LAD runs at 43 ms ( 20 Hz), comparable to recent closed-loop planners. With reasoning enabled (10 tokens), LAD operates at 10 Hz, demonstrating that language-based planning need not sacrifice latency for textual reasoning.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: LAD demonstrates strong spatio-temporal understanding when navigating a complex left-turn. In each case, it correctly identifies nearby pedestrians, crosswalks, and turning geometry, and produces a safe, smooth trajectory while articulating its high-level intent (e.g., “turning left,” “near pedestrian on crosswalk”). LAD ’s textual explanations remain consistent with the scene layout and the executed motion plan.
Figure 8: LAD handles diverse road configurations by accurately understanding lane topology, control rules, and surrounding agents. Its generated reasoning reflects this situational awareness (e.g., “on stopline stop sign,” “crossing an intersection”, “following lane without lead”), and its trajectory choices align with these high-level descriptions.
Figure 9: LAD produces safe and consistent trajectories through this complex roundabout while simultaneously articulating its high-level intent (e.g., “ego vehicle is traversing pickup-dropoff and is turning left”). In contrast, PlanTF [ 9 ] frequently hesitates or commits to suboptimal maneuvers. LAD ’s textual reasoning aligns with its chosen motion plan, providing interpretable justification for each action.
Figure 10: LAD correctly infers from the scene that the ego lane is blocked by static objects. It then selects the farther unblocked lane and executes a safe maneuver through the intersection, identifying relevant dynamic objects (“long vehicle”). In contrast, PlanTF [ 9 ] continues to follow the blocked lane or hesitates, failing to account for the static obstacles.
| PDM-Closed | +Replan | +Vocab Aug. | +Adj. Centerlines |
| 92.75 | 93.92 | 94.40 | 94.51 |
Table 4: RAD Ablation on nuPlan Test14 subset
| RAD | +Goal Opt. | +Rule Relaxation |
| 84.31 | 90.22 | 92.99 |
Table 5: RAD Ablation on InterPlan subset
| Type | Planner | Val14 (R) |
| Expert | Log Replay | 93.68 |
| Rule | IDM [ 42 ] | 79.31 |
| PDM-Closed [ 10 ] | 92.12 | |
| RAD (Ours) | 92.31 | |
| Learned | PlanTF [ 9 ] | 77.07 |
| PLUTO [ 8 ] | 80.01 |
Table 6: Val14 (Reactive) results. RAD marginally improves over PDM-Closed, confirming that dynamic replanning and lane-change capability do not degrade nominal driving quality. LAD outperforms PlanTF among learned planners, demonstrating that language supervision preserves normal driving performance.
| Planner | Controller | Val14 (R) |
| LAD | LQR | 78.40 |
| LAD | iLQR | 82.47 |
Table 7: Impact of Downstream Controller. Replacing the default LQR controller with iLQR improves LAD ’s score by 4.07, consistent with the hidden imitation gap arising due to not accounting for downstream controller [ 9 ] . Thus, controller-awareness is an important consideration for trajectory-based planners.
| Val14 | Test14-Hard | |||
| Method | Score | Human Gap | Score | Human Gap |
| PDM-Closed | 93.20 | 0.80 | 75.19 | 10.77 |
| RAD | 92.31 | 1.69 | 80.53 | 5.43 |
| Human (Expert) | 94.00 | – | 85.96 | – |
Table 8: Human-gap analysis across splits. Both RAD and PDM-Closed achieve near-human performance on Val14, but RAD maintains a significantly smaller gap on Test14-Hard, suggesting that RAD ’s lane-change and replanning capabilities generalize to long-tail scenarios without overfitting.
Figure 11: DrivingQA generation prompt. Template used to synthesize trajectory-grounded QA pairs from driving logs, for LAD ’s scalable language supervision pipeline.
More in this topic
Autonomous Driving Planning
Autonomous driving planning selects safe, feasible vehicle actions and trajectories amid traffic, uncertainty, road rules, and changing goals.
See all 384 papers