AutoAgent0: An Agentic Runtime for Safe Closed-Loop Driving

Jiageng Liu; Seth Z. Zhao; Zewei Zhou; Alejandro Huerta; Aidan Bayer-Calvert; Yuxin Bao; Haotian Lin; Robert Spataru

cs.ROOct 13, 2026

AutoAgent0: An Agentic Runtime for Safe Closed-Loop Driving

Authors: Jiageng Liu, Seth Z. Zhao, Zewei Zhou, Alejandro Huerta, Aidan Bayer-Calvert, Yuxin Bao, Haotian Lin, Robert Spataru,

Organizations: UCLA · CMU · UT Dallas · Stanford

Abstract

Safe autonomous driving requires more than accurate planning at a single frame: it demands reliable closed-loop execution in dynamic environments. Although end-to-end driving policies have advanced rapidly, the accumulated closed-loop errors, especially in unseen scenarios, can push vehicles into states from which the same policy struggles to recover. We address the challenge by treating closed-loop driving safety as a test-time execution problem. We introduce AutoAgent0, an agentic runtime and harness framework that augments existing driving experts with complementary detection skills, rule-based action verification, and feedback-driven recovery. To address inaccurate scene perception, an independent detection module constructs scene context using complementary perception skills, and a rule-based verifier checks proposed trajectories against this context before execution. When the nominal trajectories are unreliable, the orchestrator of the runtime activates Recovery Planning, where an agentic loop constructs alternative action proposals from action primitives and iteratively refines them using verification feedback. This design combines independent scene assessment with feasibility checks and online recovery, addressing both perception errors and the limited recovery capabilities of individual experts. Evaluations on Fail2Drive and NavSafe- $\infty$ demonstrate improved closed-loop robustness over single-expert execution in long-tail and safety-critical scenarios. These results demonstrate that runtime supervision can improve closed-loop robustness across multiple driving experts without retraining the underlying policies.

Figures & tables

Figure 1: Motivation and overview. Inaccurate scene context and limited generalization to off-nominal states motivate runtime supervision. AutoAgent0 combines scene refinement, trajectory verification, and recovery planning. Top: a Fail2Drive case illustrates yielding and bypass recovery. Bottom: a NavSafe- $\infty$ example compares standalone and AutoAgent0 -augmented expert.

Figure 2: Overview of the AutoAgent0 runtime. AutoAgent0 surrounds the nominal end-to-end policy with a) scene context refinement, which uses perception skills to augment the evidence available to a rule-based verifier, and b) agentic recovery planning, which generates and revises recovery planning trajectories when no nominal proposal is admitted. The verifier evaluates both nominal proposals and recovery planning trajectories before commitment to the tracking controller.

Figure 3: Agentic Recovery Planning. Rejection of nominal proposals initiates action primitive selection and parameterized trajectory generation. The agent uses scene context and verifier feedback to revise recovery planning trajectories before execution. During execution, new observations can trigger further revision, with each replacement trajectory checked before commitment.

	Inputs		In-Distribution			Generalization			Average
Method	RGB	LiDAR	DS $\uparrow$	SR (%) $\uparrow$	HM $\uparrow$	DS $\uparrow$	SR (%) $\uparrow$	HM $\uparrow$	DS $\uparrow$	SR (%) $\uparrow$	HM $\uparrow$
TCP ( Wu et al., 2022 )	✓	✗	24.7	39.1	30.3	24.5	31.4	27.5	24.6	35.3	28.9
Orion ( Fu et al., 2025 )	✓	✗	53.0	52.0	52.5	51.2	46.0	48.5	52.1	49.0	50.5
HiP-AD ( Tang et al., 2025 )	✓	✗	74.1	70.7	72.4	67.1	56.7	61.5	70.6	63.7	67.0
SimLingo ( Renz et al., 2025 )	✓	✗	82.6	79.3	80.9	71.7	55.0	62.2	77.2	67.2	71.6
AlignDrive ( Wu et al., 2026 )	✓	✗	74.7	75.7	75.2	68.6	62.7	65.5	71.7	69.2	70.4

Table 1: Benchmark Results on Fail2Drive. HM is the harmonic mean of DS and SR within each split; Average is the arithmetic mean of the two splits for each metric. Parentheses in Average indicate absolute improvements over the corresponding base method.

Method	Closed-Loop Metrics				Ability Scores
	DS $\uparrow$	SR(%) $\uparrow$	DE $\uparrow$	Comf. $\uparrow$	TC $\uparrow$	VRUC $\uparrow$	TV $\uparrow$	TI $\uparrow$
DrivoR ( Kirby et al., 2026 )	64.7	50.7	188.0	87.9	58.7	31.7	52.4	43.3
GTRS-Dense–V2-99 ( Li et al., 2025a )	64.3	49.5	163.9	95.9	52.0	40.0	54.2	36.7
AutoVLA ( Zhou et al., 2025a )	51.1	29.5	186.0	65.4	33.0	15.0	29.7	36.7
RAP ( Feng et al., 2026 )	47.7	27.3	220.8	48.4	32.3	14.2	28.8	22.2
MTDrive–SFT ( Li et al., 2026b )	48.8	31.8	173.8	94.5	30.3	15.0	35.8	44.4

Table 2: Closed-loop Evaluation on NavSafe-

\infty

. All reported values are three-seed means over the 280-scenario evaluation set. Parentheses indicate absolute improvement over the corresponding base method.

Figure 4: Scenario-wise improvements and qualitative examples. DS and SR gains from augmenting SparseDriveV2 with AutoAgent0 on (a) Fail2Drive and (b) NavSafe- $\infty$ . (c) NavSafe- $\infty$ examples comparing standalone and augmented driving behavior.

Variant	Context Refinement	Rule-based Verification	Recovery Planning	DS $\uparrow$	SR (%) $\uparrow$	HM $\uparrow$	Col. $\downarrow$	Rec./km
SparseDriveV2	–	–	–	66.5	63.4	64.7	0.445	0.0
+ Rule-based verification ‡	–	$\checkmark$	–	69.1	60.0	64.2	0.095	–
+ Scene context refinement	$\checkmark$	$\checkmark$	–	76.0	74.0	75.0	0.050	54.2
+ Recovery planning (full)	$\checkmark$	$\checkmark$	$\checkmark$	80.0	83.4	81.7	0.145	20.3
AutoAgent0 †	GT	$\checkmark$	$\checkmark$	81.4	90.3	85.6	0.110	26.2

Table 3: Ablation of the three runtime modules. HM denotes the harmonic mean of DS and SR. Col. is the mean number of collision events per route. Rec./km denotes recovery-trigger events per kilometer; standalone SparseDriveV2 has no recovery module, so its Rec./km is zero by construction. Routes reaching the 180 s simulation cap are scored as scenario timeouts (Appendix F.2 ). † Simulator ground-truth scene context replaces the output of scene context refinement. ‡ Rec./km is not available for this run.

Variant	Skill	VLM Choice	Param	Feedback	DS $\uparrow$	SR (%) $\uparrow$	Col. $\downarrow$	Rec.-SR (%) $\uparrow$
Controlled fallback	–	–	–	–	76.0	74.0	0.050	–
+ Random primitive selection	$\checkmark$	–	–	–	74.0	73.0	0.210	52.7
+ VLM skill selection	$\checkmark$	$\checkmark$	–	–	76.3	77.0	0.125	61.7
+ Parameterized recovery	$\checkmark$	$\checkmark$	$\checkmark$	–	76.3	77.3	0.200	62.5
+ Budget-matched retry	$\checkmark$	$\checkmark$	$\checkmark$	–	77.8	77.3	0.125	63.5
+ Verifier-guided refinement	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	80.0	83.4	0.145	67.4

Table 4: Recovery mechanism ablations. Skill denotes access to the primitive library; VLM-Choice denotes VLM-based recovery action selection; Param denotes parameter selection; and Feedback denotes verifier-guided pre-execution revision. Budget-matched retry and refinement each allow at most three VLM calls per recovery episode. Col. is the mean number of collision events per route. Rec.-SR measures recovery success and is undefined (–) for controlled fallback. See Appendix A.1 for configurations and evaluation details.

Method	In-Distribution			Generalization			Average
	DS $\uparrow$	SR (%) $\uparrow$	HM $\uparrow$	DS $\uparrow$	SR (%) $\uparrow$	HM $\uparrow$	DS $\uparrow$	SR (%) $\uparrow$	HM $\uparrow$
SparseDriveV2	70.3	75.0	72.6	63.1	52.5	57.3	66.7	63.8	64.9
+ RSS	73.5	66.2	69.7	65.5	45.0	53.3	69.5	55.6	61.5
+ Longitudinal MPC	69.9	55.0	61.6	65.4	42.5	51.5	67.7	48.8	56.5
+ 2D MPC	65.2	65.0	65.1	60.1	47.5	53.1	62.7	56.3	59.1
+ Uncertainty monitor	74.5	60.0	66.5	60.7	32.5	42.3	67.6	46.3	54.4

Table 5: Comparison with alternative runtime methods. Reported results cover 80 In-Distribution/Generalization route pairs, excluding calibration routes, with one rollout per route. HM is computed within each split; Average is the arithmetic mean of the two splits for each metric. Longitudinal MPC preserves the nominal path geometry, while 2D MPC jointly optimizes acceleration and steering. AESOP denotes our CARLA adaptation; AESOP + primitives uses our primitive library to provide references for branch MPC. Implementation details are provided in Appendix F.4 .

Figure 5: Runtime performance and intermediate recovery decisions. (a) DS and SR on 80 In-Distribution/Generalization route pairs, excluding calibration routes, with one rollout per route. (b) SparseDriveV2’s maneuver ends in a vehicle collision. (c) MPC brakes but later stalls and times out; AutoAgent0 detects stalled progress and executes a checked left bypass, completing the route without collisions or timeouts. Qualitative rollouts use different configurations; see Appendix F.4 .

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Recovery decisions under vehicle blockage. Qwen3.5-9B changes its actions while repeatedly misinterpreting motion and scene constraints, and the rollout ends in timeout with a collision. Opus 5 responds to unsuccessful maneuver attempts by reversing before retrying a right bypass. GPT-6 Astra initially brakes and waits, then revises its motion interpretation and selects a bypass. Both Opus and Astra complete the displayed route without collisions. Text annotations summarize the models’ stated scene interpretations. The right panel reports aggregate benchmark performance.

Figure 7: Runtime decisions in two safety-critical encounters. Rows compare SparseDriveV2, RSS, MPC, uncertainty monitoring, and AutoAgent0 across scene context, intervention trigger, selected response, and outcome. (a) Crossing animals: only AutoAgent0 completes without collisions. (b) Pedestrian in lane: RSS, MPC, and AutoAgent0 complete without collisions; SparseDriveV2 and uncertainty monitoring record a pedestrian collision. Blue outlines identify the ego vehicle; gold outlines mark hazards.

VLM backbone	DS $\uparrow$	SR (%) $\uparrow$	Col. $\downarrow$	Rec.-SR (%) $\uparrow$
Qwen3.5-9B ( Qwen Team, 2026 )	78.3	80.5	0.210	64.6
Opus 5 ( Anthropic, 2026 )	80.0	83.4	0.145	67.4
GPT-6 Astra ( OpenAI, 2026 )	82.6	83.4	0.125	71.0

Appendix

Table 6: VLM backbone comparison on Fail2Drive . Rec.-SR denotes recovery success rate.

Figure 8: Qualitative examples of complementary runtime behaviors. Left: a verified bypass resolves a construction blockage. Right: obstacle detection supports stopping before a road-image wall and resuming after it clears. The examples illustrate improvements through both active maneuvering and appropriate waiting.

Module	Calls	Mean (ms)	Median (ms)	P95 (ms)
Nominal planning	9,227	598.9	397.4	1,326.7
Perception inference	3,017	460.5	303.9	939.4
Perception update without inference	9,032	112.9	106.2	169.1
Perception fusion and refinement	3,017	178.2	169.7	344.3
Scene-context resolution	12,049	0.21	0.19	0.37
Batch trajectory verification	9,227	80.2	78.6	153.1

Appendix

Table 7: Local module wall-clock latency across 12 profiling routes. Statistics pool completed calls after excluding the first 20 control steps of each route. Timings include inter-process communication and nested local operations, so rows are not additive. VLM request latency is measured separately in Table 8 .

Model	Requests	Mean (s)	Median (s)	P95 (s)
Qwen3.5-9B	20	8.33	8.89	9.68
GPT-6 Astra	20	4.72	4.36	6.82
Opus 5	20	4.77	4.69	6.39

Appendix

Table 8: Standalone VLM request latency on the same 20 saved recovery contexts. Each model processes one request per context. Qwen runs locally on a shared L40S, while Astra and Opus use remote APIs. The measurements exclude model loading but include input preparation and response parsing.

Quantity	Statistic	SparseDriveV2	+ AutoAgent0	$\Delta$
Wall-clock (min)	Mean	15.49	15.90	$+2.6$
	Median	9.88	11.44	$+15.7$
	Total (h)	51.6	53.0	$+2.6$
Simulated (s)	Mean	62.6	57.4	$-8.2$
	Median	44.4	38.9	$-12.5$
	Routes at 180 s	19	21	–

Appendix

Table 9: Per-route runtime on the same 200 Fail2Drive routes under a matched 180 s simulation budget. The AutoAgent0 in this experiment uses the Qwen3.5-9B VLM as its backbone. Wall-clock time is in minutes and simulated time in seconds. Wall-clock per simulated second is the real time needed to simulate one second of driving; the pooled rate divides total wall-clock by total simulated time.

\Delta

is the relative change of AutoAgent0 in percent.

Primitive	Behavior	Default duration	Parameter range
brake	Brake to a full stop	1.0 s	–
wait	Remain stopped	1.0 s	–
creep	Crawl straight forward at low speed	1.5 s	speed 0.5–2.0 m/s
creep_back	Stop, then back straight up with heading preserved	3.0 s	speed 0.5–2.0 m/s
swerve_left	Swerve left around an obstacle, then stop	2.5 s	lateral 0.5–2.5 m
swerve_right	Swerve right around an obstacle, then stop	2.5 s	lateral 0.5–2.5 m

Appendix

Table 10: Recovery primitive library. Durations and parameter ranges describe the default action specifications. Actual execution may end early or use a configuration-dependent duration. A dash indicates that no parameter range is listed in the default specification.

Threshold	DS $\uparrow$	SR (%) $\uparrow$	HM $\uparrow$	Col. $\downarrow$	Timeouts $\downarrow$	Rejected (%)
0.01 (hard gates only)	75.0	70.6	72.7	0.176	4	1.8
0.40	79.5	85.3	82.3	0.118	2	0.7
0.43 (ours)	81.1	88.2	84.5	0.059	2	1.1
0.45	76.2	82.4	79.1	0.147	2	1.0
0.60 (TTC becomes a hard gate)	73.3	76.5	74.9	0.029	8	4.2
0.84 (any soft defect rejects)	64.5	64.7	64.6	0.000	12	8.6

Appendix

Table 11: Verifier threshold sweep (17 scenario families). One Base/Generalization route pair per Fail2Drive scenario family (34 routes): 17 pairs of routes are drawn uniformly at random within their families. Full AutoAgent0 runtime with SparseDriveV2, 180 s simulation cap; routes reaching the cap are scored as scenario timeouts (Appendix F.2 ). HM is the harmonic mean of DS and SR. Timeouts counts routes that reach the simulation cap. Rejected is the fraction of control ticks at which the verifier rejects the selected proposal and triggers recovery.

Parameter	Value
Admission threshold	0.43
Verification sampling interval	0.1 s
Vehicle/static-object NC horizon	1.5 s
Pedestrian veto horizon	3.0 s
Stop-conditioned horizon extension	Enabled
Reaction-time margin	0.5 s

Appendix

Table 12: Verifier configuration for the SparseDriveV2 sensor-based setting. Values are shared by the evaluated local-VLM, Opus 5, and GPT-6 Astra configurations.

Autonomous Driving Planning

Autonomous driving planning selects safe, feasible vehicle actions and trajectories amid traffic, uncertainty, road rules, and changing goals.

See all 384 papers

Reason--Imagine--Act: Closed-Loop LLM Decision Making with World Models for Autonomous Driving
May 19, 2026Zhengqi Sun, Yiwen Sun, Boxuan Liu +3Language Model-Based Control World Models
Department of Information Management, Peking University, Beijing 100871, China · School of Intelligence Science and Technology, Peking University, Beijing 100871, China · State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing 100080, China +3
Learning from Mistakes: Rollout-Retrieval Lifelong Policy Learning for Autonomous Driving
Jun 29, 2026Cheng Gong, Haoyang Wang, Chao Lu +2Autonomous Driving Planning Continual RL
School of Mechanical Engineering, Beijing Institute of Technology, Beijing 100081, China · School of Mechanical and Aerospace Engineering, Nanyang Technological University, Singapore 639798
MAPLE: Latent Multi-Agent Play for End-to-End Autonomous Driving
May 13, 2026Rajeev Yasarla, Deepti Hegde, Hsin-Pai Cheng +9End-to-End Autonomous Driving Vision-Language-Action Models
Qualcomm AI Research · Qualcomm Technologies, Inc

May 19, 2026J/K move · Enter open · S save

Reason--Imagine--Act: Closed-Loop LLM Decision Making with World Models for Autonomous Driving

Zhengqi Sun, Yiwen Sun, Boxuan Liu, Tailai Chen+2

Department of Information Management, Peking University, Beijing 100871, China · School of Intelligence Science and Technology, Peking University, Beijing 100871, China · State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing 100080, China+3

Language Model-Based Control World Models World Models for Autonomous Driving Autonomous Driving Planning Autonomous Driving Simulation

Open paper AskPDF arXiv

Large language models (LLMs) are promising for autonomous driving, but semantics-only decision policies can yield physically unsafe behavior in dynamic traffic. Existing methods either perform online language reasoning without explicit dynamics verification or use world models mainly in offline pipelines, leaving a gap between semantic intent and physical feasibility at decision time. We propose Reason--Imagine--Act (RIA), a closed-loop framework that couples an LLM reasoner with an action-conditioned world model for online safety verification. At each step, the LLM proposes an action template and candidate sub-actions, the world model performs short-horizon rollouts, and a safety scorer selects the safest executable action with feedback to the next reasoning step. Under a unified CARLA point-goal protocol (1000 episodes), RIA achieves 80.05% route completion, 51.10% arrival rate, and 0.20% collision rate. Under the same closed-loop interface, RIA consistently outperforms training-free baselines, including CARLA TM and MADA, on core closed-loop metrics. For reproducibility, code is available at https://github.com/pku-smart-city/source_code/tree/main/RIA.

Abstract

Autonomous Driving Planning

Explore similar work

Reason--Imagine--Act: Closed-Loop LLM Decision Making with World Models for Autonomous Driving

Learning from Mistakes: Rollout-Retrieval Lifelong Policy Learning for Autonomous Driving

MAPLE: Latent Multi-Agent Play for End-to-End Autonomous Driving