ADV-0: Closed-Loop Min-Max Adversarial Training for Long-Tail Robustness in Autonomous Driving

Tong Nie; Yihong Tang; Junlin He; Yuewen Mei; Jie Sun; Lijun Sun; Wei Ma; Jian Sun

cs.LGMar 16, 2026

ADV-0: Closed-Loop Min-Max Adversarial Training for Long-Tail Robustness in Autonomous Driving

Authors: Tong Nie, Yihong Tang, Junlin He, Yuewen Mei, Jie Sun, Lijun Sun, Wei Ma, Jian Sun

Organizations: The Hong Kong Polytechnic University, Hong Kong SAR, China · Tongji University, Shanghai, China · McGill University, Montreal, QC, Canada

Abstract

Deploying autonomous driving systems requires robustness against long-tail scenarios that are rare but safety-critical. While adversarial training offers a promising solution, existing methods typically decouple scenario generation from policy optimization and rely on heuristic surrogates. This leads to objective misalignment and fails to capture the shifting failure modes of evolving policies. This paper presents ADV-0, a closed-loop min-max optimization framework that treats the interaction between driving policy (defender) and adversarial agent (attacker) as a zero-sum Markov game. By aligning the attacker's utility directly with the defender's objective, we reveal the optimal adversary distribution. To make this tractable, we cast dynamic adversary evolution as iterative preference learning, efficiently approximating this optimum and offering an algorithm-agnostic solution to the game. Theoretically, ADV-0 converges to a Nash Equilibrium and maximizes a certified lower bound on real-world performance. Experiments indicate that it effectively exposes diverse safety-critical failures and greatly enhances the generalizability of both learned policies and motion planners against unseen long-tail risks.

Figures & tables

Figure 1: Illustration of the ADV-0 framework. It alternates between an Inner Loop where the adversary

\psi

evolves via IPL to track the failure frontier, and an Outer Loop where the ego

\pi_{\theta}

optimizes policy gradients against the induced adversarial distribution.

Adversary	Replay		IDM		RL Agent
Adversary	CR $\uparrow$	ER $\downarrow$	CR $\uparrow$	ER $\downarrow$	CR $\uparrow$	ER $\downarrow$
Replay	-	-	19.03%	51.75	16.80%	51.65
Heuristic	100.00%	0.00	74.70%	32.12	69.64%	24.90
CAT	90.08%	1.03	43.13%	43.47	36.84%	42.26
KING	23.28%	47.67	24.49%	49.41	21.26%	49.32
AdvTrajOpt	69.64%	3.12	26.92%	45.36	28.95%	45.10

Table 1: Safety-critical scenario generation . CR denotes the collision rate of the ego in generated scenarios, and ER denotes the ego’s cumulative return. Ego is controlled by Replay, IDM, and trained PPO policies, respectively. Metrics of ADV-0 are mean values across all training methods. See full results in Table 6 .

Figure 4: Examples of generated reward-reduced adversarial scenarios from ADV-0 . Additional cases are shown in Figure 15 .

	Val. Env.	RC $\uparrow$	Crash $\downarrow$	Reward $\uparrow$	Cost $\downarrow$
ADV-0 (w/ IPL)	Replay	0.742 $\pm$ 0.011	0.159 $\pm$ 0.019	49.56 $\pm$ 1.05	0.480 $\pm$ 0.013
	ADV-0	0.695 $\pm$ 0.011	0.289 $\pm$ 0.029	44.60 $\pm$ 0.83	0.598 $\pm$ 0.022
	CAT	0.704 $\pm$ 0.011	0.271 $\pm$ 0.025	45.68 $\pm$ 1.16	0.585 $\pm$ 0.025
	SAGE	0.699 $\pm$ 0.013	0.263 $\pm$ 0.025	45.19 $\pm$ 1.54	0.567 $\pm$ 0.028
	Heuristic	0.710 $\pm$ 0.016	0.217 $\pm$ 0.021	45.41 $\pm$ 2.08	0.552 $\pm$ 0.035
	Avg.	0.710 $\pm$ 0.012	0.240 $\pm$ 0.024	46.09 $\pm$ 1.33	0.556 $\pm$ 0.027

Table 2: Performance validation of learned policies . Results are averaged across 6 RL methods (GRPO, PPO, PPO-Lag, SAC, SAC-Lag, TD3). This table compares the performances of agents learned by different adversarial methods and evaluate on different generated scenarios. Full results are shown in Tables 8 , 8 , 10 , 10 , 12 , 12 .

Agent/Adversary	Reward ( $\uparrow$ )
Agent/Adversary	Adversary (w/o IPL)	Adversary (w/ IPL)	Decrease
Agent w/o IPL	$41.03\pm 1.87$	$39.01\pm 0.97$	$-4.92\%$
Agent w/ IPL	$44.60\pm 0.83$	$43.47\pm 1.40$	$-2.53\%$
Improvement	$+8.70\%$	$+11.43\%$
Agent/Adversary	Cost ( $\downarrow$ )
Agent/Adversary	Adversary (w/o IPL)	Adversary (w/ IPL)	Decrease

Table 3: Cross-validation of ADV-0 . Performances of agents and adversaries with/without IPL. Decrease indicates the percentage change in performance when facing an IPL-enhanced adversary compared to the baseline. Improvement indicates the percentage change when the agent is trained with IPL compared to the other.

Figure 5: Generalization gap over different training budgets.

	Reactive Traffic	Safety Margin		Stability & Comfort		Defensive Driving
	Reactive Traffic	Avg Min-TTC ( $\uparrow$ )	Avg Min-PET ( $\uparrow$ )	Mean Abs Jerk ( $\downarrow$ )	95% Jerk ( $\downarrow$ )	Near-Miss Rate ( $\downarrow$ )	RDP Violation Rate ( $\downarrow$ )
ADV-0 (w/ IPL)	$\checkmark$	$\pagecolor{myblue!20}{\boldsymbol{0.993\pm 0.189}}$	${\boldsymbol{1.150\pm 0.317}}$	${\boldsymbol{1.653\pm 0.169}}$	${\boldsymbol{5.053\pm 0.885}}$	${\boldsymbol{63.45\%\pm 4.34\%}}$	${\boldsymbol{33.70\%\pm 3.77\%}}$
ADV-0 (w/ IPL)	$\times$	${{1.527\pm 0.182}}$	${\boldsymbol{1.103\pm 0.313}}$	${\boldsymbol{1.617\pm 0.204}}$	${\boldsymbol{5.177\pm 0.953}}$	${\boldsymbol{60.97\%\pm 3.94\%}}$	${\boldsymbol{36.45\%\pm 2.85\%}}$
CAT	$\checkmark$	${0.825\pm 0.192}$	${1.017\pm 0.317}$	${1.875\pm 0.229}$	${5.422\pm 0.833}$	${83.04\%\pm 5.99\%}$	${44.80\%\pm 6.12\%}$
CAT	$\times$	${1.415\pm 0.462}$	${0.980\pm 0.327}$	${1.979\pm 0.193}$	${6.002\pm 0.931}$	${75.36\%\pm 7.86\%}$	${51.20\%\pm 4.74\%}$
Heuristic	$\checkmark$	${0.864\pm 0.200}$	$\boldsymbol{1.150\pm 0.467}$	${2.129\pm 0.237}$	${6.542\pm 0.942}$	${74.46\%\pm 5.98\%}$	${48.79\%\pm 3.65\%}$

Table 4: Robustness on mined real-world long-tailed sets. Average agent performance on four long-tail scenario categories filtered by criteria: Critical TTC (

\min\text{TTC}<0.4s

), Critical PET (

\text{PET}<1.0s

), Hard Dynamics (Acc.

<-4.0m/s^{2}

|\text{Jerk}|>4.0m/s^{3}

), and Rare Cluster (topologically sparse trajectory clusters). Metrics assess average Safety Margin (higher values indicate earlier risk detection), Stability & Comfort (lower Jerk indicates smoother control), and Defensive Driving , quantified by Near-Miss Rate (proximity without collision) and RDP Violation (percentage of time requiring deceleration

>6m/s^{2}

to avoid collision). Full results are shown in Table 15 .

Figure 6: Visualization of improved safe driving ability after being trained with ADV-0 . More examples are shown in Figure 14 .

Model	RC $\uparrow$	Crash $\downarrow$	Reward $\uparrow$	Cost $\downarrow$
PlanTF	0.628 $\pm$ 0.025	0.357 $\pm$ 0.031	35.85 $\pm$ 2.51	1.04 $\pm$ 0.04
+ ADV-0	0.674 $\pm$ 0.016	0.263 $\pm$ 0.025	41.98 $\pm$ 1.62	0.77 $\pm$ 0.02
Rel. Change	+7.46%	-26.23%	+17.11%	-25.68%
SMART	0.587 $\pm$ 0.029	0.396 $\pm$ 0.034	32.66 $\pm$ 2.85	1.15 $\pm$ 0.05
+ ADV-0	0.631 $\pm$ 0.015	0.305 $\pm$ 0.023	37.85 $\pm$ 1.70	0.92 $\pm$ 0.02
Rel. Change	+7.57%	-22.88%	+15.86%	-20.31%

Table 5: Application to learning-based planners . Performance comparisons of two SOTA trajectory planning models before and after fine-tuning using ADV-0 (GRPO). See Table 13 for details.

Figure 7: Impacts of the temperature parameter in sampling.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

ADV-0 Variation	Replay		IDM		RL Agent
ADV-0 Variation	CR $\uparrow$	Reward $\downarrow$	CR $\uparrow$	Reward $\downarrow$	CR $\uparrow$	Reward $\downarrow$
Pretrained (logit-based sampling)	$31.72\%\pm 1.06\%$	$43.31\pm 1.10$	$18.84\%\pm 1.48\%$	$50.66\pm 2.22$	$12.49\%\pm 1.06\%$	$52.12\pm 2.03$
Pretrained (energy-based sampling)	$85.09\%\pm 1.13\%$	$1.89\pm 0.20$	$40.14\%\pm 1.06\%$	$45.72\pm 0.47$	$36.30\%\pm 0.77\%$	$41.99\pm 0.33$
GRPO Finetuned	$92.61\pm 0.62\%$	$0.87\pm 0.06$	$46.34\pm 0.73\%$	$39.60\pm 0.13$	$41.95\pm 0.47\%$	$38.45\pm 0.42$
PPO Finetuned	$90.93\pm 0.69\%$	$1.01\pm 0.07$	$45.09\pm 0.63\%$	$40.69\pm 0.22$	$39.12\pm 0.38\%$	$39.63\pm 0.43$
SAC Finetuned	$91.88\pm 0.86\%$	$1.03\pm 0.08$	$46.09\pm 0.19\%$	$39.62\pm 0.24$	$41.09\pm 0.59\%$	$39.17\pm 0.48$

Appendix

Table 6: Detailed results of safety-critical scenario generation using ADV-0 .

Table 14

Table 15

Figure 10: Learning curves of TD3.

Table 17

Figure 11: Learning curves of GRPO.

Model Phase	Val. Env.	RC $\uparrow$	Crash $\downarrow$	Reward $\uparrow$	Cost $\downarrow$
PlanTF (Pretrained)	Replay	0.727 $\pm$ 0.021	0.220 $\pm$ 0.027	41.53 $\pm$ 3.10	0.76 $\pm$ 0.03
	ADV-0	0.581 $\pm$ 0.030	0.420 $\pm$ 0.035	32.11 $\pm$ 2.54	1.24 $\pm$ 0.05
	CAT	0.615 $\pm$ 0.025	0.385 $\pm$ 0.030	35.26 $\pm$ 2.16	1.09 $\pm$ 0.03
	SAGE	0.595 $\pm$ 0.028	0.407 $\pm$ 0.032	33.80 $\pm$ 2.80	1.15 $\pm$ 0.04
	Heuristic	0.620 $\pm$ 0.020	0.352 $\pm$ 0.032	36.53 $\pm$ 1.97	0.94 $\pm$ 0.03
	Avg.	0.628 $\pm$ 0.025	0.357 $\pm$ 0.031	35.85 $\pm$ 2.51	1.04 $\pm$ 0.04

Appendix

Table 13: Performance comparison of learning-based planners before and after adversarial fine-tuning using ADV-0 (GRPO).

Method	Progress	Collision	Off-road	Comfort	Speed Eff.	Total Score
Method	( + )	( - )	( - )	( - )	( - )	Total Score
PlanTF (Pretrained)	14.17 $\pm$ 1.53	-2.85 $\pm$ 0.92	-0.12 $\pm$ 0.05	-0.34 $\pm$ 0.08	-0.54 $\pm$ 0.12	10.32 $\pm$ 1.79
PlanTF (Fine-tuned)	16.45 $\pm$ 0.85	-0.89 $\pm$ 0.45	-0.05 $\pm$ 0.04	-0.24 $\pm$ 0.05	-0.64 $\pm$ 0.13	14.63 $\pm$ 0.97
Improvement	+16.09%	+68.77%	+58.33%	+29.41%	-18.52%	+41.76%
SMART (Pretrained)	13.20 $\pm$ 1.80	-3.61 $\pm$ 1.11	-0.25 $\pm$ 0.08	-1.26 $\pm$ 0.25	-0.60 $\pm$ 0.15	7.48 $\pm$ 2.14
SMART (Fine-tuned)	15.10 $\pm$ 1.10	-1.75 $\pm$ 0.37	-0.17 $\pm$ 0.05	-0.92 $\pm$ 0.19	-0.65 $\pm$ 0.11	11.61 $\pm$ 1.18

Appendix

Table 14: Detailed breakdown of the trajectory-level reward model values . Values represent the average accumulated discounted reward (over a 2.0s horizon with discount factor

\gamma=0.99

) of the best planned trajectory selected by the model across all validation steps. Reward components & weights : Progress (

w_{\text{prog}}=1.0

, longitudinal advance), Collision (

w_{\text{coll}}=20.0

, penalty for overlap with objects), Off-road (

w_{\text{off}}=5.0

, penalty for lane deviation), Comfort (

w_{\text{acc}}=0.1,w_{\text{jerk}}=0.1

, penalties for harsh dynamics), Speed Efficiency (

w_{\text{eff}}=0.2

, penalty for deviating from 10m/s). Note that the negative improvement in Speed Efficiency reflects a safety-efficiency trade-off, where the fine-tuned planner adopts a more conservative velocity profile to satisfy safety constraints.

	Scenario Category	Percentage	Reactive Traffic	Safety Margin		Stability & Comfort		Defensive Driving
	Scenario Category	Percentage	Reactive Traffic	Avg Min-TTC ( $\uparrow$ )	Avg Min-PET ( $\uparrow$ )	Mean Abs Jerk ( $\downarrow$ )	95% Jerk ( $\downarrow$ )	Near-Miss Rate ( $\downarrow$ )	RDP Violation Rate ( $\downarrow$ )
ADV-O (w/ IPL)	Critical TTC	7.40%	$\checkmark$	$0.645\pm 0.122$	$1.450\pm 0.50$	$1.853\pm 0.188$	$5.850\pm 1.252$	$65.28\%\pm 1.45\%$	$15.53\%\pm 4.16\%$
	Critical PET	3.40%		$0.492\pm 0.025$	$1.150\pm 0.20$	$1.623\pm 0.095$	$5.157\pm 0.653$	$74.57\%\pm 5.52\%$	$58.26\%\pm 3.29\%$
	Hard Dynamics	3.20%		$0.885\pm 0.157$	N/A	$1.685\pm 0.250$	$4.555\pm 0.951$	$55.40\%\pm 8.26\%$	$29.54\%\pm 4.80\%$
	Rare Cluster	5.20%		$1.951\pm 0.450$	$0.850\pm 0.25$	$1.451\pm 0.143$	$4.650\pm 0.684$	$58.55\%\pm 2.12\%$	$31.48\%\pm 2.81\%$
	Avg			$\mathbf{0.993\pm 0.189}$	$\mathbf{1.150\pm 0.317}$	$\mathbf{1.653\pm 0.169}$	$\mathbf{5.053\pm 0.885}$	$\mathbf{63.45\%\pm 4.34\%}$	$\mathbf{33.70\%\pm 3.77\%}$

Appendix

Table 15: Full results of quantitative evaluation on the unbiased long-tailed set mined from real-world data. The benchmark consists of four long-tail scenario categories filtered by strict physical thresholds: Critical TTC (

\min\text{TTC}<0.4s

), Critical PET (

\text{PET}<1.0s

), Hard Dynamics (Longitudinal Acc

<-4.0m/s^{2}

|\text{Jerk}|>4.0m/s^{3}

), and Rare Cluster (topologically sparse trajectory clusters). Reactive Traffic denotes whether background vehicles utilize IDM/MOBIL policies to interact with the agent (

\checkmark

) or strictly follow logged trajectories (

\times

). Metrics assess Safety Margin (higher values indicate earlier risk detection), Stability & Comfort (lower Jerk indicates smoother control), and Defensive Driving performance, quantified by Near-Miss Rate (hazardous proximity without collision) and RDP Violation (percentage of time requiring deceleration

>6m/s^{2}

to avoid collision).

Figure 14: Qualitative comparison of improved safe driving ability after being trained with ADV-0 . We showcase three typical safety-critical scenarios generated by the adversary: Left-turn, Sudden-brake, and Cut-in. In each column, the bottom row (Agent w/o ADV-0 ) shows the baseline agent failing to anticipate the aggressive behavior of the background traffic, resulting in collisions. In contrast, the top row (Agent w/ ADV-0 ) demonstrates that the agent trained with our framework learns robust defensive behaviors, such as yielding at intersections or decelerating in time, thereby successfully avoiding accidents.

Figure 15: Additional qualitative examples of reward-reduced adversarial scenarios from ADV-0 . In the first case (left), the adversary interrupts the straight-going ego, forcing it to deviate from the lane centerline to avoid a crash; this maneuver halts the ego’s progress, causing the cumulative reward to stagnate. In the second case (right), a collision occurs near the end of the episode, triggering a sharp drop in the cumulative reward due to the safety penalty.

Table 16: Hyperparameters for different RL algorithms used in the experiments.

Module	Parameter	Value
Backbone (DenseTNT)	Hidden Size	128
	Sub-graph Depth	3
	Global-graph Depth	1
	Trajectory Modes ( $K$ )	32
	NMS Threshold	7.2
IPL Fine-tuning	Learning Rate	5e-6

Appendix

Table 20: Hyperparameters for ADV-0 Framework and IPL Fine-tuning.

Long-Tail Autonomous Driving

See all 11 papers

Jun 12, 2026

From Attacks to Curricula: Learnability-Guided Adversarial Training for Safe Autonomous Driving

Closed-loop adversarial training improves autonomous driving safety by exposing policies to rare safety-critical scenarios. Standard pipelines first generate adversarial scenarios and then sample them for policy optimization. However, most existing frameworks remain attack-oriented: collision-driven generators often synthesize unsolvable extreme situations, which can degrade learning, while heuristic samplers ignore the evolving capability of the driving policy, causing sample inefficiency and delayed convergence. We propose AlignADV, a learnability-guided closed-loop adversarial training framework that converts adversarial scenarios into resolvable and capability-aligned curricula. First, we reformulate adversarial scenario generation as a preference alignment problem and employ direct preference optimization to guide the generator toward critical yet resolvable scenarios. Second, we introduce behavioral fingerprints to capture the intrinsic characteristics of the evolving policy and construct a multi-modal capability prediction model that estimates policy performance without expensive closed-loop simulations. By combining resolvability-aligned scenarios with capability predictions, AlignADV develops a dynamic curriculum sampling mechanism that prioritizes scenarios targeting the current policy's vulnerabilities. Experiments on the Waymo Open Motion Dataset demonstrate that AlignADV improves convergence efficiency and final performance, reducing training steps by up to 40.6 percent compared with baseline methods while lowering collision rate and improving route completion under both normal and adversarial traffic conditions. These results highlight a shift from attack-oriented scenario generation to learnability-guided policy improvement, offering a principled direction for safer and more efficient autonomous driving training. Project page: https://meiyuewen.github.io/AlignADV/.

Yuewen Mei, Tong Nie, Jie Sun +3

College of Transportation & Key Laboratory of Road and Traffic Engineering of Ministry of Education, Tongji University, Shanghai, 201804, China · Department of Civil and Environmental Engineering, The Hong Kong Polytechnic University, Hong Kong SAR, China

Adversarial Training Autonomous Driving Curriculum RL RL for Autonomous Driving Autonomous Driving Scenario Generation

Ask PDF

Apr 8, 2026

Evaluation as Evolution: Transforming Adversarial Diffusion into Closed-Loop Curricula for Autonomous Vehicles

Autonomous vehicles in interactive traffic environments are often limited by the scarcity of safety-critical tail events in static datasets, which biases learned policies toward average-case behaviors and reduces robustness. Existing evaluation methods attempt to address this through adversarial stress testing, but are predominantly open-loop and post-hoc, making it difficult to incorporate discovered failures back into the training process. We introduce Evaluation as Evolution (

E^2

), a closed-loop framework that transforms adversarial generation from a static validation step into an adaptive evolutionary curriculum. Specifically,

E^2

formulates adversarial scenario synthesis as transport-regularized sparse control over a learned reverse-time SDE prior. To make this high-dimensional generation tractable, we utilize topology-driven support selection to identify critical interacting agents, and introduce Topological Anchoring to stabilize the process. This approach enables the targeted discovery of failure cases while strictly constraining deviations from realistic data distributions. Empirically,

E^2

improves collision failure discovery by 9.01% on the nuScenes dataset and up to 21.43% on the nuPlan dataset over the strongest baselines, while maintaining low invalidity and high realism. It further yields substantial robustness gains when the resulting boundary cases are recycled for closed-loop policy fine-tuning.

Yicheng Guo, Jiaqi Liu, Chengkai Xu +2

College of Transportation, Tongji University, Shanghai 201804, China · Department of Computer Science, University of North Carolina at Chapel Hill, United States

Autonomous Driving Safety Evaluation Adversarial Scenario Generation Diffusion Models

Ask PDF

Jul 12, 2026

World Models as Adversaries: Multi-Agent Self-Play Fine-Tuning for Robust Motion Planning

Robust motion planning in dense traffic requires autonomous vehicles to interact in rare and safety-critical scenarios that are underrepresented in naturalistic driving data. Although adversarial training offers a feasible solution, existing methods often rely on external scenario generators, heuristic perturbations, or simulator-heavy rollouts, which makes them difficult to integrate with modern autoregressive planners. Here, we cast adversarially robust planner learning as a constrained min-max game and propose Adversarial World Modeling (AWM), a theoretically grounded multi-agent self-play fine-tuning framework. Since solving the exact game is intractable, AWM introduces a principled decoupled solver. In the inner minimization, the planner's predictive world model is converted into a role-conditioned adversary that learns sparse, scene-adaptive attack coalitions via counterfactual credit assignment. In the outer maximization, the ego planner optimizes a regret-aware robust best response against the frozen AWM, utilizing tail-risk weighting and reference-anchored trust regions to improve hard-case recovery while preserving nominal driving behavior. Experiments on the nuPlan and InterPlan benchmarks demonstrate that our method generates transferable adversarial interactions and yields a robust planner that achieves competitive closed-loop performance in both nominal and highly interactive long-tail scenarios. Theoretical analysis justifies the decoupled solver and the main optimization components.

Tong Nie, Yuewen Mei, Junlin He +3

The Hong Kong Polytechnic University · Tongji University · McGill University +1

Adversarial Training Self-Play RL World Model-Based Planning World Models for Autonomous Driving Motion Planning Autonomous Driving Planning+2

Ask PDF

Abstract

Long-Tail Autonomous Driving

Explore similar work

From Attacks to Curricula: Learnability-Guided Adversarial Training for Safe Autonomous Driving

Evaluation as Evolution: Transforming Adversarial Diffusion into Closed-Loop Curricula for Autonomous Vehicles

World Models as Adversaries: Multi-Agent Self-Play Fine-Tuning for Robust Motion Planning