While autonomous vehicles have achieved reliable performance within specific operating regions, their deployment to new cities remains costly and slow. A key bottleneck is the need to collect many human demonstration trajectories when adapting driving policies to new cities that differ from those seen in training in terms of road geometry, traffic rules, and interaction patterns. In this paper, we show that self-play multi-agent reinforcement learning can adapt a driving policy to a substantially different target city using only the map and meta-information, without requiring any human demonstrations from that city. We introduce NO data Map-based self-play for Autonomous Driving (NOMAD), which enables policy adaptation in a simulator constructed based on the target-city map. Using a simple reward function, NOMAD substantially improves both task success rate and trajectory realism in target cities, demonstrating an effective and scalable alternative to data-intensive city-transfer methods. Project page: https://zerlinwang.github.io/NOMAD/
Figures & tables
Figure 1: Zero-shot transfer versus NOMAD.
Figure 2: NOMAD overview . Starting from a source-city imitation policy π0 , NOMAD adapts it to a target city C using map segments m∼MC and meta-information IC . A scenario generator samples initial states and goals. The generated scenarios are loaded in a data-driven multi-agent driving simulator, yielding a simulator of C . The policy πθ is initialized from π0 and optimized via KL-regularized self-play MARL. Checkpoints along training yield an adapted policy set Π+(C) , from which practitioners select a deployment policy πdeploy to suit their preferences.
Figure 3: NOMAD consistently improves the success–realism Pareto frontier in unseen target cities. We plot mean success rate versus mean realism meta score over 5 runs for three transfer settings: (a) Boston-to-Singapore (primary), (b) Singapore-to-Boston, and (c) Singapore-to-Pittsburgh. Each dot is a NOMAD training checkpoint, colored by the cumulative number of interaction steps; the red dashed curve denotes the empirical Pareto frontier over NOMAD checkpoints. Stars denote BC reference policies, while dashed curves denote RL-based ablations initialized or trained under different data/settings.
Policy
Realism
Success
πdexpert
0.8056
88.26%
Random
0.4074
4.30%
Constant Velocity
0.6147
19.45%
π0
0.6795
42.25%
BC (Singapore)
0.7011
55.49%
NOMAD Frontier
0.7570 ∼ 0.7697
87.28 ∼ 94.27%
Table 1: NOMAD substantially improves both success and realism over zero-shot transfer and imitation baselines. We report success rate and realism meta score for representative policies in closed-loop Singapore evaluation. πdexpert denotes a policy obtained by inferring actions from logged expert trajectories and discretizing them to match the action space; its realism meta score serves as an approximate upper bound under the current POSG formulation.
Figure 4: Behavioral priors are critical for preserving kinematically realistic driving during self-play. We compare the kinematic realism score of NOMAD against self-play from scratch in the Boston-to-Singapore transfer setting. Self-play without behavioral priors struggles to learn kinematically realistic behavior, while NOMAD preserves substantially more realistic motion patterns despite lacking explicit kinematic rewards.
Transfer
Policy
Failure
Off-road only
Collision only
Both
Neither (timeout)
Boston → Singapore
π0
57.7%
33.3%
1.2%
2.7%
20.5%
NOMAD
7.7% ± 1.8
0.6% ± 0.2
0.4% ± 0.2
0.3% ± 0.3
6.5% ± 1.7
Singapore → Boston
π0
63.3%
18.8%
3.1%
8.1%
33.3%
NOMAD
13.6% ± 5.6
1.3% ± 0.3
1.5% ± 0.5
1.1% ± 0.3
9.7% ± 4.3
Table 2: NOMAD eliminates most failures in the target city. We report the failure rate (%) and its breakdown by failure mode in closed-loop evaluation for Boston-to-Singapore and Singapore-to-Boston transfer. NOMAD results are reported as mean ± standard deviation over 5 runs.
Figure 5: Qualitative comparison of the zero-shot policy and NOMAD under Singapore-to-Boston transfer. Both rows start from the same scenario and initial state. Numbers identify the controlled vehicles and gi marks the goal of vehicle i ; a controlled vehicle is drawn in red while it is in collision and in orange while it is off-road. More examples are in Appendix O .
Appendix figures & tables49 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Policy Network architecture. The policy consists of modality-specific embedding layers for ego state, surrounding agents, and road observations, followed by a shared observation encoder. The encoded features are passed to separate actor and critic branches with residual blocks. The actor outputs trajectory increments (δx,δy,δh) , while the critic predicts the scalar state value vθ .
Figure 7: Visualization of the ego-centric observation space.
Hyperparameter
Value
Observation Dim
2984
Observation Embedding Dim
64 × 3
Encoder Width
512
Dropout
0.01
Residual Block Num
1
Projection Width
256
Appendix
Table 3: Network hyperparameters.
Hyperparameter
Value
Observation Radius
100.0
Max Controllable Agents Num
64
Max Road Observation Num
200
Ego observation
1 × 6
Agents Observation
(64-1) × 6
Road Observation
200 × 13
Appendix
Table 4: Environment configurations.
Hyperparameters
Value
Dataset Scenario Num
3200
Batch Size
64
Loss Function
Cross Entropy
Learning Rate
1e-3
Weight Decay
1e-2
Scheduler
Cosine Annealing
Appendix
Table 5: Behavior Cloning hyperparameters.
Hyperparameters
Value
World Num
1,000
Batch Size
8,192
Buffer Size
131,072
Update Epoch
4
Discount Factor γ
0.99
GAE Discount Factor λ
0.95
Appendix
Table 6: PPO hyperparameters.
City
α
ρ0
θ
Boston
1.4694
0.1897
0.4252
Pittsburgh
2.1327
0.0000
0.3982
Singapore
1.2630
0.1363
0.2500
Appendix
Table 7: City-specific traffic density parameters for ρ∼Gamma(α,θ)+ρ0 (vehicles/km), where α is the shape and θ is the scale parameter.
Category
Parameter
Value
Vehicle Placement
Minimum time gap Δtgap
1.2s
Minimum distance dmin
5.0m
Static vehicle probability pstatic
0.5
Speed Distribution
Mean speed fraction vˉ/vmax
0.85
Speed std fraction σv/vmax
0.22
Goal Generation
Distance mean coefficient κμ
9.0
Appendix
Table 8: Shared scenario generation parameters.
Generated
Logged
Held-out eval
(training)
(reference)
(evaluation)
# scenes
25,600
3,200
800
Road geometry
Lane length (km)
9.9 [7.0, 13.0]
9.9 [7.0, 13.0]
10.1 [7.5, 13.0]
# lane segments
401 [283, 490]
401 [283, 490]
410 [312, 495]
# crosswalks
19 [13, 26]
19 [13, 26]
20 [14, 26]
Appendix
Table 9: Scenario statistics for Boston → Singapore. All statistics are computed on the initial scene ( t=0 ). Each entry is the median over scenes, followed in brackets by the interquartile range (IQR), i.e., the 25th and 75th percentiles, which span the middle 50% of scenes. The row marked † reports the mean instead, as its median is 0 in all sets. Generated scenes reuse the training-split maps. The classifier AUC is computed on the 9 traffic features against the held-out evaluation set and reported as mean ± standard deviation over 5 random draws.
Figure 8: Pareto frontier of success rate versus realism meta score across training checkpoints for Boston-to-Singapore transfer with small traffic density perturbation. Each point corresponds to a policy checkpoint, colored by the number of interaction steps. The yellow star denotes the zero-shot transfer policy π0 from Boston, while the lime star shows behavior cloning using Singapore trajectories. The red dashed line and points indicate the empirical Pareto frontier on this test set.
Figure 9: Pareto frontier of success rate versus realism meta score across training checkpoints for Boston-to-Singapore transfer with large traffic density perturbation. Each point corresponds to a policy checkpoint, colored by the number of interaction steps. The yellow star denotes the zero-shot transfer policy π0 from Boston, while the lime star shows behavior cloning using Singapore trajectories. The red dashed line and points indicate the empirical Pareto frontier on this test set.
Figure 10: Visualization of scenarios from different cities in the nuPlan dataset. The rectangles denote vehicles and the circles with the corresponding color denote their goals. To mirror the Singapore and make it a right-hand traffic city, we first flip the scenario across x-axis, then we change the orientation of road edges while keeping the orientation of road lanes.
Figure 11: Examples of generated scenarios on Singapore map for two different sampled map segments.
Policy
Scenarios
Realism Meta Score
Success Rate
πcexpert
Boston
0.8311
100.0%
Singapore
0.8581
100.0%
πdexpert
Boston
0.7814
85.47%
Singapore
0.8056
88.26%
Appendix
Table 10: Performance of expert policies in different cities.
Figure 12: Detailed sub-metrics in realism metric and ADE in the experiment of transferring from Boston to Singapore. Although the BC policy (at the beginning of training) achieves reasonable kinematic smoothness, it frequently violates map constraints (e.g., off-road driving), leading to a poor map-based realism meta score. In our experiments, behavior cloning serves as a canonical source-city planner. While its absolute performance depends on dataset coverage, our focus is on the performance degradation induced by the lack of supervised finetuning in the target city and the ability of NOMAD to mitigate it without demonstrations.
Figure 13: Detailed sub-metrics in realism metric and ADE in the experiment of transferring from Singapore to Boston.
Figure 14: Detailed sub-metrics in realism metric and ADE in the experiment of transferring from Singapore to Pittsburgh.
Figure 15: Detailed realism sub-metrics and ADE for behavior cloning followed by self-play MARL in Boston, evaluated zero-shot in Singapore.
Figure 16: Comparison of all metrics between self-play with and without behavioral priors. Behavioral priors play a critical role in stabilizing self-play training and guiding exploration toward human-like driving under the supervision of a simple reward function.
Figure 17: Source-city self-play is beaten by NOMAD. The dashed red curve (BC (Boston) + RL (Boston)) denotes policies trained with behavior cloning and self-play in Boston only, evaluated zero-shot in Singapore. The dashed purple curve denotes NOMAD. Source-city self-play improves over zero-shot transfer but fails to approach the Pareto frontier achieved by NOMAD, demonstrating that self-play must be grounded in the target-city map to effectively adapt across cities.
Figure 18: NOMAD closes most of the gap between zero-shot transfer and target-city demonstration-accessed policy. We compare zero-shot transfer, NOMAD (RL with generated scenarios), and a policy trained with 3,200 scenarios with logged trajectories in the target-city using the same BC+RL training protocol (BC (Singapore) + RL (Singapore)). While the policy with access to target-city demonstrations yields additional gains over NOMAD, this improvement is modest compared to the substantial gap between zero-shot transfer and NOMAD, highlighting the effectiveness of map-based self-play for city transfer.
Figure 19: Realism meta score and success rate over interaction steps on the Singapore test scenarios. Results are shown with mean and 95% confidence interval over 5 seeds.
Figure 20: Realism meta score and success rate during map-based self-play without behavioral priors in Singapore. Self-play without pretraining and regularization achieves near-perfect success but a lower realism meta score. Shaded regions indicate 95% confidence interval across 5 runs.
Figure 21: The KL regularization weight controls the success–realism trade-off during adaptation. We compare Pareto frontiers with different λKL . Smaller KL weights favor higher success at the cost of realism, while larger KL weights encourage more realistic behaviors but constrain success rate.
Figure 22: Pareto frontier of success rate versus realism meta score across training checkpoints for Singapore with log-replay sim agents. Each point corresponds to a policy checkpoint, colored by the number of interaction steps. The yellow star denotes the zero-shot transfer policy π0 from Boston, while the lime star shows behavior cloning using Singapore trajectories. The red dashed line and points indicate the empirical Pareto frontier on this test set. Frontier checkpoints consistently dominate the baselines and exhibit only a mild trade-off between task success and realism.
Figure 23: Pareto frontier of success rate versus realism meta score across training checkpoints for Boston with log-replay sim agents. Each point corresponds to a policy checkpoint, colored by the number of interaction steps. The yellow star denotes the zero-shot transfer policy π0 from Singapore, while the lime star shows behavior cloning using Boston trajectories. The red dashed line and points indicate the empirical Pareto frontier on this test set. Frontier checkpoints consistently dominate the baselines and exhibit only a mild trade-off between task success and realism.
Figure 24: Pareto frontier of success rate versus realism meta score across training checkpoints for Pittsburgh with log-replay sim agents. Each point corresponds to a policy checkpoint, colored by the number of interaction steps. The yellow star denotes the zero-shot transfer policy π0 from Singapore, while the lime star shows behavior cloning using Pittsburgh trajectories. The red dashed line and points indicate the empirical Pareto frontier on this test set. Frontier checkpoints consistently dominate the baselines and exhibit only a mild trade-off between task success and realism.
Checkpoint
Realism Self-Play
Realism Cross-Play
Success Rate Self-Play
Success Rate Cross-Play
3690
0.768 ± 0.004
0.769 ± 0.003
0.885 ± 0.011
0.885 ± 0.008
3710
0.771 ± 0.003
0.771 ± 0.003
0.873 ± 0.011
0.872 ± 0.010
3740
0.770 ± 0.003
0.769 ± 0.002
0.879 ± 0.017
0.880 ± 0.011
3930
0.768 ± 0.003
0.767 ± 0.002
0.897 ± 0.015
0.894 ± 0.013
3980
0.766 ± 0.004
0.766 ± 0.003
0.900 ± 0.014
0.902 ± 0.010
4010
0.766 ± 0.005
0.766 ± 0.003
0.901 ± 0.011
0.899 ± 0.009
Appendix
Table 11: Self-play vs cross-play summary at Pareto-frontier checkpoints (Boston → Singapore). We evaluate each selected frontier checkpoint under both self-play (paired with the same-seed partner) and cross-play (paired with different-seed partners) using n=5 independently trained seeds. We report mean ± standard deviation across pairings. Cross-play success and realism closely match self-play at the same checkpoint, indicating that NOMAD does not rely on seed-specific conventions.
Figure 25: Cross-play success matrices across Pareto-frontier checkpoints (Boston → Singapore). Each panel corresponds to one selected checkpoint on the success–realism Pareto frontier. Within each panel, entry (i,j) reports the ego success rate when pairing seed i with seed j ( n=5 total seeds). Diagonal entries are self-play (same seed paired with itself), while off-diagonals are cross-play (different seeds). Off-diagonal performance is comparable to diagonal performance across checkpoints, indicating minimal sensitivity to partner seed.
Figure 26: Cross-play realism matrices across Pareto-frontier checkpoints (Boston → Singapore). Each panel corresponds to one selected checkpoint on the success–realism Pareto frontier. Within each panel, entry (i,j) reports the ego realism meta score when pairing seed i with seed j ( n=5 total seeds). Diagonal entries are self-play (same seed paired with itself), while off-diagonals are cross-play (different seeds). Off-diagonal performance is comparable to diagonal performance across checkpoints, indicating minimal sensitivity to partner seed.
Checkpoint
Realism Self-Play
Realism Cross-Play
Success Rate Self-Play
Success Rate Cross-Play
4860
0.740 ± 0.003
0.738 ± 0.003
0.751 ± 0.077
0.749 ± 0.046
4910
0.740 ± 0.003
0.739 ± 0.003
0.724 ± 0.036
0.721 ± 0.023
4920
0.738 ± 0.004
0.737 ± 0.004
0.756 ± 0.042
0.755 ± 0.029
4940
0.738 ± 0.004
0.737 ± 0.002
0.776 ± 0.012
0.771 ± 0.012
6180
0.737 ± 0.003
0.735 ± 0.002
0.783 ± 0.089
0.783 ± 0.055
6240
0.736 ± 0.005
0.735 ± 0.004
0.828 ± 0.056
0.825 ± 0.035
Appendix
Table 12: Self-play vs cross-play summary at Pareto-frontier checkpoints (Singapore → Boston). We evaluate each selected frontier checkpoint under both self-play (paired with the same-seed partner) and cross-play (paired with different-seed partners) using n=5 independently trained seeds. We report mean ± standard deviation across pairings. Cross-play success and realism closely match self-play at the same checkpoint, indicating that NOMAD does not rely on seed-specific conventions.
Figure 27: Cross-play success matrices across Pareto-frontier checkpoints (Singapore → Boston). Each panel corresponds to one selected checkpoint on the success–realism Pareto frontier. Within each panel, entry (i,j) reports the ego success rate when pairing seed i with seed j ( n=5 total seeds). Diagonal entries are self-play (same seed paired with itself), while off-diagonals are cross-play (different seeds). Off-diagonal performance is comparable to diagonal performance across checkpoints, indicating minimal sensitivity to partner seed.
Figure 28: Cross-play realism matrices across Pareto-frontier checkpoints (Singapore → Boston). Each panel corresponds to one selected checkpoint on the success–realism Pareto frontier. Within each panel, entry (i,j) reports the ego realism meta score when pairing seed i with seed j ( n=5 total seeds). Diagonal entries are self-play (same seed paired with itself), while off-diagonals are cross-play (different seeds). Off-diagonal performance is comparable to diagonal performance across checkpoints, indicating minimal sensitivity to partner seed.
Checkpoint
Realism Self-Play
Realism Cross-Play
Success Rate Self-Play
Success Rate Cross-Play
6030
0.726 ± 0.007
0.726 ± 0.005
0.809 ± 0.029
0.809 ± 0.018
6040
0.730 ± 0.005
0.728 ± 0.005
0.796 ± 0.024
0.793 ± 0.015
6080
0.727 ± 0.006
0.727 ± 0.005
0.806 ± 0.029
0.804 ± 0.016
6340
0.728 ± 0.006
0.726 ± 0.004
0.798 ± 0.041
0.794 ± 0.024
6550
0.727 ± 0.005
0.725 ± 0.003
0.809 ± 0.025
0.812 ± 0.017
7010
0.725 ± 0.004
0.724 ± 0.004
0.843 ± 0.031
0.838 ± 0.018
Appendix
Table 13: Self-play vs cross-play summary at Pareto-frontier checkpoints (Singapore → Pittsburgh). We evaluate each selected frontier checkpoint under both self-play (paired with the same-seed partner) and cross-play (paired with different-seed partners) using n=5 independently trained seeds. We report mean ± standard deviation across pairings. Cross-play success and realism closely match self-play at the same checkpoint, indicating that NOMAD does not rely on seed-specific conventions.
Figure 29: Cross-play success matrices across Pareto-frontier checkpoints (Singapore → Pittsburgh). Each panel corresponds to one selected checkpoint on the success–realism Pareto frontier. Within each panel, entry (i,j) reports the ego success rate when pairing seed i with seed j ( n=5 total seeds). Diagonal entries are self-play (same seed paired with itself), while off-diagonals are cross-play (different seeds). Off-diagonal performance is comparable to diagonal performance across checkpoints, indicating minimal sensitivity to partner seed.
Figure 30: Cross-play realism matrices across Pareto-frontier checkpoints (Singapore → Pittsburgh). Each panel corresponds to one selected checkpoint on the success–realism Pareto frontier. Within each panel, entry (i,j) reports the ego realism meta score when pairing seed i with seed j ( n=5 total seeds). Diagonal entries are self-play (same seed paired with itself), while off-diagonals are cross-play (different seeds). Off-diagonal performance is comparable to diagonal performance across checkpoints, indicating minimal sensitivity to partner seed.
Figure 31: Pareto frontier of success rate versus realism meta score across training checkpoints for Boston-to-Pittsburgh transfer. Each point corresponds to a policy checkpoint, colored by the number of interaction steps. The yellow star denotes the zero-shot transfer policy π0 from Boston, while the lime star shows behavior cloning using Pittsburgh trajectories. The red dashed line and points indicate the empirical Pareto frontier on this test set. Frontier checkpoints consistently dominate the baselines and exhibit only a mild trade-off between task success and realism.
Policy
Scenarios
Realism Meta Score
Success Rate
πBOS0
Original BOS
0.7022
55.77%
Original SIN
0.6795
42.25%
Mirrored SIN
0.6756
43.96%
πSIN0
Original SIN
0.7011
55.49%
Original BOS
0.6585
36.67%
Mirrored BOS
0.6278
38.20%
Appendix
Table 14: Effect of map mirroring on zero-shot cross-city transfer. BOS and SIN denote Boston and Singapore scenarios. πBOS0 and πSIN0 denote behavior cloning policies trained in Boston and Singapore. Mirroring the city map yields only modest performance improvements, indicating that cross-city distribution shift extends beyond driving-side conventions and cannot be resolved by geometric transformations alone.
Dataset
Value
Test set@Singapore
0.8581
New test set@Singapore
0.8602
Test set@Boston
0.8311
New test set@Boston
0.8284
Test set@Pittsburgh
0.8339
New test set@Pittsburgh
0.8392
Appendix
Table 15: The WOSAC realism meta scores of human trajectories in different sets of scenarios.
Figure 32: Pareto frontier of success rate versus realism meta score across training checkpoints under Boston-to-Singapore transfer. Each point corresponds to a policy checkpoint, colored by the number of interaction steps. The yellow star denotes the zero-shot transfer policy π0 from Boston, while the lime star shows behavior cloning using Singapore trajectories. The red dashed line and points indicate the empirical Pareto frontier on this test set. Frontier checkpoints consistently dominate the baselines and exhibit only a mild trade-off between task success and realism. Blue triangles report the performance of selected frontier policies on a held-out Singapore test set, confirming that the Pareto dominance generalizes beyond the test data.
Figure 33: Pareto frontier of success rate versus realism meta score across training checkpoints under Singapore-to-Boston transfer. Each point corresponds to a policy checkpoint, colored by the number of interaction steps. The yellow star denotes the zero-shot transfer policy π0 from Singapore, while the lime star shows behavior cloning using Boston trajectories. The red dashed line and points indicate the empirical Pareto frontier on this test set. Frontier checkpoints consistently dominate the baselines and exhibit only a mild trade-off between task success and realism. Blue triangles report the performance of selected frontier policies on a held-out Boston test set, confirming that the Pareto dominance generalizes beyond the test data.
Figure 34: Pareto frontier of success rate versus realism meta score across training checkpoints under Singapore-to-Pittsburgh transfer. Each point corresponds to a policy checkpoint, colored by the number of interaction steps. The yellow star denotes the zero-shot transfer policy π0 from Singapore, while the lime star shows behavior cloning using Pittsburgh trajectories. The red dashed line and points indicate the empirical Pareto frontier on this test set. Frontier checkpoints consistently dominate the baselines and exhibit only a mild trade-off between task success and realism. Blue triangles report the performance of selected frontier policies on a held-out Pittsburgh test set, confirming that the Pareto dominance generalizes beyond the test data.
Figure 35: Overly conservative behavior when facing curbed parking vehicles. When facing many static parking vehicles, the vehicle 33 shows overly conservative behavior and thus fails to reach the goal in time. The gray dashed line shows the path the vehicle should follow and the red one shows the generated trajectory.
Figure 36: Being lost if initialized off-road. The vehicle 9 is initialized outside the road. At the beginning, the vehicle tries to go on road. However, as the goal is outside of the road as well, it finally turn left and drive outside of the road again.
Figure 37: Narrow-lane failure under Boston-to-Singapore transfer. Representative frames from a Boston-to-Singapore transfer scenario illustrating geometric mismatch in narrow roads. (a) Zero-shot transferred policy. (b) NOMAD-adapted policy. The zero-shot policy, trained in Boston, fails to account for the narrower lane geometry commonly found in Singapore: by tsim=2.4 s vehicles 1 and 2 have both drifted outside the drivable region, and vehicle 1 never reaches its goal. In contrast, NOMAD adapts to the target-city road geometry through map-based self-play and maintains stable lane-centering throughout the maneuver, so both vehicles reach their goals. Numbers identify the controlled vehicles and gi marks the goal of vehicle i ; a controlled vehicle is drawn in orange while it is off-road.
Figure 38: Driving-side mismatch under Boston-to-Singapore transfer. (a) Zero-shot transferred policy. (b) NOMAD-adapted policy. The zero-shot policy exhibits a systematic bias toward right-hand driving learned in the source city: vehicles 1 (moving down) and 2 (moving up) both drift to their right, toward the opposing lane, and collide head-on at tsim=3.3 s in the left-hand traffic setting of Singapore, and neither vehicle reaches its goal. In contrast, NOMAD adapts its lane-keeping behavior through map-based self-play and consistently drives on the correct side of the road, so both vehicles reach their goals. Numbers identify the controlled vehicles and gi marks the goal of vehicle i ; a controlled vehicle is drawn in red while it is in collision and in orange while it is off-road.
Figure 39: Driving-side mismatch under Singapore-to-Boston transfer (same scenario as Figure 5 ). (a) Zero-shot transferred policy. (b) NOMAD-adapted policy. The zero-shot policy retains a left-hand driving bias learned in the source city: vehicle 2 drifts into the lower lane, which belongs to oncoming traffic, and collides head-on with vehicle 1 at tsim=3.6 s, and vehicle 4 drifts to its left until it leaves the road, resulting in lane violations, collisions, and off-road behavior in the right-hand traffic setting of Boston. In contrast, NOMAD adapts its lane-keeping behavior via map-based self-play and consistently drives on the correct side of the road, so all four vehicles reach their goals. Numbers identify the controlled vehicles and gi marks the goal of vehicle i ; a controlled vehicle is drawn in red while it is in collision and in orange while it is off-road.
Figure 40: Cascading collision failure from a Boston-to-Singapore transfer scenario involving dense static obstacles. (a) Zero-shot transferred policy. (b) NOMAD-adapted policy. The scene is rotated so that the vehicles of interest travel from left to right. Under the zero-shot policy, vehicle 1 fails to execute an early rightward avoidance maneuver and collides with the first static vehicle at tsim=2.8 s. Following the collision, it initiates a right-turn corrective action, whereas a leftward maneuver is required to avoid the second obstacle, leading to a second collision at tsim=8.0 s. In contrast, NOMAD selects appropriate avoidance actions at both decision points, and all three controlled vehicles reach their goals, illustrating improved closed-loop decision making under target-city geometric and obstacle-layout shifts. Numbers identify the controlled vehicles and gi marks the goal of vehicle i ; a controlled vehicle is drawn in red while it is in collision and in orange while it is off-road.
Figure 41: Interaction mismatch frames from a Singapore-to-Boston transfer scenario involving multiple interacting vehicles. (a) Zero-shot transferred policy. (b) NOMAD-adapted policy. The zero-shot policy exhibits miscalibrated interaction behavior: on the upper-left approach, vehicle 2 travels considerably faster than vehicle 3 ahead of it (about 10 vs. 6 m/s) and rear-ends it at tsim=2.3 s, while at the intersection vehicle 6 enters without yielding to vehicle 4, causing a second collision at tsim=3.2 s. In contrast, NOMAD learns more balanced speed regulation and yielding behavior through map-based self-play: all six controlled vehicles reach their goals without collision or off-road events. Numbers identify the controlled vehicles and gi marks the goal of vehicle i ; a controlled vehicle is drawn in red while it is in collision and in orange while it is off-road.
Autonomous driving develops vehicles that perceive road scenes, predict interactions, and plan and control safe movement with limited or no human input.
Self-play reinforcement learning has recently emerged as a way to train driving policies without any human data. It uses cheap, large-scale simulations to substitute expensive, large-scale human driving demonstrations. A key limitation of this approach is that policies trained through pure self-play can learn effective but alien driving conventions incompatible with people. Previous works attempt to mitigate such behavioral misalignments through extensive reward engineering and domain randomization, which are brittle and labor-intensive. Instead of completely discarding human demonstrations, our method treats them as a regularization objective on top of a minimal safe goal-reaching reward. Like the spice in a good stew, we find that a little human data goes a long way: our method uses only 30 minutes of human demonstrations, 2500x fewer than comparable imitation learning approaches. Resulting policies coordinate with held-out human trajectories and complete training in 15 hours on a single consumer-grade GPU. Videos and full source code are available at https://spiced-self-play.com/.
Daphne Cornelisse, Julian Hunt, Zixu Zhang +4
NYU Tandon School of Engineering · NYU Courant · Princeton University +2
Autonomous driving has made remarkable progress through imitation learning with massive human demonstration data. However, a trained planner often degrades severely when applied to a new environment zero-shot, because of domain shifts in traffic regulations, road layout and driving behaviors. Therefore, adapting a trajectory planner to a new city typically requires resource-demanding local data collection with a vehicle sensor suite. In this work, we show that driving behavior can be learned from a scalable and efficient alternative. We introduce SkyDrive, a framework that utilizes drone-based traffic monitoring to provide efficient supervision for autonomous driving agents in a new environment. While vehicle-based data collection logs the ego and its surroundings, an aerial platform naturally observes many road users simultaneously over an extended field of view. As a result, every vehicle can be a data source with grounded driving behavior, effectively scaling up the amount of supervision. Based on 137 hours of aerial traffic monitoring footage, we extract 650K driving samples and construct a benchmark for trajectory planners and motion predictors. Zero-shot experiments with multiple models reveal significant cross-city domain gaps, but many of them can be alleviated by limited supervision from the sky, e.g., 30 minutes of monitoring per location. Our findings show that aerial traffic monitoring is an efficient and scalable data source for adapting autonomous driving systems in new cities. Data and code will be made publicly available.
Training autonomous driving policies through pure self-play has recently shown promising results. Following Gigaflow and Puffer- Drive, we train driving policies in a similar self-play fashion, but extend the models from MLPs to Transformers and train on the high-definition map of a real city, where we ultimately aim to deploy them. On the CARLA and Waymax benchmarks, our policies fall short of Gigaflow, and we trace the gap to specific failure modes, including reward hacking at traffic lights and a missing incentive to stop at stop signs. We further analyze which traffic rules emerge from self-play and how closely they match human driving, and we confirm that reward conditioning yields the intended diversity of driving behaviors. A demonstration of a trained policy is available at https://laursisask-ut.github.io/eccvdemo.