Self-play in high-throughput simulators yields driving policies with robust closed-loop performance, but improvement per unit of simulation diminishes as training scales. Policies learn to handle common situations early, while further rollouts repeatedly encounter unresolved failures. Post-training offers an opportunity to target these failures, but existing methods primarily evaluate alternative actions or continuations at visited states, although successful recovery may require changing driving style earlier. We propose CounterPlay, a counterfactual self-play post-training approach that backtracks from failed tasks and retries them under alternate driving styles. CounterPlay rests on three key components. First, failure-driven backtracking uses the policy's value estimates to select an earlier stored state from which to retry the task. Second, reward conditioning enables a single policy to retry the task from this state using candidate styles ranging from cautious to aggressive. Third, CounterPlay retains task-completing retries only if no other vehicle incurs a new or earlier collision or off-road event relative to the factual branch. Retries that pass verification with fresh randomness are then distilled into the policy under its deployment condition. On BehaviorBench, CounterPlay achieves state-of-the-art scores on both the Interactive and Random splits across all eight traffic regimes using 1B post-training transitions, which is just 1% of the anchor's 100B self-play training budget. Improvements over the anchor hold across all three evaluated driving styles. CounterPlay resolves a substantial fraction of the anchor's timeout cases on BehaviorBench and achieves a balance between task completion and safety that neither continued self-play nor adopting a more aggressive driving style attains.
Figures & tables
Fig. 1: The self-play policy repeatedly waits for a gap and times out. Re-sampling the same driving style from a late state fails to recover the task. CounterPlay backtracks to an earlier root and explores alternative driving styles. An aggressive retry completes the task, passes verification with fresh randomness, and is distilled under the deployment condition.
Fig. 2: Overview of CounterPlay . Based on a pretrained self-play anchor policy, Stage 1 uses critic estimates recorded along the failed trajectory to select an earlier root. Stage 2 then restores this state and simulates a factual branch under the original reward condition and retries under the candidate profiles. Stage 3 filters and ranks retries using task completion and other-vehicle outcomes relative to the factual branch. It then reruns the factual and selected branches with fresh shared random draws to verify the recovery. Stage 4 distills the verified retry into the policy under the deployment condition.
Interactive1k
Random1k
Planner
IDM
PPO
SMART
Exp.
Aggr.
Norm.
Caut.
Mix
Mean
IDM
PPO
SMART
Exp.
Aggr.
Norm.
Caut.
Mix
Mean
Anchor
48.95
66.84
59.00
55.68
68.48
60.64
49.39
59.29
58.53
62.64
69.11
63.97
62.78
71.79
70.02
66.05
68.03
66.80
PPO-Continue
50.92
65.27
56.82
53.53
69.47
61.93
51.00
60.58
58.69 ± 0.06
65.29
70.92
64.50
66.09
71.64
70.92
66.55
69.67
68.20 ± 0.13
PPO-Rewind
50.53
64.98
57.17
54.08
68.99
62.16
50.76
58.97
58.46 ± 0.21
65.63
71.12
65.03
65.54
71.77
70.86
66.83
69.35
68.27 ± 0.29
PPO-Opponent
49.90
66.33
57.33
53.91
69.37
63.04
50.42
59.11
58.68 ± 0.38
64.95
70.81
64.52
66.34
71.82
70.91
67.72
69.67
68.34 ± 0.13
SPICED † [ 16 ]
44.91
62.96
59.83
55.69
68.02
65.35
51.72
61.02
58.69 ± 0.94
69.29
73.78
67.59
62.74
75.17
72.48
67.65
70.40
69.89 ± 0.44
TABLE I: Benchmark scores per traffic regime on BehaviorBench.
Fig. 3: Qualitative results. Each pair compares the anchor (left) and CounterPlay (right) in the same scene and traffic regime. Ego trajectories are colored by elapsed time in seconds. Grey arrows show other road users’ motion. Open boxes mark ego starts and stars mark goals. Red crosses mark collisions. Circles and triangles mark timeouts and off-road events, respectively.
Variant
B
C
S
D
Score ↑
Anchor (no post-training)
–
–
–
–
58.53
PPO-Continue
✗
✗
✗
✗
58.69 ± 0.06
Backtracking
10 decisions before failure
10
✓
✓
✓
62.83 ± 0.41
30 decisions before failure
30
✓
✓
✓
65.21 ± 0.33
50 decisions before failure
50
✓
✓
✓
64.58 ± 0.39
TABLE II: Component ablations on Interactive1k.
Policy
Completed
Timeout
AF coll.
Non-AF coll.
Offroad
Interactive1k
Anchor
68.4
23.1
3.8
4.5
0.2
PPO-Continue
96.3 ± 0.11
4.9 ± 0.5
30.0 ± 3.3
9.3 ± 1.6
33.3 ± 11.1
CounterPlay
95.1 ± 0.59
45.9 ± 1.4
38.9 ± 5.1
38.0 ± 2.6
40.7 ± 12.8
Random1k
Anchor
75.6
18.5
1.9
1.8
2.2
TABLE III: Task retention and failure repair after post-training (%).
Normal
Halfway
Aggressive
Method
Score ↑
AF-C. ↓
Score ↑
AF-C. ↓
Score ↑
AF-C. ↓
Anchor
58.53
3.75
64.21
4.03
69.85
4.38
PPO-Continue
58.69 ± 0.06
2.17 ± 0.14
64.73 ± 0.72
2.75 ± 0.50
69.49 ± 0.89
3.12 ± 0.62
Aggressive distillation
65.44 ± 0.42
4.46 ± 0.35
70.07 ± 0.39
4.62 ± 0.33
66.22 ± 0.47
5.08 ± 0.41
Cautious distillation
61.18 ± 0.48
1.86 ± 0.22
65.83 ± 0.45
1.90 ± 0.24
68.94 ± 0.52
2.20 ± 0.27
CounterPlay
66.61 ± 0.14
2.57 ± 0.29
73.55 ± 0.18
1.79 ± 0.07
73.60 ± 0.75
1.92 ± 0.51
TABLE IV: Effect of ego deployment condition on Interactive1k.
Search styles
Score ↑
AF-C. ↓
Off. ↓
Goal ↑
All five styles ( CounterPlay )
66.61 ± 0.14
2.57 ± 0.29
0.67 ± 0.07
79.0 ± 0.14
Cautious
57.02 ± 0.55
2.88 ± 0.31
0.50 ± 0.12
66.5 ± 0.62
Cautious–normal midpoint
54.27 ± 0.47
2.75 ± 0.26
1.62 ± 0.21
63.5 ± 0.53
Normal
54.18 ± 0.52
3.58 ± 0.33
1.25 ± 0.18
63.5 ± 0.58
Normal–aggressive midpoint
64.56 ± 0.37
3.84 ± 0.33
1.12 ± 0.15
75.3 ± 0.44
Aggressive
65.75 ± 0.28
3.06 ± 0.27
0.88 ± 0.11
77.9 ± 0.35
TABLE V: Search with one or five driving styles on Interactive1k.
Autonomous driving develops vehicles that perceive road scenes, predict interactions, and plan and control safe movement with limited or no human input.
Training autonomous driving policies through pure self-play has recently shown promising results. Following Gigaflow and Puffer- Drive, we train driving policies in a similar self-play fashion, but extend the models from MLPs to Transformers and train on the high-definition map of a real city, where we ultimately aim to deploy them. On the CARLA and Waymax benchmarks, our policies fall short of Gigaflow, and we trace the gap to specific failure modes, including reward hacking at traffic lights and a missing incentive to stop at stop signs. We further analyze which traffic rules emerge from self-play and how closely they match human driving, and we confirm that reward conditioning yields the intended diversity of driving behaviors. A demonstration of a trained policy is available at https://laursisask-ut.github.io/eccvdemo.
Self-play reinforcement learning has recently emerged as a way to train driving policies without any human data. It uses cheap, large-scale simulations to substitute expensive, large-scale human driving demonstrations. A key limitation of this approach is that policies trained through pure self-play can learn effective but alien driving conventions incompatible with people. Previous works attempt to mitigate such behavioral misalignments through extensive reward engineering and domain randomization, which are brittle and labor-intensive. Instead of completely discarding human demonstrations, our method treats them as a regularization objective on top of a minimal safe goal-reaching reward. Like the spice in a good stew, we find that a little human data goes a long way: our method uses only 30 minutes of human demonstrations, 2500x fewer than comparable imitation learning approaches. Resulting policies coordinate with held-out human trajectories and complete training in 15 hours on a single consumer-grade GPU. Videos and full source code are available at https://spiced-self-play.com/.
Daphne Cornelisse, Julian Hunt, Zixu Zhang +4
NYU Tandon School of Engineering · NYU Courant · Princeton University +2
Open-loop imitation learning has advanced modern autonomous driving policy architectures, but closed-loop deployment remains vulnerable to policy-induced distribution shift. Existing post-training paradigms exhibit fundamental trade-offs: closed-loop RL fine-tuning provides grounded feedback from executed actions but is constrained by the sparsity of informative events, whereas counterfactual fine-tuning provides dense supervision over candidate futures but inherits bias from imperfect future estimates. We introduce Counterfactual-to-Interactive Reinforcement Fine-Tuning (CRAFT), an on-policy framework that formulates closed-loop post-training as proxy-residual optimization. CRAFT uses group-normalized counterfactual advantages as a dense proxy for real closed-loop advantages and aligns this proxy with the closed-loop world through grounded residual correction from interaction-critical events. To stabilize adaptation, CRAFT regularizes the online policy toward an EMA teacher via asymmetric KL self-distillation. Theoretically, CRAFT decomposes the real closed-loop policy gradient into proxy and residual terms under the same visited-state distribution, reducing residual variance with an aligned proxy while mitigating proxy bias through grounded residual approximation. Empirically, CRAFT achieves the strongest closed-loop gains on Bench2Drive across hierarchical planning, vision-language-action, and vocabulary-scoring architectures. Ablations, scaling behavior, stability analyses, and transfer results further validate the complementary roles of dense counterfactual proxy and grounded residual correction. Project page: https://currychen77.github.io/CRAFT.
Keyu Chen, Nanfei Ye, Yida Wang +4
School of Vehicle and Mobility, Tsinghua University · Li Auto Inc