Organizations: State Key Laboratory of Intelligent Transportation System, Key Laboratory of Autonomous Transportation Technology for Special Vehicles, Ministry of Industry and Information Technology, School of Transportation Science and Engineering, Beihang University, Beijing 100191, China. · State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China.
Learning diverse and high-fidelity traffic simulations from human driving demonstrations is crucial for autonomous driving evaluation. The recent next-token prediction (NTP) paradigm, widely adopted in large language models (LLMs), has been applied to traffic simulation and achieves iterative improvements via supervised fine-tuning (SFT). However, such methods limit active exploration of potentially valuable motion tokens, particularly in suboptimal regions. Entropy patterns provide a promising perspective for enabling exploration driven by motion token uncertainty. Motivated by this insight, we propose a novel tokenized traffic simulation policy, R1Sim, which represents an initial attempt to explore reinforcement learning based on motion token entropy patterns, and systematically analyzes the impact of different motion tokens on simulation outcomes. Specifically, we introduce an entropy-guided adaptive sampling mechanism that focuses on previously overlooked motion tokens with high uncertainty yet high potential. We further optimize motion behaviors using Group Relative Policy Optimization (GRPO), guided by a safety-aware reward design. Overall, these components enable a balanced exploration-exploitation trade-off through diverse high-uncertainty sampling and group-wise comparative estimation, resulting in realistic, safe, and diverse multi-agent behaviors. Extensive experiments on the Waymo Sim Agent benchmark demonstrate that R1Sim achieves competitive performance compared to state-of-the-art methods.
Figures & tables
Fig. 1 : The motivation of R1Sim. (a) Exploration: Compared with Top-K sampling [ 5 ] , our entropy-guided adaptive sampling explores more high-entropy motion tokens. (b) Exploitation: Compared with SFT [ 6 ] , our refined GRPO estimates group-wise advantages and selects the optimal scenario.
Fig. 2 : Temporal evolution of token entropy and its role in characterizing scene uncertainty. (a) visualizes representative low- and high-entropy motion patterns at different time steps, highlighting the interested vehicle in red. (b) shows the temporal evolution of the ranked token probability distribution and the corresponding motion token entropy, where higher entropy coincides with a flatter distribution.
Fig. 3 : An overview of R1Sim framework. Our framework follows (A) an NTP-based autoregressive formulation for sequential motion token generation. Exploration is facilitated by (B) an entropy-guided adaptive sampling strategy that allocates sampling budget according to token entropy, while exploitation is guided by (C) a token-level reward model operating in traffic simulation. Policy network is optimized using (D) GRPO, enabling learning through group relative advantage estimation.
Method
RMM (↑)
Kinematic L.(↑)
Interactive L.(↑)
Map-based L.(↑)
Collision L.(↑)
Offroad L.(↑)
UniMM [ 8 ]
0.7684
0.4913
0.8101
0.8737
0.9679
0.9506
DRoPE [ 28 ]
0.7625
0.4779
0.8065
0.8685
0.9607
0.9443
SMART-large [ 5 ]
0.7614
0.4786
0.8066
0.8648
0.9632
0.9403
KiGRAS [ 29 ]
0.7597
0.4691
0.8064
0.8658
0.9617
0.9431
BehaviorGPT [ 16 ]
0.7473
0.4333
0.7997
0.8593
0.9537
0.9349
GUMP [ 30 ]
0.7431
0.4780
0.7887
0.8359
0.9403
0.9028
TABLE I : Results on the WOMD Test Dataset
Fig. 4 : Entropy distribution of generated scenarios.
Category
Method
Waymo Easy
Waymo Hard
RMM
Δ RMM
RMM
Δ RMM
Sampling
Top-K=5
0.9039
-0.0001
0.4652
+0.1039
Top-K=32
0.9018
+0.0020
0.5354
+0.0337
Top-K=64
0.9005
+0.0033
0.5591
+0.0100
Ours
0.9038
-
0.5691
-
Optimization
SMART (IL)
0.8958
+0.0080
0.5489
+0.0202
TABLE II : Samplings and Optimizations Comparison.
Category
Sampling
Optimization
RMM (↑)
Kinematic L.(↑)
Interactive L.(↑)
Map-based L.(↑)
Baseline
Top-K=32
IL
0.7654( ± 0.02)
0.4852( ± 0.01)
0.8056( ± 0.01)
0.8737( ± 0.02)
+ Sampling
Top-K=5
IL
0.7545( ± 0.01)
0.4554( ± 0.00)
0.8013( ± 0.01)
0.8652( ± 0.01)
Top-K=64
IL
0.7629( ± 0.03)
0.4855( ± 0.01)
0.8005( ± 0.00)
0.8730( ± 0.03)
Entropy-guided
IL
0.7670( ± 0.01)
0.4870( ± 0.01)
0.8076( ± 0.01)
0.8747( ± 0.02)
+ Optimization
Top-K=32
SFT
0.7669( ± 0.01)
0.4862( ± 0.01)
0.8079( ± 0.01)
0.8745( ± 0.02)
Top-K=32
GRPO-standard
0.7619( ± 0.02)
0.4729( ± 0.01)
0.8035( ± 0.01)
0.8736( ± 0.03)
TABLE III : Ablation Study on WOMD Validation Split.
Fig. 5 : Impact of the minimum bound kmin , maximum bound kmax and sample ranges on RMM.
Fig. 6 : Qualitative comparison of closed-loop rollouts. The left panel displays the initial scenario. The right panels illustrate the temporal evolution generated by (a) CATK [ 6 ] and (b) our R1Sim. Red boxes highlight the interested agents (ID 4 and 6), while circles (orange/green) mark critical interaction behaviors.
Supervised open-loop training has been widely adopted for training traffic simulation models; however, it fails to capture the inherently dynamic, multi-agent interactions common in complex driving scenarios. We introduce RLFTSim, a reinforcement-learning-based fine-tuning framework that enhances scenario realism by aligning simulator rollouts with real-world data distributions and provides a method for distilling goal-conditioned controllability in scenario generation. We instantiate RLFTSim on top of a pre-trained simulation model, design a reward that balances fidelity and controllability, and perform comprehensive experiments on the Waymo Open Motion Dataset. Our results show improvements in realism, achieving state-of-the-art performance. Compared with other heuristic search-based fine-tuning methods, RLFTSim requires significantly fewer samples due to a proposed low-variance and dense reward signal, and it directly addresses the realism alignment issue by design. We also demonstrate the effectiveness of our approach for distilling traffic simulation controllability through goal conditioning. The project page is available at https://ehsan-ami.github.io/rlftsim.
Interactive traffic simulation is a vital world model for autonomous driving. A central challenge in long-horizon simulation is modeling sustained multi-agent interactions, which is further exacerbated by dynamic token cardinality as agents continuously enter and exit the scene. In this work, we propose that the solution lies in the synergy between the architectural inductive biases and statistical priors of large-scale sequence models, e.g., Large Language Models (LLMs). Our probing experiments reveal that the transferability of attention mechanisms and the distributional consistency between motion tokens and natural language enable small-scale, heavily frozen LLMs to rapidly adapt to traffic modeling. Building on this insight, we introduce RosettaSim, a unified framework that projects scene topology, agent states, and spawning intents into a structured autoregressive stream with variable length, achieving both strong short-term accuracy and stable long-horizon simulation fidelity. Furthermore, evaluating extended rollouts presents yet another hurdle, as one-to-one agent correspondence inevitably fades over time. To address this, we introduce Retrieval-based Traffic Evaluation (RTE), which retrieves semantically similar real-world scenarios as context-aware reference anchors. Experiments on the Waymo Open Sim Agent Challenge (WOSAC) demonstrate that RosettaSim achieves state-of-the-art performance in both short- and long-term simulation. Furthermore, RTE exhibits a stronger correlation with standard metrics (r=0.83) than existing approaches (r=0.74), indicating improved alignment with long-horizon simulation fidelity.
Realistic and diverse traffic simulation is essential to autonomous driving development. Yet prevailing benchmarks predominantly reward realism, and recent methods have optimized accordingly, leaving diversity underexplored. We introduce Flow-ERD, a multi-agent simulator that pursues realism and diversity jointly. Its backbone, Agent-Type Aware Flow Matching (AFM), couples flow matching's multi-modal expressiveness with type-specific kinematic execution. It preserves fine-grained diversity while keeping motions consistent with each agent type. A second stage, Entropy-Regularized Distillation (ERD), fine-tunes the closed-loop rollout distribution with an entropy-regularized reverse-KL objective. This mitigates covariate shift while explicitly preventing collapse onto high-density modes. We evaluate Flow-ERD with a log-free diversity metric alongside standard realism scores. Flow-ERD ranks first on the WOSAC test benchmark and dominates the realism--diversity Pareto front among reproducible baselines. Our project page is available here.