AutoWorld: Learning Multi-Agent Traffic Simulation with Self-Supervised World Models

Mozhgan Pourkeshavarz; Tianran Liu; Nicholas Rhinehart

cs.ROMar 30, 2026

AutoWorld: Learning Multi-Agent Traffic Simulation with Self-Supervised World Models

Authors: Mozhgan Pourkeshavarz, Tianran Liu, Nicholas Rhinehart

Organizations: University of Toronto, Canada

Abstract

Simulation with realistic traffic agents is essential for validating autonomous driving systems. Existing data-driven simulators learn agent behavior from higher-level abstractions such as 3D bounding boxes and polylines, inferred by upstream perception pipelines. These lossy abstractions discard sensory context that directly shapes agent behavior, limiting the distributional realism that simulation aims to reproduce. To address this limitation, we propose AutoWorld, a traffic simulation framework that grounds agent behavior in raw sensor observations through a self-supervised world model trained on LiDAR occupancy data. Given world model samples, AutoWorld constructs a coarse-to-fine predictive scene context as input to a multi-agent motion generation model. Furthermore, we designed a motion-aware latent supervision objective that enriches AutoWorld's latent representation of scene dynamics. To better exploit this latent space during inference, AutoWorld employs a cascaded Determinantal Point Process framework to guide diversity-aware sampling across both the world model and motion model. Experiments on the Waymo Sim Agents Challenge (WOSAC) demonstrate that AutoWorld achieves competitive performance, with larger gains in partially-observed scenarios where trajectory abstractions are most limited. We further show that grounding simulation in raw LiDAR through AutoWorld scales better with additional data than trajectory-only and LiDAR-conditioning baselines. Ablations confirm the contribution of each component.

Figures & tables

Figure 1: Comparison of traffic simulation approaches . (Left) Conventional traffic simulation methods. (Right) AutoWorld grounds simulation in future scene occupancies predicted by a self-supervised LiDAR world model, preserving sensory context discarded by trajectory representations.

Figure 2: Overview of AutoWorld . AutoWorld learns latent scene dynamics from unlabeled LiDAR occupancy sequences and uses predicted future occupancies to condition diffusion-based motion generation. At inference, a cascaded sampling strategy further improves use of the learned latent spaces across both scene evolution and agent behavior without additional training.

Model	Reference	RMM	Kinematic	Interactive	Map-based	minADE
LLM2AD [ 40 ]	CoRL 2025	0.7779	0.4846	0.8048	0.9109	1.2827
SMART [ 45 ]	NeurIPS 2024	0.7814	0.4854	0.8089	0.9153	1.3931
+ CAT-K [ 50 ]	CVPR 2025	0.7846	0.4931	0.8106	0.9177	1.3065
+ RLFTSim [ 5 ]	CVPR 2026	0.7857	0.4927	0.8129	0.9183	1.3252
+ SMART-R1 [ 31 ]	ICLR 2026	0.7858	0.4944	0.8110	0.9201	1.2885
+ DecompGAIL [ 10 ]	ICLR 2026	0.7864	0.4919	0.8152	0.9176	1.4209

Table 1: Results on the WOSAC leaderboard . RMM (Realism Meta Metric) is the primary ranking metric. For each metric, the best , second , and third ranked values are highlighted. The “+” denotes fine-tuning-based extensions of the SMART. Except for minADE, higher values are better.

Figure 3: Qualitative rollout comparison. SMART leads to a collision between the vehicle ( ) and pedestrian ( ), while AutoWorld maintains a realistic yielding interaction.

Figure 4: Simulation performance across occlusion levels. AutoWorld–SMART gaps are measured as differences between benchmark-normalized z-scores

(\mu\!=\!0,\sigma\!=\!1)

computed from WOSAC submissions. AutoWorld’s advantage becomes larger under heavier occlusion.

Figure 5: Scaling with different data modalities. AutoWorld scales better with additional raw LiDAR data.

Table 7

Figure 8

Conditioning	RMM $(\uparrow)$	minADE $(\downarrow)$
Future latent only	0.7562	1.4519
+ Pred. scene context	0.7618	1.4334

Table 4: Effect of coarse-to-fine predictive scene context on traffic simulation. We compare conditioning on future latent occupancies alone versus combining them with a predictive scene context, under a fixed sampling budget with IID sampling. Incorporating the predictive scene context improves overall realism.

Occlusion	Model / Statistic	RMM	Kinematic	Interactive	Map-based
Low	SMART	0.7728	0.4851	0.7948	0.9090
	AutoWorld	0.7743	0.4872	0.7965	0.9098
	Gap	+0.0015	+0.0021	+0.0017	+0.0008
	$\Delta$ z-score	+0.10	+0.08	+0.12	+0.06
Moderate	SMART	0.7694	0.4783	0.7908	0.9082
	AutoWorld	0.7741	0.4868	0.7964	0.9097

Table 5: Simulation performance across occlusion levels. We report raw scores, AutoWorld–SMART gaps ( ), and differences between benchmark-normalized z-scores

(\mu=0,\sigma=1)

computed from WOSAC submissions ( ). Within each metric, darker shades indicate larger magnitudes. AutoWorld’s advantage becomes larger under heavier occlusion.

Metric	Mean $(\mu)$	Std. Dev. $(\sigma)$
RMM	0.7768	0.0153
Kinematic	0.4811	0.0268
Interactive	0.8036	0.0143
Map-based	0.9113	0.0139

Table 6: WOSAC submission statistics used for z-score normalization. We report the mean and standard deviation computed across WOSAC submissions for each metric.

Model	N $\times$ M	RMM	Kinematic	Interactive	Map-based	minADE
Baseline	- $\times$ 32	0.7472	0.4749	0.7741	0.8682	1.4892
+ World model grounding	1 $\times$ 32	0.7610	0.4849	0.7855	0.8873	1.4369
	4 $\times$ 8	0.7597	0.4840	0.7839	0.8861	1.4256
	8 $\times$ 4	0.7618	0.4846	0.7876	0.8870	1.4334
	32 $\times$ 1	0.7523	0.4796	0.7728	0.8819	1.4178
Full AutoWorld (Cascaded Diversity; only meaningful when $N>1$ and $M>1$ )

Table 7: Effect of budget allocation between world model and motion generation.

N

and

M

denote the numbers of world-model hypotheses and motion samples per hypothesis, respectively. Allocating the full budget to either world-model hypotheses or motion samples hurts realism, while a balanced allocation achieves the best overall performance. The “ † ” denotes the model used for leaderboard submission.

Notation	Name / Description	Domain
Scene encoder
$A$	Number of agents	$\mathbb{N}$
$v_{l}$	Number of map polylines	$\mathbb{N}$
$v_{p}$	Number of sampled waypoints per polyline	$\mathbb{N}$
$v$	Total number of map vector points ( $v=v_{l}v_{p}$ )	$\mathbb{N}$
$r$	Number of traffic lights	$\mathbb{N}$

Table 9: Main notations used in AutoWorld. For functions, “Domain” refers to the domain of the function’s output.

Explore similar work

OccSim: Multi-kilometer Simulation with Long-horizon Occupancy World Models
Mar 30, 2026Tianran Liu, Shengwen Zhao, Mozhgan Pourkeshavarz +2Autonomous Driving Simulation World Models for Autonomous Driving
Learning, Embodied Autonomy, and Forecasting (LEAF) Lab, University of Toronto
HybridWorldSim: A Scalable and Controllable High-fidelity Simulator for Autonomous Driving
Nov 27, 2025Qiang Li, Yingwenqi Jiang, Tuoxi Li +173D Reconstruction Novel View Synthesis
1XPeng Motors · 2ShanghaiTech University · University of Science and Technology of China
Long-term Traffic Simulation via Structured Autoregressive Modeling
Jun 30, 2026Lingyu Xiao, Zexin Feng, Xintao YanIntelligent Transportation Systems Autonomous Driving Simulation
The University of Hong Kong, Pokfulam, Hong Kong

Mar 30, 2026J/K move · Enter open · S save

OccSim: Multi-kilometer Simulation with Long-horizon Occupancy World Models

Tianran Liu, Shengwen Zhao, Mozhgan Pourkeshavarz, Weican Li+1

Learning, Embodied Autonomy, and Forecasting (LEAF) Lab, University of Toronto

Autonomous Driving Simulation World Models for Autonomous Driving Autonomous Driving Scenario Generation World Models

Open paper AskPDF arXiv

Data-driven autonomous driving simulation has long been constrained by its heavy reliance on pre-recorded driving logs or spatial priors, such as HD maps. This fundamental dependency severely limits scalability, restricting open-ended generation capabilities to the finite scale of existing collected datasets. To break this bottleneck, we present OccSim, the first occupancy world model-driven 3D simulator. OccSim obviates the requirement for continuous logs or HD maps; conditioned only on a single initial frame and a sequence of future ego-actions, it can stably generate over 3,000 continuous frames, enabling the continuous construction of large-scale 3D occupancy maps spanning over 4 kilometers for simulation. This represents an >80x improvement in stable generation length over previous state-of-the-art occupancy world models. OccSim is powered by two modules: W-DiT based static occupancy world model and the Layout Generator. W-DiT handles the ultra-long-horizon generation of static environments by explicitly introducing known rigid transformations in architecture design, while the Layout Generator populates the dynamic foreground with reactive agents based on the synthesized road topology. With these designs, OccSim can synthesize massive, diverse simulation streams. Extensive experiments demonstrate its downstream utility: data collected directly from OccSim can pre-train 4D semantic occupancy forecasting models to achieve up to 67% zero-shot performance on unseen data, outperforming previous asset-based simulator by 11%. When scaling the OccSim dataset to 5x the size, the zero-shot performance increases to about 74%, while the improvement over asset-based simulators expands to 22.1%.

Abstract

Explore similar work

OccSim: Multi-kilometer Simulation with Long-horizon Occupancy World Models

HybridWorldSim: A Scalable and Controllable High-fidelity Simulator for Autonomous Driving

Long-term Traffic Simulation via Structured Autoregressive Modeling