Language-conditioned traffic simulation enables scalable test scenarios steered by natural language. The goal is to control selected agents while surrounding traffic reacts to their observed behaviour. For example, a vehicle instructed to cut in late should cause the following vehicle to brake only when the manoeuvre becomes observable. Yet simulators can leak private instructions across agents, causing surrounding vehicles to react before the instructed behaviour becomes observable. Such premature reactions defuse desired events and alter the ensuing closed-loop scenario. Existing benchmarks measure what agents do, not what they could have known, leaving this failure unmeasured. We formalise it as intent leakage and address it through evaluation, data and modelling. Our paired intervention test holds an instructed agent's executed trajectory fixed while varying its private instruction, isolating leakage-induced changes in surrounding agents. We also construct WOMD-PACT, an automatically generated, verifiable dataset of 531,099 WOMD scenes with timed behaviour tags with reaction episodes identified through a counterfactual ghost test and grounded private instructions. Finally, we introduce CaFe, a dual-stream diffusion-forcing simulator fine-tuned in closed loop that restricts inter-agent interactions to public beliefs, preserving legitimate anticipation without leaking private intent. Our evaluation reveals that existing conditioning designs displace 23-83% of nearby uninstructed agents through private-intent leakage, even when the instructed agent's executed motion is held fixed. CaFe eliminates measurable leakage while maintaining comparable WOSAC realism (0.763), demonstrating that causal faithfulness can be enforced without sacrificing measured simulation realism.
Figures & tables
Dataset
Base
Verif.
Private
Symm.
Inter.
Timed
ProSim-Instruct [ 37 ]
WOMD
✗
✗
✓
✗
∼
InterDrive [ 4 ]
WOMD, nuPlan
∼
✗
∼
✓
✗
WOMD-Reas. [ 21 ]
WOMD
∼
✗
✗
∼
∼
InstructWaymo [ 12 ]
WOMD
✓
–
✗
✗
✗
doPlan [ 31 ]
nuPlan
✓
–
✗
✗
∼
WOMD-PACT (ours)
WOMD
✓
✓
✓
✓
✓
Table 1: Instruction datasets for trajectory-level driving. We compare factual verification, private per-agent instructions, symmetric coverage, interaction identification and timing at behaviour onset. ∼ indicates partial support and – denotes single-agent or inapplicable settings.
Figure 2: Measuring intent leakage. We replay the instructed agent’s motion without its private instruction, compare surrounding agents across rollouts, and quantify leakage along two dimensions: the fraction of affected agents and their mean displacement.
Figure 3: CaFe. The private stream conditions each agent on its own instruction, while a language-blind pass of the frozen backbone predicts public beliefs from committed motion. In inter-agent attention, an agent reads its own private features and the public beliefs of others, never their private futures or instructions. After each one-second commit, both streams update from the new observable history. The example on the right shows how public beliefs change as an agent’s motion reveals its intention.
Leakage
Realism ↑
Accuracy and safety ↓
Method
Affected (%) ↓
Displacement (m) ↓
Meta
Kin.
Inter.
Map
minADE
Coll. (%)
Off-road (%)
1 s
4 s
8 s
1 s
4 s
8 s
ECoSim [ 6 ]
0.1
7.1
22.6
1.54
1.46
3.08
0.7685
0.4642
0.8032
0.8977
1.507
3.49
11.54
Backbone (no text)
n/a
n/a
n/a
n/a
n/a
n/a
0.7599
0.4493
0.7935
0.8940
1.417
4.34
11.68
Inter-agent attention [ 16 , 4 , 6 ]
0.1
11.2
26.7
0.78
1.40
2.77
0.7628
0.4624
0.7903
0.8990
1.362
5.82
12.22
Broadcast instructions [ 37 , 4 ]
22.2
69.2
82.7
0.87
2.32
4.14
0.7651
0.4694
0.7917
0.9000
1.435
6.06
12.18
Table 2: Intent leakage and realism on 1,000 WOMD-PACT validation scenes with 32 rollouts each. Variants reproduce the information routes of the cited works on a shared backbone with identical training data, sampler and leak-free guidance, isolating what each agent can observe. ECoSim [ 6 ] is evaluated as published as an external autoregressive baseline. Affected is the percentage of nearby moving uninstructed agents displaced by more than 0.5 m when instructed agents replay identical motion with their instructions removed, evaluated on the 746 eligible scenes (750 for ECoSim). Displacement is the mean displacement among affected agents. Realism and its components follow WOSAC 2025. n/a means that the leakage test is inapplicable (no-text backbone) or that mean displacement is undefined because no agents were affected. Bold indicates the best result among variants sharing our backbone.
Method
Clause (%) ↑
Reacts (%) ↑
B hits A (%) ↓
Meta ↑ (guidance-free)
No language
38.0
81.7
5.8
0.7494
Single stream
40.9
82.4
5.3
0.7466
History only
35.8
79.5
9.8
0.7238
CaFe (ours)
40.8
81.7
5.8
0.7481
Table 3: Instruction success and replay response quality. Clause satisfaction and responder metrics use eight matched rollouts per item. Responder metrics use fixed-motion replay with unprompted B .
Method
Guide
Meta ↑
Coll. ↓
Off-road ↓
Backbone
No
0.7494
5.6
13.7
Yes
0.7599
4.34
11.68
Single stream
No
0.7466
8.8
14.8
Yes
0.7628
5.82
12.22
CaFe (Ours)
No
0.7481
7.1
14.5
Yes
0.7631
4.81
12.10
Table 4: Effect of guidance. WOSAC 2025 realism, collision and off-road rates without and with guidance. Guided CaFe has 0.0% affected agents at 8 s.
Autonomous driving scenario generation creates realistic, controllable traffic situations for training and testing vehicles, especially in rare and safety-critical cases.
Closed-loop traffic simulation remains challenging because it must generate interactive multi-agent behaviors that are scene-consistent and controllable throughout rollout. Prior diffusion-based approaches achieve strong realism, but their computational cost can hinder deployment in time-constrained replanning loops for autonomous vehicle planning and simulation. We present a diffusion-based scenario generation framework conditioned on instance-centric scene context and multimodal proposal priors, with optional test-time guidance for shaping safety-critical behaviors. A compact action-latent representation and proposal-based initialization improve sampling efficiency and reduce per-step runtime without retraining. Experiments on the Waymo Open Motion Dataset demonstrate a favorable balance among realism, safety, and controllability across diverse interactive scenarios, while showing that test-time guidance enables systematic trade-offs among competing objectives.
Shubham Vaijanath Phoolari, Aleyna Kara, Christoph Lauer +1
Authors are with CARIAD SE, Germany. · Author is with the Institute of Automotive Engineering (FZD), TU Darmstadt, Germany. · Author is with the Technical University of Munich, Germany.
Using offline datasets to evaluate conversational agents often fails to cover rare scenarios or to support testing new policies. This has motivated the use of controllable user simulators for targeted, counterfactual evaluation, typically implemented by prompting or fine-tuning large language models. In this work, we formalize controllable simulation as a causal inference problem. By bridging natural language evaluation with off-policy evaluation methodology, we show that the standard practice of training simulators via supervised fine-tuning on post-hoc trajectory labels yields a structurally biased model. Specifically, these labels are inextricably coupled to the data-generating behavior policy, injecting a look-ahead bias that breaks causal consistency. Furthermore, we prove that under policy shift this failure causes the variance of evaluation metrics to explode geometrically, a phenomenon we term controllability collapse. To restore causal consistency, we establish theoretical conditions for accurate simulation and propose practical training mitigations: a priori controls, step-wise dynamic controls, and direct policy-conditioned learning. Empirical evaluation confirms that while standard global controls distort conversational distributions and collapse behavioral diversity, our causally grounded simulators eliminate look-ahead bias, preserve natural variance, and exhibit robust zero-shot generalization to unseen agent behaviors.
Supervised open-loop training has been widely adopted for training traffic simulation models; however, it fails to capture the inherently dynamic, multi-agent interactions common in complex driving scenarios. We introduce RLFTSim, a reinforcement-learning-based fine-tuning framework that enhances scenario realism by aligning simulator rollouts with real-world data distributions and provides a method for distilling goal-conditioned controllability in scenario generation. We instantiate RLFTSim on top of a pre-trained simulation model, design a reward that balances fidelity and controllability, and perform comprehensive experiments on the Waymo Open Motion Dataset. Our results show improvements in realism, achieving state-of-the-art performance. Compared with other heuristic search-based fine-tuning methods, RLFTSim requires significantly fewer samples due to a proposed low-variance and dense reward signal, and it directly addresses the realism alignment issue by design. We also demonstrate the effectiveness of our approach for distilling traffic simulation controllability through goal conditioning. The project page is available at https://ehsan-ami.github.io/rlftsim.