FlowSim: Rendering-Free Closed-Loop Simulation for Evaluating End-to-End Driving Systems

Hanfeng Wu; Alvaro Marcos-Ramiro; Michael Schmidt; Christoph Stiller

cs.CVOct 13, 2026

FlowSim: Rendering-Free Closed-Loop Simulation for Evaluating End-to-End Driving Systems

Authors: Hanfeng Wu, Alvaro Marcos-Ramiro, Michael Schmidt, Christoph Stiller

Organizations: Karlsruhe Institute of Technology Munich, Germany · BMW Group Munich, Germany · Karlsruhe Institute of Technology Karlsruhe, Germany

Abstract

Closed-loop evaluation of end-to-end (E2E) driving systems typically relies on generating sensor observations and re-processing them through the perception encoder, introducing substantial computational cost and a sim2real gap between simulated and real sensor observations. In this paper, we ask a more fundamental question: is image-space simulation necessary for closed-loop evaluation of E2E driving systems? We propose FlowSim, a rendering-free simulation paradigm that predicts the internal latent representation directly from a previous latent and target-frame symbolic scene conditions. The predicted latent is then processed by the original frozen downstream modules, eliminating image rendering and perception re-encoding. Explicit cross-frame agent association improves latent transition prediction and rollout stability, while variable-horizon simulation enables long-horizon prediction from a common real anchor without accumulating intermediate rollout errors. Experiments on nuScenes and NavSim demonstrate that FlowSim preserves downstream behavior competitively with, and often exceeds image-space simulation in source-to-reference agreement across variable simulation horizons and both dense and sparse latent representations. By predicting the latent directly, both instantiations of FlowSim simulate each step in 0.04--0.06s, substantially faster than the image-space simulation methods they are compared against. These results establish FlowSim as an efficient and practical alternative to conventional image-space simulation methods.

Figures & tables

Panel of Figure 1 — Fig. 1: Comparison of our simulation paradigm ( FlowSim ) with 2 existing closed-loop simulation protocols for evaluating E2E driving systems, each producing a latent $\hat{Z}_{t}$ that is sourced by the same frozen downstream heads. (i) Neural reconstruction renders the target frame from the explicit scene conditions (agent boxes, ego pose) into pixels, which the frozen encoder re-encodes into $\hat{Z}_{t}$ . (ii) A world model likewise produces pixels for the same encoder, conditioned on its own previous images $I_{t-1}$ together with the scene conditions. (iii) FlowSim is conditioned on the previous latent $Z_{t-1}$ and the same scene conditions, yet predicts the latent $\hat{Z}_{t}$ directly , bypassing image rendering and re-encoding.

Fig. 2: Overview of FlowSim . Top: Training. A frozen perception encoder extracts the source and target latents, $Z_{t-1}$ and $Z_{t}$ , from real sensor observations. Given $Z_{t-1}$ and target-frame symbolic scene conditions, FlowSim predicts $\hat{Z}_{t}$ and is trained to reconstruct the target latent $Z_{t}$ . The same formulation is instantiated with a residual U-Net for UniAD’s dense BEV representation and a conditional flow-matching model for TransFuser’s sparse latent representation. Bottom: Simulation protocols. FlowSim supports both autoregressive simulation and variable-horizon simulation. In the latter, multiple future horizons are simulated directly from a common real latent anchor, preserving the real-to-simulated transition used during training and avoiding intermediate autoregressive drift. The predicted latent is passed directly to the frozen downstream stack to obtain the ego trajectory. Both simulation protocols can be coupled with a symbolic scene simulator that updates the ego and surrounding agents from the planned trajectory, enabling closed-loop simulation without image rendering.

Fig. 3: Associated dynamic-agent conditioning. Each target-frame agent is augmented with information from its corresponding source-frame agent. The resulting descriptor combines the target state, source state, relative motion, and a validity flag, explicitly exposing cross-frame correspondence and temporal dynamics to the latent simulator. New agents without a valid correspondence are retained with an invalid flag rather than removed.

		Detection		Map		Motion		Planning		Inference
Source	$\Delta t$	IoU $\uparrow$	cADE $\downarrow$	Lane $\uparrow$	Drive. $\uparrow$	ADE $\downarrow$	FDE $\downarrow$	ADE $\downarrow$	FDE $\downarrow$	Time (s) $\downarrow$	VRAM (GB) $\downarrow$
SplatAD (gt hist.) [ 10 ]	$0.5$ s	0.75	1.86	0.48	0.84	6.03	11.03	0.31	0.64	0.13	4.5
DreamForge (gt hist.) [ 14 ]		0.65	2.32	0.48	0.85	6.35	11.57	0.30	0.61	92.3	38.2
Warp ( $R_{t}{=}0$ )		0.57	1.89	0.46	0.82	5.87	10.72	0.62	1.14	< 0.01	-
FlowSim (VH, assoc.)		0.68	1.69	0.48	0.86	5.96	10.88	0.53	0.97	0.04	3.0
SplatAD (gen. hist.)	$1.5$ s	0.55	4.65	0.35	0.73	8.13	14.87	0.61	1.19	0.13	4.5

TABLE I: Multi-horizon downstream agreement with UniAD under the protocol in Sec. IV-C . FlowSim uses real-anchored latent prediction; DreamForge and SplatAD are image-space sources, and warp is a training-free ego-motion baseline. SplatAD does not extrapolate temporally, so its gt-history/generated-history results are repeated for reference.

\dagger

indicates the CPU offload, because of insufficient VRAM on A100(40GB). Bold indicates the best result at each horizon.

		Segmentation		Detection		Planning		Inference
Source	$n$	Pix. $\uparrow$	mIoU $\uparrow$	Box L2 $\downarrow$	Head. $\downarrow$	ADE $\downarrow$	FDE $\downarrow$	Time (s) $\downarrow$
OmniRe [ 8 ]	–	0.77	0.42	5.78	0.84	0.61	1.60	0.45
FlowSim (AR, no assoc.)	1	0.89	0.64	2.36	0.48	0.44	1.17	0.06
	2	0.86	0.59	2.67	0.58	0.59	1.56	0.12
	3	0.85	0.56	2.76	0.62	0.67	1.77	0.18
	4	0.85	0.55	2.76	0.62	0.71	1.89	0.24

TABLE II: Downstream agreement with TransFuser on NavSim navtest . We compare the image-space OmniRe reconstruction with FlowSim under autoregressive (AR) and real-anchored variable-horizon (VH) simulation, with and without agent association. Each step corresponds to a

0.5

s keyframe (

n{=}4

denotes

2.0

s). Bold marks the best FlowSim result at each horizon. Time denotes cumulative source-generation time to reach each horizon.

Fig. 4: Qualitative comparison of image-space and latent-space simulation on nuScenes, at a short ( $\Delta t{=}0.5$ s, left) and a long ( $\Delta t{=}4.0$ s, right) horizon. Each row shows, for a given source, the surround-view image inputs (or GT reference), the resulting BEV feature, and UniAD ’s downstream detection/lane/drivable-area/planning outputs decoded from it. The green and red boxes are GT annotations (agent conditions) and predicted results respectively. DreamForge and SplatAD must render an image and re-encode it through UniAD ’s frontend; FlowSim predicts the BEV latent directly and is rendering-free. At $\Delta t{=}0.5$ s all sources remain close to GT, but at $\Delta t{=}4.0$ s DreamForge and SplatAD accumulate visible rendering and re-encoding artifacts that degrade the BEV feature and downstream outputs, while FlowSim ’s latent and decoded outputs stay closer to GT.

Fig. 5: Qualitative single-frame agreement on NavSim navtest (two scenes with $\Delta t=0.5s$ ). Top row: front-view camera crops used to obtain the source latent for the real GT frame and for OmniRe ’s independently rendered frame. Bottom row: corresponding TransFuser -predicted BEV segmentation, detected agent boxes, and planning results for GT, FlowSim without and with agent association, and OmniRe .

Fig. 6: Counterfactual closed-loop rollout illustrated in Sec. V-C . We show four sampled steps of the $5.5$ s rollout; each panel shows the front-view cameras together with the BEV segmentation decoded from the logged real latent ( Real ) and from FlowSim ’s closed-loop simulated latent ( Sim ).

		Detection		Map		Planning
Protocol	$\Delta t$	IoU $\uparrow$	cADE $\downarrow$	Lane $\uparrow$	Driv. $\uparrow$	ADE $\downarrow$	FDE $\downarrow$
FlowSim (AR, no assoc.)	$0.5$ s	0.69	1.41	0.49	0.86	0.61	1.19
	$1.5$ s	0.58	2.13	0.36	0.77	1.25	2.45
	$2.5$ s	0.54	2.77	0.28	0.70	1.60	3.10
	$4.0$ s	0.46	4.80	0.19	0.61	1.97	3.76
FlowSim (AR, assoc.)	$0.5$ s	0.64	1.76	0.49	0.86	0.48	0.92

TABLE III: FlowSim rollout ablation with UniAD at four horizons: two autoregressive variants (with/without associated agents) that re-feed each generated latent, and the variable-horizon variant reproduced from Table I . Bold marks the best per horizon.

Explore similar work

DreamStream: Towards Policy-Oriented Generative Simulation for End-to-End Driving
Sep 22, 2026Ziyang Leng, Sicheng Mo, Seth Z. Zhao +4Autonomous Driving Benchmarks End-to-End Autonomous Driving
University of California, Los Angeles · Toyota Research Institute
RECAST: From Log Replay to Closed-Loop Driving Simulation with View-Complete Actors
Sep 25, 2026Zijun Zhao, Liewen Liao, Kang Shen +2Autonomous Driving Simulation 3D Asset Generation
Global Institute of Future Technology, Shanghai Jiao Tong University, Shanghai 200240, China · School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, and the Key Laboratory of System Control and Information Processing, Ministry of Education of China, Shanghai 200240, China
HorizonDrive: Self-Corrective Autoregressive World Model for Long-horizon Driving Simulation
May 12, 2026Conglang Zhang, Yifan Zhan, Qingjie Wang +10World Model Learning Long-Horizon Video Generation
Wuhan University · The University of Tokyo · Horizon Robotics +3

Sep 22, 2026J/K move · Enter open · S save

DreamStream: Towards Policy-Oriented Generative Simulation for End-to-End Driving

Ziyang Leng, Sicheng Mo, Seth Z. Zhao, Haoyuan Cai+3

University of California, Los Angeles · Toyota Research Institute

Autonomous Driving Benchmarks End-to-End Autonomous Driving Autoregressive Video Generation Autonomous Driving Simulation

Open paper AskPDF arXiv

Faithfully evaluating end-to-end driving policies in simulation requires observations that are not merely photo-realistic, but preserve the scene features a policy relies on to make decisions. Existing platforms, however, exhibit a sim-to-real visual gap that corrupts policy perception, undermining their ability to assess a policy's closed-loop decision-making. To this end, we propose DreamStream, a generative, closed-loop simulator that achieves policy-oriented fidelity using a simulator-grounded autoregressive video model. Our video model is distilled from a large pretrained video model via traffic layout guidance, varying visual appearance while preserving policy-relevant features such as scenario layout and the temporal consistency of dynamic objects. We further observe that perceptual metrics like FID misrank how well these features are preserved. To tackle this, we introduce FD

π

, a new multi-representation metric that measures the sim-to-real gap as the Fréchet distance over scene-context features from public E2E policies. Under FD

π

, DreamStream improves over the strongest prior closed-loop simulator by

1.6\times

on nuScenes and

4.7\times

on NAVSIM, and induces the least perturbation to policy's perceptual observability. Based on DreamStream, we construct Navhard-CL benchmark, which turns non-reactive real-world benchmark NAVSIM into interactive testing environments with adversarial driving behaviors and weather variations. This benchmark exposes many failure modes of driving policies, such as scorer bias and lack of recovery behaviors, that prior closed-loop benchmarks overlook. Code and data are available at https://github.com/VAIL-UCLA/DreamStream.

Abstract

Explore similar work

DreamStream: Towards Policy-Oriented Generative Simulation for End-to-End Driving

RECAST: From Log Replay to Closed-Loop Driving Simulation with View-Complete Actors

HorizonDrive: Self-Corrective Autoregressive World Model for Long-horizon Driving Simulation