RECAST: From Log Replay to Closed-Loop Driving Simulation with View-Complete Actors

Zijun Zhao; Liewen Liao; Kang Shen; Songan Zhang; Ming Yang

cs.ROSep 25, 2026

RECAST: From Log Replay to Closed-Loop Driving Simulation with View-Complete Actors

Authors: Zijun Zhao, Liewen Liao, Kang Shen, Songan Zhang, Ming Yang

Organizations: Global Institute of Future Technology, Shanghai Jiao Tong University, Shanghai 200240, China · School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, and the Key Laboratory of System Control and Information Processing, Ministry of Education of China, Shanghai 200240, China

Abstract

Closed-loop driving simulation requires rendered observations to remain reliable as the ego vehicle and surrounding actors move beyond their recorded trajectories, exposing views absent from the source log. Existing data-driven simulators reconstruct dynamic actors from sparse observations, which can result in rendering artifacts under these viewpoint changes. We introduce RECAST (REconstructing Controllable Actors for Simulation and Testing), a 3D Gaussian Splatting framework that generates a view-complete actor from a single segmented vehicle observation in a driving log and registers the generated actor in the reconstructed scene. RECAST supports planner-in-the-loop rendering under controlled ego-actor interactions. To adapt an image-to-3D prior to real vehicles, we further introduce RECAR, a dataset of approximately 20K real vehicles with 600K background-free RGBA images spanning diverse vehicle colors and types. We use two-stage adaptation to improve vehicle generation from real driving-log observations. At the actor level, RECAST reduces $\mathrm{FD}_{\mathrm{incep}}$ from 9.788 to 7.992 relative to unadapted TRELLIS. At the scene level, under actor motion beyond logged trajectories, RECAST reduces $\mathrm{FD}_{\mathrm{incep}}$ from 129.35 to 112.10 and increases $\mathrm{CLIP}_{\mathrm{margin}}$ ( $\times1000$ ) from 0.14 to 3.47 relative to Street Gaussians. We demonstrate planner-in-the-loop simulation with the image-conditioned planner GTRS-Dense. Compared with native Street Gaussians actors, RECAST increases the no-collision (NC) rate from 22.2% (12/54) to 63.0% (34/54) and the mean minimum predicted time-to-collision (TTC) from 0.798 s to 2.150 s. These experiments show that RECAST supports closed-loop planner evaluation under controlled ego-actor interactions beyond log replay. Visit our project page at https://zijunkr.github.io/RECAST/

Figures & tables

Fig. 1: Actor appearance beyond log replay. Actor motion beyond the logged trajectory exposes previously unseen sides of the vehicle, revealing rendering artifacts in Street Gaussians (StreetGS) [ 9 ] and OmniRe [ 7 ] . RECAST generates a view-complete actor from a logged image and preserves coherent vehicle appearance under these viewpoint changes. Ego Actor

Fig. 2: RECAST overview. Multi-view pseudo-targets from RECAR support vehicle-domain and driving-log-robust (log-robust) adaptation. Given a single segmented driving-log image, the adapted model generates a view-complete Gaussian actor, which is registered using the logged 3D box and LiDAR points and then inserted into the reconstructed background. During closed-loop simulation, the ego controller executes the planner’s predicted trajectory, while the actor controller follows the specified motion. Updated states and camera poses provide the next planner input. Ego Actor

Dataset	Vehicles	Type	Views	Resolution	BG-free	Regular
SRN-Car [ 31 ]	2.1K	Synthetic	250	$128{\times}128$	$\checkmark$	$\checkmark$
Objaverse-XL-Car [ 32 ]	11.4K	Synthetic	–	–	$\checkmark$	$\checkmark$
MVMC [ 33 ]	576	Real	10	$600{\times}450$	$\times$	$\times$
3DRealCar [ 30 ]	2.5K	Real	200	$1920{\times}1440$	$\times$	$\times$
MAD-CARS [ 12 ]	70K	Real	85	$1920{\times}1080$	$\times$	$\times$
RECAR	20K	Real	30	$1200{\times}800$	$\checkmark$	$\checkmark$

TABLE I: Comparison of vehicle-centric datasets.

Fig. 3: RECAR dataset. Examples of representative multi-view RGBA vehicle images, together with a bar chart illustrating the vehicle color distribution and a pie chart illustrating the vehicle type distribution.

Method	NC (%) $\uparrow$	DAC (%) $\uparrow$	TTC (s) $\uparrow$	$R_{c}$ (%) $\uparrow$	COM (%) $\uparrow$
StreetGS [ 9 ]	12/54 (22.2%)	53/54 (98.1%)	0.798	16.82	91.20
w/ TRELLIS [ 25 ]	33/54 (61.1%)	53/54 (98.1%)	2.115	16.81	94.68
w/ AH [ 14 ]	30/54 (55.6%)	54/54 (100%)	2.070	17.11	94.91
OmniRe [ 7 ]	19/54 (35.2%)	53/54 (98.1%)	1.178	16.67	89.12
w/ TRELLIS [ 25 ]	32/54 (59.3%)	52/54 (96.3%)	2.139	16.72	93.87
w/ AH [ 14 ]	33/54 (61.1%)	53/54 (98.1%)	2.119	16.44	94.44

TABLE II: Image-conditioned closed-loop evaluation with GTRS-Dense [ 37 ] .

Fig. 4: Scene-level qualitative comparison under controlled interactions on Waymo. We evaluate four target-actor motion settings. All methods share the same ego–actor trajectories within each scenario. Ego Actor

Scene quality
Method	$\mathrm{FD}_{\mathrm{incep}}\downarrow$	$\mathrm{CLIP}_{\mathrm{margin}}\uparrow$	$\mathrm{VQA}_{\mathrm{margin}}\uparrow$
StreetGS [ 9 ]	129.35	0.14	-294.91
w/ TRELLIS [ 25 ]	112.84	2.86	-172.36
w/ AH [ 14 ]	110.00	1.95	-156.42
OmniRe [ 7 ]	138.52	0.04	-314.18
w/ TRELLIS [ 25 ]	121.47	2.93	-216.82

TABLE III: Scene-level quantitative comparison under controlled interactions on Waymo.

Fig. 5: Actor-level qualitative comparison of single-image vehicle generation. The first three examples are from driving-log crops, while the last two examples are from 3DRealCar [ 30 ] . AH denotes Asset Harvester [ 14 ] , and Ours w/o II denotes our model without driving-log-robust adaptation.

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	$\mathrm{CLIP}_{\mathrm{img}}\uparrow$	$\mathrm{FD}_{\mathrm{incep}}\downarrow$	$\mathrm{FD}_{\mathrm{dinov2}}\downarrow$
Real3D [ 21 ]	13.19	0.5691	0.3223	68.17	91.196	1282.01
LN3Diff [ 22 ]	18.95	0.8843	0.1105	63.03	93.288	1455.05
3DTopia-XL [ 23 ]	15.54	0.8085	0.1817	59.90	166.294	1299.59
SAR3D [ 24 ]	15.39	0.8231	0.1588	66.55	64.042	746.99
AH [ 14 ]	19.22	0.8792	0.1044	78.29	38.368	183.08
TRELLIS [ 25 ]	21.02	0.9077	0.0603	78.22	9.788	43.89

TABLE IV: Actor-level quantitative generation results on held-out RECAR vehicles.

Explore similar work

Apr 20, 2026

Asset Harvester: Extracting 3D Assets from Autonomous Driving Logs for Simulation

Closed-loop simulation is a core component of autonomous vehicle (AV) development, enabling scalable testing, training, and safety validation before real-world deployment. Neural scene reconstruction converts driving logs into interactive 3D environments for simulation, but it does not produce complete 3D object assets required for agent manipulation and large-viewpoint novel-view synthesis. To address this challenge, we present Asset Harvester, an image-to-3D model and end-to-end pipeline that converts sparse, in-the-wild object observations from real driving logs into complete, simulation-ready assets. Rather than relying on a single model component, we developed a system-level design for real-world AV data that combines large-scale curation of object-centric training tuples, geometry-aware preprocessing across heterogeneous sensors, and a robust training recipe that couples sparse-view-conditioned multiview generation with 3D Gaussian lifting. Within this system, SparseViewDiT is explicitly designed to address limited-angle views and other real-world data challenges. Together with hybrid data curation, augmentation, and self-distillation, this system enables scalable conversion of sparse AV object observations into reusable 3D assets.

Tianshi Cao, Jiawei Ren, Yuxuan Zhang +12

NVIDIA

3D Asset Generation Sparse-View 3D Reconstruction Autonomous Driving Simulation

Ask PDF

Jul 15, 2026

Instant NuRec: Feed-Forward 3D Gaussian Reconstruction for Driving Scene Simulation

3D simulation platforms are critical for autonomous driving because they enable end-to-end policy evaluation, thereby reducing development costs and improving safety. In recent years, neural simulation has become predominant, with methods such as NuRec playing a central role; however, these methods remain relatively slow and typically require per-scene tuning. In this work, we present Instant NuRec, a feed-forward neural reconstruction model that turns a short multi-view driving log into a fully simulatable 3D Gaussian Splatting (3DGS) world in a single forward pass. The model accepts multi-view input from a calibrated camera rig and emits a layered output consisting of static and dynamic 3DGS layers, a sky cubemap, and per-camera ISP corrections, while providing native support for non-pinhole camera models via 3DGUT. It reconstructs a 10-20-second multi-camera scene in roughly 1.5 seconds and achieves a PSNR on the Waymo Open Dataset that is 2.01 dB above the strongest evaluated baseline. Instant NuRec is deeply integrated into NuRec and is compatible with AlpaSim for closed-loop simulation.

NVIDIA, :, Jiahui Huang +14

NVIDIA†

3D Gaussian Splatting Multi-View 3D Reconstruction Dynamic Scene Reconstruction Feed-Forward 3D Reconstruction Autonomous Driving Simulation

Ask PDF

Jul 7, 2026

Point as Skeleton: Accumulated Point Cloud Enhanced Autoregressive Generation for Closed-Loop Autonomous Driving Simulation

Evaluating end-to-end autonomous driving (E2E-AD) remains challenging, as existing driving simulation methods often trade off closed-loop interactivity (e.g., CARLA) and real-world visual fidelity (e.g., nuScenes). We present Point as Skeleton, a generative sensor simulation framework for state-updated autoregressive driving video generation, in which an autoregressive generator synthesizes visual observations from step-wise updated ego states, actor states, scene maps, and point-cloud skeleton conditions. To support closed-loop rollout, we introduce Reset-and-Roll, which adapts rolling diffusion inference to simulation by preventing future-conditioned latent states from being committed across simulation steps. To stabilize error accumulation during step-wise autoregressive rollout, we introduce point-cloud skeletons that decouple foreground and background assets and project them into camera-view painted-point and template-depth conditions, providing appearance and geometric cues. We further implement a nuPlan-based renderer-level closed-loop generative interface for evaluating generation under ego deviations from the original log. Experiments on nuScenes and nuPlan show that Point as Skeleton improves autoregressive generation quality during closed-loop rollout, demonstrating its potential for visually faithful closed-loop driving simulation. The code is available at https://github.com/krauwu/point-as-skeleton.

Songbur Wong, Xiaosong Jia, Junqi You +12

1Shanghai Jiao Tong University · 3Fudan University · 4Shanghai Artificial Intelligence Laboratory +3

Autoregressive Video Generation Autonomous Driving Simulation Autonomous Driving Scenario Generation

Ask PDF