SceneFactory-3D: Lifting 2D Traffic Scenes into 3D Physical Counterfactuals for Scalable Physically Grounded Safety Evaluation

Yicheng Zhu; Linfeng Tian; Tianmu Zhao; Yang Chen; Fan Zuo; Tao Li; Zilin Bian

cs.ROOct 2, 2026

SceneFactory-3D: Lifting 2D Traffic Scenes into 3D Physical Counterfactuals for Scalable Physically Grounded Safety Evaluation

Authors: Yicheng Zhu, Linfeng Tian, Tianmu Zhao, Yang Chen, Fan Zuo, Tao Li, Zilin Bian

Organizations: Rochester Institute of Technology · City University of Hong Kong · New York University

Abstract

Scalable driving simulators typically execute vehicle commands using prescribed behavioral or kinematic rules, overlooking the physics of tire-road interfaces, thereby limiting their ability to capture how adverse road and environmental conditions alter vehicle execution and propagate through traffic. To address this limitation, we present SceneFactory-3D, a GPU-batched, physics-grounded multi-agent driving simulator. Vehicles execute acceleration and steering commands via suspension- and friction-limited forces evaluated at each wheel-contact point. Spatially varying friction, per-world 3D heightfields, gravity, and rigid contact consistently govern wheel motion and chassis collisions. Per-world terrain isolation and GPU batching enable SceneFactory-3D to run matched physical counterfactuals in parallel: traffic scenario setup and vehicle controllers remain fixed while only the road condition changes, enabling the resulting closed-loop effects to be evaluated across parallel worlds. To demonstrate the advantage of the SceneFactory-3D-enabled counterfactual evaluation, we conduct an empirical study on vehicle controllers' sensitivity to road conditions. We study three learned-policy families on 1,024 matched 12-vehicle worlds per condition, and two classical planners on a shared 32-world subset, across 21 friction and grade conditions. When friction drops from 1.0 to 0.18, the share of vehicles that clear the work zone safely falls by 6 to 90 percentage points across learned policies (18-19 for classical planners), and near-collision situations become more frequent for every learned policy. Code: https://github.com/SmallWorldLab/SceneFactory_3D

Figures & tables

Figure 1: SceneFactory-3D overview. (A) Capability levels of representative simulators on six axes (inner ring 0 to outer ring 2; criteria in Appendix A ). SceneFactory-3D is the only system shown at the top level for parallel worlds, interactive traffic, force-based execution, spatial road physics and matched interventions; Chrono offers more vehicle and terrain detail. (B) One planar scene is instantiated as matched 3D worlds that differ only in road physics, so each changed world is a counterfactual of the reference. (C) Each counterfactual is compared with its reference on task outcome, interaction safety and vehicle response.

Figure 2: Controller construction and matched evaluation. (A) Each of four nominal PPO parents is kept frozen (Frozen L0) or continued on terrain-diverse training scenes without (L0-T) or with (L1) a friction and grade preview. (B) All controllers share route and traffic inputs, the actuation bridge and batched physics. (C) The held-out closure task runs on a reference road and on matched counterfactual roads with a low-friction patch, a slope, or combined.

		Valid passage					$P(\mathrm{TTC}$	Speed
Regime	Worlds	Ref.	$\Delta\,\mu{=}.35$	$\Delta\,\mu{=}.18$	$\Delta\,\mu{\leq}.08$	$\Delta$ grade	${<}1.5\,\mathrm{s})$	(m/s)
Tier 2: 1,024 worlds per checkpoint; mean over 4 training seeds (min, max over seeds)
Frozen L0	1,024	.81 (.28, .99)	$\downarrow$ .27 ( $\downarrow$ .70, $\downarrow$ .01)	$\downarrow$ .57 ( $\downarrow$ .90, $\downarrow$ .21)	$\downarrow$ .69 ( $\downarrow$ .93, $\downarrow$ .12)	$\downarrow$ .05 ( $\downarrow$ .10, +.00)	.06 (.01, .14)	16.3 (14.7, 17.5)
L0-T	1,024	.87 (.53, 1.00)	$\downarrow$ .12 ( $\downarrow$ .36, $\downarrow$ .00)	$\downarrow$ .49 ( $\downarrow$ .83, $\downarrow$ .07)	$\downarrow$ .70 ( $\downarrow$ .89, $\downarrow$ .42)	$\downarrow$ .03 ( $\downarrow$ .13, +.00)	.15 (.01, .51)	14.9 (12.0, 20.1)
L1	1,024	.86 (.48, 1.00)	$\downarrow$ .05 ( $\downarrow$ .18, $\downarrow$ .00)	$\downarrow$ .26 ( $\downarrow$ .47, $\downarrow$ .06)	$\downarrow$ .65 ( $\downarrow$ .93, $\downarrow$ .35)	$\downarrow$ .06 ( $\downarrow$ .15, +.00)	.07 (.01, .20)	14.8 (11.0, 19.9)
Tier 1: classical controllers on the first 32 Tier-2 worlds (learned rows: Table 18 )

Table 1: Valid passage on the reference road and its change (

\Delta

) on the same worlds at

\mu_{\mathrm{surface}}=0.35

0.18

\{0.08,0.05\}

(mean) and the most influential grade (

\downarrow

: decrease); TTC: share of vehicles below 1.5 s (reference). Tier 2: seed mean (range over seeds).

Figure 3: Friction sweep on 1,024 matched worlds per policy. Thick lines: family means; thin: seeds (dashed: seed 3); shaded: extreme levels. (A) Valid passage. (B) Change from each policy’s own reference. (C) Share of vehicles with TTC below 1.5 s before their first failure. (D) Where passage is lost at

\mu_{\mathrm{surface}}=0.18

, by lane.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

System	Vehicle execution	Surrounding traffic	Local friction	Non-flat road	Parallel execution
SUMO	behavioral models	same	n.v.	n.v.	n.v.
Waymax	kinematic bicycle or position delta	same	no tire model	n.v.	accelerator
GPUDrive	kinematic bicycle (default)	same	no tire model	n.v.	GPU, $>10^{6}$ steps/s
CommonRoad	planar models incl. dynamic single-track; friction-circle checks	scenario agents	n.v.	n.v.	n.v.
CARLA	engine vehicle physics	physics; optional hybrid mode teleports distant vehicles	friction triggers	n.v.	n.v.
SMARTS	Bullet for vehicles of interest	SUMO	n.v.	n.v.	n.v.

Appendix

Table 2: Capabilities as documented in the sources cited in Section 2 and this appendix ( Lopez et al., 2018 ; Gulino et al., 2023 ; Kazemkhani et al., 2025 ; CommonRoad, n.d. ; Dosovitskiy et al., 2017 ; CARLA, n.d.b ; Zhou et al., 2021 ; Cai et al., 2020 ; Li et al., 2023b ; Project Chrono Development Team, n.d. ; Project Chrono, n.d. ; Kaufeld et al., 2025 ; Zhu et al., 2026 ) . n.v.: not verified here.

Aspect	SceneFactory	SceneFactory-3D
Batching	Isaac Lab, many worlds per GPU	Inherited; Newton XPBD backend
Vehicle	Articulated PhysX body, no explicit tire model	Rigid chassis, raycast suspension, per-wheel tire forces (Sec. 4.2 )
Friction	One coefficient per world, from weather	Spatial patches evaluated at each wheel contact
Road geometry	Waymo topologies, flattened to $z=0$	Per-world heightfields; procedural road in this study
Policy interface	Goal-reaching RL with weather context	Canonical routes shared by classical and learned controllers
Comparison	Friction-aware vs. friction-blind policy	Matched reference/changed worlds with recorded checks

Appendix

Table 3: What SceneFactory-3D inherits from and changes relative to SceneFactory ( Zhu et al., 2026 ) . Entries for SceneFactory follow its paper.

Mass $m$	1400 kg	Spring rate $k$	34,000 N/m
Length $\times$ width	$4.25\times 1.78$ m	Damping $c$	3,400 N s/m
Wheelbase $L$ / track $t$	2.62 / 1.54 m	Wheel inertia $I_{w}$	1.3 kg m 2
Wheel radius $R$	0.32 m	Drive torque $T_{\max}$ (total)	2,200 N m
Suspension rest length $l_{0}$	0.38 m	Power $P_{\max}$	135 kW
Grip factor $\mu_{\mathrm{vehicle}}$	1.05	Brake torque $T_{b}$ (per wheel)	1,800 N m
Steering limit $\delta_{\max}$	0.55 rad	Drag area $C_{d}A$	0.65 m 2

Appendix

Table 4: Parameters of the benchmark vehicle model.

Condition	Wheel $\mu$	Brake (m/s 2 )	Stop (m)	Coast (m/s 2 )
Reference	1.00	10.02	12.5	$-0.18$
$\mu_{\mathrm{surface}}=0.70$	0.70	7.23	17.3	—
$\mu_{\mathrm{surface}}=0.50$	0.50	5.17	24.2	—
$\mu_{\mathrm{surface}}=0.35$	0.35	3.63	34.5	—
$\mu_{\mathrm{surface}}=0.25$	0.25	2.60	48.1	—
$\mu_{\mathrm{surface}}=0.18$	0.18	1.88	66.5	—

Appendix

Table 5: Controller-free vehicle probes of all 21 conditions on the frozen final-benchmark code. Wheel-contact

\mu

is sampled under the four tire contacts every step; slope is measured from the vehicle’s own height profile. Brake: mean deceleration under a full brake command from 16 m/s inside the changed region, averaged while the front axle remains on it. Coast: zero command from 12 m/s through the constant-grade section; the reference coast value (

-0.18

m/s 2 ) reflects rolling and drivetrain resistance. Stop: distance from 16 m/s; “exit” gives the speed at which the vehicle leaves the 140 m region without stopping. These are internal consistency checks, not real-vehicle calibration.

Controller	Decision mechanism	Terrain training	Preview
IDM-17	IDM longitudinal law + Stanley route tracking	—	No
PDM-Closed-17	Published PDM proposals/scoring + LQR tracking	—	No
Frozen L0	Route-conditioned policy, nominal roads	No	No
L0-T	Frozen L0 continued on terrain-diverse roads	Yes	No
L1	As L0-T, with physical preview inputs	Yes	Yes

Appendix

Table 6: Controller regimes. Each regime drives all 12 vehicles of a world, with the same canonical routes, traffic state and vehicle model. Preview is privileged, noise-free physical state sampled along the route.

Block	Width	Content (scale)
Ego	9	speed (12), lateral speed (5), yaw rate (1), steering angle (0.55), previous acceleration and steering actions (1), cross-track error (2), heading error (1), route curvature (0.02)
Route preview	12	route points 2, 5, 10, 15, 20, 30 m ahead, forward and left (30)
Acceleration	1	speed change over the last decision divided by 0.1 s (4)
Route leader	6	presence, gap (50), relative speed (12), lateral offset (2), sine and cosine of relative heading
Neighbors	60	six nearest vehicles within 40 m, each: valid, $x$ (30), $y$ (10), $v_{x}$ (12), $v_{y}$ (5), sine and cosine of relative heading, distance (30), route-arc offset (30), lateral offset (5)
Semantics	24	held at zero

Appendix

Table 7: Learned-policy observation, ego frame, divided by the listed scale.

Variable	Continuation training	Final benchmark
Friction mode	none 40%, road band 40%, lane band 10%, whole road 10%	closing-lane patch
Friction value	bands $\mu_{\mathrm{surface}}\sim U(0.2,0.7)$ ; whole road $U(0.35,0.8)$	0.70 to 0.05
Patch length	25–70 m (25% add a second band)	140 m
Grade	half flat, half rolling fields: three plane waves, wavelength 90–220 m, maximum grade 4–9%	constant $+4\%$ to $-8\%$ with 20 m transitions
Road and task	procedural single routes, following, platoons, lane shifts, merges, crossings	straight two-lane closure
Vehicles per world	1 (first stage) or 6	12

Appendix

Table 8: Training support of the terrain-diverse continuation versus the final benchmark.

Study	Task and controllers	Sample	Horizon	Role
Probes	One vehicle, fixed commands, 21 conditions	one run per condition	to stop or exit	verification (App. C )
Development	Held-out training-family scenes; three learned families, 4 seeds, plus nominalized preview	256 scenes $\times$ 6 vehicles per cell	35 s	preliminary evidence
Final Tier 1	Closure task; all five regimes; 21 conditions	32 worlds $\times$ 12 vehicles	66 s	cross-controller comparison
Final Tier 2	Closure task; three learned families, 4 seeds; 21 conditions	1,024 worlds $\times$ 12 vehicles	66 s	learned-controller statistics

Appendix

Table 9: Studies, their unit of analysis and their role.

Condition	Controller	Success	Failure	Speed (m/s)
Nominal	Frozen L0	$0.991\pm 0.008$	$0.002\pm 0.003$	$12.37\pm 0.84$
	L0-T	$0.993\pm 0.007$	$0.001\pm 0.002$	$11.99\pm 1.40$
	L1	$0.994\pm 0.003$	$0.002\pm 0.001$	$11.81\pm 1.63$
	L1 (nominal preview)	$0.994\pm 0.003$	$0.002\pm 0.001$	$11.81\pm 1.63$
Road band $\mu=0.50$	Frozen L0	$0.977\pm 0.014$	$0.009\pm 0.006$	$12.16\pm 0.79$
	L0-T	$0.987\pm 0.011$	$0.003\pm 0.003$	$11.80\pm 1.33$

Appendix

Table 10: Preliminary controller-development evaluation: mean

\pm

standard deviation over four training seeds. Success and failure are route-task metrics defined in Appendix F . Speed is included to expose operating-point differences. These are not final two-tier benchmark results.

	Vehicle-steps/s (millions)			21 terrains	Loop (s)	Peak GPU	Efficiency
Worlds	no log	log	log+safety	log+safety	log+safety	(GB)	log+safety
64	0.15	0.13	0.11	0.10	27.0	0.56	1.00
128	0.30	0.25	0.23	0.19	27.0	1.10	1.00
256	0.59	0.51	0.45	0.39	26.9	2.18	1.00
512	1.15	0.99	0.89	0.75	27.3	4.35	0.99
1,024	2.24	1.92	1.71	1.40	28.4	8.67	0.95

Appendix

Table 11: Throughput and memory against world count (means of three repeats; repeat standard deviations are at most 2% of each mean). Efficiency is throughput relative to 64 worlds divided by the ideal ratio

N/64

Figure 4: Throughput (A) and peak GPU memory (B) against the number of worlds on one GPU.

Condition	Same-batch reruns	Across batch sizes 64–1,024
	vehicle flips (max)	vehicle flips	worlds changed	$\|\Delta$ mean VP $\|$ (max)
Reference	0 of 768	11–28 of 768	6–14 of 64	0.010
$\mu=0.18$	2 of 768	70–112 of 768	22–34 of 64	0.029

Appendix

Table 12: Batch-size sensitivity on 64 common worlds (768 vehicles).

Checkpoint	Ref.	..70	..50	..35	..25	..18	..12	..08	..05	$+4\%$	$-4.5\%$	$-8\%$	$-4.5\%$ ,.18	$+3\%$ ,.18
Frozen L0 s0	0.98	$-$ 0.09	$-$ 0.34	$-$ 0.70	$-$ 0.89	$-$ 0.90	$-$ 0.87	$-$ 0.86	$-$ 0.82	+0.00	$-$ 0.00	$-$ 0.01	$-$ 0.90	$-$ 0.89
Frozen L0 s1	0.99	$-$ 0.00	$-$ 0.02	$-$ 0.17	$-$ 0.48	$-$ 0.85	$-$ 0.93	$-$ 0.94	$-$ 0.92	+0.01	$-$ 0.04	$-$ 0.09	$-$ 0.90	$-$ 0.79
Frozen L0 s2	0.98	$-$ 0.00	$-$ 0.00	$-$ 0.01	$-$ 0.08	$-$ 0.30	$-$ 0.66	$-$ 0.85	$-$ 0.87	+0.00	+0.00	+0.00	$-$ 0.47	$-$ 0.20
Frozen L0 s3	0.28	$-$ 0.05	$-$ 0.16	$-$ 0.20	$-$ 0.22	$-$ 0.21	$-$ 0.17	$-$ 0.14	$-$ 0.11	+0.08	$-$ 0.06	$-$ 0.10	$-$ 0.20	$-$ 0.21
L0-T s0	0.98	$-$ 0.01	$-$ 0.02	$-$ 0.10	$-$ 0.52	$-$ 0.83	$-$ 0.86	$-$ 0.86	$-$ 0.84	+0.00	$-$ 0.00	$-$ 0.01	$-$ 0.91	$-$ 0.78
L0-T s1	1.00	$-$ 0.00	+0.00	$-$ 0.03	$-$ 0.19	$-$ 0.60	$-$ 0.86	$-$ 0.88	$-$ 0.89	+0.00	+0.00	$-$ 0.00	$-$ 0.71	$-$ 0.48

Appendix

Table 13: Tier-2 valid passage on the reference road and its change for selected conditions, per checkpoint (1,024 worlds). Bold: the 95% paired world-bootstrap interval excludes zero. Friction columns give

\mu_{\mathrm{surface}}

; grade columns the slope; the last two are compositions with

\mu_{\mathrm{surface}}=0.18

Family	$\mu$	VP closing	VP open	coll.	off-road	incurs.	own term.	censored	world reset	open on patch
Frozen L0	Ref.	0.797	0.819	0.016	0.002	0.000	0.004	0.168	0.254	0.011
	.35	0.518	0.557	0.026	0.031	0.002	0.007	0.393	0.489	0.004
	.18	0.198	0.276	0.042	0.055	0.004	0.015	0.641	0.823	0.004
	.08	0.025	0.178	0.092	0.056	0.021	0.019	0.697	0.951	0.009
L0-T	Ref.	0.872	0.876	0.010	0.000	0.000	0.001	0.115	0.247	0.002
	.35	0.731	0.769	0.032	0.020	0.000	0.001	0.194	0.260	0.001

Appendix

Table 14: Composition of vehicle outcomes (seed means, Tier 2). VP: valid passage by lane. The next five columns are the first failure of the vehicle itself or censoring by another vehicle’s reset, as shares of all vehicles; no pass: valid at the horizon without crossing. World reset: share of worlds reset before the horizon by a termination other than a map-edge exit. Open on patch: open-lane vehicles with any wheel contact on the patch.

Contrast	Seed	Ref.	..50	..35	..25	..18	..12	..08	..05	$-8\%$
L1 $-$ L0-T	s0	$-$ 0.02	+0.01	+0.07	+0.47	+0.72	+0.45	+0.27	+0.15	$-$ 0.14
	s1	+0.00	+0.00	+0.02	+0.13	+0.13	$-$ 0.04	$-$ 0.05	$-$ 0.03	$-$ 0.00
	s2	+0.00	$-$ 0.00	$-$ 0.00	$-$ 0.00	+0.01	$-$ 0.16	+0.02	$-$ 0.07	+0.00
	s3	$-$ 0.06	+0.10	+0.18	+0.12	+0.05	+0.04	+0.06	+0.07	+0.03
L0-T $-$ Frozen L0	s0	$-$ 0.01	+0.32	+0.60	+0.37	+0.07	+0.01	+0.00	$-$ 0.03	+0.00
	s1	+0.00	+0.02	+0.14	+0.29	+0.25	+0.07	+0.05	+0.03	+0.09

Appendix

Table 15: Seed-paired family contrasts in valid passage (arms share parent seed

i

). Reference column: difference in level; other columns: difference in the change relative to each arm’s own reference. Bold: world-bootstrap interval excludes zero. L0-T

-

Frozen L0 includes additional training.

Checkpoint	$-$ 4.5%, $\mu$ =0..35	$-$ 4.5%, $\mu$ =0..18	+3.0%, $\mu$ =0..35	+3.0%, $\mu$ =0..18
Frozen L0 s0	$-$ 0.12	+0.00	+0.11	+0.01
Frozen L0 s1	$-$ 0.09	$-$ 0.01	+0.05	+0.06
Frozen L0 s2	$-$ 0.02	$-$ 0.16	+0.00	+0.10
Frozen L0 s3	+0.06	+0.07	$-$ 0.05	$-$ 0.07
L0-T s0	$-$ 0.12	$-$ 0.07	+0.02	+0.05
L0-T s1	$-$ 0.01	$-$ 0.11	+0.01	+0.12

Appendix

Table 16: Interaction contrasts of grade–friction compositions,

\mathrm{VP}(g,\mu)-\mathrm{VP}(\mu)-\mathrm{VP}(g)+\mathrm{VP}(\mathrm{ref})

, paired by world. Negative: the composition loses more than the sum of the single losses. Bold: world-bootstrap interval excludes zero.

Figure 5: Grade sweep, drawn as Figure 3 . (A) Change in valid passage. (B) Change in the share of vehicles with TTC below 1.5 s. (C) Change in closing-lane speed where the grade begins; L1 slows before downhill grades.

Indicator (1–2 s earlier)	Family	$\mu=0$ .25	$\mu=0$ .18	$\mu=0$ .12	$\mu=0$ .08	$\mu=0$ .05
braking need $>$ residual capacity	Frozen L0	(2/14)	29 [12, 51]; 12/56	29 [19, 42]; 25/84	21 [13, 34]; 24/102	18 [12, 27]; 33/150
	L0-T	(2/4)	(2/25)	21 [10, 40]; 14/66	15 [10, 22]; 34/165	11 [8, 14]; 53/340
	L1	(0/3)	(1/11)	11 [3, 25]; 5/50	13 [8, 18]; 26/149	11 [8, 15]; 36/212
TTC $<1.5$ s	Frozen L0	26 [14, 44]; 21/174	25 [16, 36]; 33/193	36 [29, 45]; 99/310	30 [23, 38]; 89/303	21 [16, 27]; 69/294
	L0-T	30 [21, 40]; 65/687	37 [28, 50]; 82/452	29 [23, 36]; 113/473	24 [20, 28]; 167/634	19 [16, 23]; 175/744
	L1	39 [18, 70]; 27/367	15 [7, 26]; 22/444	17 [12, 23]; 68/510	22 [18, 27]; 175/700	19 [16, 23]; 184/740

Appendix

Table 17: Hazard ratio of a vehicle’s own failure given the indicator 1–2 s earlier, with 95% world-bootstrap interval and exposed failures / exposed vehicle-seconds; cells with fewer than three exposed failures show counts only. Rich subset, seeds pooled.

Figure 6: One matched pair (Frozen L0 seed 0, scenario 100050): reference road (grey) and

\mu_{\mathrm{surface}}=0.25

patch (blue, shaded). (A) All 12 vehicles; focal vehicle bold, first failure marked. (B, C) Focal vehicle’s speed and lateral position until its failure.

Checkpoint	Ref.	..70	..50	..35	..25	..18	..12	..08	..05	$+4\%$	$-4.5\%$	$-8\%$	$-4.5\%$ ,.18	$+3\%$ ,.18
IDM-17	0.68	+0.01	+0.01	$-$ 0.01	$-$ 0.04	$-$ 0.18	$-$ 0.31	$-$ 0.43	$-$ 0.49	$-$ 0.08	+0.04	+0.05	$-$ 0.20	$-$ 0.18
PDM-Closed-17	1.00	+0.00	+0.00	$-$ 0.06	$-$ 0.16	$-$ 0.19	$-$ 0.32	$-$ 0.46	$-$ 0.48	$-$ 0.04	$-$ 0.01	$-$ 0.01	$-$ 0.36	$-$ 0.14
Frozen L0 s0	0.98	$-$ 0.10	$-$ 0.36	$-$ 0.62	$-$ 0.84	$-$ 0.91	$-$ 0.89	$-$ 0.84	$-$ 0.81	+0.01	$-$ 0.01	$-$ 0.01	$-$ 0.91	$-$ 0.90
Frozen L0 s1	0.99	+0.00	$-$ 0.03	$-$ 0.06	$-$ 0.46	$-$ 0.86	$-$ 0.90	$-$ 0.96	$-$ 0.92	+0.00	$-$ 0.01	$-$ 0.08	$-$ 0.88	$-$ 0.80
Frozen L0 s2	0.98	$-$ 0.02	+0.01	$-$ 0.03	$-$ 0.06	$-$ 0.24	$-$ 0.64	$-$ 0.84	$-$ 0.87	$-$ 0.01	+0.02	+0.02	$-$ 0.42	$-$ 0.15
Frozen L0 s3	0.26	$-$ 0.03	$-$ 0.13	$-$ 0.18	$-$ 0.20	$-$ 0.18	$-$ 0.14	$-$ 0.12	$-$ 0.08	+0.11	$-$ 0.05	$-$ 0.10	$-$ 0.16	$-$ 0.20

Appendix

Table 18: Tier-1 valid passage (32 worlds) on the reference road and its change for selected conditions. Bold: the 95% paired world-bootstrap interval excludes zero. Learned rows are the first 32 worlds of the Tier-2 cells.

Explore similar work

SceneFactory: GPU-Accelerated Multi-Agent Driving Simulation with Physics-Based Vehicle Dynamics
May 8, 2026Yicheng Zhu, Yang Chen, Tao Li +1GPU Acceleration Vehicle Dynamics
Rochester Institute of Technology · City University of Hong Kong
TerrainForge: Physics-Grounded Road Geometry Editing for Counterfactual Autonomous Driving
Oct 2, 2026Yang Chen, Yicheng Zhu, zhenning Li +2Autonomous Driving Scenario Generation Autonomous Driving Simulation
Rochester Institute of Technology · University of Macau · City University of Hong Kong
ScenarioControl: Vision-Language Controllable Vectorized Latent Scenario Generation
Apr 18, 2026Lili Gao, Yanbo Xu, William Koch +8VLMs for Autonomous Driving 3D Scene Generation
1Torc Robotics · 2Princeton University · 3Mila

May 8, 2026J/K move · Enter open · S save

SceneFactory: GPU-Accelerated Multi-Agent Driving Simulation with Physics-Based Vehicle Dynamics

Yicheng Zhu, Yang Chen, Tao Li, Zilin Bian

Rochester Institute of Technology · City University of Hong Kong

GPU Acceleration Vehicle Dynamics Physics-Based Simulation Autonomous Driving Simulation Autonomous Driving Scenario Generation Multi-Agent Reinforcement Learning

Open paper AskPDF arXiv

Autonomous-driving simulators typically trade physical fidelity for scalable parallelism. Physics-based platforms such as CARLA and MetaDrive provide articulated vehicle dynamics and contact, but their non-vectorized interfaces make batched training difficult. GPU-batched systems such as Waymax and GPUDrive scale to hundreds of scenarios by replacing rigid-body physics with simplified kinematic models, omitting tire--road interaction, suspension, contact dynamics, and road-condition-dependent friction. We introduce SceneFactory, a GPU-vectorized platform for procedural scene construction, physics-based multi-agent simulation, and RL in autonomous-driving environments. Built on NVIDIA Isaac Sim + Isaac Lab, SceneFactory represents worlds and agents as batched tensors: control, observations, rewards, resets, and policy inference run as GPU tensor operations over the Isaac Lab tensor API. SceneFactory converts Waymo Open Motion Dataset road topologies into simulation-ready USD worlds, runs many worlds concurrently on one GPU, populates each with multiple articulated PhysX vehicles, and maps precipitation and road-surface type to PhysX material friction coefficients. With GPU vectorization, SceneFactory achieves up to 127

\times

higher throughput than a non-vectorized PhysX baseline on the same GPU and physics solver, reaching 19,250 controlled-agent simulation steps per second at 256 worlds

\times

16 agents. Cross-simulator transfer reveals an asymmetric dynamics gap: physics-grounded RL policies transfer to a simplified kinematic bicycle model with 99.5% success, whereas reverse transfer drops to 47.3%. Under wet-road friction, friction-aware policies reduce mean peak DRAC from 58.7 to 27.8,m/s

^2

without sacrificing goal reach. SceneFactory shows that scalable autonomous-driving training need not discard articulated rigid-body dynamics or physically grounded road-condition variation.

Abstract

Explore similar work

SceneFactory: GPU-Accelerated Multi-Agent Driving Simulation with Physics-Based Vehicle Dynamics

TerrainForge: Physics-Grounded Road Geometry Editing for Counterfactual Autonomous Driving

ScenarioControl: Vision-Language Controllable Vectorized Latent Scenario Generation