TerrainForge: Physics-Grounded Road Geometry Editing for Counterfactual Autonomous Driving

Yang Chen; Yicheng Zhu; zhenning Li; Tao Li; Zilin Bian

cs.CVOct 2, 2026

TerrainForge: Physics-Grounded Road Geometry Editing for Counterfactual Autonomous Driving

Authors: Yang Chen, Yicheng Zhu, zhenning Li, Tao Li, Zilin Bian

Organizations: Rochester Institute of Technology · University of Macau · City University of Hong Kong

Abstract

Road geometry (e.g., crests, sags, and speed humps) and surface conditions (e.g., wet or icy pavement) affect how vehicles move, what drivers and onboard cameras observe, and how much clearance remains between vehicles. Editing these properties in a driving scene therefore requires corresponding changes in vehicle motion. Capturing these differences in a driving video requires a road edit to propagate to vehicle motion, camera viewpoint, and the clearance between vehicles. We present TerrainForge, a framework for generating road geometry-focused counterfactuals from reconstructed multi-vehicle driving episodes. A unified road model connects scene deformation with four-wheel vehicle dynamics, allowing crests, sags, speed humps, and friction changes to propagate through vehicle motion, camera viewpoint, and inter-vehicle clearance. Vehicle dynamics are evaluated against CarSim, and prescribed road geometry is verified in reconstructed Waymo scenes. Across 18 episodes, leaving surrounding vehicles on their recorded trajectories instead of recomputing their responses produces median peak differences in predicted ego-lead distance of 1.52 m for crests and 1.41 m for sags. We further simulate the ego response to 15,758 road edits across 983 braking episodes, pairing each edit with its safety outcomes relative to an unedited replay. These pairs train a first-stage screening surrogate that takes the original driving context and candidate road-edit parameters as input and predicts the resulting change in the ego's terminal gap. On held-out scenes, this prediction achieves 22-40% lower mean absolute error than predicting no change, so candidates can be screened cheaply before the full multi-vehicle rollout.

Figures & tables

Figure 1: TerrainForge edits the road while preserving traffic context, propagating each intervention to scene geometry, vehicle motion, and inter-vehicle clearance.

Figure 2: Overview of TerrainForge. TerrainForge converts a reconstructed real-world driving episode into an editable, physics-grounded counterfactual environment by recovering per-vehicle driving actions and allowing interventions to road geometry and surface conditions. Each intervention jointly determines the reconstructed scene geometry and the physical excitation experienced by all vehicles. Replaying the recovered vehicle controls on the intervened road produces synchronized vehicle trajectories, camera observations, rendered videos, and physical states, enabling paired analysis of road-induced changes in motion and inter-vehicle safety outcomes.

Experiment	Control input fixed across arms	Vehicle motion	Camera
CarSim (§ 4.1 )	authored speed, steering, torque; identical in both simulators	1 vehicle, re-solved	none
ZOD (§ 4.2 )	measured speed and accelerations	half-car on the measured path	measured pose
Full chain, 109409 (§ 4.3 )	frozen inferred torque	ego + 17 moving vehicles, path replayed	dynamic
Multi-vehicle bank (§ 4.3 )	frozen inferred torque	353 moving vehicles (ego included), path replayed	none
Video comparison (§ 4.4 )	frames: none; response panel: speed tracking of recorded speed	frames: recorded schedule; panel: ego re-solved	kinematic
Surrogate bank (§ 4.5 )	frozen inferred torque and steering	ego re-solved; target as recorded	none

Table 1: Primary experimental protocols. Path replay fixes the recorded plan-view path while longitudinal and vertical motion are re-integrated.

Figure 3: Representative CarSim validation on a crest, SAG, and speed hump. Columns show longitudinal speed, body pitch, and total normal load normalized by static load. Our frozen model closely tracks CarSim 2022.1 on the raw

5\,\mathrm{ms}

trajectories; the hump case additionally includes braking and steering.

Panel of Figure 4 — Figure 4: Multi-vehicle counterfactual propagation. Left: Waymo 109409 under flat, crest, and sag edits, with the corresponding ego–lead gap change for all-vehicle and ego-only propagation. Right: peak disagreement $D$ over the 18 valid ego–lead episodes for the primary $2\%$ edits; bars denote medians. Stress results are in Appendix E.1 .

Figure 5: Comparison with video editing models on Waymo 150623. (a) Frames at the steepest SAG stations; dashed lines mark the flat-road horizon. (b) VGGT-recovered camera height versus the authored profile. (c) Camera-trajectory consistency across conditioning modes. Only full-frame depth reaches the target at one setting, while altering the recorded scene.

Edit	No-change MAE	Ours MAE	Ratio	$\rho$
SAG	$0.451$	$0.270$	$0.60$ [ $0.41$ , $0.76$ ]	$0.89$ [ $0.78$ , $0.96$ ]
Hump	$0.730$	$0.569$	$0.78$ [ $0.44$ , $0.89$ ]	$0.81$ [ $0.72$ , $0.89$ ]

Table 2: Terminal-gap prediction on held-out scenes. MAE is in meters; ratio is relative to predicting no change;

\rho

is Spearman rank correlation. Brackets denote

95\%

scene-bootstrap intervals.

Appendix figures & tables32 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Rain and snow on vertical-alignment edits of Waymo scene 150623. Rows: crest and sag ( $\pm 6\%$ , $K=7.5\,\mathrm{m}/\%$ ). Columns: reconstruction, terrain edit, edit with rain, and edit with snow, all at the same frame.

Experiment	Re-simulated	Rendered as objects	Safety quantity
Waymo 109409 (video)	ego and $17$ moving vehicles; $5$ parked vehicles replayed	$19$ of $22$ tracked vehicles †	ego–lead gap; all-pair overlap check
Multi-vehicle bank ( $20$ episodes)	$353$ moving vehicles (ego included); parked vehicles, pedestrians, cyclists not modeled	none	ego–lead gap
Obstacle bank	ego only; target replayed	none	ego–target $d_{\min}$ , $\Delta G$ , collision
Lateral bank	ego only	none	peak lateral and load quantities

Appendix

Table 3: Object scope of each experiment. “Replayed” objects follow their recorded trajectories in both arms of a comparison.

Experiment	Control input fixed across arms	Vehicle motion	Camera	Checks
Scene 22 (App. E )	prescribed: $36\,$ km/h, $1000\,$ N m brake from $5.34\,$ s	ego re-solved; lead static	dynamic	one braking mechanism
Lateral edit (§ 4.3 )	frozen inferred torque and steering	ego, re-solved	none	vertical $\rightarrow$ lateral coupling
Screening, lateral (App. H )	frozen inferred torque and steering; $\mu$ varied	ego, re-solved	none	labels for the surrogate

Appendix

Table 4: Settings of the secondary experiments, in the format of Table 1 .

Suite	Cases	Role	Continuous error (RMSE)	Passed
Grade-zero regression	19	R	event pitch mean $0.035^{\circ}$ , max $0.052^{\circ}$ ; speed $0.023\,$ m/s	19/19
Development	5	F	lateral position $0.030$ – $0.157\,$ m; pitch $0.041$ – $0.068^{\circ}$	5/5
Steered holdout	3	E	lateral position $0.035$ – $0.110\,$ m; pitch $0.046$ – $0.067^{\circ}$ ; speed $0.010$ – $0.020\,$ m/s	3/3
Low speed	4	E	speed $0.002$ – $0.005\,$ m/s; pitch $0.050$ – $0.066^{\circ}$ ; lateral position $\leq 0.28\,$ m	4/4
Vertical alignment	4	E	speed $0.004$ – $0.007\,$ m/s; pitch $0.051$ – $0.061^{\circ}$ ; heave $0.7$ – $2.0\,$ mm	4/4
Scene-supported SAG	4	E	speed $\leq 0.001\,$ m/s; pitch $0.064$ – $0.067^{\circ}$ ; lateral position $2.5$ – $10.4\,$ mm	4/4

Appendix

Table 5: CarSim comparison suites. Errors are RMSE ranges over the cases of a suite on raw

5\,

ms traces; a case passes when every pre-declared accuracy, integrity, and numerical-convergence gate is met. All suites score the coupled vehicle model except grade-zero, which is inherited from muSync-GS and scores its half-car.

Case	$G$ (%)	$\mu$	$L$ (m)	$v$ (m/s)	$\theta$ ( ∘ )	$\beta$ (rad)	$F_{z}$ (N)
1	4	0.55	59.74	0.00077	0.0636	0.00083	14.59
2	6	0.40	20.00	0.00080	0.0652	0.00050	15.32
3	2	0.70	59.74	0.00019	0.0669	0.00020	11.11
4	5	0.85	74.68	0.00118	0.0667	0.00016	15.06
Mean	–	–	–	0.00073	0.0656	0.00042	14.02

Appendix

Table 6: RMSE over the full horizon on the four scene-supported SAG cases.

Figure 7: Scene-supported SAG holdout. (a) Gate margin (error over limit) for all four cases and eight metrics. (b)–(f) Case 2: authored SAG profile and raw-time agreement in speed, pitch, sideslip, and four-wheel normal loads.

Figure 8: The failed stress case. Traces agree until both simulators leave the forward-motion domain; our model then holds the vehicle at rest while CarSim reverses, and the resulting reverse-motion sideslip wrap explains the full-horizon metric failure.

Case	$y$ RMSE [m]	Heading RMSE [mrad]	Yaw-rate NRMSE	Roll NRMSE
Sunken, brake-turn	$0.075$	$6.3$	$0.017$	$0.095$
Hump, brake-turn	$0.110$	$4.3$	$0.022$	$0.093$
SAG, sinusoidal steer	$0.035$	$1.2$	$0.032$	$0.096$
Gate	$1.5$	$50$	$0.20$	$0.25$

Appendix

Table 7: Lateral accuracy on the three holdouts (raw

5\,

ms traces, full horizon). The last row gives the pass gate.

Figure 9: Lateral response on two steered holdouts (solid: hump with braking and turning; dotted: sunken road with braking and turning). Black is CarSim and orange is our model. The second panel shows our model minus CarSim.

Figure 10: Measured vehicle world-frame pitch and predictions from a flat-road assumption and the recovered road profile on a held-out ZOD sequence. The LiDAR geometry check is in Figure 12 .

Figure 11: ZOD error (RMSE) by event type for world-frame pitch, road-relative pitch, and heave, averaged over the recordings that contain each type ( $10$ , $10$ , $12$ , $12$ , $5$ , and $3$ recordings for flat, downhill, uphill, curve with grade, SAG, and crest). Types come from road geometry and were fixed before any response was inspected.

	Test (3)			All (13)
Quantity	Flat	Kin.	Full	Flat	Kin.	Full
World-frame pitch ( ∘ )	1.027	0.236	0.258	0.594	0.160	0.196
Road-relative pitch ( ∘ )	0.140	0.145	0.168	0.120	0.119	0.136
Heave (mm)	2.35	2.49	3.11	2.60	2.70	3.42

Appendix

Table 8: ZOD prediction error (RMSE), averaged over recordings. Test is the pre-declared split of

3

recordings; All pools all

13

Figure 12: The OXTS-recovered road against an independent LiDAR ground plane. (a) Error of the flat-road assumption and of the OXTS-recovered road relative to the LiDAR ground pitch, by event type (recordings per type in brackets); LiDAR covers $79$ of the $96$ events. (b) The test event with the highest geometry score that has LiDAR.

Figure 13: Authored against recovered road geometry in the deformed Gaussian scenes of ten Waymo scenes. (a) Apex depth of

88

vertical curves. (b) Residual RMS of the recovered profile for each scene (range over its edits) against the same scene with a flat edit, which sets the floor of the measurement. (c) Height of

30

humps.

	crest $2\%$	sag $2\%$	crest $8\%$	sag $8\%$
Traversers responding	$257/266$	$257/266$	$261/266$	$261/266$
Non-traversers changed	$0/87$	$0/87$	$0/87$	$0/87$
Median peak $\|\Delta v\|$ of traversers (m/s)	$0.35$	$0.34$	$1.44$	$1.35$
Median onset delay (s)	$0.1$	$0.1$	$0.1$	$0.1$
Median $D$ (m)	$1.52$ [ $1.31$ , $2.08$ ]	$1.41$ [ $1.23$ , $1.90$ ]	$6.79$ [ $5.84$ , $9.65$ ]	$5.13$ [ $4.49$ , $7.25$ ]
90th percentile of $D$ (m)	$3.43$	$3.07$	$17.11$	$10.57$

Appendix

Table 9: Twenty-episode bank. Response rows:

353

valid vehicles in

20

episodes. Pair rows:

18

ego–lead pairs, the same episodes in every arm.

D

: largest disagreement between the all-vehicles and ego-only gap changes (Eq. 22 ). Brackets:

95\%

episode-bootstrap intervals.

Figure 14: All vehicles responding versus the ego alone, on the bank’s ego–lead pairs. (a)

D

, one point per episode; bars: medians; dotted:

0.5

and

1\,\mathrm{m}

. (b) Gap change over time for three episodes chosen by a rule fixed before plotting (smallest non-zero, median, and 90th-percentile

D

at crest

2\%

); solid: all respond, dashed: ego only. (c) Change of the minimum gap when all vehicles respond (symmetric-log axis; the row at zero is episodes whose closest approach precedes the edit).

Figure 15: Every vehicle of a real Waymo drive on the edited road. (a) Speed change of each vehicle that reaches the edit, aligned at the time it does. (b) Largest speed change per vehicle; the nine on the right never reach the edit. (c) Change of total normal load against the closed-form curvature term

v^{2}H^{\prime\prime}(s)/g

, over all frames inside the edit. (d) Change of the gap from the ego to the vehicle ahead when all vehicles respond (solid) and when the vehicle ahead is replayed as recorded (dashed).

Figure 16: Prescribed-input case on Scene 22 (ego only, static lead vehicle). Left: the same rendered instant ( $t=9.5\,\mathrm{s}$ ) under the four road arms, from the dynamic camera. Right: gap to the lead vehicle under the same prescribed input ( $36\,\mathrm{km/h}$ , $1000\,\mathrm{N\,m}$ brake from $5.34\,\mathrm{s}$ , onset chosen to expose the difference); the legend gives each arm’s minimum ego–lead gap. Gaps are computed on the metric rollout poses.

Figure 17: Where our sag and crest are visible. Left: signed vertical image motion against the flat render of the same frame, overlaid on the flat frame (grey: too smooth to measure). The textured far field moves together. Right: far-field shift against the authored road pitch, per frame; the fits are parallel lines whose offsets (about $\mp 7\,\mathrm{px}$ ) are consistent with the height parallax of a camera sitting up to $1.28\,\mathrm{m}$ lower or higher.

Figure 18: Text-only comparison on the Scene 22 clip. Columns: the sag and the crest at their steepest positions. Rows: the flat input; three generators that receive it with a text description and no target geometry; and TerrainForge. The dashed yellow line is the flat-road horizon, identical on every tile. Generator tiles show SSIM against the input, which rewards not editing; our tiles show the authored grade and the far-field shift against the flat render.

Case	VACE	Cosmos	LTX	Ours (px)
Sag $\pm 8\%$	copy, $0.94$	regenerated, $0.69$	regenerated, $0.73$	$-47$ / $+34$
Crest $\pm 8\%$	copy, $0.94$	regenerated, $0.69$	regenerated, $0.73$	$+45$ / $-34$
Uphill $+8\%$	copy, $0.94$	regenerated, $0.70$	regenerated, $0.73$	$+3$ / $+32$
Downhill $-8\%$	copy, $0.94$	regenerated, $0.70$	regenerated, $0.73$	$-2$ / $-32$

Appendix

Table 10: Outcome per case. Generators: what the output shows, and SSIM against the input (mean over every fourth frame). copy : the lane region stays within

1.25\times

the adjacent-frame difference of the input and the far field does not move; regenerated : the street content is replaced. Ours: far-field shift against the flat render at

s=20

and

52\,\mathrm{m}

, in pixels.

Weight	Diff. /255	SSIM	RMSE (m)	Corr.	Ratio	Rule	Shift (px)
$0.3$	$64.8$	$0.33$	$0.38$	$0.72$	$0.70$	not met	$-1$ / $-8$
$0.6$	$44.6$	$0.41$	$0.10$	$0.98$	$1.04$	met	$-22$ / $+29$
$1.0$ (selected)	$40.1$	$0.42$	$0.72$	$0.75$	$3.31$	not met	$-6$ / $+36$

Appendix

Table 11: Cosmos with depth over the whole frame, Waymo 150623 sag, one seed. Difference and SSIM are against the flat source outside the ego-lane mask; the last three columns come from the recovered camera height (the rule requires RMSE

\leq 0.28

, correlation

\geq 0.70

, ratio

0.6

–

1.4

); the last column is the far-field shift at

s=20

and

52\,\mathrm{m}

in pixels. Ours:

-46

and

+36

Scene	Case	VACE	Cosmos	LTX	Generators, max (px)	Ours (px)
150623	Sag	$0.90$	$0.72$	$0.69$	$1.4$	$-46$ / $+36$
	Crest	$0.88$	$0.69$	$0.69$	$1.5$	$+44$ / $-36$
177619	Sag	$0.94$	$0.66$	$0.75$	$1.2$	$-47$ / $+36$
	Crest	$0.94$	$0.65$	$0.75$	$1.1$	$+46$ / $-37$
398895	Sag	$0.88$	$0.62$	$0.53$	$1.8$	$-37$ / $+29$
	Crest	$0.88$	$0.61$	$0.53$	$1.9$	$+44$ / $-29$

Appendix

Table 12: Sag and crest (

\pm 8\%

) on three further Waymo scenes. SSIM of each generator against the input (mean over every fourth frame); largest far-field shift of any of the three generators against the input over all frames; and our far-field shift against the flat render at

s=20

and

52\,\mathrm{m}

Figure 19: Text-only comparison on Waymo 177619, in the layout of Figure 18 . The columns are at $s=20$ and $52\,\mathrm{m}$ .

Figure 20: Text-only comparison on Waymo 398895, in the layout of Figure 18 . The columns are at $s=20$ and $52\,\mathrm{m}$ .

Figure 21: Vehicle response of the frozen model to the authored sag and crest under the same driver intent, plotted against route station. Solid: body pitch from the coupled model; dashed: road pitch

\arctan H^{\prime}

, which is what a kinematic camera sees. The gap between them is the suspension response. Total normal load is the sum over the four wheels divided by static weight. The generators have no such channel.

Figure 22: The sustained $\pm 8\%$ uphill and downhill, in the layout of Figure 18 . The columns are at $s=20\,\mathrm{m}$ , where the grade has just been reached, and $36\,\mathrm{m}$ .

Figure 23: Vehicle response to the sustained

\pm 8\%

grades, in the layout of Figure 21 . The speed axis is wider than there, and the plot is cut at

s=72\,\mathrm{m}

Quantity	Edit	Group	$2\%$	$4\%$	$8\%$
Lateral acceleration [m/s 2 ]	crest	curve	$-0.0041$	$-0.0081$	$-0.0156$ [ $-0.0203$ , $-0.0130$ ]
	crest	control	$-0.0024$	$-0.0043$	$-0.0089$ [ $-0.0122$ , $-0.0047$ ]
	sag	curve	$+0.0041$	$+0.0085$	$+0.0180$ [ $0.0149$ , $0.0228$ ]
	sag	control	$+0.0024$	$+0.0049$	$+0.0100$ [ $0.0074$ , $0.0127$ ]
Minimum load ratio	crest	curve	$-0.0217$	$-0.0543$	$-0.1309$ [ $-0.1477$ , $-0.1041$ ]
	crest	control	$-0.0250$	$-0.0641$	$-0.1466$ [ $-0.1986$ , $-0.0920$ ]

Appendix

Table 13: Lateral response to the road-geometry edit: median paired delta (edit minus baseline) of the edit-window peak, with

95\%

scene-bootstrap intervals for

g=8\%

. Curve:

142

scenes; control:

35

Target	Edit	No change	Closed-form prior	Boosted trees	MAE ratio	$\rho$
$\Delta G$	sag	$0.451$	$0.469$	$0.270$	$0.60$ [ $0.41$ , $0.76$ ]	$0.89$ [ $0.78$ , $0.96$ ]
$\Delta G$	hump	$0.730$	$1.043$	$0.569$	$0.78$ [ $0.44$ , $0.89$ ]	$0.81$ [ $0.72$ , $0.89$ ]
$\Delta d_{\min}$	sag	$0.028$	$0.097$	$0.027$	$0.97$ [ $0.85$ , $1.06$ ]	$0.21$ [ $0.05$ , $0.37$ ]
$\Delta d_{\min}$	hump	$0.363$	$0.634$	$0.360$	$0.99$ [ $0.91$ , $1.02$ ]	$0.30$ [ $0.14$ , $0.44$ ]

Appendix

Table 14: Predicting the effect of a candidate edit on held-out scenes, all targets and baselines. MAE in meters over all test edits; ratio is MAE over that of predicting no change;

\rho

is the Spearman rank correlation between predicted and simulated change, pooled over test edits of all scenes. Brackets:

95\%

scene-bootstrap intervals. Test edits:

1{,}684

sag (

109

scenes),

568

hump (

142

scenes).

Model	AP	Recall@20%
Random ranking	0.20	0.23
Geometry-only logistic	0.35	0.44
Full-context logistic	0.75	0.75
Boosted trees	0.85	0.80

Appendix

Table 15: Obstacle-intervention effect detection on the scene-disjoint test split. AP and Recall@20% are reported for the same labels used in the main text.

Peak quantity	Static trees	Summary trees	Sequence model
Sideslip angle	$0.06$	$0.97$ [ $0.92$ , $0.99$ ]	$0.92$ [ $0.85$ , $0.95$ ]
Heading error	$-0.06$	$0.62$ [ $0.19$ , $0.83$ ]	$0.76$ [ $0.64$ , $0.86$ ]
Yaw-rate error	$0.20$	$0.75$ [ $0.40$ , $0.92$ ]	$0.94$ [ $0.90$ , $0.97$ ]
Lateral acceleration	$0.25$	$0.72$ [ $0.56$ , $0.81$ ]	$0.89$ [ $0.84$ , $0.92$ ]
Roll angle	$0.58$	$0.92$ [ $0.88$ , $0.95$ ]	$0.95$ [ $0.92$ , $0.96$ ]
Roll rate	$0.29$	$0.65$ [ $0.40$ , $0.81$ ]	$0.87$ [ $0.82$ , $0.91$ ]

Appendix

Table 16: Lateral screening,

R^{2}

533

test rollouts from

178

scenes, with

95\%

scene-bootstrap intervals. Static: scene-level descriptors only. Summary: the sequence model’s own pre-simulation inputs reduced to per-rollout maximum,

95

th percentile, and mean.

Explore similar work

SceneFactory-3D: Lifting 2D Traffic Scenes into 3D Physical Counterfactuals for Scalable Physically Grounded Safety Evaluation
Oct 2, 2026Yicheng Zhu, Linfeng Tian, Tianmu Zhao +4Autonomous Driving Simulation Autonomous Driving Safety Evaluation
Rochester Institute of Technology · City University of Hong Kong · New York University
Physics-Aware 3D Gaussian Editing for Driving Scene Generation
May 25, 2026Feng Zhou, Jian Zhang, Yuhang Sun +53D Gaussian Splatting 3D Scene Editing
School of Artificial Intelligence, Jilin University, Changchun, China. · China FAW Group Co., Ltd., Changchun, China. · National Key Laboratory of Automotive Chassis Integration and Bionics, Jilin University, Changchun, China.
TrafficImag: A Benchmark for Counterfactual Roadside Traffic Video Generation
Sep 25, 2026Xiangyu Li, Tianyi Wang, Zhihao Dou +2Video Generation Evaluation Causal Counterfactual Generation
Fariborz Maseeh Department of Civil, Architectural, and Environmental Engineering, The University of Texas at Austin, Austin, TX 78712, USA · Department of Electrical and Computer Engineering, Duke University, Durham, NC 27708, USA

Oct 2, 2026J/K move · Enter open · S save

SceneFactory-3D: Lifting 2D Traffic Scenes into 3D Physical Counterfactuals for Scalable Physically Grounded Safety Evaluation

Yicheng Zhu, Linfeng Tian, Tianmu Zhao, Yang Chen+3

Rochester Institute of Technology · City University of Hong Kong · New York University

Autonomous Driving Simulation Autonomous Driving Safety Evaluation Physics-Based Simulation Autonomous Driving Vehicle Dynamics

Open paper AskPDF arXiv

Scalable driving simulators typically execute vehicle commands using prescribed behavioral or kinematic rules, overlooking the physics of tire-road interfaces, thereby limiting their ability to capture how adverse road and environmental conditions alter vehicle execution and propagate through traffic. To address this limitation, we present SceneFactory-3D, a GPU-batched, physics-grounded multi-agent driving simulator. Vehicles execute acceleration and steering commands via suspension- and friction-limited forces evaluated at each wheel-contact point. Spatially varying friction, per-world 3D heightfields, gravity, and rigid contact consistently govern wheel motion and chassis collisions. Per-world terrain isolation and GPU batching enable SceneFactory-3D to run matched physical counterfactuals in parallel: traffic scenario setup and vehicle controllers remain fixed while only the road condition changes, enabling the resulting closed-loop effects to be evaluated across parallel worlds. To demonstrate the advantage of the SceneFactory-3D-enabled counterfactual evaluation, we conduct an empirical study on vehicle controllers' sensitivity to road conditions. We study three learned-policy families on 1,024 matched 12-vehicle worlds per condition, and two classical planners on a shared 32-world subset, across 21 friction and grade conditions. When friction drops from 1.0 to 0.18, the share of vehicles that clear the work zone safely falls by 6 to 90 percentage points across learned policies (18-19 for classical planners), and near-collision situations become more frequent for every learned policy. Code: https://github.com/SmallWorldLab/SceneFactory_3D

Abstract

Explore similar work

SceneFactory-3D: Lifting 2D Traffic Scenes into 3D Physical Counterfactuals for Scalable Physically Grounded Safety Evaluation

Physics-Aware 3D Gaussian Editing for Driving Scene Generation

TrafficImag: A Benchmark for Counterfactual Roadside Traffic Video Generation