BodyBridge: Grounded Body Layers for Sim-to-Real Translation of Vulnerable Road Users

Mark Azer; Fabian B. Flohr

cs.CVOct 17, 2026

BodyBridge: Grounded Body Layers for Sim-to-Real Translation of Vulnerable Road Users

Authors: Mark Azer, Fabian B. Flohr

Organizations: Intelligent Vehicles Lab Munich University of Applied Sciences

Abstract

Simulation-to-real driving video diffusion models can translate photorealistic scenes, but they typically represent pedestrians and cyclists, vulnerable road users (VRUs), as generic object classes. This is insufficient for vulnerable road users whose safety-relevant appearance depends on articulated pose and identity. We present BodyBridge and Grounded Body Layers (GBL), a simulator-derived, pixel-aligned representation that combines scene geometry, canonical per-pixel body correspondence, and exact VRU masks. GBL provides explicit articulation to the video diffusion model, emphasizes VRU regions during control-branch post-training, and supports additive composition with scene-geometry control at inference. It also enables per-joint evaluation of generated VRUs and a failure taxonomy for identity errors. On CARLA-to-real translation, BodyBridge improves near-range [email protected] by 22% over the released depth-conditioned baseline. Replacing normal-map body control with canonical correspondence reduces body flips by 18%, while composition reduces near-range facing flips by 27%. These results show that explicit body grounding complements scene-level control and improves pose-faithful VRU generation. All code, data, and model checkpoints will be made publicly available to support reproducible, pedestrian-aware autonomous driving research.

Figures & tables

Figure 2: BodyBridge: Grounded body control for sim-to-real video generation. (1) Grounded Body Layers encode scene geometry, canonical body correspondence, and the VRU mask from simulation. (2) The correspondence layer, encoded in RGB, trains a dedicated body-control branch, with the mask weighting the training objective. (3) At inference, body and released geometry control are composed to generate the translated video. Green denotes evaluation ground truth only, amber the objective and its weighting, and blue BodyBridge-specific and trained components; padlocks indicate frozen modules.

Figure 3: Qualitative sim-to-real results with BodyBridge. Examples across diverse scenes and pedestrian configurations show appearance translation while retaining the simulator-specified VRU pose and scene placement. Green boxes mark the highlighted VRUs; insets compare the source simulator crop ( Sim ) with the corresponding BodyBridge generation ( BB ).

Method	Recov. $\uparrow$	PCK all $\uparrow$	Near $\uparrow$	NMJE $\downarrow$	KID $\downarrow$
Cosmos depth	.504	.308	.666	.380	217.2
BodyBridge	.493	.373	.812	.282	139.3

Table 1: BodyBridge vs. scene-geometry control. Pose fidelity is evaluated against exact simulator ground truth; KID VRU is reported as

\times 10^{3}

Body representation
Variant	Recov. $\uparrow$	PCK all $\uparrow$	Near $\uparrow$	NMJE $\downarrow$
Edge (training-free)	.212	.136	.492	.360
2D skeleton	.320	.184	.595	.461
Normal map	.414	.310	.796	.281
Canonical corr.	.431	.325	.805	.291
Stage 2: post-training

Table 2: Component ablation. Effects of body representation, weighted post-training, and geometry-body composition.

Figure 4: Generation-induced VRU pose failures by failure type. Near-range failures are shown relative to the untranslated sim-RGB estimator floor. Identity-related failures-flip, switch, and merge-account for most of the excess errors, while canonical correspondence reduces the flip component compared with the normal-map representation.

Figure 5: Near-range VRU facing-direction errors. Points show front/back flip rates among recovered instances; error bars denote 95% clip-bootstrap CIs and the green line marks the untranslated sim-RGB estimator floor. Canonical correspondence reduces facing flips relative to the normal map, both alone (

-21.6\%

) and when composed with depth (

-27.2\%

Figure S1 : Matched qualitative examples of taxonomy-defined articulation failures. Columns show representative flip , switch , and stump cases. The top row shows the corresponding simulator ground truth, the middle row the released Cosmos depth-control baseline, and the bottom row BodyBridge. The examples illustrate how the taxonomy manifests visually under matched source states. Reported failure frequencies are computed over the full evaluation set rather than from these examples.

Configuration	s42	s43	s44	Mean
Cosmos depth	.666	.696	.655	.673
Normal map	.796	.793	.789	.793
Canonical corr.	.805	.802	.795	.800
Depth + normal map	.808	.819	.791	.806
BodyBridge	.812	.828	.803	.814

Table S1 : Inference-seed replication. Near-range [email protected] for three inference seeds with trained checkpoints held fixed.

Training run	Near PCK $\uparrow$	Flip red. $\uparrow$	Facing red. $\uparrow$
Original	.805	18.4%	21.6%
Replica 1	.771	24.2%	17.4%
Replica 2	.779	20.8%	13.0%

Table S2 : Training-run replication. Near-range [email protected] and relative identity-error reductions for three independently trained canonical-correspondence branches. The reduction columns compare canonical correspondence with normal-map conditioning under the same evaluation protocol.

Configuration	All $\uparrow$	Near $\uparrow$	Mid $\uparrow$	Far $\uparrow$
Cosmos depth	.504	.906	.640	.229
Normal map	.414	.943	.561	.082
Canonical corr.	.431	.938	.575	.109
Depth + normal map	.486	.955	.663	.146
BodyBridge	.493	.943	.677	.167

Table S3 : Distance-stratified instance recovery. Fraction of scorable GT VRUs recovered under the top-down evaluation protocol.

Threshold	$\Delta$ PCK	95% CI
$\alpha=.5$	+.146	[+.097, +.195]
$\alpha=.2$	+.148	[+.101, +.187]
$\alpha=.1$	+.069	[+.045, +.088]

Table S4 : PCK-threshold sensitivity. Paired near-range PCK difference between BodyBridge and the released depth-control baseline. Positive values favor BodyBridge.

Distance	IoU $\uparrow$	RMS [px] $\downarrow$	RMS [% height] $\downarrow$
Near	.685 [.652,.710]	3.1	3.7
Mid	.645 [.628,.659]	1.5	3.8
Far	.581 [.564,.597]	0.8	3.3

Table S5 : Registration and conditioning-geometry audit. Visible-mask IoU compares the registered body render with the simulator actor mask. Joint reprojection RMS is reported in pixels and relative to apparent body height.

Configuration	Near PCK $\uparrow$	$\Delta$ [email protected] $\uparrow$	$\Delta$ AP@[.5:.95] $\uparrow$
BodyBridge	.812	$-.0095$	$-.0051$
Depth	.666	$-.0137$	$-.0068$
Normal map	.738	$-.0113$	$-.0077$
Edge	.492	$-.0233$	$-.0114$
2D skeleton	.595	$-.0261$	$-.0139$

Table S6 : Pose fidelity and downstream detection. Near-range [email protected] is measured on each generated-data variant. Detector gains are means over six YOLO11-s training seeds relative to the corresponding same-seed real-only baseline. All variants use identical augmentation frames and GT boxes. Rows are ordered by

\Delta

AP@[.5:.95].

Sim-to-Real Transfer

Simulation-to-reality transfer adapts models and policies trained in simulation for reliable performance in physical or otherwise real-world settings.

See all 316 papers

NeuralSRender: G-Buffer Guided Video Diffusion for Sim-to-Real Rendering
Oct 21, 2026Yipeng Lu, Youqi Liao, Zhenyu Wang +2Sim-to-Real Transfer Synthetic-to-Real Domain Adaptation
Wuhan University, Wuhan, China
GenAssets: Generating in-the-wild 3D Assets in Latent Space
Apr 24, 2026Ze Yang, Jingkang Wang, Haowei Zhang +33D Asset Generation Latent Diffusion Models
Waabi · University of Toronto
Controllable and Photorealistic Pedestrian Risky Motion Generation for End-to-End Driving Safety Evaluation
Oct 5, 2026Siyuan Liu, Miao Li, Haibao Yu +4Pedestrian Behavior Modeling End-to-End Autonomous Driving
School of Vehicle and Mobility, Tsinghua University · Carnegie Mellon University · State Key Laboratory of Intelligent Green Vehicle and Mobility, Tsinghua University

Oct 21, 2026J/K move · Enter open · S save

NeuralSRender: G-Buffer Guided Video Diffusion for Sim-to-Real Rendering

Yipeng Lu, Youqi Liao, Zhenyu Wang, Yuning Peng+1

Wuhan University, Wuhan, China

Sim-to-Real Transfer Synthetic-to-Real Domain Adaptation Video Diffusion Models Controllable Video Generation Geometric Consistency in Video Generation

Open paper AskPDF arXiv

The scarcity of high-quality annotated visual data remains a critical bottleneck for autonomous-driving perception. Simulators provide training data at scale, but the limited realism of synthetic videos compromises perception performance in real-world applications. To address this limitation, we propose NeuralSRender, a sim-to-real video translation framework that enhances the photorealism of synthetic videos while maintaining geometric consistency with the source scenes. Built on a pretrained video diffusion transformer (DiT), NeuralSRender uses a lightweight projector to inject geometric and material priors from G-buffers, and LoRA enables parameter-efficient adaptation. The model is trained in two stages: it first learns G-buffer conditioning from synthetic videos with accurate G-buffers, then adapts to real-world appearance using real videos with estimated G-buffers. During the second stage, an additional LoRA module is trained while the projector and first-stage LoRA remain frozen to reduce interference from estimation noise. NeuralSRender introduces only 1.45% parameter overhead and can be trained on consumer-grade GPUs with 24 GB of memory. To support training and evaluation, we introduce GenReal, a large-scale driving video dataset comprising synthetic and real-world videos with corresponding G-buffers. Experiments demonstrate improved photorealism with competitive geometric consistency. Training perception models on our translated videos instead of raw simulator renders improves detection

\mathrm{AP}_{50}

on nuImages from 35.1 to 40.6 and segmentation mIoU on Cityscapes from 32.4 to 40.8. Our code, data, and model weights are available at: https://github.com/WHU-USI3DV/NeuralSRender.

Abstract

Sim-to-Real Transfer

Explore similar work

NeuralSRender: G-Buffer Guided Video Diffusion for Sim-to-Real Rendering

GenAssets: Generating in-the-wild 3D Assets in Latent Space

Controllable and Photorealistic Pedestrian Risky Motion Generation for End-to-End Driving Safety Evaluation