Ontology-Guided Diffusion for Zero-Shot Visual Sim2Real Transfer

Mohamed Youssef; Mayar Elfares; Anna-Maria Meer; Matteo Bortoletto; Andreas Bulling

cs.CVMar 19, 2026

Ontology-Guided Diffusion for Zero-Shot Visual Sim2Real Transfer

Authors: Mohamed Youssef, Mayar Elfares, Anna-Maria Meer, Matteo Bortoletto, Andreas Bulling

Organizations: University of Stuttgart, Germany · Fraunhofer IPA, Germany

Abstract

Bridging the simulation-to-reality (sim2real) gap remains challenging as labelled real-world data is scarce. Existing diffusion-based approaches rely on unstructured prompts or statistical alignment, which do not capture the structured factors that make images look real. We introduce Ontology- Guided Diffusion (OGD), a neuro-symbolic zero-shot sim2real image translation framework that represents realism as structured knowledge. OGD decomposes realism into an ontology of interpretable traits -- such as lighting and material properties -- and encodes their relationships in a knowledge graph. From a synthetic image, OGD infers trait activations and uses a graph neural network to produce a global embedding. In parallel, a symbolic planner uses the ontology traits to compute a consistent sequence of visual edits needed to narrow the realism gap. The graph embedding conditions a pretrained instruction-guided diffusion model via cross-attention, while the planned edits are converted into a structured instruction prompt. Across benchmarks, our graph-based embeddings better distinguish real from synthetic imagery than baselines, and OGD outperforms state-of-the-art diffusion methods in sim2real image translations. Overall, OGD shows that explicitly encoding realism structure enables interpretable, data-efficient, and generalisable zero-shot sim2real transfer.

Figures & tables

Figure 1: Ontology-Guided Diffusion (OGD) explicitly models visual realism to bridge the sim2real gap. Unlike current instruction-guided diffusion that relies on unstructured prompts, OGD decomposes realism into structured traits organised in a knowledge graph that models causal relationships, and uses a symbolic PDDL planner to generate coherent editing actions. Conditioning the diffusion model on this structured guidance produces more realistic and consistent translations from synthetic to real images, enabling interpretable and data-efficient sim2real transfer.

Figure 2: Overview of the proposed ontology-guided diffusion framework. A synthetic image is first mapped to realism trait probabilities using supervised MLP heads trained on frozen CLIP features. Traits are propagated through a static realism knowledge graph using a GNN to obtain node-level realism embeddings. Differences between synthetic and target realism states are converted into symbolic transformation plans via PDDL. The symbolic plan and graph embeddings jointly condition a diffusion-based image editing model.

Panel of Figure 3 — Figure 3: Qualitative sim-to-real results.

Method	Accuracy (%)	ROC–AUC
CLIP features	59.6	0.88
End-to-end CNN	76.5	0.68
Traits meta-classifier	97.0	0.90
Ours (Traits + GNN meta-classifier)	98.4	0.99

Table 1: Real vs. synthetic classification performance.

Method	TraitDist $\downarrow$	LPIPS $\downarrow$	SSIM $\uparrow$
I-Pix2Pix	0.702 $\pm$ 0.413	0.315 $\pm$ 0.247	0.534 $\pm$ 0.332
ControlNet	3.533 $\pm$ 0.717	0.530 $\pm$ 0.116	0.286 $\pm$ 0.150
Ours	0.593 $\pm$ 0.509	0.275 $\pm$ 0.220	0.562 $\pm$ 0.307

Table 2: Quantitative sim-to-real image translation performance.

Configuration	TraitDist $\downarrow$	LPIPS $\downarrow$	SSIM $\uparrow$
Full model (ours)	0.693	0.275	0.562
– w/o realism graph	0.842	0.312	0.524
– w/o symbolic planning	0.781	0.298	0.538
– w/o KG alignment loss	0.756	0.287	0.546

Table 3: Ablation study on sim-to-real image translation.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Source Trait (Observable / Inferable)		Target Trait	Rel.	Supporting Evidence
Visually Observable and Scene-Inferable Trait Relationships
Lighting: Uniform	$\Rightarrow$	Shadows: Present	opposes	Diffuse illumination eliminates cast shadows as spatial cues ( Ferwerda, 2003 ) .
Shadows: Present	$\Rightarrow$	Scene Consistency: Object Interaction	supports	Cast shadows convey spatial relationships between objects ( Ferwerda, 2003 ) .
Optical & Sensor Artifacts: Chromatic Aberration	$\Rightarrow$	Edge & Geometry: Perfect Geometry	opposes	Color fringing along high-contrast edges disrupts geometric precision ( Carlson et al., 2018 ) .
Optical & Sensor Artifacts: Blur or Depth of Field	$\Rightarrow$	Optical & Sensor Artifacts: Noise Present	supports	Blur and noise commonly co-occur as visible image degradations ( Carlson et al., 2018 ) .
Optical & Sensor Artifacts: Noise Present	$\Rightarrow$	Optical & Sensor Artifacts: Compression Artifacts	supports	Noise interacts with compression, producing visible artifacts ( Elmquist and Negrut, 2021 ) .

Appendix

Table 4: A part of the paper-supported trait–trait relationships restricted to properties that are visually observable or inferable from a single image by a strong vision–language model. Trait names correspond directly to canonical knowledge graph identifiers.

Figure 4: Visualization of part of the realism knowledge graph generated using Neo4j Browser. Nodes correspond to realism traits, while signed edges encode supportive (positive) and opposing (negative) relationships derived from graphics and perception literature.

Panel of Figure 5 — Figure 5: Qualitative sim-to-real results.

Panel of Figure 6 — Figure 6: Qualitative sim-to-real results on ablation study

Sim-to-Real Transfer

Simulation-to-reality transfer adapts models and policies trained in simulation for reliable performance in physical or otherwise real-world settings.

See all 316 papers

NeuralSRender: G-Buffer Guided Video Diffusion for Sim-to-Real Rendering
Oct 21, 2026Yipeng Lu, Youqi Liao, Zhenyu Wang +2Sim-to-Real Transfer Synthetic-to-Real Domain Adaptation
Wuhan University, Wuhan, China
A Hybrid Approach for Closing the Sim2real Appearance Gap in Game Engine Synthetic Datasets
May 4, 2026Stefanos PasiosImage-to-Image Translation Sim-to-Real Transfer
independent researcher
Wavelet Phase Diffusion for Structurally and Semantically Consistent Sim-to-Real Translation
Jul 8, 2026Kaiwen Wang, Frank Bieder, Yinzhe Shen +3Synthetic-to-Real Domain Adaptation Image-to-Image Translation
Karlsruhe Institute of Technology · FZI Research Center for Information Technology

Oct 21, 2026J/K move · Enter open · S save

NeuralSRender: G-Buffer Guided Video Diffusion for Sim-to-Real Rendering

Yipeng Lu, Youqi Liao, Zhenyu Wang, Yuning Peng+1

Wuhan University, Wuhan, China

Sim-to-Real Transfer Synthetic-to-Real Domain Adaptation Video Diffusion Models Controllable Video Generation Geometric Consistency in Video Generation

Open paper AskPDF arXiv

The scarcity of high-quality annotated visual data remains a critical bottleneck for autonomous-driving perception. Simulators provide training data at scale, but the limited realism of synthetic videos compromises perception performance in real-world applications. To address this limitation, we propose NeuralSRender, a sim-to-real video translation framework that enhances the photorealism of synthetic videos while maintaining geometric consistency with the source scenes. Built on a pretrained video diffusion transformer (DiT), NeuralSRender uses a lightweight projector to inject geometric and material priors from G-buffers, and LoRA enables parameter-efficient adaptation. The model is trained in two stages: it first learns G-buffer conditioning from synthetic videos with accurate G-buffers, then adapts to real-world appearance using real videos with estimated G-buffers. During the second stage, an additional LoRA module is trained while the projector and first-stage LoRA remain frozen to reduce interference from estimation noise. NeuralSRender introduces only 1.45% parameter overhead and can be trained on consumer-grade GPUs with 24 GB of memory. To support training and evaluation, we introduce GenReal, a large-scale driving video dataset comprising synthetic and real-world videos with corresponding G-buffers. Experiments demonstrate improved photorealism with competitive geometric consistency. Training perception models on our translated videos instead of raw simulator renders improves detection

\mathrm{AP}_{50}

on nuImages from 35.1 to 40.6 and segmentation mIoU on Cityscapes from 32.4 to 40.8. Our code, data, and model weights are available at: https://github.com/WHU-USI3DV/NeuralSRender.

Abstract

Sim-to-Real Transfer

Explore similar work

NeuralSRender: G-Buffer Guided Video Diffusion for Sim-to-Real Rendering

A Hybrid Approach for Closing the Sim2real Appearance Gap in Game Engine Synthetic Datasets

Wavelet Phase Diffusion for Structurally and Semantically Consistent Sim-to-Real Translation