SynthRender and I-AsSET: Open-Source Framework and Dataset for Bidirectional Sim-Real Transfer in Industrial Object Perception

Jose Moises Araya-Martinez; Thushar Tom; Adrián Sanchis Reig; Pablo Rey Valiente; Jens Lambrecht; Jörg Krüger

cs.CVFeb 24, 2026

SynthRender and I-AsSET: Open-Source Framework and Dataset for Bidirectional Sim-Real Transfer in Industrial Object Perception

Authors: Jose Moises Araya-Martinez, Thushar Tom, Adrián Sanchis Reig, Pablo Rey Valiente, Jens Lambrecht, Jörg Krüger

Organizations: Technical University Berlin, Industrial Automation Technology, Berlin, Germany · Mercedes-Benz AG, Future Manufacturing Technologies, Sindelfingen, Germany · Technical University Braunschweig, Institute for Cognitive Robotics, Braunschweig, Germany

Abstract

Object perception is fundamental for tasks such as robotic material handling and quality inspection. However, modern supervised deep-learning models require large annotated datasets for robust automation under semi-uncontrolled conditions; a major barrier for widespread deployment with proprietary industrial parts. We address this through an integrated framework combining synthetic data generation and structured empirical evaluation for systematic investigation of bidirectional sim-to-real transfer. Our method integrates 2D-to-3D Reality-to-Simulation techniques for 3D asset creation from physical parts with programmatic Guided Domain Randomization (GDR) via SynthRender, an open-source synthetic image generation framework. Structured ablation studies across multiple benchmarks quantify the impact of individual rendering design choices, yielding practical guidelines for data-efficient synthetic training. To support evaluation under realistic industrial conditions, we introduce Industrial Assets for Sim-to-Real Evaluation and Transfer (I-AsSET), a 32-class dataset with diverse textures, intra-class variation, strong inter-class similarities, and 19,672 annotations, providing both CAD models and reconstructed meshes for bidirectional sim-to-real benchmarking. Across three industrial benchmarks, the proposed framework achieves highly competitive performance, reaching 98.7% mAP@50 on a public robotics dataset, 97.9% mAP@50 on an automotive benchmark, and 95.1% mAP@50 on I-AsSET.

Figures & tables

Real-Image Subset

Train Set: Synthetic Images

Visual Complexity

Image Quality

Modalities

Dataset Name

Availability

# Classes

# Images

Instances ∗

Light Var.

Pose Var.

Inter-Class Ambiguity

Material Variation

Background Clutter

Highest RGB Resolution

Depth Acc. at 0.5 m

# Images

PBR Materials

RGB

Depth

3D CADs

T-LESS [ 34 ]

Public

48,960

108,432

3264

\times

2448

\sim

1.2 mm ∗∗

50,000

✓ ∗

✓

SIP15-OD [ 74 ]

Proprietary

321

877

4032

\times

3024

—

\leq

8,000

✓

\times

\times

Automotive [ 4 ]

Proprietary

225

4320

\times

3240

—

\leq

1,000

✓

\times

\times

SORDI.ai [ 1 ]

Public

111

—

1,191,893

✓

\times

\times

Table 1: Comparison of datasets for sim-to-real industrial object perception. Special attention is given to public datasets with dynamic light and pose conditions, as well as a high degree of inter-class similarity. Possible scores across these categories are low , medium , high , and very high variability, relative to other state-of-the-art datasets. Also a high diversity in the data modalities can support future sim-to-real benchmarking with novel technologies, such as the bidirectional sim-real gap exploration offered by I-AsSET . The instances variable reports the total number of annotated object occurrences across the listed real-image subset.

System	Engine	Features	Auto. Annotations	Focus	Sim-to-Real Functions
SynthDet [ 41 ]	Unity [ 70 ]	Domain Randomization	C D – SM BB – –	Consumer	Unity Perception, ML-Agents
CARLA [ 20 ]	Unreal [ 23 ]	Urban Sim, Weather	C D P SM BB – L	Driving	Python API, ROS bridge
Isaac Sim [ 54 ]	Omniverse	DR/GDR, Physics	C D P SM BB N L	Industrial	Omniverse Replicator, ROS
CAD2Render [ 52 ]	Unity (HDRP)	DR, GPU-accelerated	C D – SM – N –	Manufacturing	GUI Generator, custom pass
SynosIs [ 30 ]	Appleseed [ 2 ]	Procedural Textures	C – – SM – – –	Inspection	Parametric control
Kubric [ 31 ]	Blender + PyBullet	Physics, scalable video	C D P SM BB N –	General vision	Python API, distributed generation

Table 2: Technical comparison of the proposed SynthRender against related synthetic data generation systems and simulation applications with sim-to-real integration. The modalities category comprises automatic annotations and scene information that can be generated with each system. They are abbreviated as color (C), depth (D), pose annotations (P), semantic masks (SM), bounding boxes (BB), surface normals (N), and LiDAR data (L).

Figure 1: Proposed framework for benchmarking and ablation studies, comprising five stages: data acquisition, DA via 2D-to-3D physical scene reconstruction and manual assignment, different DR settings generate separate synthetic train-set variants using SynthRender, training across multiple object detectors, and evaluation on the introduced I-AsSET dataset for bidirectional sim-to-real gap benchmarking.

Figure 2: 3D asset and texture generation as DA approaches. The four pipelines cover three geometry paradigms: manual modeling, 3DGS , and TRELLIS; and four texture sources: manual PBR , MeshyAI, 3DGS, or TRELLIS.

Feature	SCAP [ 6 ]	SynMfg [ 74 ]	BProc [ 19 ]	Isaac [ 54 ]	SynthR. [ 7 ]
Open source	No	Yes	Yes	No	Yes
Guided DR	Yes	No	No	Yes	Yes
Workflow automation	Partial	Partial	Yes	Yes	Yes
Configurable pipeline	Partial	Yes	Yes	Yes	Yes
Keyframe-based generation	No	No	Partial	Partial	Yes
Batch rendering	No	No	Partial	Yes	Yes

Table 3: Positioning of SynthRender with respect to related synthetic data generation systems. BProc: BlenderProc; Isaac: NVIDIA Isaac Sim, built on the Omniverse platform; SynthR.: SynthRender (ours). The comparison describes the documented functionality of each system rather than ranking their overall capabilities; “No” does not imply that a feature could not be implemented through custom extensions.

Figure 3: Functional diagram of SynthRender, illustrating the input, generation, and output stages.

Parameter	Range	Unit	Description
General Settings
Output format	$\{\mathrm{seg,rgb,norm}\}$	–	Output channels
Physics	$\{0,1\}$	–	Physics simulation
Background light	$s_{\mathrm{bg}}\geq 0$	–	Env. light intensity
Anchor Spawn
Center	$\mathbf{c}_{a}\in\mathbb{R}^{3}$	m	Anchor center

Table 4: SynthRender available parameters for DR and GDR. The superscripts

\mathrm{min}

and

\mathrm{max}

denote the user-defined lower and upper bounds of each parameter, respectively.

Panel of Figure 4 — Figure 4: I-AsSET objects comparison. (a) Synthetic renders of the 32 industrial CAD models used in this study, annotated with their corresponding class IDs. (b) The real physical components arranged in similar positions.

ID	Class Name	Source	Family
1	C_O_Ring_L	Custom-Modeled	Mechanical
2	C_O_Ring_M	Custom-Modeled	Mechanical
3	C_O_Ring_S	Custom-Modeled	Mechanical
4	C_Plastic_Washer_L	Custom-Modeled	Mechanical
5	C_Plastic_Washer_S	Custom-Modeled	Mechanical
6	C_Steel_Ball_L	Custom-Modeled	Mechanical

Table 5: Object classes, source and system-level family.

Domain	Total	Image Composition
Domain	Total	1 Obj.	1x Inst.	2x Inst.
Controlled	101	64	21	16
Sunlight	67	32	20	15
Backgrounds	100	0	49	51
Robot	240	0	120	120
Total	508	96	210	202

Table 6: Distribution of I-AsSET test images across environmental domains and object densities (Single Object / Single Instance / Double Instance).

Panel of Figure 5 — Figure 5: Representative sample images from the I-AsSET dataset showcasing diverse real-world test scenes across our four different domains. Ground truth object detection bounding boxes are overlaid to demonstrate the density, scale variations, and annotation quality in highly cluttered scenarios.

Hyperparameter	Value
Epochs	500
Train image size	1024 $\times$ 1024
Batch size	16
Learning rate	0.01
Patience	30

Table 7: Training hyperparameters used across our experiments.

Additive Ablation

Reverse Ablation

Baseline

RGB

RGB + Exp

Physics

Phy+RGB+EXP

P.+R.+E.+Intri.

All + Random Texture

All minus Intri.

All minus Exp

All minus RGB

RandTex + Phy

All minus Physics

RandTex + RGB

RandTex + Exp

RandTex Only

Scale

✓

Position & Orientation

✓

Background Texture

✓

RGB Light

✗

✓

✗

✓

✗

✓

✗

Exponential Light

✗

✓

✗

✓

✗

✓

✗

✓

✗

✓

✗

Table 8: Ablation study configurations for domain randomization in SynthRender. Only parameters that vary across ablation configurations are shown. Fixed parameters, such as camera to object distance, are constrained to realistic ranges representative of the target evaluation domain. The parameters listed in Table 4 are constrained to a defined variation range. ✓ indicates the feature is enabled; ✗ indicates it is disabled. The Additive Ablation category builds features incrementally, while Reverse Ablation removes specific features from the full setup or adds them individually to the target texture randomization.

Split	Notation	Domain	Images
Training	$\mathcal{D}_{\text{train}}^{\text{syn}}$	Synthetic	3,200
Validation	$\mathcal{D}_{\text{val}}^{\text{syn}}$	Synthetic	800
Total synthetic	$\mathcal{D}^{\text{syn}}$	Synthetic	4,000
Test	$\mathcal{D}_{\text{test}}^{\text{real}}$	Real	508

Table 9: Dataset partitioning used for sim-to-real evaluation on I-AsSET .

Figure 6: Sim-to-real transfer on three SOTA object detection families and model sizes trained on the I-AsSET synthetic baseline.

Figure 7: End-to-end runtime comparison between single-GPU local execution and four-GPU cloud execution for SynthRender datasets on the automotive, robotics, and I-AsSET benchmarks. Runtime is divided into rendering, annotation generation, and detector training.

Figure 8: Ablation experiments on the automotive [ 4 ] and robotics [ 36 ] benchmarks.

Figure 9: Additive ablation study on I-AsSET using multiple SynthRender’s configurations as per Table 8 . Mean mAP values and respective 95% confidence intervals are presented.

Figure 10: Reverse ablation study on I-AsSET with SynthRender parameters. Mean mAP values and respective 95% confidence intervals are presented.

Figure 11: Effect of the synthetic training-set size (200–3200 images) on real-world test performance using YOLO11m models for our two best synthetic configurations, Random Texture and Manual Texture . Solid lines report mAP@50 and dashed lines mAP@50:95, and color distinguishes the two configurations.

Dataset	mAP	Number of real images
Dataset	mAP	0	1	3	5	10
Random Texture	@50	95.1	97.1	97.9	98.5	98.8
Random Texture	@50:95	82.9	85.9	88.1	89.3	89.7
Manual Texture	@50	92.5	95.9	97.5	98.3	98.7
Manual Texture	@50:95	80.3	84.9	87.5	89.2	90.1

Table 10: Few-shot results on I-AsSET with 4000 synthetic images. Zero-shot uses only synthetic data; few-shot setups add 1–10 real samples to the training dataset.

Background	Target Objects		mAP@		Detection Metrics
	Geometry	Texture	50	50:95	Precision	Recall
Randomized	Manual CAD	Randomized	$\mathbf{95.1}^{95.2}_{95.0}$	$\mathbf{82.9}^{82.9}_{82.8}$	$92.6^{92.8}_{92.3}$	$91.6^{91.9}_{91.3}$
	Manual CAD	Manual PBR	$92.5^{92.9}_{92.1}$	$80.3^{80.7}_{80.0}$	$92.1^{92.4}_{91.9}$	$87.2^{87.6}_{86.8}$
	3DGS Reconstr.	3DGS Reconstr.	$86.6^{86.9}_{86.2}$	$75.2^{75.4}_{74.8}$	$85.0^{85.6}_{84.4}$	$83.1^{83.4}_{82.7}$
	3DGS Reconstr.	Randomized.	$92.8^{93.2}_{92.4}$	$80.6^{80.9}_{80.3}$	$90.6^{90.9}_{90.3}$	$87.9^{88.2}_{87.6}$
	Manual CAD	MeshyAI GenAI	$85.9^{86.1}_{85.7}$	$74.6^{74.8}_{74.4}$	$83.7^{84.2}_{83.2}$	$78.9^{79.3}_{78.3}$

Table 11: Comparison of real-to-sim DA and randomization strategies. Values reported as mean

{}^{\text{95\% CI upper}}_{\text{95\% CI lower}}

over

n=6

independent runs (percentile bootstrap).

		mAP@		Detection Metrics		Train Conditions
Dataset	Method	50	50:95	Prec.	Rec.	Model	# Img	Res.	Texture
Robotics [ 36 ]	Zhu [ 74 ]	96.1	–	–	–	YOLOv8	4k	$720^{2}$	Rand. PBR
Robotics [ 36 ]	Ours	$\mathbf{98.7}^{98.9}_{98.5}$	$\mathbf{68.2}^{68.4}_{68.0}$	$96.1^{97.4}_{94.7}$	$97.3^{97.7}_{96.9}$	YOLOv8	4k	$720^{2}$	Rand. PBR
Automotive [ 4 ]	Araya-Martinez [ 4 ]	–	75.0	–	–	YOLOv8	900	$512^{2}$	Manual
	Araya-Martinez [ 9 ]	91.3	78.4	–	–	YOLOv8	900	$512^{2}$	Manual
	Ours	$\mathbf{95.5}^{95.8}_{95.1}$	$\mathbf{81.0}^{82.7}_{79.5}$	$93.3^{95.4}_{91.6}$	$92.2^{93.9}_{91.0}$	YOLOv8	900	$512^{2}$	Manual

Table 12: State-of-the-art comparison on fully-synthetic training of robotics [ 36 ] , automotive [ 4 ] and proposed I-AsSET benchmarks. Automotive and Robotics experiments use 512x512 and 720×720 respectively to match the resolution of the compared prior work. Values for Ours are reported as mean

{}^{\text{95\% CI upper}}_{\text{95\% CI lower}}

over

n=6

independent runs.

Figure 12: Scale discrimination on the single-object test images, overlaid with LayerCAM evidence maps [ 42 ] for the predicted class. Warm colors mark the image regions supporting the prediction. The ground-truth box is solid green and the predicted box is dashed, green where the class is correct and red where the size is mis-predicted. Top row: parts from the same size-ambiguous family imaged at different sizes, all correctly classified, showing that the detector recovers absolute scale without any relative-size reference. Bottom row: the residual within-family size confusions; the ground-truth and predicted boxes coincide, so localization is correct and only the size class flips. In every panel the evidence is confined to the part and concentrates on small local structures rather than on the surrounding background.

Figure 13: Per-class performance comparison of I-AsSET objects after training with manual and randomized textures.

Figure 14: Per-class confusion matrix on the I-AsSET test set comparing the two texture-assignment strategies. Each cell is split diagonally: the upper-left triangle is the randomized-texture model and the lower-right triangle is the manually-textured model, shaded by instance count on a shared logarithmic scale to keep sparse (rare) confusions visible. The final background row and column report, per model, the missed detections and the false detections; their counts are printed in each background cell (randomized top-left, manual bottom-right). The pronounced diagonal reflects the high accuracy of both models.

Sim-to-Real Transfer

Simulation-to-reality transfer adapts models and policies trained in simulation for reliable performance in physical or otherwise real-world settings.

See all 295 papers

Curating Synthetic Data for Task-Specific Visual Perception
Sep 29, 2026Saptarshi Neil Sinha, Paul Julius Kühn, Michael WeinmannSynthetic-to-Real Domain Adaptation Synthetic Data Generation
Fraunhofer IGD, 64283 Darmstadt, Germany · Delft University of Technology, 2628 CD Delft, Netherlands
ManiTwin: Scaling Data-Generation-Ready Digital Object Dataset to 100K
Mar 17, 2026Kaixuan Wang, Tianxing Chen, Jiawei Liu +133D Asset Generation Synthetic Data Generation for Robotics
The University of Hong Kong · Xspark AI · Shenzhen University +7
A Synthetic-Driven Vision System for Assembly Step Recognition
Jun 30, 2026Hui Zhang, Xuanang Lei, Rui Wang +2Synthetic-to-Real Domain Adaptation Industrial Inspection
1ETH Zurich, Switzerland · 2inspire AG, Switzerland

Sep 29, 2026J/K move · Enter open · S save

Curating Synthetic Data for Task-Specific Visual Perception

Saptarshi Neil Sinha, Paul Julius Kühn, Michael Weinmann

Fraunhofer IGD, 64283 Darmstadt, Germany · Delft University of Technology, 2628 CD Delft, Netherlands

Synthetic-to-Real Domain Adaptation Synthetic Data Generation Training Data Curation

Open paper AskPDF arXiv

Synthetic data are most valuable where general-purpose datasets cannot provide the domain-specific priors a task requires, and where manual annotation is expensive, imprecise, or infeasible. In this article we argue that the central question for specialized vision systems is not how to generate more data, but which data to generate. We therefore discuss curated synthetic data, whose scene content, appearance variations, sensing characteristics, and annotations are deliberately designed around a given task. We examine three complementary curation paradigms. Procedural rendering offers explicit control over scene parameters and the annotations follow by construction. Physically-based simulation encodes the mechanism behind an observed effect and yields exactly aligned supervision pairs. Generative AI learns sensor-specific appearance from small real seed sets and attains plausible realism, though it remains prone to hallucination and to inaccurate annotation. These paradigms are illustrated with examples from industrial surface defect detection, restoration of degraded digitized autochrome plates, and 6DoF pose estimation from RGB and event data. Using these examples, we analyze different data regimes and training strategies that combine synthetic and real data across these paradigms. We conclude that curated synthetic data are best understood as a complement to real observations, and that hybrid pipelines combining controllable supervision with learned appearance are the most promising direction for reliable sim-to-real transfer.

Abstract

Sim-to-Real Transfer

Explore similar work

Curating Synthetic Data for Task-Specific Visual Perception

ManiTwin: Scaling Data-Generation-Ready Digital Object Dataset to 100K

A Synthetic-Driven Vision System for Assembly Step Recognition