SynthRender and I-AsSET: Open-Source Framework and Dataset for Bidirectional Sim-Real Transfer in Industrial Object Perception

Jose Moises Araya-Martinez; Thushar Tom; Adrián Sanchis Reig; Pablo Rey Valiente; Jens Lambrecht; Jörg Krüger

cs.CVFeb 24, 2026

SynthRender and I-AsSET: Open-Source Framework and Dataset for Bidirectional Sim-Real Transfer in Industrial Object Perception

Authors: Jose Moises Araya-Martinez, Thushar Tom, Adrián Sanchis Reig, Pablo Rey Valiente, Jens Lambrecht, Jörg Krüger

Organizations: Technical University Berlin, Industrial Automation Technology, Berlin, Germany · Mercedes-Benz AG, Future Manufacturing Technologies, Sindelfingen, Germany · Technical University Braunschweig, Institute for Cognitive Robotics, Braunschweig, Germany

Abstract

Object perception is fundamental for tasks such as robotic material handling and quality inspection. However, modern supervised deep-learning models require large annotated datasets for robust automation under semi-uncontrolled conditions; a major barrier for widespread deployment with proprietary industrial parts. We address this through an integrated framework combining synthetic data generation and structured empirical evaluation for systematic investigation of bidirectional sim-to-real transfer. Our method integrates 2D-to-3D Reality-to-Simulation techniques for 3D asset creation from physical parts with programmatic Guided Domain Randomization (GDR) via SynthRender, an open-source synthetic image generation framework. Structured ablation studies across multiple benchmarks quantify the impact of individual rendering design choices, yielding practical guidelines for data-efficient synthetic training. To support evaluation under realistic industrial conditions, we introduce Industrial Assets for Sim-to-Real Evaluation and Transfer (I-AsSET), a 32-class dataset with diverse textures, intra-class variation, strong inter-class similarities, and 19,672 annotations, providing both CAD models and reconstructed meshes for bidirectional sim-to-real benchmarking. Across three industrial benchmarks, the proposed framework achieves highly competitive performance, reaching 98.7% mAP@50 on a public robotics dataset, 97.9% mAP@50 on an automotive benchmark, and 95.1% mAP@50 on I-AsSET.

Figures & tables

Real-Image Subset

Train Set: Synthetic Images

Visual Complexity

Image Quality

Modalities

Dataset Name

Availability

# Classes

# Images

Instances ∗

Light Var.

Pose Var.

Inter-Class Ambiguity

Material Variation

Background Clutter

Highest RGB Resolution

Depth Acc. at 0.5 m

# Images

PBR Materials

RGB

Depth

3D CADs

T-LESS [ 34 ]

Public

48,960

108,432

3264

\times

2448

\sim

1.2 mm ∗∗

50,000

✓ ∗

✓

SIP15-OD [ 74 ]

Proprietary

321

877

4032

\times

3024

—

\leq

8,000

✓

\times

\times

Automotive [ 4 ]

Proprietary

225

4320

\times

3240

—

\leq

1,000

✓

\times

\times

SORDI.ai [ 1 ]

Public

111

—

1,191,893

✓

\times

\times

Table 1: Comparison of datasets for sim-to-real industrial object perception. Special attention is given to public datasets with dynamic light and pose conditions, as well as a high degree of inter-class similarity. Possible scores across these categories are low , medium , high , and very high variability, relative to other state-of-the-art datasets. Also a high diversity in the data modalities can support future sim-to-real benchmarking with novel technologies, such as the bidirectional sim-real gap exploration offered by I-AsSET . The instances variable reports the total number of annotated object occurrences across the listed real-image subset.

System	Engine	Features	Auto. Annotations	Focus	Sim-to-Real Functions
SynthDet [ 41 ]	Unity [ 70 ]	Domain Randomization	C D – SM BB – –	Consumer	Unity Perception, ML-Agents
CARLA [ 20 ]	Unreal [ 23 ]	Urban Sim, Weather	C D P SM BB – L	Driving	Python API, ROS bridge
Isaac Sim [ 54 ]	Omniverse	DR/GDR, Physics	C D P SM BB N L	Industrial	Omniverse Replicator, ROS
CAD2Render [ 52 ]	Unity (HDRP)	DR, GPU-accelerated	C D – SM – N –	Manufacturing	GUI Generator, custom pass
SynosIs [ 30 ]	Appleseed [ 2 ]	Procedural Textures	C – – SM – – –	Inspection	Parametric control
Kubric [ 31 ]	Blender + PyBullet	Physics, scalable video	C D P SM BB N –	General vision	Python API, distributed generation

Table 2: Technical comparison of the proposed SynthRender against related synthetic data generation systems and simulation applications with sim-to-real integration. The modalities category comprises automatic annotations and scene information that can be generated with each system. They are abbreviated as color (C), depth (D), pose annotations (P), semantic masks (SM), bounding boxes (BB), surface normals (N), and LiDAR data (L).

Figure 1: Proposed framework for benchmarking and ablation studies, comprising five stages: data acquisition, DA via 2D-to-3D physical scene reconstruction and manual assignment, different DR settings generate separate synthetic train-set variants using SynthRender, training across multiple object detectors, and evaluation on the introduced I-AsSET dataset for bidirectional sim-to-real gap benchmarking.

Figure 2: 3D asset and texture generation as DA approaches. The four pipelines cover three geometry paradigms: manual modeling, 3DGS , and TRELLIS; and four texture sources: manual PBR , MeshyAI, 3DGS, or TRELLIS.

Feature	SCAP [ 6 ]	SynMfg [ 74 ]	BProc [ 19 ]	Isaac [ 54 ]	SynthR. [ 7 ]
Open source	No	Yes	Yes	No	Yes
Guided DR	Yes	No	No	Yes	Yes
Workflow automation	Partial	Partial	Yes	Yes	Yes
Configurable pipeline	Partial	Yes	Yes	Yes	Yes
Keyframe-based generation	No	No	Partial	Partial	Yes
Batch rendering	No	No	Partial	Yes	Yes

Table 3: Positioning of SynthRender with respect to related synthetic data generation systems. BProc: BlenderProc; Isaac: NVIDIA Isaac Sim, built on the Omniverse platform; SynthR.: SynthRender (ours). The comparison describes the documented functionality of each system rather than ranking their overall capabilities; “No” does not imply that a feature could not be implemented through custom extensions.

Figure 3: Functional diagram of SynthRender, illustrating the input, generation, and output stages.

Parameter	Range	Unit	Description
General Settings
Output format	$\{\mathrm{seg,rgb,norm}\}$	–	Output channels
Physics	$\{0,1\}$	–	Physics simulation
Background light	$s_{\mathrm{bg}}\geq 0$	–	Env. light intensity
Anchor Spawn
Center	$\mathbf{c}_{a}\in\mathbb{R}^{3}$	m	Anchor center

Table 4: SynthRender available parameters for DR and GDR. The superscripts

\mathrm{min}

and

\mathrm{max}

denote the user-defined lower and upper bounds of each parameter, respectively.

Panel of Figure 4 — Figure 4: I-AsSET objects comparison. (a) Synthetic renders of the 32 industrial CAD models used in this study, annotated with their corresponding class IDs. (b) The real physical components arranged in similar positions.

ID	Class Name	Source	Family
1	C_O_Ring_L	Custom-Modeled	Mechanical
2	C_O_Ring_M	Custom-Modeled	Mechanical
3	C_O_Ring_S	Custom-Modeled	Mechanical
4	C_Plastic_Washer_L	Custom-Modeled	Mechanical
5	C_Plastic_Washer_S	Custom-Modeled	Mechanical
6	C_Steel_Ball_L	Custom-Modeled	Mechanical

Table 5: Object classes, source and system-level family.

Domain	Total	Image Composition
Domain	Total	1 Obj.	1x Inst.	2x Inst.
Controlled	101	64	21	16
Sunlight	67	32	20	15
Backgrounds	100	0	49	51
Robot	240	0	120	120
Total	508	96	210	202

Table 6: Distribution of I-AsSET test images across environmental domains and object densities (Single Object / Single Instance / Double Instance).

Panel of Figure 5 — Figure 5: Representative sample images from the I-AsSET dataset showcasing diverse real-world test scenes across our four different domains. Ground truth object detection bounding boxes are overlaid to demonstrate the density, scale variations, and annotation quality in highly cluttered scenarios.

Hyperparameter	Value
Epochs	500
Train image size	1024 $\times$ 1024
Batch size	16
Learning rate	0.01
Patience	30

Table 7: Training hyperparameters used across our experiments.

Additive Ablation

Reverse Ablation

Baseline

RGB

RGB + Exp

Physics

Phy+RGB+EXP

P.+R.+E.+Intri.

All + Random Texture

All minus Intri.

All minus Exp

All minus RGB

RandTex + Phy

All minus Physics

RandTex + RGB

RandTex + Exp

RandTex Only

Scale

✓

Position & Orientation

✓

Background Texture

✓

RGB Light

✗

✓

✗

✓

✗

✓

✗

Exponential Light

✗

✓

✗

✓

✗

✓

✗

✓

✗

✓

✗

Table 8: Ablation study configurations for domain randomization in SynthRender. Only parameters that vary across ablation configurations are shown. Fixed parameters, such as camera to object distance, are constrained to realistic ranges representative of the target evaluation domain. The parameters listed in Table 4 are constrained to a defined variation range. ✓ indicates the feature is enabled; ✗ indicates it is disabled. The Additive Ablation category builds features incrementally, while Reverse Ablation removes specific features from the full setup or adds them individually to the target texture randomization.

Split	Notation	Domain	Images
Training	$\mathcal{D}_{\text{train}}^{\text{syn}}$	Synthetic	3,200
Validation	$\mathcal{D}_{\text{val}}^{\text{syn}}$	Synthetic	800
Total synthetic	$\mathcal{D}^{\text{syn}}$	Synthetic	4,000
Test	$\mathcal{D}_{\text{test}}^{\text{real}}$	Real	508

Table 9: Dataset partitioning used for sim-to-real evaluation on I-AsSET .

Figure 6: Sim-to-real transfer on three SOTA object detection families and model sizes trained on the I-AsSET synthetic baseline.

Figure 7: End-to-end runtime comparison between single-GPU local execution and four-GPU cloud execution for SynthRender datasets on the automotive, robotics, and I-AsSET benchmarks. Runtime is divided into rendering, annotation generation, and detector training.

Figure 8: Ablation experiments on the automotive [ 4 ] and robotics [ 36 ] benchmarks.

Figure 9: Additive ablation study on I-AsSET using multiple SynthRender’s configurations as per Table 8 . Mean mAP values and respective 95% confidence intervals are presented.

Figure 10: Reverse ablation study on I-AsSET with SynthRender parameters. Mean mAP values and respective 95% confidence intervals are presented.

Figure 11: Effect of the synthetic training-set size (200–3200 images) on real-world test performance using YOLO11m models for our two best synthetic configurations, Random Texture and Manual Texture . Solid lines report mAP@50 and dashed lines mAP@50:95, and color distinguishes the two configurations.

Dataset	mAP	Number of real images
Dataset	mAP	0	1	3	5	10
Random Texture	@50	95.1	97.1	97.9	98.5	98.8
Random Texture	@50:95	82.9	85.9	88.1	89.3	89.7
Manual Texture	@50	92.5	95.9	97.5	98.3	98.7
Manual Texture	@50:95	80.3	84.9	87.5	89.2	90.1

Table 10: Few-shot results on I-AsSET with 4000 synthetic images. Zero-shot uses only synthetic data; few-shot setups add 1–10 real samples to the training dataset.

Background	Target Objects		mAP@		Detection Metrics
	Geometry	Texture	50	50:95	Precision	Recall
Randomized	Manual CAD	Randomized	$\mathbf{95.1}^{95.2}_{95.0}$	$\mathbf{82.9}^{82.9}_{82.8}$	$92.6^{92.8}_{92.3}$	$91.6^{91.9}_{91.3}$
	Manual CAD	Manual PBR	$92.5^{92.9}_{92.1}$	$80.3^{80.7}_{80.0}$	$92.1^{92.4}_{91.9}$	$87.2^{87.6}_{86.8}$
	3DGS Reconstr.	3DGS Reconstr.	$86.6^{86.9}_{86.2}$	$75.2^{75.4}_{74.8}$	$85.0^{85.6}_{84.4}$	$83.1^{83.4}_{82.7}$
	3DGS Reconstr.	Randomized.	$92.8^{93.2}_{92.4}$	$80.6^{80.9}_{80.3}$	$90.6^{90.9}_{90.3}$	$87.9^{88.2}_{87.6}$
	Manual CAD	MeshyAI GenAI	$85.9^{86.1}_{85.7}$	$74.6^{74.8}_{74.4}$	$83.7^{84.2}_{83.2}$	$78.9^{79.3}_{78.3}$

Table 11: Comparison of real-to-sim DA and randomization strategies. Values reported as mean

{}^{\text{95\% CI upper}}_{\text{95\% CI lower}}

over

n=6

independent runs (percentile bootstrap).

		mAP@		Detection Metrics		Train Conditions
Dataset	Method	50	50:95	Prec.	Rec.	Model	# Img	Res.	Texture
Robotics [ 36 ]	Zhu [ 74 ]	96.1	–	–	–	YOLOv8	4k	$720^{2}$	Rand. PBR
Robotics [ 36 ]	Ours	$\mathbf{98.7}^{98.9}_{98.5}$	$\mathbf{68.2}^{68.4}_{68.0}$	$96.1^{97.4}_{94.7}$	$97.3^{97.7}_{96.9}$	YOLOv8	4k	$720^{2}$	Rand. PBR
Automotive [ 4 ]	Araya-Martinez [ 4 ]	–	75.0	–	–	YOLOv8	900	$512^{2}$	Manual
	Araya-Martinez [ 9 ]	91.3	78.4	–	–	YOLOv8	900	$512^{2}$	Manual
	Ours	$\mathbf{95.5}^{95.8}_{95.1}$	$\mathbf{81.0}^{82.7}_{79.5}$	$93.3^{95.4}_{91.6}$	$92.2^{93.9}_{91.0}$	YOLOv8	900	$512^{2}$	Manual

Table 12: State-of-the-art comparison on fully-synthetic training of robotics [ 36 ] , automotive [ 4 ] and proposed I-AsSET benchmarks. Automotive and Robotics experiments use 512x512 and 720×720 respectively to match the resolution of the compared prior work. Values for Ours are reported as mean

{}^{\text{95\% CI upper}}_{\text{95\% CI lower}}

over

n=6

independent runs.

Figure 12: Scale discrimination on the single-object test images, overlaid with LayerCAM evidence maps [ 42 ] for the predicted class. Warm colors mark the image regions supporting the prediction. The ground-truth box is solid green and the predicted box is dashed, green where the class is correct and red where the size is mis-predicted. Top row: parts from the same size-ambiguous family imaged at different sizes, all correctly classified, showing that the detector recovers absolute scale without any relative-size reference. Bottom row: the residual within-family size confusions; the ground-truth and predicted boxes coincide, so localization is correct and only the size class flips. In every panel the evidence is confined to the part and concentrates on small local structures rather than on the surrounding background.

Figure 13: Per-class performance comparison of I-AsSET objects after training with manual and randomized textures.

Figure 14: Per-class confusion matrix on the I-AsSET test set comparing the two texture-assignment strategies. Each cell is split diagonally: the upper-left triangle is the randomized-texture model and the lower-right triangle is the manually-textured model, shaded by instance count on a shared logarithmic scale to keep sparse (rare) confusions visible. The final background row and column report, per model, the missed detections and the false detections; their counts are printed in each background cell (randomized top-left, manual bottom-right). The pronounced diagonal reflects the high accuracy of both models.

Sim-to-Real Transfer

Simulation-to-reality transfer adapts models and policies trained in simulation for reliable performance in physical or otherwise real-world settings.

See all 300 papers

Sep 29, 2026

Curating Synthetic Data for Task-Specific Visual Perception

Synthetic data are most valuable where general-purpose datasets cannot provide the domain-specific priors a task requires, and where manual annotation is expensive, imprecise, or infeasible. In this article we argue that the central question for specialized vision systems is not how to generate more data, but which data to generate. We therefore discuss curated synthetic data, whose scene content, appearance variations, sensing characteristics, and annotations are deliberately designed around a given task. We examine three complementary curation paradigms. Procedural rendering offers explicit control over scene parameters and the annotations follow by construction. Physically-based simulation encodes the mechanism behind an observed effect and yields exactly aligned supervision pairs. Generative AI learns sensor-specific appearance from small real seed sets and attains plausible realism, though it remains prone to hallucination and to inaccurate annotation. These paradigms are illustrated with examples from industrial surface defect detection, restoration of degraded digitized autochrome plates, and 6DoF pose estimation from RGB and event data. Using these examples, we analyze different data regimes and training strategies that combine synthetic and real data across these paradigms. We conclude that curated synthetic data are best understood as a complement to real observations, and that hybrid pipelines combining controllable supervision with learned appearance are the most promising direction for reliable sim-to-real transfer.

Saptarshi Neil Sinha, Paul Julius Kühn, Michael Weinmann

Fraunhofer IGD, 64283 Darmstadt, Germany · Delft University of Technology, 2628 CD Delft, Netherlands

Synthetic-to-Real Domain Adaptation Synthetic Data Generation Training Data Curation

Ask PDF

Mar 17, 2026

ManiTwin: Scaling Data-Generation-Ready Digital Object Dataset to 100K

Learning in simulation provides a useful foundation for scaling robotic manipulation capabilities. However, this paradigm often suffers from a lack of data-generation-ready digital assets, in both scale and diversity. In this work, we present ManiTwin, an automated and efficient pipeline for generating data-generation-ready digital object twins. Our pipeline transforms a single image into simulation-ready and semantically annotated 3D asset, enabling large-scale robotic manipulation data generation. Using this pipeline, we construct ManiTwin-100K, a dataset containing 100K high-quality annotated 3D assets. Each asset is equipped with physical properties, language descriptions, functional annotations, and verified manipulation proposals. Experiments demonstrate that ManiTwin provides an efficient asset synthesis and annotation workflow, and that ManiTwin-100K offers high-quality and diverse assets for manipulation data generation, random scene synthesis, and VQA data generation, establishing a strong foundation for scalable simulation data synthesis and policy learning. Our webpage is available at https://manitwin.github.io/.

Kaixuan Wang, Tianxing Chen, Jiawei Liu +13

The University of Hong Kong · Xspark AI · Shenzhen University +7

3D Asset Generation Synthetic Data Generation for Robotics Digital Twins

Ask PDF

Jun 30, 2026

A Synthetic-Driven Vision System for Assembly Step Recognition

Quality control in industrial assembly is essential, and real-time monitoring of the assembly process is crucial for preventing costly defects and ensuring production reliability. Vision-based automated inspection offers a powerful solution for such real-time monitoring. However, due to the specialized industrial components and processes, training these models typically relies on task-specific real-world data, which is costly and labor-intensive to collect and annotate. In this paper, we propose a system that automatically generates realistic assembly sequences and further trains real-time inspection models using the synthetic data. It can be efficiently applied to a given task within an hour, requiring only CAD models and simple step descriptions. Focusing on practical challenges, our system integrates a physics-based motion generation module to capture the variance of different human assembly, designs domain-randomized rendering to deal with the environmental complexity and variation, and employs an object-detection-based step recognition module for robust sim-to-real transfer, leading to 92.4% accuracy on a real-world assembly case with 46.7%, 15.8% and 61.2% performance improvement, respectively. Overall, our system provides a practical solution for industrial assembly inspection without requiring expensive real-world data collection and annotation, with the effectiveness validated on real industrial assembly tasks.

Hui Zhang, Xuanang Lei, Rui Wang +2

1ETH Zurich, Switzerland · 2inspire AG, Switzerland

Synthetic-to-Real Domain Adaptation Industrial Inspection Synthetic Data Generation for Robotics

Ask PDF

Abstract

Sim-to-Real Transfer

Explore similar work

Curating Synthetic Data for Task-Specific Visual Perception

ManiTwin: Scaling Data-Generation-Ready Digital Object Dataset to 100K

A Synthetic-Driven Vision System for Assembly Step Recognition