Curating Synthetic Data for Task-Specific Visual Perception

Saptarshi Neil Sinha; Paul Julius Kühn; Michael Weinmann

cs.CVSep 29, 2026

Curating Synthetic Data for Task-Specific Visual Perception

Authors: Saptarshi Neil Sinha, Paul Julius Kühn, Michael Weinmann

Organizations: Fraunhofer IGD, 64283 Darmstadt, Germany · Delft University of Technology, 2628 CD Delft, Netherlands

Abstract

Synthetic data are most valuable where general-purpose datasets cannot provide the domain-specific priors a task requires, and where manual annotation is expensive, imprecise, or infeasible. In this article we argue that the central question for specialized vision systems is not how to generate more data, but which data to generate. We therefore discuss curated synthetic data, whose scene content, appearance variations, sensing characteristics, and annotations are deliberately designed around a given task. We examine three complementary curation paradigms. Procedural rendering offers explicit control over scene parameters and the annotations follow by construction. Physically-based simulation encodes the mechanism behind an observed effect and yields exactly aligned supervision pairs. Generative AI learns sensor-specific appearance from small real seed sets and attains plausible realism, though it remains prone to hallucination and to inaccurate annotation. These paradigms are illustrated with examples from industrial surface defect detection, restoration of degraded digitized autochrome plates, and 6DoF pose estimation from RGB and event data. Using these examples, we analyze different data regimes and training strategies that combine synthetic and real data across these paradigms. We conclude that curated synthetic data are best understood as a complement to real observations, and that hybrid pipelines combining controllable supervision with learned appearance are the most promising direction for reliable sim-to-real transfer.

Figures & tables

Figure 1: Workflow for curated synthetic data in task specific multimodal scene understanding. Real world data from different sensing modalities is combined with curated synthetic data that embeds task specific prior knowledge. Models are trained using synthetic and real data, with the real data share varying from 0 to 100% or used for fine tuning. The trained models are then transferred to real applications through sim to real transfer, including surface defect detection and object pose estimation.

Figure 2: Curation of synthetic data utilizing various approaches. Curation embeds task-specific priors through three complementary simulation paradigms. Procedural rendering and physically-based simulation yield annotations that are exact by construction but reproduce only explicitly modelled variations, whereas simulation utilizing generative AI learns realistic texture, contrast and sensor statistics from a small real seed set at the cost of annotation accuracy.

Panel of Figure 3 — Figure 3: Effect of adding real images to procedurally rendered defect data. Detection performance (mAP@50) on the real toy-Ferrari test split [ 14 ] for increasing fractions of real training data. Real-only training (red squares) collapses below a real share of $50\,\%$ , whereas mixing in curated synthetic renderings (blue circles) sustains high performance with only $10\,\%$ real images and already exceeds the full real-only baseline (dash-dotted) at $50\,\%$ . Fine-tuning from synthetic weights (green $\star$ ) is best overall, but its margin over mixed training at $100\,\%$ real data is small ( $+0.016$ YOLO26, $+0.008$ LW-DETR).

	Procedural rendering	Physical process	Generative AI
Specification	Scene program	Process model	Learned from seeds
Annotation	Exact	Exact	Approximate
Realism	Asset-limited	Inherited from input	Learned from domain
Diversity	Sampled parameters	Model expressiveness	Seed set
Limitation	Throughput	Model design	Annotation quality, hallucinations

Table 1: Curation paradigms and their properties.

Panel of Figure 4 — Figure 4: Effect of adding real images to generatively synthesized defect data. Solid curves report AP 50 on the real test split as the real share of a fixed-size training set grows from $0\,\%$ (synthetic only) to $100\,\%$ (real only), while dashed lines mark the best union setting, in which synthetic images are added on top of the complete real set. (Left) On BSData (pitting), AP 50 saturates beyond a real share of $50\,\%$ , so synthetic images can replace about half of the annotation budget at a cost of $\leq 0.02$ AP 50 , and all union lines exceed the real-only endpoint ( $+0.007$ YOLOv26, $+0.027$ LW-DETR, $+0.017$ YOLOX-S). (Right) On the near-saturated MSD scratch subset ( $0.995$ real only) the substitution trend holds, but the union line drops slightly below the baseline ( $-0.005$ ), indicating that generated data helps mainly where the real task is not already solved.

Figure 5: Synthetic autochrome greening defects generated using a signal processing approach [ 19 ]

Figure 6: Autochrome greening defects generated utilizing generative AI [ 20 ]

Figure 7: Evaluation of the YOLOX 6D Pose model trained exclusively on synthetic data. (a) Fraction of test instances whose ADD S error is below the specified threshold, averaged over classes. (b) Inference time divided into the forward pass and non maximum suppression for the NVIDIA RTX 3090 and the Jetson Orin Nano.

Figure 8: Synthetic data for 6DoF strawberry pose estimation. Rendered RGB image with instance mask, 3D cuboid, and 2D bounding box annotations [ 22 ] .

Explore similar work

SynSur: An end-to-end generative pipeline for synthetic industrial surface defect generation and detection
Apr 29, 2026Paul Julius Kühn, Mika Pommeranz, Arjan Kuijper +1Defect Detection Synthetic Data
Fraunhofer IGD, 64283 Darmstadt, Germany
A Real-Calibrated Synthetic-First Data Engine
May 10, 2026Yukang Shen, Zhiguo Liu, Yingshu Li +1Synthetic Data Medical Image Generation
Kennesaw State University, Marietta, GA, USA. · Department of Computer Science, Georgia State University, Atlanta, GA, USA.
From Unity Simulation to Diffusion-Based Augmentation: Quantifying Dataset Balance for Robust Object Detection
Sep 29, 2026Mohamed Benkedadra, Aissa Saoudi, Maxime Gloesener +2Object Detection Generative Adversarial Network
ILIA, Université de Mons Mons, Belgium · Université Polytechnique Hauts-de-France Valenciennes, France · DeepILIA, ILIA, Université de Mons Mons, Belgium +1

Apr 29, 2026, cs.CVJ/K move · Enter open · S save

SynSur: An end-to-end generative pipeline for synthetic industrial surface defect generation and detection

Paul Julius Kühn, Mika Pommeranz, Arjan Kuijper, Saptarshi Neil Sinha

Fraunhofer IGD, 64283 Darmstadt, Germany

Defect Detection Synthetic Data Defect Synagent Data Generation Pipelines

Open paper AskPDF arXiv

The bottleneck in learning-based industrial defect detection is often limited not by model capacity, but by the scarcity of labeled defect data: defects are rare, annotations are expensive, and collecting balanced training sets is slow. We present an end-to-end pipeline for synthetic defect generation and annotation, combining Vision-Language-Model-based prompts, LoRA-adapted diffusion, mask-guided inpainting, and sample filtering with automatic label derivation, and demonstrates the potential of real data with realistic synthetic samples to overcome data scarcity. The evaluation is conducted on, a challenging dataset of pitting defects on ball screw drives, and then on a subset of the Mobile phone screen surface defect segmentation dataset (MSD) dataset to test cross-domain transfer. Beyond downstream detector performance, we analyze key stages of the pipeline, including prompt construction, LoRA selection, and sample filtering with DreamSim and CLIPScore, to understand which synthetic samples are both realistic and useful. Experiments with YOLOv26, YOLOX, and LW-DETR show that synthetic-only training does not replace real data. When combined with real data, synthetic defects can preserve performance and yield modest gains in selected BSData training regimes. The MSD transfer study shows that the overall pipeline structure carries over to a second industrial inspection domain, while also highlighting the importance of domain-specific adaptation and annotation-quality control. Overall, the paper provides an end-to-end assessment of diffusion-based industrial defect synthesis and shows that its strongest value lies in strengthening scarce real datasets rather than substituting for them.

Abstract

Explore similar work

SynSur: An end-to-end generative pipeline for synthetic industrial surface defect generation and detection

A Real-Calibrated Synthetic-First Data Engine

From Unity Simulation to Diffusion-Based Augmentation: Quantifying Dataset Balance for Robust Object Detection