Semantically-Guided Domain Randomization for Industrial Object Detection in Low-Image-Budget Regimes

Jose Moises Araya-Martinez; Gautham Mohan; Jens Lambrecht

cs.CVSep 22, 2026

Semantically-Guided Domain Randomization for Industrial Object Detection in Low-Image-Budget Regimes

Authors: Jose Moises Araya-Martinez, Gautham Mohan, Jens Lambrecht

Organizations: Electrical Engineering and Computer Science TU Berlin Berlin, Germany · Electrical Engineering University of Stuttgart Stuttgart, Germany · Institute for Cognitive Robotics TU Braunschweig Braunschweig, Germany

Abstract

Retraining visual perception pipelines in High-Mix, Low-Volume (HMLV) automotive manufacturing must be carried out under tight annotation, energy, and time budgets, yet most Synthetic Data Generation (SDG) strategies still operate in the thousands of images. This work evaluates Semantically-Guided Domain Randomization (S-GDR), an annotation-free adaptation pipeline that couples Vision-Language Model (VLM)-based semantic captioning of a small unannotated real reference set with diffusion-based background synthesis (Stable Diffusion XL (SDXL) conditioned by ControlNet and IP-Adapter) and mask-based object composition. On an automotive multi-object detection benchmark and with a fixed budget of 200 synthetic training images, S-GDR reaches mAP50-95 = 0.739 on a real held-out test set, outperforming a domain-randomized render baseline (mAP50-95 = 0.697) as well as brightness filtering, perceptual hashing, CycleGAN style transfer, and unguided diffusion variants sharing the same 200-image budget. These initial observations position S-GDR as a promising annotation- free alternative for extreme data-scarcity regimes.

Figures & tables

Fig. 1 : Comparison of SDG paradigms. DR randomizes simulation parameters in an open loop; DA incorporates domain knowledge via manual tuning; SAL closes the loop with a fixed, non-semantic evaluator on synthetic data; the proposed S-GDR replaces the fixed evaluator with VLM -driven semantic feedback over real reference data

\mathcal{D}_{\text{ref}}

and conditions a generative model on the resulting caption to produce context-aware training images.

Work	Strategy	DA Features	Real Context	Sem. Eval.	Train Set
Tobin et al. [ 5 ]	DR	None	✗	✗	$\geq$ 5k
Tremblay et al. [ 6 ]	DR	Low	✗	✗	$\geq$ 2.5k
Prakash et al. [ 3 ]	DA	Struct.	✗	✗	$\geq$ 1k
Mayershofer et al. [ 9 ]	DA	Low + struct.	✗	✗	$\geq$ 2.5k
SynthRender [ 10 ]	DA	Low + struct.	✗	✗	$\geq$ 400
Zhu et al. [ 11 ]	SAL	Low (synth.)	✗	✗	$\geq$ 4k

TABLE I : Representative SDG approaches for object detection. ✓ = present; ✗ = absent. Adaptation features: low = color/lighting/texture; struct. = geometry/layout; sem. = natural-language semantics.

Panel of Figure 3 — Fig. 2 : Representative samples from the three dataset partitions used in this work: (a) domain-randomized rendered training image ( $\mathcal{D}_{\text{train}}$ ); (b) real test image ( $\mathcal{D}_{\text{test}}$ ), used solely for evaluation; (c) real reference image ( $\mathcal{D}_{\text{ref}}$ ), used as semantic context source for S-GDR augmentation.

Fig. 3 : Sample images produced by the S-GDR pipeline. (a) Original domain-randomized synthetic image. (b) Generatively augmented background. (c) Final training image with semantically contextualized background and original target objects re-inserted via mask-based composition.

Fig. 4 : Per-class AP and overall mAP (

\text{mAP}_{50\text{-}95}

) on the real test set

\mathcal{D}_{\text{test}}

of the automotive benchmark of [ 17 ] , for the six configurations of Section III-C . YOLOv8 was trained on 200 synthetic images per configuration. Bars: per-class AP ; red line: overall

\text{mAP}_{50\text{-}95}

. Scores are single-run values.

Domain Randomization

Domain randomization trains models across deliberately varied simulated or synthetic environments to improve performance in unseen real-world domains.

See all 25 papers

Diffuse the object, keep its label: curating detector training data from a few unlabeled photographs via VLM-built 3D vegetation scenes
Aug 10, 2026Mario Malizia, Marnix Enting, Rob Haelterman +1Unsupervised Domain Adaptation Data Augmentation
Royal Military Academy, Brussels, Belgium · KU Leuven, Leuven, Belgium · Flanders Make, Heverlee, Belgium
Class-specific diffusion models improve military object detection in a low-data domain
Apr 20, 2026Ella P. Fokkinga, Jan Erik van Woerden, Thijs A. Eker +4Synthetic Data Augmentation Diffusion Model Guidance
TNO - Intelligent Imaging, Oude Waalsdorperweg 63, the Hague, the Netherlands
SynSur: An end-to-end generative pipeline for synthetic industrial surface defect generation and detection
Apr 29, 2026Paul Julius Kühn, Mika Pommeranz, Arjan Kuijper +1Synthetic Data Augmentation Industrial Anomaly Detection
Fraunhofer IGD, 64283 Darmstadt, Germany

Aug 10, 2026J/K move · Enter open · S save

Diffuse the object, keep its label: curating detector training data from a few unlabeled photographs via VLM-built 3D vegetation scenes

Mario Malizia, Marnix Enting, Rob Haelterman, Ken Hasselmann

Royal Military Academy, Brussels, Belgium · KU Leuven, Leuven, Belgium · Flanders Make, Heverlee, Belgium

Unsupervised Domain Adaptation Data Augmentation Synthetic Data Generation Diffusion-Based Data Augmentation

Open paper AskPDF arXiv

Labeled images of small objects hidden in vegetation are scarce, and detectors trained on them generalize poorly across sites. Rather than reusing labels collected at another site, we synthesize labeled training images from a handful of unlabeled photographs of the deployment site itself. A vision--language model generates a coarse 3D vegetation scene from one photograph; placing 3D object meshes in the scene yields bounding boxes, segmentation masks, and per-instance occlusion directly from the scene geometry, without manual annotation. A lightweight adapter fine-tuned on the photographs conditions a diffusion pass that re-textures the renders, and a graded mask-lock sets how much diffusion may touch the object itself. In our runs this grade was the most influential curation choice: lightly diffusing the object improves minority-class recall over fully protecting its pixels, while unrestricted diffusion dissolves it. Trained on these images, a standard detector matched or exceeded its counterpart trained on a larger labeled dataset of real images from a different site, consistently across seeds on a humanitarian-demining benchmark; the comparison is thus unsupervised site adaptation from a handful of photographs against conventional cross-site label reuse. In our ablations the gains were largely insensitive to the photograph and crop budgets, and in-domain accuracy did not predict cross-site performance.

Abstract

Domain Randomization

Explore similar work

Diffuse the object, keep its label: curating detector training data from a few unlabeled photographs via VLM-built 3D vegetation scenes

Class-specific diffusion models improve military object detection in a low-data domain

SynSur: An end-to-end generative pipeline for synthetic industrial surface defect generation and detection