From Unity Simulation to Diffusion-Based Augmentation: Quantifying Dataset Balance for Robust Object Detection

Mohamed Benkedadra; Aissa Saoudi; Maxime Gloesener; Sidi Ahmed Mahmoudi; Matei Mancas

cs.CVSep 29, 2026

From Unity Simulation to Diffusion-Based Augmentation: Quantifying Dataset Balance for Robust Object Detection

Authors: Mohamed Benkedadra, Aissa Saoudi, Maxime Gloesener, Sidi Ahmed Mahmoudi, Matei Mancas

Organizations: ILIA, Université de Mons Mons, Belgium · Université Polytechnique Hauts-de-France Valenciennes, France · DeepILIA, ILIA, Université de Mons Mons, Belgium · ISIA Lab, Université de Mons Mons, Belgium

Abstract

Modern computer vision models achieve high accuracy when trained on large-scale annotated datasets. In critical domains such as construction safety monitoring, data collection is costly, hazardous, and ethically constrained. This paper presents a systematic study comparing two complementary data generation paradigms, (1) Unity Simulation-based rendering and (2) Controllable Diffusion-based generation (CIA), for object detection under real data-scarce conditions. A unified experimental framework enables controlled dataset mixing across real, simulated, and generative sources, while maintaining identical model and training settings. Quantitative evaluation using Precision, Recall, mAP, and custom $Δ$ -metrics, reveals that neither simulation nor generative augmentation alone achieves optimal transferability. Unity-only training yields an mAP@0.5 drop of $-50\%$ relative to real data, while CIA-only training shows a milder $-16.5\%$ degradation. Hybrid compositions significantly improve performance, with the 90% real + 10% Unity configuration achieving the best overall mAP@0.5 of $62.68\%$ ( $+7.64\%$ over baseline), and the 90% real + 10% CIA configuration maximizing precision at $74.45\%$ . Results demonstrate that limited synthetic inclusion enhances generalization, while excessive substitution induces domain drift.

Figures & tables

Figure 1 . Overview of the proposed hybrid data pipeline. CIA generates photorealistic augmentations from real images using Diffusion, while Unity3D renders synthetic scenes with automatic annotations. The fused dataset $\mathcal{D}_{\text{mix}}$ balances realism and diversity, reducing the sim-to-real gap, and improving model robustness.

Figure 2 . Representative samples from the three dataset sources used in this study. Top: real construction-site images from MOCS. Middle: CIA-generated variants using diffusion-based controllable augmentation. Bottom: Unity-rendered synthetic scenes produced through domain randomization.

Group	Real	CIA	Unity	Precision	Recall	mAP@0.5	mAP@0.5:0.95	Fitness
Baselines
Real only	$100$	$0$	$0$	0.7266	0.4828	0.5504	0.3873	0.4036
CIA only	$0$	$100$	$0$	$0.5923$	$0.3566$	$0.3851$	$0.2551$	$0.2681$
Unity only	$0$	$0$	$100$	$0.3004$	$0.0558$	$0.0447$	$0.0260$	$0.0279$
Real–CIA Mixes
	$90$	$10$	$0$	0.7445	0.5063	0.5792	0.4076	0.4248

Table 1 . Comprehensive performance comparison across all dataset composition experiments. Each configuration reports Precision, Recall, mAP@0.5, mAP@0.5:0.95, and Fitness, evaluated on a held-out real test set. Groups correspond to different mixing regimes between Real, Unity-simulated, and CIA-generated data. Green-highlighted row indicates the best overall result, while Blue-highlighted rows indicate the best results within each group. Purple-bordered cells mark the overall best value per metric, across all configurations.

Panel of Figure 3 — Figure 3 . Non-normalized confusion matrices with a shared color scale. Raw counts expose the dominant error mode (FN→background) and the reduction achieved by small Unity/CIA mixes.

Figure 4 . Comparison of $\Delta_{\text{sim\textrightarrow real}}$ and $\Delta_{\text{gen\textrightarrow real}}$ across increasing synthetic data ratios

Explore similar work

Class-specific diffusion models improve military object detection in a low-data domain
Apr 20, 2026Ella P. Fokkinga, Jan Erik van Woerden, Thijs A. Eker +4Synthetic Image Detection Object Detection
TNO - Intelligent Imaging, Oude Waalsdorperweg 63, the Hague, the Netherlands
Training-Free Metrics for Synthetic Object Detection Data: A Proxy for Detector Performance
Jun 18, 2026Myeongseok Nam, Donghun Yeo, Seungwook KimSynthetic Image Detection Synthetic Training Data
GenGenAI, South Korea
A Real-Calibrated Synthetic-First Data Engine
May 10, 2026Yukang Shen, Zhiguo Liu, Yingshu Li +1Synthetic Data Medical Image Generation
Kennesaw State University, Marietta, GA, USA. · Department of Computer Science, Georgia State University, Atlanta, GA, USA.

Apr 20, 2026, cs.CVJ/K move · Enter open · S save

Class-specific diffusion models improve military object detection in a low-data domain

Ella P. Fokkinga, Jan Erik van Woerden, Thijs A. Eker, Sebastiaan P. Snel+3

TNO - Intelligent Imaging, Oude Waalsdorperweg 63, the Hague, the Netherlands

Synthetic Image Detection Object Detection Detection Performance Synthetic Training Data Diffusion Models Image Generation

Open paper AskPDF arXiv

Diffusion-based image synthesis has emerged as a promising source of synthetic training data for AI-based object detection and classification. In this work, we investigate whether images generated with diffusion can improve military vehicle detection under low-data conditions. We fine-tuned the text-to-image diffusion model FLUX.1 [dev] using LoRA with only 8 or 24 real images per class across 15 vehicle categories, resulting in class-specific diffusion models, which were used to generate new samples from automatically generated text prompts. The same real images were used to fine-tune the RF-DETR detector for a 15-class object detection task. Synthetic datasets generated by the diffusion models were then used to further improve detector performance. Importantly, no additional real data was required, as the generative models leveraged the same limited training samples. FLUX-generated images improved detection performance, particularly in the low-data regime (up to +8.0% mAP

_{50}

with 8 real samples). To address the limited geometric control of text prompt-based diffusion, we additionally generated structurally guided synthetic data using ControlNet with Canny edge-map conditioning, yielding a FLUX-ControlNet (FLUX-CN) dataset with explicit control over viewpoint and pose. Structural guidance further enhanced performance when data is scarce (+4.1% mAP

_{50}

with 8 real samples), but no additional benefit was observed when more real data is available. This study demonstrates that object-specific diffusion models are effective for improving military object detection in a low-data domain, and that structural guidance is most beneficial when real data is highly limited. These results highlight generative image data as an alternative to traditional simulation pipelines for the training of military AI systems.

Abstract

Explore similar work

Class-specific diffusion models improve military object detection in a low-data domain

Training-Free Metrics for Synthetic Object Detection Data: A Proxy for Detector Performance

A Real-Calibrated Synthetic-First Data Engine