Do Generative Metrics Predict YOLO Performance? An Evaluation Across Models, Augmentation Ratios, and Dataset Complexity

Vasile Marian; Yong-Bin Kang; Alexander Buddery

cs.CVFeb 20, 2026

Do Generative Metrics Predict YOLO Performance? An Evaluation Across Models, Augmentation Ratios, and Dataset Complexity

Authors: Vasile Marian, Yong-Bin Kang, Alexander Buddery

Organizations: School of Mechanical and Mining Engineering, The University of Queensland, Level 4, Mansergh Shaw Building (45), St Lucia, QLD 4072, Australia · Swinburne University of Technology, John Street, Hawthorn, VIC 3122, Australia

Abstract

Synthetic images are increasingly used to augment object-detection training sets, but reliably evaluating a synthetic dataset before training remains difficult: standard global generative metrics (e.g., FID) often do not predict downstream detection mAP. We present a controlled evaluation of synthetic augmentation for YOLOv11 across three single-class detection regimes -- Traffic Signs (sparse/near-saturated), Cityscapes Pedestrian (dense/occlusion-heavy), and COCO PottedPlant (multi-instance/high-variability). We benchmark six GAN-, diffusion-, and hybrid-based generators over augmentation ratios from 10% to 150% of the real training split, and train YOLOv11 both from scratch and with COCO-pretrained initialization, evaluating on held-out real test splits ([email protected]:0.95). For each dataset-generator-augmentation configuration, we compute pre-training dataset metrics under a matched-size bootstrap protocol, including (i) global feature-space metrics in both Inception-v3 and DINOv2 embeddings and (ii) object-centric distribution distances over bounding-box statistics. Synthetic augmentation yields substantial gains in the more challenging regimes (up to +7.6% and +30.6% relative mAP in Pedestrian and PottedPlant, respectively) but is marginal in Traffic Signs and under pretrained fine-tuning. To separate metric signal from augmentation quantity, we report both raw and augmentation-controlled (residualized) correlations with multiple-testing correction, showing that metric-performance alignment is strongly regime-dependent and that many apparent raw associations weaken after controlling for augmentation level.

Figures & tables

Figure 1: Overview of the four-stage pipeline: dataset curation & baselines; synthetic generation & labeling; YOLO training/evaluation; metric–performance analysis.

Dataset	# imgs	Target class mapping	Final size	Regime tag
Cityscapes Pedestrian	2195	person+rider+sitting $\rightarrow$ pedestrian	$768{\times}768$	dense / occlusion-heavy
Traffic Signs	2568	all sign classes $\rightarrow$ traffic sign	$768{\times}768$	sparse / near-saturated
COCO PottedPlant	2568	potted plant $\rightarrow$ potted plant	$512{\times}512$	multi-instance / diverse

Table 1: Summary of the three evaluation datasets after preprocessing and curation. All experiments use an approximately 70/15/15 train/val/test split; full curation details and additional examples are in Appendix A .

Figure 2: Representative preprocessed samples from the three datasets used in this study: (a) Cityscapes Pedestrian, (b) Traffic Signs, and (c) COCO PottedPlant.

Figure 3: YOLOv11 From-scratch training-time validation mAP@0.50:0.95 vs. augmentation ratio. Each curve corresponds to one generator; the horizontal line indicates the real-only baseline. Curves report the best validation mAP observed during training (from Ultralytics training logs). — Figure 3: YOLOv11 From-scratch training-time validation [email protected]:0.95 vs. augmentation ratio. Each curve corresponds to one generator; the horizontal line indicates the real-only baseline. Curves report the best validation mAP observed during training (from Ultralytics training logs).

Dataset	Regime	Baseline	Best	$\boldsymbol{\Delta}$	$\boldsymbol{\Delta(\%)}$	Best gen. @ Aug
Cityscapes Pedestrian	From-scratch	0.4562	0.4910	+0.0348	+7.6%	DiffusionGAN@75%
COCO PottedPlant	From-scratch	0.1599	0.2088	+0.0489	+30.6%	DiffusionGAN@100%
Traffic Signs	From-scratch	0.9297	0.9434	+0.0137	+1.5%	DiT@25%
Cityscapes Pedestrian	Pretrained	0.6430	0.6516	+0.0086	+1.3%	ADM@25%
COCO PottedPlant	Pretrained	0.4919	0.4980	+0.0061	+1.2%	DiffusionGAN@10%
Traffic Signs	Pretrained	0.9949	0.9949	+0.0000	+0.0%	StyleGAN2-ADA@25%

Table 2: Best held-out test evaluation [email protected]:0.95 (real-only baseline vs. best augmented) for each dataset and initialization regime, including absolute and relative gains. Held-out test results are obtained by running Ultralytics val on the unseen real test split using the best checkpoint from training (selected by validation mAP).

Figure 4: Residual Spearman correlations between synthetic-data metrics and YOLOv11 mAP@0.50:0.95 for the From-Scratch regime, controlling for augmentation ratio. Asterisks denote BH–FDR corrected significance at $q<0.05$ ( Benjamini and Hochberg, 1995 ) . — Figure 4: Residual Spearman correlations between synthetic-data metrics and YOLOv11 [email protected]:0.95 for the From-Scratch regime, controlling for augmentation ratio. Asterisks denote BH–FDR corrected significance at $q<0.05$ ( Benjamini and Hochberg, 1995 ) .

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Additional real samples from Cityscapes Pedestrian . Scenes are dense with frequent occlusion and multiple small objects.

Figure 6: Additional real samples from Traffic Signs . Most images contain a single dominant object with minimal overlap.

Figure 7: Additional real samples from COCO PottedPlant. Images often contain multiple instances with wide scale variation (including many small objects) and occasional overlaps, with highly diverse indoor/outdoor backgrounds.

Dataset	Inst./img	% small	Mean area	Mean IoU
Cityscapes Pedestrian	$5.08\pm 2.80$	67.80%	$0.014\pm 0.008$	$0.041\pm 0.054$
Traffic Signs	$2.95\pm 1.62$	8.91%	$0.032\pm 0.023$	$0.008\pm 0.023$
COCO PottedPlant	$1.80\pm 1.14$	36.79%	$0.113\pm 0.168$	$0.009\pm 0.029$

Appendix

Table 3: Regime statistics on real training splits after preprocessing. “Small” denotes boxes with normalized area

<0.01

(i.e.,

<1\%

of image area). Mean pairwise IoU is computed per image over all box pairs (images with fewer than two boxes contribute IoU

=0

), then averaged over images.

Figure 8: YOLOv11 Pretrained training-time validation mAP@0.50:0.95 vs. augmentation ratio (per-generator curves). Curves report the best validation mAP observed during training (from Ultralytics training logs); the horizontal line indicates the real-only baseline. — Figure 8: YOLOv11 Pretrained training-time validation [email protected]:0.95 vs. augmentation ratio (per-generator curves). Curves report the best validation mAP observed during training (from Ultralytics training logs); the horizontal line indicates the real-only baseline.

Figure 9: YOLOv11 From-scratch held-out test evaluation mAP@0.50:0.95 vs. augmentation ratio (Ultralytics val on the real test split). Each curve corresponds to one generator; the horizontal line indicates the real-only baseline. — Figure 9: YOLOv11 From-scratch held-out test evaluation [email protected]:0.95 vs. augmentation ratio (Ultralytics val on the real test split). Each curve corresponds to one generator; the horizontal line indicates the real-only baseline.

Figure 10: YOLOv11 Pretrained held-out test evaluation mAP@0.50:0.95 vs. augmentation ratio (Ultralytics val on the real test split). Each curve corresponds to one generator; the horizontal line indicates the real-only baseline. — Figure 10: YOLOv11 Pretrained held-out test evaluation [email protected]:0.95 vs. augmentation ratio (Ultralytics val on the real test split). Each curve corresponds to one generator; the horizontal line indicates the real-only baseline.

Data	Init.	Item	Value
COCO	Pretr.	Base mAP	$0.4932\!\pm\!0.0019$
		DiffGAN@10%	$0.4852\!\pm\!0.0133$
		$\Delta$ (Aug $-$ Base)	$-0.0080\!\pm\!0.0151$
		CI 95 on $\Delta$	$[-0.0454,\,0.0294]$
Cityscapes	Pretr.	Base mAP	$0.6352\!\pm\!0.0107$
		ADM@25%	$0.6352\!\pm\!0.0085$

Appendix

Table 4: Seed robustness on held-out [email protected]:0.95 (seeds 0/1/2). Values are mean

\pm

std.

\Delta

is (Aug

-

Base) per-seed, summarized as mean

\pm

std. CI 95 uses Student-

t

(df=2).

Figure 11: Residualized scatter plots ( From-Scratch ): selected object-centric distribution metrics vs. held-out test mAP residuals (augmentation-ratio trend removed). Rows correspond to object-count Wasserstein, complexity Wasserstein, and difficult-object-ratio difference ( raw $=|r_{\text{real}}-r_{\text{syn}}|\geq 0$ ); columns correspond to datasets. Dashed line is an OLS fit in residual space; Spearman $\rho$ is reported per panel. These plots are diagnostic: they visualize outliers and generator-specific clustering that a single correlation coefficient can obscure.

Panel of Figure 12 — Figure 12: From-Scratch: Full-suite Spearman metric–mAP correlations. Left: residual Spearman correlations after controlling for augmentation ratio. Right: raw Spearman correlations (no augmentation control). Stars indicate BH–FDR significance at $q<0.05$ .

Panel of Figure 13 — Figure 13: Pretrained: Full-suite Spearman metric–mAP correlations. Left: residual Spearman correlations after controlling for augmentation ratio. Right: raw Spearman correlations (no augmentation control). Stars indicate BH–FDR significance at $q<0.05$ .

Regime	Dataset	Best metric (dir.)	Avg $\tau$	Top-1	Avg regret
From-scratch	Pedestrian	fls_overfit_inception ( $\downarrow$ )	0.156	0/3	0.0028
From-scratch	PottedPlant	kd_value_inception ( $\downarrow$ )	0.422	3/3	0.0000
From-scratch	Traffic Signs	authpct_inception ( $\uparrow$ )	0.467	1/3	0.0079
Pretrained	Pedestrian	fls_overfit_inception ( $\downarrow$ )	0.200	1/3	0.0045
Pretrained	PottedPlant	fd_dino ( $\downarrow$ )	-0.022	1/3	0.0052
Pretrained	Traffic Signs	fls_overfit_dino ( $\downarrow$ )	0.467	1/3	0.0001

Appendix

Table 6: Decision-oriented screening summary (averaged over budgets 25%, 50%, and 100%). Per dataset/regime we report the single shortlisted metric with the lowest average regret. Arrows indicate the ranking direction used for screening (

\downarrow

= lower is better,

\uparrow

= higher is better).

Synthetic Data Evaluation

Synthetic data evaluation measures whether generated examples preserve the properties needed for a stated use, including fidelity, privacy, and utility.

See all 235 papers

Jun 18, 2026

Training-Free Metrics for Synthetic Object Detection Data: A Proxy for Detector Performance

Synthetic images are increasingly used to augment scarce real data for object detection. However, not all synthetic sets help equally, and the only way to know a set's value is to train a detector on it, which is slow and demands dense annotation. We ask whether a training-free metric can instead rank candidate synthetic training sets by their downstream utility. Existing image-set metrics such as FID, KID, and MMD compare two feature distributions with a single global statistic, which we show is mis-specified for detection-data selection in two ways: it is blind to per-image composition (object count, box scale, class mix), and even at fixed composition its global averaging washes out the appearance differences that separate high-mAP pools from low-mAP ones. We propose Conditional-Composition Domain Match (CCDM), which converts any feature-space distance into a composition-stratified comparison, matching candidate and target within metadata-defined strata without training a detector. On COCO and VisDrone-DET, the best CCDM variant ranks 19 candidate training sets in strong agreement with YOLOv8 mAP (Spearman \r{ho} = 0.97 and 0.96), outperforming FID, KID, and MMD. Furthermore, CCDM holds when reference metadata comes from detector pseudo-labels rather than ground-truth boxes.

Myeongseok Nam, Donghun Yeo, Seungwook Kim

GenGenAI, South Korea

Distribution Matching Object Detection Synthetic Data Evaluation Synthetic Data Selection

Ask PDF

Jun 1, 2026

Train, Test, Re-evaluate: Schedule-Sensitive Evaluation of Generative Data for Hand Detection

Generated (or synthetic) image data is increasingly used to augment or replace real training datasets when target imagery is scarce, expensive, or biased. For hand detection, particularly in occupational safety settings, public datasets mostly contain bare hands. This under-represents the variation in hand appearance introduced by gloves, tattoos, jewelry, and other personal protective equipment, creating a distribution shift that safety-critical applications encounter at deployment. We test whether generative inpainting, editing only the hand region of a real photograph to introduce accessories, can close this shift gap. On a paired dataset of real images and their synthetic counterparts, we train YOLOv8n hand detectors under six training-and-scheduling regimes (Experiments A-F, three random seeds each), evaluate every detector on a real test set and on a real-gloves-only test split, and report the mean average precision (mAP) at two overlap thresholds ([email protected] and [email protected]:0.95) along with paired statistical tests. A two-stage experiment: train on real U synthetic data, then fine-tune the resulting weights on real-only at a lower learning rate, increases [email protected] compared to the real-only baseline model on the standard real test set, and improves the real-gloves out-of-distribution gap. Another three-stage experiment preserves box-tightness best, reaching the highest [email protected]:0.95 of any other experiment in the study. The synthetic-data utility for safety-critical hand detection is determined by the training procedure, and simple multi-stage experiments extract substantial real-deployment benefit from inpainted accessory data.

Atmika Bhardwaj, Silvia Vock, Nico Steckhan

Federal Institute for Occupational Safety and Health (BAuA), Germany

Synthetic-to-Real Domain Adaptation Synthetic Data Augmentation Synthetic Data Generation Object Detection Synthetic Data Evaluation

Ask PDF

Sep 29, 2026

From Unity Simulation to Diffusion-Based Augmentation: Quantifying Dataset Balance for Robust Object Detection

Modern computer vision models achieve high accuracy when trained on large-scale annotated datasets. In critical domains such as construction safety monitoring, data collection is costly, hazardous, and ethically constrained. This paper presents a systematic study comparing two complementary data generation paradigms, (1) Unity Simulation-based rendering and (2) Controllable Diffusion-based generation (CIA), for object detection under real data-scarce conditions. A unified experimental framework enables controlled dataset mixing across real, simulated, and generative sources, while maintaining identical model and training settings. Quantitative evaluation using Precision, Recall, mAP, and custom

Δ

-metrics, reveals that neither simulation nor generative augmentation alone achieves optimal transferability. Unity-only training yields an [email protected] drop of

-50\%

relative to real data, while CIA-only training shows a milder

-16.5\%

degradation. Hybrid compositions significantly improve performance, with the 90% real + 10% Unity configuration achieving the best overall [email protected] of

62.68\%

(

+7.64\%

over baseline), and the 90% real + 10% CIA configuration maximizing precision at

74.45\%

. Results demonstrate that limited synthetic inclusion enhances generalization, while excessive substitution induces domain drift.

Mohamed Benkedadra, Aissa Saoudi, Maxime Gloesener +2

ILIA, Université de Mons Mons, Belgium · Université Polytechnique Hauts-de-France Valenciennes, France · DeepILIA, ILIA, Université de Mons Mons, Belgium +1

Synthetic-to-Real Domain Adaptation Synthetic Data Augmentation Diffusion-Based Image Generation Object Detection

Ask PDF

Abstract

Synthetic Data Evaluation

Explore similar work

Training-Free Metrics for Synthetic Object Detection Data: A Proxy for Detector Performance

Train, Test, Re-evaluate: Schedule-Sensitive Evaluation of Generative Data for Hand Detection

From Unity Simulation to Diffusion-Based Augmentation: Quantifying Dataset Balance for Robust Object Detection