Do Generative Metrics Predict YOLO Performance? An Evaluation Across Models, Augmentation Ratios, and Dataset Complexity

Vasile Marian; Yong-Bin Kang; Alexander Buddery

cs.CVFeb 20, 2026

Do Generative Metrics Predict YOLO Performance? An Evaluation Across Models, Augmentation Ratios, and Dataset Complexity

Authors: Vasile Marian, Yong-Bin Kang, Alexander Buddery

Organizations: School of Mechanical and Mining Engineering, The University of Queensland, Level 4, Mansergh Shaw Building (45), St Lucia, QLD 4072, Australia · Swinburne University of Technology, John Street, Hawthorn, VIC 3122, Australia

Abstract

Synthetic images are increasingly used to augment object-detection training sets, but reliably evaluating a synthetic dataset before training remains difficult: standard global generative metrics (e.g., FID) often do not predict downstream detection mAP. We present a controlled evaluation of synthetic augmentation for YOLOv11 across three single-class detection regimes -- Traffic Signs (sparse/near-saturated), Cityscapes Pedestrian (dense/occlusion-heavy), and COCO PottedPlant (multi-instance/high-variability). We benchmark six GAN-, diffusion-, and hybrid-based generators over augmentation ratios from 10% to 150% of the real training split, and train YOLOv11 both from scratch and with COCO-pretrained initialization, evaluating on held-out real test splits ([email protected]:0.95). For each dataset-generator-augmentation configuration, we compute pre-training dataset metrics under a matched-size bootstrap protocol, including (i) global feature-space metrics in both Inception-v3 and DINOv2 embeddings and (ii) object-centric distribution distances over bounding-box statistics. Synthetic augmentation yields substantial gains in the more challenging regimes (up to +7.6% and +30.6% relative mAP in Pedestrian and PottedPlant, respectively) but is marginal in Traffic Signs and under pretrained fine-tuning. To separate metric signal from augmentation quantity, we report both raw and augmentation-controlled (residualized) correlations with multiple-testing correction, showing that metric-performance alignment is strongly regime-dependent and that many apparent raw associations weaken after controlling for augmentation level.

Figures & tables

Figure 1: Overview of the four-stage pipeline: dataset curation & baselines; synthetic generation & labeling; YOLO training/evaluation; metric–performance analysis.

Dataset	# imgs	Target class mapping	Final size	Regime tag
Cityscapes Pedestrian	2195	person+rider+sitting $\rightarrow$ pedestrian	$768{\times}768$	dense / occlusion-heavy
Traffic Signs	2568	all sign classes $\rightarrow$ traffic sign	$768{\times}768$	sparse / near-saturated
COCO PottedPlant	2568	potted plant $\rightarrow$ potted plant	$512{\times}512$	multi-instance / diverse

Table 1: Summary of the three evaluation datasets after preprocessing and curation. All experiments use an approximately 70/15/15 train/val/test split; full curation details and additional examples are in Appendix A .

Figure 2: Representative preprocessed samples from the three datasets used in this study: (a) Cityscapes Pedestrian, (b) Traffic Signs, and (c) COCO PottedPlant.

Figure 3: YOLOv11 From-scratch training-time validation mAP@0.50:0.95 vs. augmentation ratio. Each curve corresponds to one generator; the horizontal line indicates the real-only baseline. Curves report the best validation mAP observed during training (from Ultralytics training logs). — Figure 3: YOLOv11 From-scratch training-time validation [email protected]:0.95 vs. augmentation ratio. Each curve corresponds to one generator; the horizontal line indicates the real-only baseline. Curves report the best validation mAP observed during training (from Ultralytics training logs).

Dataset	Regime	Baseline	Best	$\boldsymbol{\Delta}$	$\boldsymbol{\Delta(\%)}$	Best gen. @ Aug
Cityscapes Pedestrian	From-scratch	0.4562	0.4910	+0.0348	+7.6%	DiffusionGAN@75%
COCO PottedPlant	From-scratch	0.1599	0.2088	+0.0489	+30.6%	DiffusionGAN@100%
Traffic Signs	From-scratch	0.9297	0.9434	+0.0137	+1.5%	DiT@25%
Cityscapes Pedestrian	Pretrained	0.6430	0.6516	+0.0086	+1.3%	ADM@25%
COCO PottedPlant	Pretrained	0.4919	0.4980	+0.0061	+1.2%	DiffusionGAN@10%
Traffic Signs	Pretrained	0.9949	0.9949	+0.0000	+0.0%	StyleGAN2-ADA@25%

Table 2: Best held-out test evaluation [email protected]:0.95 (real-only baseline vs. best augmented) for each dataset and initialization regime, including absolute and relative gains. Held-out test results are obtained by running Ultralytics val on the unseen real test split using the best checkpoint from training (selected by validation mAP).

Figure 4: Residual Spearman correlations between synthetic-data metrics and YOLOv11 mAP@0.50:0.95 for the From-Scratch regime, controlling for augmentation ratio. Asterisks denote BH–FDR corrected significance at $q<0.05$ ( Benjamini and Hochberg, 1995 ) . — Figure 4: Residual Spearman correlations between synthetic-data metrics and YOLOv11 [email protected]:0.95 for the From-Scratch regime, controlling for augmentation ratio. Asterisks denote BH–FDR corrected significance at $q<0.05$ ( Benjamini and Hochberg, 1995 ) .

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Additional real samples from Cityscapes Pedestrian . Scenes are dense with frequent occlusion and multiple small objects.

Figure 6: Additional real samples from Traffic Signs . Most images contain a single dominant object with minimal overlap.

Figure 7: Additional real samples from COCO PottedPlant. Images often contain multiple instances with wide scale variation (including many small objects) and occasional overlaps, with highly diverse indoor/outdoor backgrounds.

Dataset	Inst./img	% small	Mean area	Mean IoU
Cityscapes Pedestrian	$5.08\pm 2.80$	67.80%	$0.014\pm 0.008$	$0.041\pm 0.054$
Traffic Signs	$2.95\pm 1.62$	8.91%	$0.032\pm 0.023$	$0.008\pm 0.023$
COCO PottedPlant	$1.80\pm 1.14$	36.79%	$0.113\pm 0.168$	$0.009\pm 0.029$

Appendix

Table 3: Regime statistics on real training splits after preprocessing. “Small” denotes boxes with normalized area

<0.01

(i.e.,

<1\%

of image area). Mean pairwise IoU is computed per image over all box pairs (images with fewer than two boxes contribute IoU

=0

), then averaged over images.

Figure 8: YOLOv11 Pretrained training-time validation mAP@0.50:0.95 vs. augmentation ratio (per-generator curves). Curves report the best validation mAP observed during training (from Ultralytics training logs); the horizontal line indicates the real-only baseline. — Figure 8: YOLOv11 Pretrained training-time validation [email protected]:0.95 vs. augmentation ratio (per-generator curves). Curves report the best validation mAP observed during training (from Ultralytics training logs); the horizontal line indicates the real-only baseline.

Figure 9: YOLOv11 From-scratch held-out test evaluation mAP@0.50:0.95 vs. augmentation ratio (Ultralytics val on the real test split). Each curve corresponds to one generator; the horizontal line indicates the real-only baseline. — Figure 9: YOLOv11 From-scratch held-out test evaluation [email protected]:0.95 vs. augmentation ratio (Ultralytics val on the real test split). Each curve corresponds to one generator; the horizontal line indicates the real-only baseline.

Figure 10: YOLOv11 Pretrained held-out test evaluation mAP@0.50:0.95 vs. augmentation ratio (Ultralytics val on the real test split). Each curve corresponds to one generator; the horizontal line indicates the real-only baseline. — Figure 10: YOLOv11 Pretrained held-out test evaluation [email protected]:0.95 vs. augmentation ratio (Ultralytics val on the real test split). Each curve corresponds to one generator; the horizontal line indicates the real-only baseline.

Data	Init.	Item	Value
COCO	Pretr.	Base mAP	$0.4932\!\pm\!0.0019$
		DiffGAN@10%	$0.4852\!\pm\!0.0133$
		$\Delta$ (Aug $-$ Base)	$-0.0080\!\pm\!0.0151$
		CI 95 on $\Delta$	$[-0.0454,\,0.0294]$
Cityscapes	Pretr.	Base mAP	$0.6352\!\pm\!0.0107$
		ADM@25%	$0.6352\!\pm\!0.0085$

Appendix

Table 4: Seed robustness on held-out [email protected]:0.95 (seeds 0/1/2). Values are mean

\pm

std.

\Delta

is (Aug

-

Base) per-seed, summarized as mean

\pm

std. CI 95 uses Student-

t

(df=2).

Figure 11: Residualized scatter plots ( From-Scratch ): selected object-centric distribution metrics vs. held-out test mAP residuals (augmentation-ratio trend removed). Rows correspond to object-count Wasserstein, complexity Wasserstein, and difficult-object-ratio difference ( raw $=|r_{\text{real}}-r_{\text{syn}}|\geq 0$ ); columns correspond to datasets. Dashed line is an OLS fit in residual space; Spearman $\rho$ is reported per panel. These plots are diagnostic: they visualize outliers and generator-specific clustering that a single correlation coefficient can obscure.

Panel of Figure 12 — Figure 12: From-Scratch: Full-suite Spearman metric–mAP correlations. Left: residual Spearman correlations after controlling for augmentation ratio. Right: raw Spearman correlations (no augmentation control). Stars indicate BH–FDR significance at $q<0.05$ .

Panel of Figure 13 — Figure 13: Pretrained: Full-suite Spearman metric–mAP correlations. Left: residual Spearman correlations after controlling for augmentation ratio. Right: raw Spearman correlations (no augmentation control). Stars indicate BH–FDR significance at $q<0.05$ .

Regime	Dataset	Best metric (dir.)	Avg $\tau$	Top-1	Avg regret
From-scratch	Pedestrian	fls_overfit_inception ( $\downarrow$ )	0.156	0/3	0.0028
From-scratch	PottedPlant	kd_value_inception ( $\downarrow$ )	0.422	3/3	0.0000
From-scratch	Traffic Signs	authpct_inception ( $\uparrow$ )	0.467	1/3	0.0079
Pretrained	Pedestrian	fls_overfit_inception ( $\downarrow$ )	0.200	1/3	0.0045
Pretrained	PottedPlant	fd_dino ( $\downarrow$ )	-0.022	1/3	0.0052
Pretrained	Traffic Signs	fls_overfit_dino ( $\downarrow$ )	0.467	1/3	0.0001

Appendix

Table 6: Decision-oriented screening summary (averaged over budgets 25%, 50%, and 100%). Per dataset/regime we report the single shortlisted metric with the lowest average regret. Arrows indicate the ranking direction used for screening (

\downarrow

= lower is better,

\uparrow

= higher is better).

Synthetic Data Evaluation

Synthetic data evaluation measures whether generated examples preserve the properties needed for a stated use, including fidelity, privacy, and utility.

See all 235 papers

Training-Free Metrics for Synthetic Object Detection Data: A Proxy for Detector Performance
Jun 18, 2026Myeongseok Nam, Donghun Yeo, Seungwook KimDistribution Matching Object Detection
GenGenAI, South Korea
Train, Test, Re-evaluate: Schedule-Sensitive Evaluation of Generative Data for Hand Detection
Jun 1, 2026Atmika Bhardwaj, Silvia Vock, Nico SteckhanSynthetic-to-Real Domain Adaptation Synthetic Data Augmentation
Federal Institute for Occupational Safety and Health (BAuA), Germany
From Unity Simulation to Diffusion-Based Augmentation: Quantifying Dataset Balance for Robust Object Detection
Sep 29, 2026Mohamed Benkedadra, Aissa Saoudi, Maxime Gloesener +2Synthetic-to-Real Domain Adaptation Synthetic Data Augmentation
ILIA, Université de Mons Mons, Belgium · Université Polytechnique Hauts-de-France Valenciennes, France · DeepILIA, ILIA, Université de Mons Mons, Belgium +1

Jun 18, 2026J/K move · Enter open · S save

Training-Free Metrics for Synthetic Object Detection Data: A Proxy for Detector Performance

Myeongseok Nam, Donghun Yeo, Seungwook Kim

GenGenAI, South Korea

Distribution Matching Object Detection Synthetic Data Evaluation Synthetic Data Selection

Open paper AskPDF arXiv

Synthetic images are increasingly used to augment scarce real data for object detection. However, not all synthetic sets help equally, and the only way to know a set's value is to train a detector on it, which is slow and demands dense annotation. We ask whether a training-free metric can instead rank candidate synthetic training sets by their downstream utility. Existing image-set metrics such as FID, KID, and MMD compare two feature distributions with a single global statistic, which we show is mis-specified for detection-data selection in two ways: it is blind to per-image composition (object count, box scale, class mix), and even at fixed composition its global averaging washes out the appearance differences that separate high-mAP pools from low-mAP ones. We propose Conditional-Composition Domain Match (CCDM), which converts any feature-space distance into a composition-stratified comparison, matching candidate and target within metadata-defined strata without training a detector. On COCO and VisDrone-DET, the best CCDM variant ranks 19 candidate training sets in strong agreement with YOLOv8 mAP (Spearman \r{ho} = 0.97 and 0.96), outperforming FID, KID, and MMD. Furthermore, CCDM holds when reference metadata comes from detector pseudo-labels rather than ground-truth boxes.

Abstract

Synthetic Data Evaluation

Explore similar work

Training-Free Metrics for Synthetic Object Detection Data: A Proxy for Detector Performance

Train, Test, Re-evaluate: Schedule-Sensitive Evaluation of Generative Data for Hand Detection

From Unity Simulation to Diffusion-Based Augmentation: Quantifying Dataset Balance for Robust Object Detection