cs.CVMar 30, 2026

Synthetic Designed Experiments for Diagnosing Vision Model Failure

Authors: Krisanu Sarkar

Organizations: Indian Institute of Technology Bombay, Mumbai, India

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Current synthetic data pipelines for computer vision generate images without diagnosing what the downstream model actually needs. This open-loop paradigm treats synthetic data as cheap real data, randomly sampling the generator's output space and hoping to cover the model's failure modes. We argue this fundamentally misuses synthetic data's unique property: the controllable, independent variation of scene factors.Drawing on the statistical theory of Design of Experiments (DoE), we propose Synthetic Designed Experiments for Representational Sufficiency (SDRS). SDRS treats the downstream model as a black-box system and the synthetic generator as an experimental apparatus. Using fractional factorial designs, SDRS efficiently audits a model's factor-sensitivity profile via ANOVA decomposition. It classifies failures into two actionable types: Type I gaps (coverage failures on underrepresented factor levels) and Type II gaps (reliance on spurious nuisance dependencies). The audit then prescribes targeted synthetic data to address each gap type. We validate SDRS on three experiments: (1) a controlled diagnostic on dSprites with planted biases, where the audit correctly identifies both gap types and targeted data improves accuracy from 49.9% to 79.0%; (2) a dense segmentation task on procedural scenes, where detecting background-complexity shortcuts and applying targeted data improves mIoU from 0.948 to 0.998; and (3) an entanglement detection experiment showing that the ANOVA audit identifies cross-factor contamination in imperfect generators. Finally, we show that per-factor invariance penalties can transfer sensitivity between factors, identifying an open problem for representation-level correction.

Figures & tables

 Before After 
FactorFFSig. FFSig.Gap Type
shape219.7 8.3(semantic)
scale5.8 26.3–
orientation45.9 20.3Type I
posX2.1 1.0n.s.Type II
posY1.5  1.5 –
Table 1: Experiment 1: ANOVA audit results (audit-validation split). Holm-adjusted significance before and after correction.
MethodAccuracyData Source
No synthetic data0.499–
SDRS (task + invariance)0.751SDRS-diagnosed
Targeted Data (task loss only)0.790SDRS-diagnosed
Random synthetic †0.824Oracle (balanced)
Domain randomization †0.851Oracle (balanced)
† Samples from full balanced data; see text.
Table 2: Experiment 1: Held-out classification accuracy. “Targeted Data” uses the same diagnosed samples with task loss only. “Random” and “DR” sample from the full balanced dSprites distribution and are reported as oracle references.
Figure 1: Experiment 1: Controlled diagnostic validation on dSprites. The audit identifies a posX shortcut (Type II) and an orientation coverage gap (Type I). After targeted correction, posX becomes non-significant and orientation sensitivity is reduced by 55.8%, with improved held-out accuracy. The in-figure diagnostic table illustrates the theoretical taxonomy; operational assignment follows the priority rule in Section 3.3 .
Figure 2: Experiment 2: Dense prediction on procedural scenes. SDRS reduces the dominant bg_complex shortcut (89.8 →\rightarrow 9.8), closes the occlusion gap, and improves held-out mIoU from 0.948 to 0.998. Task-loss-only training on the same diagnosed data reaches 0.9995, revealing sensitivity transfer under invariance regularization.
 Before After 
FactorFFSig. FFSig.Gap
light_dir5.8 1.8n.s.Type II
light_int10.3 97.6↑\uparrow
bg_complex89.8 9.8Type II
obj_material1.5n.s. 0.1n.s.–
cam_angle2.2n.s. 134.5↑\uparrow
Table 3: Experiment 2: ANOVA audit and segmentation mIoU. Audit computed on disjoint audit-validation data; significance is Holm-adjusted.
MethodmIoU
No synthetic data0.948
SDRS (task + invariance)0.998
Targeted Data (task loss only)0.9995
Random synthetic †1.000
Domain randomization †1.000
† Uniformly sampled factor space.
Table 4: Experiment 2: Held-out segmentation mIoU. SDRS improves mIoU from 0.948 to 0.998. Task-loss-only training on the same diagnosed data achieves 0.9995.
Figure 3: Experiment 3: Detecting generator entanglement. Comparing perfect and entangled generators, the audit detects style →\rightarrow size leakage: style rises from 2.4 to 7.1 while size drops from 11.4 to 3.6.
FactorFF (Perfect)FF (Entangled)ΔF\Delta F
shape8.74.6−4.1-4.1
color1.00.8−0.1-0.1
size11.43.6−7.9-7.9
style2.47.1+4.7+4.7
position3.23.2+0.0\phantom{+}0.0
Table 5: Experiment 3: Entanglement detection. ANOVA FF -statistics for perfect vs. entangled generators (audit-validation split).
Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

 Exp 1Exp 2Exp 3
Learning rate3×10−43\times 10^{-4}1×10−31\times 10^{-3}1×10−31\times 10^{-3}
Batch size2563264
Epochs (biased)153015
Epochs (correction)1530–
λ\lambda (Eq. ( 3 ))0.50.5–
Sensitivity margin3.03.0–
Appendix
Table S6: Training hyperparameters across experiments.
Experiment 1
FactorFbeforeF_{\text{before}}FafterF_{\text{after}}Δ%\Delta\%Note
shape219.78.3−96.2-96.2semantic
scale5.826.3+355.9+355.9↑\uparrow transfer
orient.45.920.3−55.8-55.8Type I corrected
posX2.11.0−50.8-50.8Type II corrected
posY1.51.5−2.4-2.4unchanged
Appendix
Table S7: Complete ANOVA FF -statistics before and after SDRS correction. Factors marked ↑\uparrow exhibit sensitivity transfer.
More in this topic

Synthetic Data Evaluation

Synthetic data evaluation measures whether generated examples preserve the properties needed for a stated use, including fidelity, privacy, and utility.

See all 235 papers

Explore similar work

CardsList
Jul 2, 2026

Diagnosing Aerial-View Object Detectors with Foundational Image Generative Models

Recent advances in large-scale image generative models enable photorealistic scene synthesis with controllable attributes. Beyond data augmentation, their potential as diagnostic tools for trained vision systems remains unexplored in the aerial and remote sensing domains. We introduce a synthetic diagnostic framework for aerial-view vehicle detection that combines text-guided generation, attribute-controlled editing, and automated attribute verification to construct a controllable synthetic testbed. This enables fine-grained evaluation of pretrained detectors under diverse scene types and environmental conditions that are difficult to isolate in real datasets. Across three detection architectures and three real aerial datasets, synthetic scene-wise performance trends closely match real-world weaknesses. Guided by these diagnostics, targeted supplementation with small real datasets from the identified weak categories yields improvements of up to 13% AP50 while requiring substantially fewer additional samples than non-targeted augmentation. Our results show that controlled synthetic probing can predict real-domain performance gaps and guide efficient data collection. The proposed diagnostic framework is modular and can incorporate alternative generative or vision-language models as capabilities evolve. Our code and datasets are available here: https://humansensinglab.github.io/AVODDiag/

Stanislav Panev, Minhyek Jeon, Vaishnavi Khindkar +5

Carnegie Mellon University, USA · DEVCOM Army Research Laboratory, USA · Florida State University, USA

Synthetic Data AugmentationSynthetic Data GenerationAerial Object Detection
AskPDF
Sep 29, 2026

Curating Synthetic Data for Task-Specific Visual Perception

Synthetic data are most valuable where general-purpose datasets cannot provide the domain-specific priors a task requires, and where manual annotation is expensive, imprecise, or infeasible. In this article we argue that the central question for specialized vision systems is not how to generate more data, but which data to generate. We therefore discuss curated synthetic data, whose scene content, appearance variations, sensing characteristics, and annotations are deliberately designed around a given task. We examine three complementary curation paradigms. Procedural rendering offers explicit control over scene parameters and the annotations follow by construction. Physically-based simulation encodes the mechanism behind an observed effect and yields exactly aligned supervision pairs. Generative AI learns sensor-specific appearance from small real seed sets and attains plausible realism, though it remains prone to hallucination and to inaccurate annotation. These paradigms are illustrated with examples from industrial surface defect detection, restoration of degraded digitized autochrome plates, and 6DoF pose estimation from RGB and event data. Using these examples, we analyze different data regimes and training strategies that combine synthetic and real data across these paradigms. We conclude that curated synthetic data are best understood as a complement to real observations, and that hybrid pipelines combining controllable supervision with learned appearance are the most promising direction for reliable sim-to-real transfer.

Saptarshi Neil Sinha, Paul Julius Kühn, Michael Weinmann

Fraunhofer IGD, 64283 Darmstadt, Germany · Delft University of Technology, 2628 CD Delft, Netherlands

Synthetic-to-Real Domain AdaptationSynthetic Data GenerationTraining Data Curation
AskPDF
Feb 23, 2026

When Pretty Isn't Useful: Investigating Why Modern Text-to-Image Models Fail as Reliable Training Data Generators

Recent text-to-image (T2I) diffusion models produce visually stunning images and demonstrate excellent prompt following. But do they perform well as synthetic vision data generators? In this work, we revisit the promise of synthetic data as a scalable substitute for real training sets and uncover a surprising performance regression. We generate large-scale synthetic datasets using state-of-the-art T2I models released between 2022 and 2025, train standard classifiers solely on this synthetic data, and evaluate them on real test data. Despite observable advances in visual fidelity and prompt adherence, classification accuracy on real test data consistently declines with newer T2I models as training data generators. Our analysis reveals a hidden trend: These models collapse to a narrow, aesthetic-centric distribution that undermines diversity and real data distribution coverage. Overall, our findings challenge a growing assumption in vision research, namely that progress in generative realism implies progress in data realism. We thus highlight an urgent need to rethink the capabilities of modern T2I models as reliable training data generators.

Krzysztof Adamkiewicz, Brian Bernhard Moser, Stanislav Frolov +3

1RPTU University Kaiserslautern-Landau, Kaiserslautern, Germany · 2German Research Center for Artificial Intelligence (DFKI), Kaiserslautern, Germany

Synthetic-to-Real Domain AdaptationDistribution ShiftSynthetic Data GenerationT2I Diffusion ModelsSynthetic Data Evaluation
AskPDF