Current synthetic data pipelines for computer vision generate images without diagnosing what the downstream model actually needs. This open-loop paradigm treats synthetic data as cheap real data, randomly sampling the generator's output space and hoping to cover the model's failure modes. We argue this fundamentally misuses synthetic data's unique property: the controllable, independent variation of scene factors.Drawing on the statistical theory of Design of Experiments (DoE), we propose Synthetic Designed Experiments for Representational Sufficiency (SDRS). SDRS treats the downstream model as a black-box system and the synthetic generator as an experimental apparatus. Using fractional factorial designs, SDRS efficiently audits a model's factor-sensitivity profile via ANOVA decomposition. It classifies failures into two actionable types: Type I gaps (coverage failures on underrepresented factor levels) and Type II gaps (reliance on spurious nuisance dependencies). The audit then prescribes targeted synthetic data to address each gap type. We validate SDRS on three experiments: (1) a controlled diagnostic on dSprites with planted biases, where the audit correctly identifies both gap types and targeted data improves accuracy from 49.9% to 79.0%; (2) a dense segmentation task on procedural scenes, where detecting background-complexity shortcuts and applying targeted data improves mIoU from 0.948 to 0.998; and (3) an entanglement detection experiment showing that the ANOVA audit identifies cross-factor contamination in imperfect generators. Finally, we show that per-factor invariance penalties can transfer sensitivity between factors, identifying an open problem for representation-level correction.
Figures & tables
Before
After
Factor
F
Sig.
F
Sig.
Gap Type
shape
219.7
8.3
(semantic)
scale
5.8
26.3
–
orientation
45.9
20.3
Type I
posX
2.1
1.0
n.s.
Type II
posY
1.5
1.5
–
Table 1: Experiment 1: ANOVA audit results (audit-validation split). Holm-adjusted significance before and after correction.
Method
Accuracy
Data Source
No synthetic data
0.499
–
SDRS (task + invariance)
0.751
SDRS-diagnosed
Targeted Data (task loss only)
0.790
SDRS-diagnosed
Random synthetic †
0.824
Oracle (balanced)
Domain randomization †
0.851
Oracle (balanced)
† Samples from full balanced data; see text.
Table 2: Experiment 1: Held-out classification accuracy. “Targeted Data” uses the same diagnosed samples with task loss only. “Random” and “DR” sample from the full balanced dSprites distribution and are reported as oracle references.
Figure 1: Experiment 1: Controlled diagnostic validation on dSprites. The audit identifies a posX shortcut (Type II) and an orientation coverage gap (Type I). After targeted correction, posX becomes non-significant and orientation sensitivity is reduced by 55.8%, with improved held-out accuracy. The in-figure diagnostic table illustrates the theoretical taxonomy; operational assignment follows the priority rule in Section 3.3 .
Figure 2: Experiment 2: Dense prediction on procedural scenes. SDRS reduces the dominant bg_complex shortcut (89.8 → 9.8), closes the occlusion gap, and improves held-out mIoU from 0.948 to 0.998. Task-loss-only training on the same diagnosed data reaches 0.9995, revealing sensitivity transfer under invariance regularization.
Before
After
Factor
F
Sig.
F
Sig.
Gap
light_dir
5.8
1.8
n.s.
Type II
light_int
10.3
97.6
↑
bg_complex
89.8
9.8
Type II
obj_material
1.5
n.s.
0.1
n.s.
–
cam_angle
2.2
n.s.
134.5
↑
Table 3: Experiment 2: ANOVA audit and segmentation mIoU. Audit computed on disjoint audit-validation data; significance is Holm-adjusted.
Method
mIoU
No synthetic data
0.948
SDRS (task + invariance)
0.998
Targeted Data (task loss only)
0.9995
Random synthetic †
1.000
Domain randomization †
1.000
† Uniformly sampled factor space.
Table 4: Experiment 2: Held-out segmentation mIoU. SDRS improves mIoU from 0.948 to 0.998. Task-loss-only training on the same diagnosed data achieves 0.9995.
Figure 3: Experiment 3: Detecting generator entanglement. Comparing perfect and entangled generators, the audit detects style → size leakage: style rises from 2.4 to 7.1 while size drops from 11.4 to 3.6.
Factor
F (Perfect)
F (Entangled)
ΔF
shape
8.7
4.6
−4.1
color
1.0
0.8
−0.1
size
11.4
3.6
−7.9
style
2.4
7.1
+4.7
position
3.2
3.2
+0.0
Table 5: Experiment 3: Entanglement detection. ANOVA F -statistics for perfect vs. entangled generators (audit-validation split).
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Exp 1
Exp 2
Exp 3
Learning rate
3×10−4
1×10−3
1×10−3
Batch size
256
32
64
Epochs (biased)
15
30
15
Epochs (correction)
15
30
–
λ (Eq. ( 3 ))
0.5
0.5
–
Sensitivity margin
3.0
3.0
–
Appendix
Table S6: Training hyperparameters across experiments.
Experiment 1
Factor
Fbefore
Fafter
Δ%
Note
shape
219.7
8.3
−96.2
semantic
scale
5.8
26.3
+355.9
↑ transfer
orient.
45.9
20.3
−55.8
Type I corrected
posX
2.1
1.0
−50.8
Type II corrected
posY
1.5
1.5
−2.4
unchanged
Appendix
Table S7: Complete ANOVA F -statistics before and after SDRS correction. Factors marked ↑ exhibit sensitivity transfer.
Synthetic data evaluation measures whether generated examples preserve the properties needed for a stated use, including fidelity, privacy, and utility.