Adaptive Adversarial Augmentation for Controllable Face Synthesis
Authors: Saransh Suri, Shivang Agarwal, Mayank Vatsa, Richa Singh
Organizations: Manipal University Jaipur, India · Indian Institute of Technology Jodhpur, India · Birla Institute of Technology and Science Pilani Dubai Campus, Dubai, United Arab Emirates
Synthetic data provides a scalable alternative to real-world datasets for training face recognition models, particularly under challenging conditions such as low resolution, occlusion, and masks. Yet, most approaches lack diversity and fail to generalize effectively. We propose Ensemble Feedback Controllable Synthesis (EFCS), a guided framework that generates diverse and challenging samples while preserving visual realism. EFCS expands distributional variability, often reflected in higher FID and KID scores compared to single-feedback and random synthesis, while maintaining high precision. Recognition models trained on EFCS data consistently outperform baselines across multiple benchmarks, showing improved generalization to real-world scenarios. Furthermore, we introduce an analytically motivated formulation linking perturbation-induced difficulty, sample utility, and performance degradation, offering principled insights into balancing synthetic data complexity for optimal training. Together, these contributions establish EFCS as an effective and analytically grounded approach for bridging the gap between synthetic and real datasets.
Figures & tables
Figure 1: Illustration highlighting the necessity of adaptive synthesis. Controlled perturbations (hard samples) introduce targeted complexity to synthetic data, enhancing model robustness beyond traditional synthesis methods.
Figure 2: Qualitative comparison of synthetic samples: (I) constant degradation, (II) single feedback synthesis [ 17 ] , and (III) proposed EFCS method.
Figure 3: Overview of the proposed EFCS framework showing synthesis, perturbation, and ensemble feedback.
Dataset
Identities
Images/ID
Total Images
CASIA-WebFace (original)
10,575
∼ 47 avg
494,414
Real-only subset
10,575
20
211,500
EFCS Synthetic
10,575
20
211,500
Mixed (Real + EFCS)
10,575
40
423,000
Table 1: Dataset statistics used for EFCS training. Synthetic images are generated per identity using the EFCS synthesis model.
Sample Type
Avg. SER-FIQ
Original
0.7776
Synthesized
0.7338
Single Feedback [ 17 ]
0.7264
EFCS (ours)
0.7277
Table 2: Average SER-FIQ comparison across synthesis strategies (higher is better).
Table 5: Core evaluation on custom test sets (R1/R5 and TAR@{1e-5,1e-4,1e-3}, %).
Figure 4: Effect of synthesis noise and difficulty on training utility and recognition performance. Results averaged across ArcFace, AdaFace, and ElasticFace on CASIA-WebFace. Moderate perturbation improves utility, while excessive noise causes degradation.
Model
Training strategy
M1 (EFCS)
Scratch on EFCS
M2 (Mix-50)
Scratch on EFCS+Real (50%)
M3 (Mix-33)
Scratch on EFCS+Real (33%)
M4 (Mix-25)
Scratch on EFCS+Real (25%)
M5 (Real)
Scratch on Real
M6 (Pretr.+EFCS)
Pretrained → finetune EFCS
Table 6: Training settings of evaluated models.
LFW
CFP-FP
AgeDB
CALFW
CPLFW
CFP-FF
Model
Acc
1e-4
1e-3
Acc
1e-4
1e-3
Acc
1e-4
1e-3
Acc
1e-4
1e-3
Acc
1e-4
1e-3
Acc
1e-4
1e-3
M1
98.03
29.81
89.52
87.10
0.07
0.73
87.95
5.76
27.22
89.90
14.90
50.80
81.97
22.50
26.20
97.49
1.90
19.02
M2
98.68
29.15
92.33
90.57
0.18
1.80
90.70
9.56
33.13
91.28
6.02
51.70
84.65
5.65
37.70
98.51
3.25
32.53
M3
98.65
27.98
90.78
90.27
0.23
2.30
90.73
10.22
38.37
91.40
14.34
57.87
84.55
7.64
31.30
98.41
3.30
33.28
M4
98.83
30.39
94.97
90.21
0.52
5.20
91.00
6.04
43.03
92.03
15.51
54.53
84.20
6.32
29.18
98.29
3.40
33.95
M5
98.88
30.72
94.60
91.33
0.38
3.80
90.98
13.70
48.63
92.08
15.60
62.00
85.03
6.35
39.10
98.41
3.60
36.02
Table 7: Verification performance (%) on six public benchmarks. Metrics: accuracy (Acc) and TAR@FAR.