SynthFaces: Balanced Large Scale Human Dataset

Sergey Karpukhin; Denis Bobkov; Kirill Morozov; Aibek Alanov

cs.CVOct 29, 2026

SynthFaces: Balanced Large Scale Human Dataset

Authors: Sergey Karpukhin, Denis Bobkov, Kirill Morozov, Aibek Alanov

Organizations: FusionBrain Lab, AXXX Moscow, Russia · HSE University Moscow, Russia · MSU Moscow, Russia

Abstract

The pursuit of fair and equitable AI in human-centric tasks is critically hindered by the inherent biases and privacy risks of real-world datasets, which offer imbalanced representations of identity and often contain sensitive personal information. Consequently, state-of-the-art generative models trained on this skewed data exhibit unfair performance, particularly in identity-preserving generation. In this work we propose a pipeline for controllable generation of dataset with diverse faces for generative models training and finetuning. We achieve this by expanding foundational Arc2Face diffusion model with a flow matching generative model in the manifold of identity embeddings. We demonstrate that such approach achieves both goals of generative fidelity and label diversity not present in real human-centric datasets. As a result we collect and further contextualize a highly balanced identity-centric dataset of humans named SynthFaces. By design, our released images and identities are synthetic, reducing direct reliance on real personal images; we empirically check identity separation with a nearest-neighbor non-leakage metric. We purpose it for training and tuning of fair models for various downstream tasks, and showcase its usability in finetuning an existing adapter-based method for personalized generation.

Figures & tables

Type	Dataset	Captions	Balanced labels	ID-grouped	Private	HQ
Raw Images	VGG-Face, WebFace	no	partially	yes	no	partially
HQ Captioned	Cosmic, DeepFashion	yes	no	no	no	yes
Face Recognition	DCFace, HyperFace	no	no	yes	partially	no
	SynthFaces (ours)	yes	yes	yes	yes	yes

Table 1: High level comparison of dataset types, in columns Caption denotes that images are paired with prompts, Balanced labels that dataset is balanced over selected demographic labels, ID-grouped that multiple images per person are provided and HQ suggests that images are high resolution and visually high quality

Figure 1: Distribution of labels across different categories in some of conventional datasets with humans. Dashed lines denotes mean value across labels, color highlights values above mean

Figure 2: Dataset multi-stage collection pipeline.

Figure 3: Left - Conditional identity flow between uniform points on sphere and ArcFace embeddings used for training, constrained to tangent hyperplanes, right - inference scheme together with Arc2Face at stage 2, trained ID-flow samples identity embedding with given control signals and Arc2Face samples image

Category	Method	FID-i $\downarrow$	FID-e $\downarrow$	Prec. $\uparrow$	Rec. $\uparrow$	F1 $\uparrow$	ID-NL $\uparrow$
Naive	uniform	96.53	0.277	0.011	0.95	0.02	–
Naive	*interpolation	30.59	0.035	0.765	0.622	0.68	0.43
	*PCA	51.18	0.048	0.29	0.88	0.44	0.76
Model	StyleGAN	45.8	0.111	0.711	0.173	0.28	–
	Arc2Face	37.3	0.118	0.701	0.185	0.29	–
	SDXL-Photomaker	73.9	0.207	0.73	0.01	0.02	–

Table 2: Baseline comparison of Stages 1 and 2. Lower FID is better and higher Precision/Recall/F1/ID-NL scores are better. ID-NL (ID-NonLeak) measures average nearest-neighbor separation from real training identities.

Figure 4: Distribution of different label categories data generation by ID-Flow. Compared to conventional datasets, our method offers significant data balance for all major characteristics of persons

Method	$KL$
FFHQ	0.449
SHHQ	0.533
EasyPortait	0.508
CosmicHQ-1.0	0.576
HyperFace-10k	0.49
PCA	0.75

Table 3: Label diversity

Figure 5: Diverse samples from ID-Flow. Dataset was sampled with uniform label controls, ensuring greatest variance of identity representations

Figure 6: Data filtering pipeline from Stage 2 to contextualization - in order to minimize amount of corrupted images with faces we 1) remove images in long tails of statistics 2) detect images with artifacts via vision-language model 3) threshold pairwise similarity, ensuring variability of identities in all categories, resulting in the end with approximately 150,000 final images

	F1	Precision	Recall	FID e	FID i	KL-age	KL-race	KL-gender
Before	0.496	0.425	0.596	0.163	65.59	0.000108	0.000042	0.000001
After	0.444	0.359	0.580	0.127	57.66	0.121146	0.006530	0.003619

Table 4: Comparison of metrics before and after filtering Stage 2 images. F1/Precision/Recall/FID and KL divergence same as in previous tables, KL is split for separate categories analysis. Note that metrics are computed with skewed test set of face images

Method	Label KL $\downarrow$	ID-div. $\uparrow$
Flux.2 T2I, unconditional	0.891	0.257
Flux.2 T2I, balanced prompts	0.101	0.201
ID-Flow (ours)	0.037	0.751

Table 5: Text-conditioned Flux.2 sampling versus ID-Flow (

N{=}1000

each). Label KL is the mean KL divergence from a uniform distribution over age, race, and gender; ID-div. is mean nearest-neighbor cosine distance.

	FaceSize	IR	CLIP
Cosmic-HQ	0.22	-0.69	31.20
FluxSynID [ 10 ]	0.61	1.54	34.07
SynthFaces	0.29	1.38	33.04

Table 6: Comparison of HQ dataset metrics with SynthFaces - our dataset offers superior quality over Cosmic-HQ, while allowing richer contextualization without overly large faces

Figure 7: SynthFaces dataset sample examples - after filtration and caption matching identities are contextualized with Flux-based methods, resulting in total 150,000 high-quality contextualized images with approximately 10 images per identity

Figure 8: Face similarity measured across different categories of IpAdapter-FaceID before and after tuning on data generated by ID-flow.

	FairBench		Context
Method	ID	CLIP-T	ID	CLIP-T
SD-1.5 + FaceID	68.63	32.91	28.81	53.56
SD-1.5 + FaceID (+ tuned)	71.31	33.27	28.49	51.56

Table 7: Effect of tuning pipelines with our dataset, reporting average metric values

Model	Acc Disp $\downarrow$	SER max @1% $\downarrow$	SER max @0.1% $\downarrow$
Vec2Face	0.087	1.107	1.057
HyperFace	0.068	1.143	1.101
SynthFaces-500K (ours)	0.043	1.074	1.011

Table 8: BFW fairness aggregate (

\downarrow

fairer). Bold = best.

Model	LFW	CFP-FP	AgeDB-30	CALFW	CPLFW	Mean
Vec2Face-HSFace300K	2.217	1.296	1.560	1.185	1.397	1.531
HyperFace-10K-LDM	2.005	1.096	1.176	1.041	1.085	1.280
SynthFaces-500K (ours)	1.197	1.062	1.054	1.170	1.003	1.097

Table 9: Maximum racial SER at FPR = 0.1% on standard FR benchmarks (

\downarrow

fairer, 1.0 = perfect parity). Bold = best.

\star

= SynthFaces-500K is fairest.

Domain	Model	Avg acc $\uparrow$	$\overline{\Delta}$ $\downarrow$
HQ	Baseline	0.551	0.183
HQ	SynthFaces-HQ (ours)	0.700	0.151
non-HQ	Baseline	0.645	0.176
non-HQ	Vec2Face	0.559	0.263
non-HQ	SynthFaces-nonHQ (ours)	0.797	0.138

Table 10: Domain-aligned VQA results Avg acc = mean over gender/race/age (

\uparrow

);

\overline{\Delta}\,\downarrow

fairer. Bold = best per domain.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Left - high level architecture of the ID-flow model, with conceptual parts corresponding to encoder/decoder architecture, Right - architecture of Modulated ResBlock , which consists of parts inspired by DiT architectures, but without attention mechanisms, orange color highlights trainable parameters

block_num	hidden_dim	c_embed dim	attention	activation	optimizer	schedule
2, 4, 6	128, 256, 512	128, 256, 512,	yes, no	LeakyRelu Sigmoid SoftSign GeLu	SGD, Adam, AdamW	const, linear warmup

Appendix

Table 12: Ablated parameters of model

Figure 10: Trade-off of inference quality of the model embedding samples between fidelity and variability of the samples. We change guidance scale values in range from 1.0 to 8.0

Figure 11: Examples removed by image-statistics filtering. Low entropy corresponds to heavily stylized samples; low saturation identifies grayscale images, while high saturation exposes unnatural colors and lighting artifacts.

Figure 12: Distribution of similarities within data before and after filtering w.r.t. label of data. Each section correspond to distribution with certain race label attached - effect of filtering eliminates tails present for certain labels

Baseline	ID-diversity
Real train data	0.305
HyperFace	0.306
ID-flow samples (before filtering)	0.355
ID-flow samples (after filtering)	0.295

Appendix

Table 13: Embedding diversity measured as mean cosine distance to closest neighbor within data. After filtering Stage 2 faces match in quality filtered label data that was used for training and HyperFace baseline

Figure 13: VLM-filtering output examples of stage 2 - VLM is prompted to visually analyze face in the image, its alignment with labels, detect any artifacts and reason about them

Figure 14: Comparison of various models and checkpoints of Flux-2 and Qwen-IE for contextualization stage. It can be seen that only Flux-2 [dev]- turbo achieves both preservation and fidelity. Images generated with same prompt ”Realistic image of man from image 1, in glasses reading book at home”

Figure 15: Face similarity metric unfairness for different age categories in identity preserving generation - measured preservation across labels is far from balanced

Figure 16: Finetuning results for Photomaker - FairBench benchmark shows average improvement of quality, while also balancing performance between categories

Figure 17: More samples from SynthFaces dataset, including non-square resolution images cropped around faces. Each column corresponds to images of the same synthetic identity.

	LFW		CFP-FP		AgeDB-30		CALFW		CPLFW		Mean
Model	Acc	AUC	Acc	AUC	Acc	AUC	Acc	AUC	Acc	AUC	Acc	AUC
SynthFaces-500K	0.913	0.971	0.727	0.803	0.713	0.791	0.791	0.867	0.679	0.741	0.764	0.835
Vec2Face	0.993	0.999	0.913	0.969	0.945	0.984	0.946	0.973	0.877	0.927	0.935	0.970
HyperFace	0.983	0.998	0.880	0.945	0.855	0.931	0.879	0.944	0.825	0.891	0.884	0.942

Appendix

Table 15: Global accuracy and AUC on standard FR benchmarks (

\uparrow

better). SynthFaces-500K’s lower accuracy reflects its earlier training stage.

Model	LFW	CFP-FP	AgeDB-30	CALFW	CPLFW	Mean
SynthFaces-500K (ours)	1.482	1.133	1.117	1.087	1.040	1.172
Vec2Face-HSFace300K	1.536	1.347	1.670	1.196	1.344	1.419
HyperFace-10K-LDM	2.061	1.155	1.686	1.032	1.137	1.414

Appendix

Table 16: Maximum racial SER at FPR = 1% on standard FR benchmarks (

\downarrow

fairer).

\star

= SynthFaces-500K is fairest.

Synthetic Data Generation

Synthetic data generation creates artificial examples, simulations, and datasets to train, test, or protect machine-learning systems.

See all 629 papers

Identity-Conditioned Latent Consistency Distillation for Face Synthesis
Aug 31, 2026Tiago Kienen Chaves, Bernardo Biesseck, David MenottiSynthetic Data Augmentation Latent Diffusion Models
Department of Informatics, Federal University of Paraná, Curitiba, Brazil · Federal Institute of Mato Grosso (IFMT), Pontes e Lacerda, MT, Brazil
SteerFace: Debiasing Synthetic Face Generation via Adaptive Residue Perturbation
May 29, 2026Yuxi Mi, Qiuyang Yuan, Jianqing Xu +5Image Generation Identity-Preserving Image Generation
Fudan University Shanghai, China · Youtu Lab, Tencent Shanghai, China · WeChat Pay Lab33, Tencent Shenzhen, China
WithEveryone: Unified Planning and Identity Grounding for Group Image Generation
Aug 20, 2026Hengyuan Xu, Qixun Wang, Yiji Cheng +5Subject-Driven Image Generation Image Generation
1Fudan University · 2Hunyuan, Tencent · 3The University of Hong Kong

Aug 31, 2026J/K move · Enter open · S save

Identity-Conditioned Latent Consistency Distillation for Face Synthesis

Tiago Kienen Chaves, Bernardo Biesseck, David Menotti

Department of Informatics, Federal University of Paraná, Curitiba, Brazil · Federal Institute of Mato Grosso (IFMT), Pontes e Lacerda, MT, Brazil

Synthetic Data Augmentation Latent Diffusion Models Diffusion Model Distillation Identity-Preserving Image Generation Diffusion Model Inference Acceleration

Open paper AskPDF arXiv

Diffusion models have achieved strong results in high-fidelity image synthesis, but their iterative sampling process makes large-scale generation computationally expensive. This limitation is especially relevant when generating synthetic face datasets for face recognition, where a large number of subjects with many samples in different poses, expressions, ages, etc., are required. In this work, we show that identity-conditioned face synthesis can be performed at a substantially lower computational cost by a latent Consistency Model with few iterations, without compromising image quality. For training, we distill knowledge from the foundation Diffusion Model Arc2Face (teacher) by adapting its original text-to-image pipeline to an embedding-to-face setting, replacing textual prompts with ArcFace identity embeddings. Our distilled model (student) generates identity-conditioned face images with an average inference time of 0.4819 seconds per image, compared with 2.102 seconds for Arc2Face, resulting in a 4.36

\times

speed-up. Quantitative results, based on FID scores, show that the distilled model remains competitive with Arc2Face across all evaluation protocols. On 100k generated images, it achieves near-parity on CelebA (13.921 vs. 12.928) and outperforms the teacher on WebFace42M (9.317 vs. 9.802). Further evaluations on Synth-500 and AgeDB show a moderate performance gap for the former but comparable results for the latter. These results indicate that Arc2Face can be accelerated through task-specific latent consistency distillation while preserving high image quality for large-scale synthetic face generation. Our proposal is publicly available at https://github.com/UFPR-IPASP-PR/FaceRec-IdentityConsistency.

Abstract

Synthetic Data Generation

Explore similar work

Identity-Conditioned Latent Consistency Distillation for Face Synthesis

SteerFace: Debiasing Synthetic Face Generation via Adaptive Residue Perturbation

WithEveryone: Unified Planning and Identity Grounding for Group Image Generation