Organizations: Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China · School of Computer Science and Technology, Beijing Jiaotong University, Beijing 100044, China · School of Computer Science and Engineering, South China University of Technology, Guangzhou 510006, China · Intelligent Game and Decision Lab, Beijing 100071, China
Although diffusion-based methods have substantially improved the controllability of multimodal face synthesis, their semantic alignment remains suboptimal because most existing approaches rely on implicit latent-space objectives to model the relationship between denoising variables and multimodal conditions. Such implicit modeling is often insufficient to enforce precise correspondence between synthesized faces and conditional inputs, especially under long-tailed semantic mask distributions where rare attributes receive weak optimization signals. To address these limitations, we propose EC\textsuperscript{2}Face, a multimodal face synthesis framework that improves semantic alignment through explicit semantic supervision and distribution-aware optimization. First, we introduce Explicit Conditional Consistency Guidance (ECCG), which imposes direct consistency supervision in pixel space by decoding an approximate reverse estimate of the clean latent and explicitly aligning the synthesized image with textual descriptions and semantic masks. A temporal dynamic modulation function is further designed to adapt the supervision strength according to the timestep-dependent reliability of reverse estimation. Second, we propose Long-Tail Adaptive Flow Matching (LAFM), which reweights spatial optimization signals based on semantic attribute frequency, with normalized weights to maintain numerical stability during training. Importantly, all additional modules are used only during training and introduce no extra inference overhead. Extensive experiments show that EC\textsuperscript{2}Face consistently outperforms competitive baselines in both generation quality and semantic alignment, achieving a 29.38% improvement in mask accuracy on rare attributes.
Figures & tables
Fig. 1 : Common failure cases of existing multimodal face synthesis methods under long-tailed semantic masks. Rare attributes such as earrings and necklaces are often under-modeled or entirely omitted, even when they are explicitly specified in the mask condition.
Fig. 2 : Overview of EC 2 Face: A training framework devised to enhance precise semantic alignment in multimodal face synthesis. Building upon Latent-Space Multimodal Modeling, this framework incorporates two pivotal improvements—Explicit Conditional Consistency Guidance and Long-Tail Adaptive Flow Matching—which markedly boost the alignment accuracy between synthesized facial images and the input textual descriptions as well as semantic masks.
Fig. 3 : Reliability of one-step reverse estimation decreases as the timestep t increases.
Fig. 4 : Long-tail distribution phenomenon of predefined mask attributes in the MM-CelebA-HQ dataset.
Fig. 5 : Qualitative comparison with state-of-the-art methods under joint text and mask guidance. EC 2 Face generates images with better multimodal consistency, especially for fine-grained attributes such as hats, earrings, and necklaces.
Methods
Multimodal Conditional Consistency
Visual Quality
CLIP-S (%, ↑ )
Mask mAcc (%, ↑ )
LPIPS ( ↓ )
NIQE ( ↓ )
OAA
HFA
NFA
RFA
TediGAN [ 32 ]
23.90
66.58
91.14
69.18
0.61
0.49
5.19
UaC [ 35 ]
25.52
65.37
79.54
68.27
18.19
0.57
6.47
PixelFace+ [ 33 ]
26.16
82.11
93.96
86.80
27.81
0.56
6.24
CoDiffusion [ 54 ]
24.51
61.51
85.34
62.83
5.26
0.58
4.46
TABLE I : Quantitative comparison with state-of-the-art methods on MM-CelebA-HQ. EC 2 Face achieves the best performance in both text consistency and mask consistency, while also delivering competitive image quality.
Methods
Multimodal Conditional Consistency
Visual Quality
CLIP-S (%, ↑ )
Mask mAcc(%, ↑ )
LPIPS ( ↓ )
NIQE ( ↓ )
OAA
HFA
NFA
RFA
TediGAN [ 32 ]
25.03
64.63
91.40
66.25
0.56
0.64
5.28
UaC [ 35 ]
26.97
67.61
82.52
70.23
20.73
0.67
6.39
PixelFace+ [ 33 ]
26.60
83.36
93.72
88.24
30.99
0.67
5.93
CoDiffusion [ 54 ]
23.03
58.35
82.80
59.31
3.22
0.68
6.79
TABLE II : Quantitative results of EC 2 Face and competing methods on MM-FFHQ. The proposed method consistently achieves superior performance, demonstrating strong cross-dataset robustness and generalization ability.
Fig. 6 : Generative diversity of EC 2 Face under fixed text and mask conditions. By varying the random seed, the model produces diverse face images while preserving semantic consistency with the given conditions.
Fig. 7 : EC 2 Face showcases remarkable robustness in uni-modal (mask- or text-driven) facial synthesis.
Fig. 8 : The use of LAFM significantly enhances the generation accuracy of rare attributes.
LAFM
Mask mAcc(%, ↑ )
OAA
HFA
NFA
RFA
✗
86.14
95.13
88.62
52.04
✓
87.64
95.18
89.41
61.12 ( +9.08 )
TABLE III : Ablation study on the effectiveness of LAFM. Introducing LAFM significantly improves mask alignment performance, especially for rare attributes.
ECCG
CLIP-S (%, ↑ )
Mask mAcc(%, ↑ )
Ltext
Lmask
OAA
HFA
NFA
RFA
✗
✗
26.12
87.64
95.18
89.41
61.12
✗
✓
26.26
88.59
96.50
90.14
62.68
✓
✗
27.22
87.45
95.36
89.16
60.60
✓
✓
27.05
88.64
96.56
90.18
62.78
TABLE IV : Ablation study on ECCG. Both Ltext and Lmask improve semantic alignment in their respective modalities, and their joint use simultaneously improves overall multimodal consistency.
β(t)
CLIP-S(%, ↑ )
mAcc(%, ↑ )
LPIPS( ↓ )
NIQE( ↓ )
≡1
28.45
88.04
0.57
3.88
Dynamic
27.05
88.64
0.51
3.55
TABLE V : Ablation study on temporal dynamic modulation. We compare the full model with a variant using a fixed modulation strength, i.e., β(t)≡1 .
α1
CLIP-S(%, ↑ )
LPIPS( ↓ )
26.26
0.504
0.5
27.05
0.510
1.0
27.99
0.526
2.0
28.81
0.545
TABLE VI : Influence of the hyperparameter α1 .
α2
mAcc(%, ↑ )
LPIPS( ↓ )
87.45
0.515
0.5
88.64
0.510
1.0
90.15
0.519
2.0
91.03
0.535
TABLE VII : Influence of the hyperparameter α2 .
Fig. 9 : Visual comparison of generated faces with and without temporal dynamic modulation. Dynamic modulation of β(t) produces more natural and visually realistic results under the same random seed.
Fig. 10 : More visual comparisons with state-of-the-art methods: The facial images synthesized by EC 2 Face significantly outperform competing methods in terms of multimodal consistency. Please zoom in for a better view.
Fig. 11 : More examples of generated diversity: While maintaining consistency with the given conditions (mask and text), EC 2 Face demonstrates high diversity in non-specified attributes such as identity, skin tone, and so on.
Fig. 12 : More examples of unimodal-driven facial synthesis: When confronted with incomplete conditions, such as being provided with only a mask or only text, EC 2 Face is still capable of generating visually plausible facial images while exhibiting good consistency and diversity.
Multimodal generation creates coordinated text, images, audio, video, 3D, and other outputs from mixed inputs while preserving meaning and consistency.
Recent multimodal face generation models address the spatial control limitations of text-to-image diffusion models by augmenting text-based conditioning with spatial priors such as segmentation masks, sketches, or edge maps. This multimodal fusion enables controllable synthesis aligned with both high-level semantic intent and low-level structural layout. However, most existing approaches typically extend pre-trained text-to-image pipelines by appending auxiliary control modules or stitching together separate uni-modal networks. These ad hoc designs inherit architectural constraints, duplicate parameters, and often fail under conflicting modalities or mismatched latent spaces, limiting their ability to perform synergistic fusion across semantic and spatial domains. We introduce MMFace-DiT, a unified dual-stream diffusion transformer engineered for synergistic multimodal face synthesis. Its core novelty lies in a dual-stream transformer block that processes spatial (mask/sketch) and semantic (text) tokens in parallel, deeply fusing them through a shared Rotary Position-Embedded (RoPE) Attention mechanism. This design prevents modal dominance and ensures strong adherence to both text and structural priors to achieve unprecedented spatial-semantic consistency for controllable face generation. Furthermore, a novel Modality Embedder enables a single cohesive model to dynamically adapt to varying spatial conditions without retraining. MMFace-DiT achieves a 40% improvement in visual fidelity and prompt alignment over six state-of-the-art multimodal face generation models, establishing a flexible new paradigm for end-to-end controllable generative modeling. The code and dataset are available on our project page: https://vcbsl.github.io/MMFace-DiT/
Diffusion models have achieved strong results in high-fidelity image synthesis, but their iterative sampling process makes large-scale generation computationally expensive. This limitation is especially relevant when generating synthetic face datasets for face recognition, where a large number of subjects with many samples in different poses, expressions, ages, etc., are required. In this work, we show that identity-conditioned face synthesis can be performed at a substantially lower computational cost by a latent Consistency Model with few iterations, without compromising image quality. For training, we distill knowledge from the foundation Diffusion Model Arc2Face (teacher) by adapting its original text-to-image pipeline to an embedding-to-face setting, replacing textual prompts with ArcFace identity embeddings. Our distilled model (student) generates identity-conditioned face images with an average inference time of 0.4819 seconds per image, compared with 2.102 seconds for Arc2Face, resulting in a 4.36× speed-up. Quantitative results, based on FID scores, show that the distilled model remains competitive with Arc2Face across all evaluation protocols. On 100k generated images, it achieves near-parity on CelebA (13.921 vs. 12.928) and outperforms the teacher on WebFace42M (9.317 vs. 9.802). Further evaluations on Synth-500 and AgeDB show a moderate performance gap for the former but comparable results for the latter. These results indicate that Arc2Face can be accelerated through task-specific latent consistency distillation while preserving high image quality for large-scale synthetic face generation. Our proposal is publicly available at https://github.com/UFPR-IPASP-PR/FaceRec-IdentityConsistency.
Tiago Kienen Chaves, Bernardo Biesseck, David Menotti
Department of Informatics, Federal University of Paraná, Curitiba, Brazil · Federal Institute of Mato Grosso (IFMT), Pontes e Lacerda, MT, Brazil
Strong semantic representations improve the convergence and generation quality of diffusion and flow models. Existing approaches largely rely on external models, which require separate training, operate on misaligned objectives, and exhibit unexpected scaling behavior. We argue that this dependence arises from the model's training objective, which poses a denoising task with little incentive to learn semantic representations. We introduce Self-Flow: a self-supervised flow matching paradigm that integrates representation learning within the generative framework. Our key mechanism, Dual-Timestep Scheduling, applies heterogeneous noise levels across tokens, creating an information asymmetry that forces the model to infer missing information from corrupted inputs. This drives learning strong representations alongside generative capabilities without external supervision. Our method generalizes across modalities and enables multi-modal training while following expected scaling laws, achieving superior image, video, and audio generation.