Long-Tail Adaptive Flow Matching with Explicit Conditional Consistency Guidance for Precise Multimodal Face Synthesis

Yushe Cao; Xuechao Zou; Xing Xi; Dianxi Shi; Chun Yu; Junliang Xing

cs.CVSep 14, 2026

Long-Tail Adaptive Flow Matching with Explicit Conditional Consistency Guidance for Precise Multimodal Face Synthesis

Authors: Yushe Cao, Xuechao Zou, Xing Xi, Dianxi Shi, Chun Yu, Junliang Xing

Organizations: Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China · School of Computer Science and Technology, Beijing Jiaotong University, Beijing 100044, China · School of Computer Science and Engineering, South China University of Technology, Guangzhou 510006, China · Intelligent Game and Decision Lab, Beijing 100071, China

Abstract

Although diffusion-based methods have substantially improved the controllability of multimodal face synthesis, their semantic alignment remains suboptimal because most existing approaches rely on implicit latent-space objectives to model the relationship between denoising variables and multimodal conditions. Such implicit modeling is often insufficient to enforce precise correspondence between synthesized faces and conditional inputs, especially under long-tailed semantic mask distributions where rare attributes receive weak optimization signals. To address these limitations, we propose EC\textsuperscript{2}Face, a multimodal face synthesis framework that improves semantic alignment through explicit semantic supervision and distribution-aware optimization. First, we introduce Explicit Conditional Consistency Guidance (ECCG), which imposes direct consistency supervision in pixel space by decoding an approximate reverse estimate of the clean latent and explicitly aligning the synthesized image with textual descriptions and semantic masks. A temporal dynamic modulation function is further designed to adapt the supervision strength according to the timestep-dependent reliability of reverse estimation. Second, we propose Long-Tail Adaptive Flow Matching (LAFM), which reweights spatial optimization signals based on semantic attribute frequency, with normalized weights to maintain numerical stability during training. Importantly, all additional modules are used only during training and introduce no extra inference overhead. Extensive experiments show that EC\textsuperscript{2}Face consistently outperforms competitive baselines in both generation quality and semantic alignment, achieving a 29.38% improvement in mask accuracy on rare attributes.

Figures & tables

Fig. 1 : Common failure cases of existing multimodal face synthesis methods under long-tailed semantic masks. Rare attributes such as earrings and necklaces are often under-modeled or entirely omitted, even when they are explicitly specified in the mask condition.

Fig. 2 : Overview of EC 2 Face: A training framework devised to enhance precise semantic alignment in multimodal face synthesis. Building upon Latent-Space Multimodal Modeling, this framework incorporates two pivotal improvements—Explicit Conditional Consistency Guidance and Long-Tail Adaptive Flow Matching—which markedly boost the alignment accuracy between synthesized facial images and the input textual descriptions as well as semantic masks.

Fig. 3 : Reliability of one-step reverse estimation decreases as the timestep $t$ increases.

Fig. 4 : Long-tail distribution phenomenon of predefined mask attributes in the MM-CelebA-HQ dataset.

Fig. 5 : Qualitative comparison with state-of-the-art methods under joint text and mask guidance. EC 2 Face generates images with better multimodal consistency, especially for fine-grained attributes such as hats, earrings, and necklaces.

Methods	Multimodal Conditional Consistency					Visual Quality
	CLIP-S (%, $\uparrow$ )	Mask mAcc (%, $\uparrow$ )				LPIPS ( $\downarrow$ )	NIQE ( $\downarrow$ )
	CLIP-S (%, $\uparrow$ )	OAA	HFA	NFA	RFA	LPIPS ( $\downarrow$ )	NIQE ( $\downarrow$ )
TediGAN [ 32 ]	23.90	66.58	91.14	69.18	0.61	0.49	5.19
UaC [ 35 ]	25.52	65.37	79.54	68.27	18.19	0.57	6.47
PixelFace+ [ 33 ]	26.16	82.11	93.96	86.80	27.81	0.56	6.24
CoDiffusion [ 54 ]	24.51	61.51	85.34	62.83	5.26	0.58	4.46

TABLE I : Quantitative comparison with state-of-the-art methods on MM-CelebA-HQ. EC 2 Face achieves the best performance in both text consistency and mask consistency, while also delivering competitive image quality.

Methods	Multimodal Conditional Consistency					Visual Quality
	CLIP-S (%, $\uparrow$ )	Mask mAcc(%, $\uparrow$ )				LPIPS ( $\downarrow$ )	NIQE ( $\downarrow$ )
	CLIP-S (%, $\uparrow$ )	OAA	HFA	NFA	RFA	LPIPS ( $\downarrow$ )	NIQE ( $\downarrow$ )
TediGAN [ 32 ]	25.03	64.63	91.40	66.25	0.56	0.64	5.28
UaC [ 35 ]	26.97	67.61	82.52	70.23	20.73	0.67	6.39
PixelFace+ [ 33 ]	26.60	83.36	93.72	88.24	30.99	0.67	5.93
CoDiffusion [ 54 ]	23.03	58.35	82.80	59.31	3.22	0.68	6.79

TABLE II : Quantitative results of EC 2 Face and competing methods on MM-FFHQ. The proposed method consistently achieves superior performance, demonstrating strong cross-dataset robustness and generalization ability.

Fig. 6 : Generative diversity of EC 2 Face under fixed text and mask conditions. By varying the random seed, the model produces diverse face images while preserving semantic consistency with the given conditions.

Panel of Figure 9 — Fig. 7 : EC 2 Face showcases remarkable robustness in uni-modal (mask- or text-driven) facial synthesis.

Fig. 8 : The use of LAFM significantly enhances the generation accuracy of rare attributes.

LAFM	Mask mAcc(%, $\uparrow$ )
LAFM	OAA	HFA	NFA	RFA
✗	86.14	95.13	88.62	52.04
✓	87.64	95.18	89.41	61.12 ( +9.08 )

TABLE III : Ablation study on the effectiveness of LAFM. Introducing LAFM significantly improves mask alignment performance, especially for rare attributes.

ECCG		CLIP-S (%, $\uparrow$ )	Mask mAcc(%, $\uparrow$ )
$\mathcal{L}_{\text{text}}$	$\mathcal{L}_{\text{mask}}$	CLIP-S (%, $\uparrow$ )	OAA	HFA	NFA	RFA
✗	✗	26.12	87.64	95.18	89.41	61.12
✗	✓	26.26	88.59	96.50	90.14	62.68
✓	✗	27.22	87.45	95.36	89.16	60.60
✓	✓	27.05	88.64	96.56	90.18	62.78

TABLE IV : Ablation study on ECCG. Both

\mathcal{L}_{\text{text}}

and

\mathcal{L}_{\text{mask}}

improve semantic alignment in their respective modalities, and their joint use simultaneously improves overall multimodal consistency.

$\beta(t)$	CLIP-S(%, $\uparrow$ )	mAcc(%, $\uparrow$ )	LPIPS( $\downarrow$ )	NIQE( $\downarrow$ )
$\equiv 1$	28.45	88.04	0.57	3.88
Dynamic	27.05	88.64	0.51	3.55

TABLE V : Ablation study on temporal dynamic modulation. We compare the full model with a variant using a fixed modulation strength, i.e.,

\beta(t)\equiv 1

$\alpha_{1}$	CLIP-S(%, $\uparrow$ )	LPIPS( $\downarrow$ )
	26.26	0.504
0.5	27.05	0.510
1.0	27.99	0.526
2.0	28.81	0.545

TABLE VI : Influence of the hyperparameter

\alpha_{1}

$\alpha_{2}$	mAcc(%, $\uparrow$ )	LPIPS( $\downarrow$ )
	87.45	0.515
0.5	88.64	0.510
1.0	90.15	0.519
2.0	91.03	0.535

TABLE VII : Influence of the hyperparameter

\alpha_{2}

Fig. 9 : Visual comparison of generated faces with and without temporal dynamic modulation. Dynamic modulation of $\beta(t)$ produces more natural and visually realistic results under the same random seed.

Fig. 10 : More visual comparisons with state-of-the-art methods: The facial images synthesized by EC 2 Face significantly outperform competing methods in terms of multimodal consistency. Please zoom in for a better view.

Fig. 11 : More examples of generated diversity: While maintaining consistency with the given conditions (mask and text), EC 2 Face demonstrates high diversity in non-specified attributes such as identity, skin tone, and so on.

Fig. 12 : More examples of unimodal-driven facial synthesis: When confronted with incomplete conditions, such as being provided with only a mask or only text, EC 2 Face is still capable of generating visually plausible facial images while exhibiting good consistency and diversity.

Multimodal Generation

Multimodal generation creates coordinated text, images, audio, video, 3D, and other outputs from mixed inputs while preserving meaning and consistency.

See all 243 papers

MMFace-DiT: A Dual-Stream Diffusion Transformer for High-Fidelity Multimodal Face Generation
Mar 30, 2026Bharath Krishnamurthy, Ajita RattaniMultimodal Generation Controllable Image Generation
University of North Texas, Denton, TX, USA
Identity-Conditioned Latent Consistency Distillation for Face Synthesis
Aug 31, 2026Tiago Kienen Chaves, Bernardo Biesseck, David MenottiSynthetic Data Augmentation Latent Diffusion Models
Department of Informatics, Federal University of Paraná, Curitiba, Brazil · Federal Institute of Mato Grosso (IFMT), Pontes e Lacerda, MT, Brazil
Self-Supervised Flow Matching for Scalable Multi-Modal Synthesis
Mar 6, 2026Hila Chefer, Patrick Esser, Dominik Lorenz +5Flow-Based Generative Modeling Flow Matching
Black Forest Labs · MIT

Mar 30, 2026J/K move · Enter open · S save

MMFace-DiT: A Dual-Stream Diffusion Transformer for High-Fidelity Multimodal Face Generation

Bharath Krishnamurthy, Ajita Rattani

University of North Texas, Denton, TX, USA

Multimodal Generation Controllable Image Generation Multimodal Diffusion Models Diffusion Transformer Multimodal Fusion

Open paper AskPDF arXiv

Recent multimodal face generation models address the spatial control limitations of text-to-image diffusion models by augmenting text-based conditioning with spatial priors such as segmentation masks, sketches, or edge maps. This multimodal fusion enables controllable synthesis aligned with both high-level semantic intent and low-level structural layout. However, most existing approaches typically extend pre-trained text-to-image pipelines by appending auxiliary control modules or stitching together separate uni-modal networks. These ad hoc designs inherit architectural constraints, duplicate parameters, and often fail under conflicting modalities or mismatched latent spaces, limiting their ability to perform synergistic fusion across semantic and spatial domains. We introduce MMFace-DiT, a unified dual-stream diffusion transformer engineered for synergistic multimodal face synthesis. Its core novelty lies in a dual-stream transformer block that processes spatial (mask/sketch) and semantic (text) tokens in parallel, deeply fusing them through a shared Rotary Position-Embedded (RoPE) Attention mechanism. This design prevents modal dominance and ensures strong adherence to both text and structural priors to achieve unprecedented spatial-semantic consistency for controllable face generation. Furthermore, a novel Modality Embedder enables a single cohesive model to dynamically adapt to varying spatial conditions without retraining. MMFace-DiT achieves a 40% improvement in visual fidelity and prompt alignment over six state-of-the-art multimodal face generation models, establishing a flexible new paradigm for end-to-end controllable generative modeling. The code and dataset are available on our project page: https://vcbsl.github.io/MMFace-DiT/

Abstract

Multimodal Generation

Explore similar work

MMFace-DiT: A Dual-Stream Diffusion Transformer for High-Fidelity Multimodal Face Generation

Identity-Conditioned Latent Consistency Distillation for Face Synthesis

Self-Supervised Flow Matching for Scalable Multi-Modal Synthesis