cs.CVMar 30, 2026
SaveMMFace-DiT: A Dual-Stream Diffusion Transformer for High-Fidelity Multimodal Face Generation
Organizations: University of North Texas, Denton, TX, USA
Abstract
Recent multimodal face generation models address the spatial control limitations of text-to-image diffusion models by augmenting text-based conditioning with spatial priors such as segmentation masks, sketches, or edge maps. This multimodal fusion enables controllable synthesis aligned with both high-level semantic intent and low-level structural layout. However, most existing approaches typically extend pre-trained text-to-image pipelines by appending auxiliary control modules or stitching together separate uni-modal networks. These ad hoc designs inherit architectural constraints, duplicate parameters, and often fail under conflicting modalities or mismatched latent spaces, limiting their ability to perform synergistic fusion across semantic and spatial domains. We introduce MMFace-DiT, a unified dual-stream diffusion transformer engineered for synergistic multimodal face synthesis. Its core novelty lies in a dual-stream transformer block that processes spatial (mask/sketch) and semantic (text) tokens in parallel, deeply fusing them through a shared Rotary Position-Embedded (RoPE) Attention mechanism. This design prevents modal dominance and ensures strong adherence to both text and structural priors to achieve unprecedented spatial-semantic consistency for controllable face generation. Furthermore, a novel Modality Embedder enables a single cohesive model to dynamically adapt to varying spatial conditions without retraining. MMFace-DiT achieves a 40% improvement in visual fidelity and prompt alignment over six state-of-the-art multimodal face generation models, establishing a flexible new paradigm for end-to-end controllable generative modeling. The code and dataset are available on our project page: https://vcbsl.github.io/MMFace-DiT/
Figures & tables
Figure 2: Overview of the MMFace-DiT Generation Pipeline. Our model operates in a VAE’s latent space. During a forward pass, a noisy latent image is converted into a sequence of image tokens. Concurrently, a text prompt is encoded into text tokens by a CLIP encoder, which also produces pooled embeddings for global conditioning. A global conditioning vector, , is formed by combining embeddings from the timestep, the text caption, and our novel Modality Embedder, which processes a flag indicating the spatial condition type. The image tokens, text tokens, and are then processed by our core transformer block, which predicts either the noise (DDPM) or the velocity (RFM). The final image is produced by unpatchifying the output tokens and decoding them using the VAE.
Figure 3: Architecture of the MMFace-DiT Block. The block processes image and text tokens in parallel, modulated by a global conditioning vector ( ) via AdaLN. A shared RoPE Attention layer acts as the central fusion mechanism for deep cross-modal interaction. Following attention and MLP operations, each stream is processed and controlled by a gated residual connection.
Figure 4: Qualitative comparison of text-and-mask-conditioned face generation. Our MMFace-DiT excels at integrating complex textual attributes (e.g., the fourth row with gold earrings and high bun ) with the mask’s structure, producing photorealistic and coherent outputs where prior methods introduce artifacts or fail to adhere to the prompt.
| Method | FID | LPIPS | SSIM | ACC | mIoU | CLIP | Dist. | LLM Sc. |
| TediGAN [ 39 ] | ||||||||
| ControlNet [ 46 ] | ||||||||
| UAC [ 6 ] | ||||||||
| CD [ 14 ] | ||||||||
| DDGI [ 11 ] | ||||||||
| MM2Latent [ 26 ] |
Table 1: Quantitative results for Text + Mask conditioned face generation. Our MMFace-DiT variants include Ours (D) , trained with diffusion-based DDPM objectives, and Ours (F) , trained using flow-matching objectives. Both substantially outperform all baselines across perceptual quality and text-image alignment metrics. Best results are in bold .
Figure 5: Qualitative comparison of text-and-sketch-conditioned face generation. Our MMFace-DiT excels at balancing the strong structural priors of a sketch with nuanced textual descriptions (e.g., the fifth row with graying hair and dark blue shirt ), producing photorealistic outputs with high textural fidelity while prior methods introduce color artifacts, blur, or semantic inconsistencies.
| Method | FID | LPIPS | SSIM | CLIP | Dist. | LLM Sc. |
| TediGAN [ 39 ] | ||||||
| ControlNet [ 46 ] | ||||||
| UAC [ 6 ] | ||||||
| DDGI [ 11 ] | ||||||
| MM2Latent [ 26 ] | ||||||
| Ours (D) | 0.72 | 31.56 | 0.68 |
Table 2: Quantitative results for Text + Sketch conditioned face generation. Our MMFace-DiT includes Ours (D) (diffusion-based DDPM) and Ours (F) (flow-matching) variants.
Figure 6: Qualitative comparison of VAE backbones integrated into MMFace-DiT. The choice of VAE highlights a clear trade-off between statistical fidelity and perceptual quality. While some VAEs introduce visual artifacts such as oversaturation (SD3) or color desaturation (SDXL), Flux consistently yields the most perceptually faithful outputs with superior color accuracy and natural texture.
| Components | Semantic Metrics | Spatial Metrics | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Model | ME | DS | RoPE | VAE | FID | LPIPS | CLIP | SSIM | ACC | mIoU |
| Model-1 | SD2 | 44.52 | 0.486 | 24.53 | 0.44 | 86.65 | 44.86 | |||
| Model-2 | ✓ | SD2 | 40.49 | 0.366 | 24.31 | 0.46 | 87.91 | 46.34 | ||
| Model-3 | ✓ | ✓ | SD2 | 35.61 | 0.367 | 29.69 | 0.49 | 90.79 | 48.91 | |
| Model-4 | ✓ | ✓ | ✓ | SD2 | 33.77 | 0.326 | 31.42 | 0.50 | 92.29 | 50.05 |
| Model-5 | ✓ | ✓ | ✓ | Flux | 27.95 | 0.340 | 31.69 | 0.51 | 93.95 | 49.16 |
Table 3: Ablation study on core model components with spatial metrics. We incrementally add our innovations, demonstrating the impact of the Modality Embedder (ME), Dual-Stream (DS) design, Rotary Position Embedding (RoPE) Attention, and the final VAE choice. Spatial metrics (SSIM, ACC, mIoU) show simultaneous improvement with semantic metrics, proving mitigation of modality dominance.
| VAE | Dim. | Scaling | FID (M) | LPIPS (M) | FID (S) | LPIPS (S) |
|---|---|---|---|---|---|---|
| SD2 | ||||||
| SDXL | 0.337 | |||||
| Qwen | – | |||||
| SD3 | 20.73 | 17.13 | ||||
| Flux | 0.239 |
Table 4: Ablation study on VAE architectures. We evaluate the reconstruction and generative quality of our model using five different pre-trained VAEs. (M) refers to mask-conditioned and (S) to sketch-conditioned generation.
| Hyperparameter | Value (Stage 1: 256x256) | Value (Stage 2: 512x512 Fine-tuning) |
|---|---|---|
| Model Architecture | ||
| Hidden Size (D) | 1152 | 1152 |
| Depth (Transformer Blocks) | 28 | 28 |
| Attention Heads | 16 | 16 |
| Patch Size | 2 | 2 |
| Input Channels | 32 (16 for latent, 16 for condition) | 32 (16 for latent, 16 for condition) |
Table 1: Detailed hyperparameters for the two-stage progressive training of our MMFace-DiT model. Stage 1 trains the model from scratch at 256x256 resolution, and Stage 2 fine-tunes the resulting checkpoint at 512x512 resolution.
Figure 1: Demonstration of Disentangled Fine-Grained Attribute Control. Our MMFace-DiT exhibits exceptional disentangled control over the synthesis process. Each row is generated from a single, fixed segmentation mask, where we systematically vary a single keyword in the text prompt. The model accurately synthesizes diverse attributes—including color (hats, hair), expression (smiling, sad), gender, and even semantic concepts like background details, showcasing our model’s advanced capability for precise, text-guided semantic generation.
Figure 2: Disentangled Attribute Control via Sketch-Conditioned Generation. MMFace-DiT exhibits fine-grained disentanglement in multimodal synthesis when guided by sketch-based spatial priors. Each row is generated from a single fixed sketch while systematically varying a single textual attribute (e.g., hair color, shirt color, eye color). The model precisely follows the specified text-based edits while preserving identity, expression, and geometric consistency dictated by the sketch. This demonstrates MMFace-DiT’s capability for precise semantic integration with strong geometric priors.
Figure 3: Mask-Conditioned Synthesis with Diffusion and Flow Paradigms. This figure showcases the high-quality performance of our MMFace-DiT model when trained under both Diffusion and Rectified Flow (Flow) objectives. Each example is generated from an identical segmentation mask (far left) and text prompt (far right), demonstrating the model’s robust ability to synthesize diverse and realistic portraits that align with both spatial and semantic guidance. Both training paradigms yield excellent results, successfully interpreting complex attributes like hair style, expression, and accessories. Notably, the Flow-based model often exhibits a particularly refined level of photorealism, producing images with remarkably consistent lighting, skin texture, and fine-grained detail.
Figure 4: Sketch-Conditioned Synthesis with Diffusion and Flow Paradigms. This figure highlights the versatility of the MMFace-DiT architecture in translating artistic sketches into photorealistic faces, comparing results from both the Diffusion and Flow training frameworks. Conditioned on the same input sketch (far left) and text prompt (far right), both models successfully preserve the core identity, pose, and expression of the sketch while integrating the specified textual attributes. This demonstrates the model’s strong multimodal capabilities regardless of the training objective. The Flow-based generations, in particular, show a strong proficiency in maintaining structural fidelity to the sketch, resulting in outputs that seamlessly blend the artistic input with photorealistic rendering.
Figure 5: Qualitative VAE Ablation with Mask-Conditioning. This figure provides a visual comparison of five different VAE backbones integrated into our MMFace-DiT framework, using segmentation masks as spatial guidance. For each row, every model receives the identical text prompt and mask. The results illustrate a clear trade-off between statistical fidelity and perceptual realism. While models like SD3 produce sharp outputs, they often introduce an artificial glossiness and oversaturate skin tones. SD2 and SDXL can lead to desaturated or less vibrant results. In contrast, the Flux VAE consistently delivers the most balanced and photorealistic portraits, excelling in color accuracy, natural skin texture, and fine-detail preservation. These qualitative findings strongly establish Flux as the superior backbone for generating high-fidelity, artifact-free portraits.
Figure 6: Qualitative VAE Ablation with Sketch-Conditioning. This figure showcases the performance of the five VAE backbones when conditioned on artistic sketches. Each model generates an image from the same input sketch and text prompt, testing its ability to preserve geometric structure while synthesizing photorealistic detail. The comparison highlights key differences in semantic adherence and realism. For example, several models fail to accurately render specified features like the striking green eyes in the first row. Although SD3 captures details sharply, its outputs can appear airbrushed. The Flux model demonstrates a superior synthesis capability, faithfully translating the sketch’s identity and expression while accurately integrating nuanced textual details. This visual evidence aligns with our quantitative results, where Flux achieved the best perceptual quality (LPIPS) for sketch-conditioned generation.
Figure 7: Efficacy of Rich Textual Conditioning in Mask-Guided Synthesis. This figure illustrates the substantial qualitative gain achieved through our VLM-powered data enrichment. Column (A) relies on the original, sparse annotations, which frequently result in flat lighting or visual artifacts (e.g., the unnatural skin texture in row 2). Conversely, Column (B) utilizes our comprehensive descriptions on the exact same segmentation masks. The enriched prompts empower the model to generate intricate accessories and textures that were previously absent, such as elegant silver earrings (row 1) or a dark suit (row 4). Furthermore, specific stylistic attributes like red lipstick (row 5) and environmental context like a softly lit indoor setting (row 2) are rendered with high fidelity, demonstrating that detailed semantic guidance is essential for resolving ambiguity in mask-to-image generation.
Figure 8: Efficacy of Rich Textual Conditioning in Sketch-Based Synthesis. This figure evaluates the impact of our VLM-enriched annotations on sketch-to-image generation. Column (A) presents results using the original, brief captions, which typically yield generic attributes and unspecified settings. In contrast, Column (B) utilizes our comprehensive descriptions with the exact same input sketches. The enriched prompts enable the model to render precise semantic details—ranging from specific environmental contexts like a plain beige background (row 2) or lush green backdrop (row 6), to fine-grained facial features such as striking blue eyes (row 6) and material textures like a straw hat (row 2). Crucially, this semantic enrichment significantly improves photorealism and scene composition while strictly adhering to the structural constraints provided by the input sketch.
More in this topic
Multimodal Generation
Multimodal generation creates coordinated text, images, audio, video, 3D, and other outputs from mixed inputs while preserving meaning and consistency.
See all 243 papers