Diffusion-Based Makeup Transfer with Facial Region-Aware Makeup Features

Zheng Gao; Debin Meng; Yunqi Miao; Zhensong Zhang; Songcen Xu; Ioannis Patras; Jifei Song

cs.CVMar 20, 2026

Diffusion-Based Makeup Transfer with Facial Region-Aware Makeup Features

Authors: Zheng Gao, Debin Meng, Yunqi Miao, Zhensong Zhang, Songcen Xu, Ioannis Patras, Jifei Song

Organizations: Queen Mary University of London · Huawei London Research Center

Abstract

Current diffusion-based makeup transfer methods commonly use the makeup information encoded by off-the-shelf foundation models (e.g., CLIP) as condition to preserve the makeup style of reference image in the generation. Although effective, these works mainly have two limitations: (1) foundation models pre-trained for generic tasks struggle to capture makeup styles; (2) the makeup features of reference image are injected to the diffusion denoising model as a whole for global makeup transfer, overlooking the facial region-aware makeup features (i.e., eyes, mouth, etc) and limiting the regional controllability for region-specific makeup transfer. To address these, in this work, we propose Facial Region-Aware Makeup features (FRAM), which has two stages: (1) makeup CLIP fine-tuning; (2) identity and facial region-aware makeup injection. For makeup CLIP fine-tuning, unlike prior works using off-the-shelf CLIP, we synthesize annotated makeup style data using GPT-o3 and text-driven image editing model, and then use the data to train a makeup CLIP encoder through self-supervised and image-text contrastive learning. For identity and facial region-aware makeup injection, we construct before-and-after makeup image pairs from the edited images in stage 1 and then use them to learn to inject identity of source image and makeup of reference image to the diffusion denoising model for makeup transfer. Specifically, we use learnable tokens to query the makeup CLIP encoder to extract facial region-aware makeup features for makeup injection, which is learned via an attention loss to enable regional control. As for identity injection, we use a ControlNet Union to encode source image and its 3D mesh simultaneously. The experimental results verify the superiority of our regional controllability and our makeup transfer performance. Code is available at https://github.com/zaczgao/Facial_Region-Aware_Makeup.

Figures & tables

Figure 1: Applications of our method FRAM . Global makeup transfer copies the makeup style of reference image to source image. Region-specific makeup transfer combines the regional makeup styles (skin, eyes and mouth) of three reference images.

Figure 2: Overview of our FRAM . It has two stages: (1) makeup CLIP fine-tuning; (2) identity and facial region-aware makeup injection. For stage 1, we synthesize annotated makeup style data using GPT-o3 and a SOTA text-driven image editing model ( e.g . , FLUX.1-Kontext-dev [ 17 ] ), and then use it to learn a makeup CLIP vision encoder $E_{\text{mu}}$ . For stage 2, we construct before-and-after makeup image pairs from the edited images in stage 1 and use them to learn to transfer the makeup style of reference image ${\mathbf{x}}^{r}$ to source image ${\mathbf{x}}^{s}$ , i.e . , inject identity and makeup information to the diffusion denoising model $\epsilon_{\theta}$ . For makeup injection, we extract facial region makeup embeddings $\{{\mathbf{f}}_{n}\}^{N}_{n=1}$ by querying our makeup CLIP encoder $E_{\text{mu}}$ and inject them to $\epsilon_{\theta}$ via cross-attention. For identity injection, we adopt a ControlNet Union [ 41 ] to encode pixel-level identity information of source image ${\mathbf{x}}^{s}$ and structure guidance from 3D mesh ${\mathbf{x}}^{m}$ simultaneously.

Figure 3: Affine vs . blend face and non-face region .

Method	MT [ 18 ]					Wild-MT [ 14 ]					CPM-real [ 24 ]
Method	CSD $\uparrow$	ID $\uparrow$	SSIM $\uparrow$	L2-M $\downarrow$	Aes $\uparrow$	CSD $\uparrow$	ID $\uparrow$	SSIM $\uparrow$	L2-M $\downarrow$	Aes $\uparrow$	CSD $\uparrow$	ID $\uparrow$	SSIM $\uparrow$	L2-M $\downarrow$	Aes $\uparrow$
GAN-based
CSD-MT [ 34 ]	0.434	0.585	0.424	0.146	4.50	0.428	0.585	0.430	0.124	4.41	0.418	0.522	0.410	0.128	4.41
Diffusion-based
MAD [ 27 ]	0.328	0.535	0.805	0.003	4.26	0.401	0.541	0.808	0.002	4.18	0.317	0.491	0.788	0.003	4.18
Gorgeous [ 30 ]	0.417	0.652	0.896	0.003	4.70	0.472	0.628	0.887	0.002	4.64	0.399	0.624	0.871	0.002	4.61

Table 1: Quantitative results . The red / blue value indicates the top/second-ranked result, respectively.

Figure 5: Region-specific makeup transfer and qualitative ablation on attention loss . The visualization of the cross-attention maps corresponding to the facial region makeup embeddings is next to the generated image.

Figure 6: Qualitative results for cross-domain faces . Note that the reference in the last row is generated by GPT-Image-1.

Panel of Figure 7 — Figure 7: Qualitative ablation on makeup/identity injection .

CSD-MT	SHMT	Stable-Makeup	Ours
20.5%	5%	16%	58.5%

Table 2: User study . The best selection ratio (

\%

) is reported.

SSL	Text	Acc $\uparrow$	CSD $\uparrow$	ID $\uparrow$	SSIM $\uparrow$	L2-M $\downarrow$	Aes $\uparrow$
✕	✕	61.7	0.461	0.459	0.799	0.003	4.88
✓	✕	80.3	0.506	0.425	0.786	0.003	4.86
✕	✓	86.9	0.508	0.406	0.776	0.003	4.79
✓	✓	88.2	0.528	0.429	0.797	0.003	4.95

Table 3: Ablation on CLIP fine-tuning objectives . SSL is self-supervised contrastive learning. Text is image-text contrastive learning. Acc is K-nearest neighbour (KNN) classification accuracy on the makeup style data, where K is 5. The first row denotes off-the-shelf CLIP [ 25 ] pre-trained for generic tasks.

LoRA	Pixel	3D	CSD $\uparrow$	ID $\uparrow$	SSIM $\uparrow$	L2-M $\downarrow$	Aes $\uparrow$
✓	✓	✕	0.510	0.403	0.775	0.003	4.88
✓	✕	✓	0.587	0.048	0.333	0.003	5.57
✕	✓	✓	0.467	0.652	0.810	0.003	4.55
✓	✓	✓	0.528	0.429	0.797	0.003	4.95

Table 4: Ablation on identity and makeup injection modules . LoRA is the LoRA fine-tuning of denoising model cross-attention layers for fusing the makeup features. Pixel / 3D is the pixel/3D mesh guidance for the identity encoder, respectively.

Diff	Attn	CSD $\uparrow$	ID $\uparrow$	SSIM $\uparrow$	L2-M $\downarrow$	Aes $\uparrow$
✓	✕	0.518	0.416	0.799	0.003	4.90
✓	✓	0.528	0.429	0.797	0.003	4.95

Table 5: Ablation on attention loss . Diff is the diffusion loss. Attn is the attention loss.

Diffusion-Based Image Editing

Diffusion-based image editing changes image content, style, or attributes while aiming to preserve the parts users want to keep.

See all 225 papers

MagicMakeup: A Region-Controllable Diffusion Transformer for High-Fidelity Makeup-Transfer
Jul 23, 2026Ziyi Wang, Siming Zheng, Yang Yang +5Diffusion Transformer Image Style Transfer
State Key Lab of CAD&CG, Zhejiang University, China · vivo BlueImage Lab, vivo Mobile Communication Co., Ltd., China
MakeupMirror: Improving Facial Attribute Preservation in Diffusion Models for Makeup Transfer
Jun 18, 2026Nefeli Andreou, Angel Martínez-González, Sabine Sternig +3Image Style Transfer Diffusion-Based Image Editing
Amazon
Anchoring on Reality: Breaking the Pseudo-Target Ceiling in Makeup Transfer
Jun 30, 2026Bo Wei, Xianhui Lin, Yi Dong +8Image Style Transfer Unpaired Image-to-Image Translation
1Harbin Institute of Technology · 2vivo BlueImage Lab, vivo Mobile Communication Co., Ltd · 3Nanjing University

Jul 23, 2026J/K move · Enter open · S save

MagicMakeup: A Region-Controllable Diffusion Transformer for High-Fidelity Makeup-Transfer

Ziyi Wang, Siming Zheng, Yang Yang, Shusong Xu+4

State Key Lab of CAD&CG, Zhejiang University, China · vivo BlueImage Lab, vivo Mobile Communication Co., Ltd., China

Diffusion Transformer Image Style Transfer Diffusion-Based Image Editing Image Editing

Open paper AskPDF arXiv

Makeup-transfer applies the reference makeup to the source face while preserving the source identity. Despite advances in full-face editing by diffusion-based methods, strong regional controllability, makeup fidelity, and identity preservation remain challenging. The reasons are (i) pixel-to-attention misalignment that causes spillover into non-target areas and weakens regional control; (ii) unclear transfer/preservation concept separation under two-image conditioning, leading to coupling between makeup attributes and identity; and (iii) the lack of a high-resolution dataset that is identity-consistent and region-labeled for fine-grained supervision. In this paper, we propose MagicMakeup, a diffusion transformer-based framework for region-controllable and high-fidelity makeup transfer, built on spatial constraints and concept disentanglement. To enable precise region-specific editing while preserving identity, we propose Token-Aligned Region Gating, which aligns pixel masks with attention and applies region-specific logit gating. To clarify the concepts of transfer and preservation, we further introduce Cross-Modal Perception Guidance, which aligns text and image features to enhance cross-modal concept perception. We also design a pipeline for the generation of 1024 x 1024 data pairs through region-specific makeup removal and establish a unified benchmark in synthetic and real settings. Extensive quantitative and qualitative experiments show that MagicMakeup improves regional controllability, makeup fidelity, and identity preservation, with strong robustness across styles, races, and poses.

Abstract

Diffusion-Based Image Editing

Explore similar work

MagicMakeup: A Region-Controllable Diffusion Transformer for High-Fidelity Makeup-Transfer

MakeupMirror: Improving Facial Attribute Preservation in Diffusion Models for Makeup Transfer

Anchoring on Reality: Breaking the Pseudo-Target Ceiling in Makeup Transfer