Diffusion-Based Makeup Transfer with Facial Region-Aware Makeup Features

Zheng Gao; Debin Meng; Yunqi Miao; Zhensong Zhang; Songcen Xu; Ioannis Patras; Jifei Song

cs.CVMar 20, 2026

Diffusion-Based Makeup Transfer with Facial Region-Aware Makeup Features

Authors: Zheng Gao, Debin Meng, Yunqi Miao, Zhensong Zhang, Songcen Xu, Ioannis Patras, Jifei Song

Organizations: Queen Mary University of London · Huawei London Research Center

Abstract

Current diffusion-based makeup transfer methods commonly use the makeup information encoded by off-the-shelf foundation models (e.g., CLIP) as condition to preserve the makeup style of reference image in the generation. Although effective, these works mainly have two limitations: (1) foundation models pre-trained for generic tasks struggle to capture makeup styles; (2) the makeup features of reference image are injected to the diffusion denoising model as a whole for global makeup transfer, overlooking the facial region-aware makeup features (i.e., eyes, mouth, etc) and limiting the regional controllability for region-specific makeup transfer. To address these, in this work, we propose Facial Region-Aware Makeup features (FRAM), which has two stages: (1) makeup CLIP fine-tuning; (2) identity and facial region-aware makeup injection. For makeup CLIP fine-tuning, unlike prior works using off-the-shelf CLIP, we synthesize annotated makeup style data using GPT-o3 and text-driven image editing model, and then use the data to train a makeup CLIP encoder through self-supervised and image-text contrastive learning. For identity and facial region-aware makeup injection, we construct before-and-after makeup image pairs from the edited images in stage 1 and then use them to learn to inject identity of source image and makeup of reference image to the diffusion denoising model for makeup transfer. Specifically, we use learnable tokens to query the makeup CLIP encoder to extract facial region-aware makeup features for makeup injection, which is learned via an attention loss to enable regional control. As for identity injection, we use a ControlNet Union to encode source image and its 3D mesh simultaneously. The experimental results verify the superiority of our regional controllability and our makeup transfer performance. Code is available at https://github.com/zaczgao/Facial_Region-Aware_Makeup.

Figures & tables

Figure 1: Applications of our method FRAM . Global makeup transfer copies the makeup style of reference image to source image. Region-specific makeup transfer combines the regional makeup styles (skin, eyes and mouth) of three reference images.

Figure 2: Overview of our FRAM . It has two stages: (1) makeup CLIP fine-tuning; (2) identity and facial region-aware makeup injection. For stage 1, we synthesize annotated makeup style data using GPT-o3 and a SOTA text-driven image editing model ( e.g . , FLUX.1-Kontext-dev [ 17 ] ), and then use it to learn a makeup CLIP vision encoder $E_{\text{mu}}$ . For stage 2, we construct before-and-after makeup image pairs from the edited images in stage 1 and use them to learn to transfer the makeup style of reference image ${\mathbf{x}}^{r}$ to source image ${\mathbf{x}}^{s}$ , i.e . , inject identity and makeup information to the diffusion denoising model $\epsilon_{\theta}$ . For makeup injection, we extract facial region makeup embeddings $\{{\mathbf{f}}_{n}\}^{N}_{n=1}$ by querying our makeup CLIP encoder $E_{\text{mu}}$ and inject them to $\epsilon_{\theta}$ via cross-attention. For identity injection, we adopt a ControlNet Union [ 41 ] to encode pixel-level identity information of source image ${\mathbf{x}}^{s}$ and structure guidance from 3D mesh ${\mathbf{x}}^{m}$ simultaneously.

Figure 3: Affine vs . blend face and non-face region .

Method	MT [ 18 ]					Wild-MT [ 14 ]					CPM-real [ 24 ]
Method	CSD $\uparrow$	ID $\uparrow$	SSIM $\uparrow$	L2-M $\downarrow$	Aes $\uparrow$	CSD $\uparrow$	ID $\uparrow$	SSIM $\uparrow$	L2-M $\downarrow$	Aes $\uparrow$	CSD $\uparrow$	ID $\uparrow$	SSIM $\uparrow$	L2-M $\downarrow$	Aes $\uparrow$
GAN-based
CSD-MT [ 34 ]	0.434	0.585	0.424	0.146	4.50	0.428	0.585	0.430	0.124	4.41	0.418	0.522	0.410	0.128	4.41
Diffusion-based
MAD [ 27 ]	0.328	0.535	0.805	0.003	4.26	0.401	0.541	0.808	0.002	4.18	0.317	0.491	0.788	0.003	4.18
Gorgeous [ 30 ]	0.417	0.652	0.896	0.003	4.70	0.472	0.628	0.887	0.002	4.64	0.399	0.624	0.871	0.002	4.61

Table 1: Quantitative results . The red / blue value indicates the top/second-ranked result, respectively.

Figure 5: Region-specific makeup transfer and qualitative ablation on attention loss . The visualization of the cross-attention maps corresponding to the facial region makeup embeddings is next to the generated image.

Figure 6: Qualitative results for cross-domain faces . Note that the reference in the last row is generated by GPT-Image-1.

Panel of Figure 7 — Figure 7: Qualitative ablation on makeup/identity injection .

CSD-MT	SHMT	Stable-Makeup	Ours
20.5%	5%	16%	58.5%

Table 2: User study . The best selection ratio (

\%

) is reported.

SSL	Text	Acc $\uparrow$	CSD $\uparrow$	ID $\uparrow$	SSIM $\uparrow$	L2-M $\downarrow$	Aes $\uparrow$
✕	✕	61.7	0.461	0.459	0.799	0.003	4.88
✓	✕	80.3	0.506	0.425	0.786	0.003	4.86
✕	✓	86.9	0.508	0.406	0.776	0.003	4.79
✓	✓	88.2	0.528	0.429	0.797	0.003	4.95

Table 3: Ablation on CLIP fine-tuning objectives . SSL is self-supervised contrastive learning. Text is image-text contrastive learning. Acc is K-nearest neighbour (KNN) classification accuracy on the makeup style data, where K is 5. The first row denotes off-the-shelf CLIP [ 25 ] pre-trained for generic tasks.

LoRA	Pixel	3D	CSD $\uparrow$	ID $\uparrow$	SSIM $\uparrow$	L2-M $\downarrow$	Aes $\uparrow$
✓	✓	✕	0.510	0.403	0.775	0.003	4.88
✓	✕	✓	0.587	0.048	0.333	0.003	5.57
✕	✓	✓	0.467	0.652	0.810	0.003	4.55
✓	✓	✓	0.528	0.429	0.797	0.003	4.95

Table 4: Ablation on identity and makeup injection modules . LoRA is the LoRA fine-tuning of denoising model cross-attention layers for fusing the makeup features. Pixel / 3D is the pixel/3D mesh guidance for the identity encoder, respectively.

Diff	Attn	CSD $\uparrow$	ID $\uparrow$	SSIM $\uparrow$	L2-M $\downarrow$	Aes $\uparrow$
✓	✕	0.518	0.416	0.799	0.003	4.90
✓	✓	0.528	0.429	0.797	0.003	4.95

Table 5: Ablation on attention loss . Diff is the diffusion loss. Attn is the attention loss.

Diffusion-Based Image Editing

Diffusion-based image editing changes image content, style, or attributes while aiming to preserve the parts users want to keep.

See all 225 papers

Jul 23, 2026

MagicMakeup: A Region-Controllable Diffusion Transformer for High-Fidelity Makeup-Transfer

Makeup-transfer applies the reference makeup to the source face while preserving the source identity. Despite advances in full-face editing by diffusion-based methods, strong regional controllability, makeup fidelity, and identity preservation remain challenging. The reasons are (i) pixel-to-attention misalignment that causes spillover into non-target areas and weakens regional control; (ii) unclear transfer/preservation concept separation under two-image conditioning, leading to coupling between makeup attributes and identity; and (iii) the lack of a high-resolution dataset that is identity-consistent and region-labeled for fine-grained supervision. In this paper, we propose MagicMakeup, a diffusion transformer-based framework for region-controllable and high-fidelity makeup transfer, built on spatial constraints and concept disentanglement. To enable precise region-specific editing while preserving identity, we propose Token-Aligned Region Gating, which aligns pixel masks with attention and applies region-specific logit gating. To clarify the concepts of transfer and preservation, we further introduce Cross-Modal Perception Guidance, which aligns text and image features to enhance cross-modal concept perception. We also design a pipeline for the generation of 1024 x 1024 data pairs through region-specific makeup removal and establish a unified benchmark in synthetic and real settings. Extensive quantitative and qualitative experiments show that MagicMakeup improves regional controllability, makeup fidelity, and identity preservation, with strong robustness across styles, races, and poses.

Ziyi Wang, Siming Zheng, Yang Yang +5

State Key Lab of CAD&CG, Zhejiang University, China · vivo BlueImage Lab, vivo Mobile Communication Co., Ltd., China

Diffusion Transformer Image Style Transfer Diffusion-Based Image Editing Image Editing

Ask PDF

Jun 18, 2026

MakeupMirror: Improving Facial Attribute Preservation in Diffusion Models for Makeup Transfer

Makeup transfer models enable fun augmented reality (AR) experiences as well as virtual try-on (VTO) for online makeup shopping. While recent state-of-the-art diffusion based solutions such as Stable-Makeup dramatically improve the accuracy and realism of makeup transfer, they still face limitations in identity and skin color preservation, making production-level VTO for makeup shopping unrealistic. In this work, we propose MakeupMirror, a diffusion-based approach to makeup transfer that makes significant progress towards preserving facial features and skin tone. We introduce several technical innovations over Stable-Makeup: (1) integration of facial geometry conditioning with ControlNets to maintain facial fidelity; (2) region-specific makeup transfer control to enable precise makeup application across facial regions such as skin, eyes and lips; (3) skin tone-based makeup transfer modulation that prevent skin tone alteration in cross-subject transfer scenarios; and (4) integration of a Levenberg-Marquardt Langevin sampler to speed up inference while maintaining generation quality. Our experiments on CPM-Real, Makeup Wild, and (herein newly collected, more diverse) MakeupSelfies datasets show that MakeupMirror improves relative facial recognition similarity by +60%, reduces relative skin tone difference by -50% over Stable-Makeup, with a latency of 0.7s, while achieving expert acceptance rate of 94% across core facial identity preservation criteria.

Nefeli Andreou, Angel Martínez-González, Sabine Sternig +3

Amazon

Image Style Transfer Diffusion-Based Image Editing Conditional Diffusion Models

Ask PDF

Jun 30, 2026

Anchoring on Reality: Breaking the Pseudo-Target Ceiling in Makeup Transfer

Makeup transfer applies a reference cosmetic style to a source face while preserving its identity and geometry. However, this task is severely hindered by the lack of real paired training data. Current methods rely on either weak priors or synthetic pseudo-targets from large-scale editing models. These paradigms provide suboptimal guidance, often leading to degraded fine-grained details, synthetic artifacts, and identity drift. To this end, we propose Anchoring on Reality Makeup Transfer (ART), a two-stage framework with a reality-anchored refinement cycle. In Stage I, the model is initialized with pseudo-targets to establish basic semantic alignment and global makeup placement. Crucially, Stage II shifts supervision from pseudo-targets to the real reference, reconstructing it from its bare-skin counterpart through a differentiable cycle that penalizes any omitted detail and overrides synthetic artifacts. Furthermore, we introduce MakeupFaces2K (MF2K), the first 2K-resolution in-the-wild makeup portrait dataset comprising 8,573 images. Extensive experiments demonstrate that our method achieves superior makeup fidelity, strong background stability, and robust identity preservation, especially for complex makeup styles.

Bo Wei, Xianhui Lin, Yi Dong +8

1Harbin Institute of Technology · 2vivo BlueImage Lab, vivo Mobile Communication Co., Ltd · 3Nanjing University

Image Style Transfer Unpaired Image-to-Image Translation

Ask PDF

Abstract

Diffusion-Based Image Editing

Explore similar work

MagicMakeup: A Region-Controllable Diffusion Transformer for High-Fidelity Makeup-Transfer

MakeupMirror: Improving Facial Attribute Preservation in Diffusion Models for Makeup Transfer

Anchoring on Reality: Breaking the Pseudo-Target Ceiling in Makeup Transfer