PixelSmile: Toward Fine-Grained Facial Expression Editing

Jiabin Hua; Hengyuan Xu; Aojie Li; Wei Cheng; Gang Yu; Xingjun Ma; Yu-Gang Jiang

cs.CVMar 26, 2026

PixelSmile: Toward Fine-Grained Facial Expression Editing

Authors: Jiabin Hua, Hengyuan Xu, Aojie Li, Wei Cheng, Gang Yu, Xingjun Ma, Yu-Gang Jiang

Organizations: Fudan University · StepFun

Abstract

Fine-grained facial expression editing has long been limited by intrinsic semantic overlap. To address this, we construct the Flex Facial Expression (FFE) dataset with continuous affective annotations and establish FFE-Bench to evaluate structural confusion, editing accuracy, linear controllability, and the trade-off between expression editing and identity preservation. We propose PixelSmile, a diffusion framework that disentangles expression semantics via fully symmetric joint training. PixelSmile combines intensity supervision with contrastive learning to produce stronger and more distinguishable expressions, achieving precise and stable linear expression control through textual latent interpolation. Extensive experiments demonstrate that PixelSmile achieves superior disentanglement and robust identity preservation, confirming its effectiveness for continuous, controllable, and fine-grained expression editing, while naturally supporting smooth expression blending.

Figures & tables

Figure 2: Observation of Expression Semantic Overlap. Inherent expression overlap causes systematic confusion across human annotators, recognition models, and generative models (top). We resolve this via the FFE dataset (bottom left) and PixelSmile framework (bottom right), utilizing continuous supervision and symmetric training for disentangled editing.

Figure 3: Framework Overview. (1) Inference Stage . We interpolate between the neutral and target expression embeddings in textual latent space using a controllable coefficient $\alpha$ , enabling continuous adjustment of expression intensity. (2) Training Stage . We adopt a joint fully symmetric training framework. Specifically, we sample a source image $P_{\mathrm{src}}$ and a confusing expression pair $(P_{a},P_{b})$ to construct a triplet. We first treat $P_{a}$ as the positive and $P_{b}$ as the negative to compute a joint loss, and then swap their roles to compute it again, yielding a symmetric training objective. The joint loss consists of three components: a Flow-Matching loss for intensity alignment, a contrastive loss for expression separation, and an identity preservation loss to maintain subject consistency.

Figure 4: Quantitative Evaluation of Linear Control Methods . Comparison of the trade-off between ID similarity and expression score across different models. PixelSmile achieves an optimal balance, providing a wider expression manipulation range while preserving identity fidelity.

Table 4

Figure 5: Qualitative Comparison with General Editing Models. PixelSmile produces clearer expression changes while preserving facial identity, whereas existing editing models either weaken expression editing or degrade identity consistency.

Figure 6: Qualitative Comparison with Linear Control Models. PixelSmile achieves smooth and monotonic expression transitions while preserving facial identity, whereas existing control methods either produce unstable responses or sacrifice identity consistency. The figure illustrates two representative expressions: happy (top row) and surprised (bottom row).

Figure 7: Ablation on identity loss. Without ID loss, large expression intensities cause identity drift in hairstyle and skin texture. Our full method preserves identity consistently.

Figure 8: Ablation on symmetric contrastive learning. Both w/o Contrastive Loss and w/o Symmetric Framework suffer from expression confusion, while our full method achieves precise expression disentanglement. The upper three rows show angry and disgust, and the lower three rows show fear and surprised.

	Ablation	mSCR $\downarrow$	ACC-6 $\uparrow$	ACC-12 $\uparrow$	CLS-6 $\uparrow$	CLS-12 $\uparrow$	HES $\uparrow$	ID Sim $\uparrow$
Loss	w/o Contrastive Loss	0.2725	0.6471	0.5889	0.6978	0.5889	0.4500	0.7018
	w/o ID Loss	0.0550	0.8824	0.6500	0.8215	0.6874	0.4451	0.5749
	w/o Sym. Frame.	0.1350	0.7843	0.4700	0.7939	0.6488	0.4253	0.6402
Constraint	w/ Log-Ratio Constraint	0.1750	0.8039	0.5300	0.7917	0.6546	0.4933	0.6943
Constraint	w/ Hinge Constraint	0.0950	0.8824	0.6600	0.7997	0.7228	0.4758	0.6280
Data	MEAD	0.2125	0.7647	0.4700	0.7047	0.6187	0.4235	0.5735

Table 3: Ablation Study . Best, second best, and third best results are indicated by , , and respectively.

Figure 9: Training dynamics of symmetric contrastive learning. The asymmetric variant reduces loss faster in early training but leads to higher structural confusion, while the symmetric framework achieves lower and more stable mSCR.

Figure 10: User study results. We show the trade-off between identity preservation and continuity of editing, annotated by human annotators. The size of the points indicates the HES scores of human annotators.

Figure 11: Additional linear expression editing results. We show the remaining ten expressions across both real and anime domains. The top row shows results on real images, while the bottom row shows results on anime images. Expression intensity increases from left to right for each expression.

Figure 12: Expression Blending Results. Visualizing compositional facial expressions generated by smoothly blending multiple emotional categories in PixelSmile.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 13 — Figure 13: Statistical distributions of annotated data in FFE. The results provide insights into the underlying data characteristics across real-world and anime domains.

Panel of Table 4 — Table 4: Human Dataset Annotation Prompt Template.

Panel of Figure 14 — Figure 14: Visualization of appearance-related textual descriptions in FFE. The visualizations highlight the distribution and diversity of annotations across real-world and anime domains.

Panel of Table 5 — Table 5: Anime Dataset Annotation Prompt Template.

Panel of Table 6 — Table 6: Facial Expression Scoring Prompt Template. The same prompt is applied to both human and anime domains, highlighting the domain-agnostic nature of our scoring pipeline.

Image Editing

Image editing changes selected visual content while preserving the parts of an image that should remain unchanged.

See all 315 papers

FED-Bench: A Cross-Granular Benchmark for Disentangled Evaluation of Facial Expression Editing
Mar 31, 2026Fengjian Xue, Xuecheng Wu, Heli Sun +8Image Editing Image Editing Evaluation
Xi’an Jiaotong University · Northwestern Polytechnical University · National University of Singapore +2
High-Fidelity 3D Facial Avatar Synthesis with Controllable Fine-Grained Expressions
Mar 16, 2026Yikang He, Jichao Zhang, Wei Wang +23D Avatar Generation
Institute of Information Science, Beijing Jiaotong University, Beijing, China. · School of Computer Science, Ocean University of China, Qingdao, China. · Department of Information Engineering and Computer Science (DISI), University of Trento, Italy.
ExpressEdit: Fast Editing of Stylized Facial Expressions with Diffusion Models in Photoshop
Apr 3, 2026Kenan Tang, Jiasheng Guo, Jeffrey Lin +1Image Editing Diffusion-Based Image Editing
University of California, Santa Barbara

Mar 31, 2026J/K move · Enter open · S save

FED-Bench: A Cross-Granular Benchmark for Disentangled Evaluation of Facial Expression Editing

Fengjian Xue, Xuecheng Wu, Heli Sun, Yunyun Shi+7

Xi’an Jiaotong University · Northwestern Polytechnical University · National University of Singapore+2

Image Editing Image Editing Evaluation

Open paper AskPDF arXiv

Facial expression image editing requires fine-grained control to strictly preserve human identity and background while precisely manipulating expression. However, existing editing benchmarks primarily focus on general scenarios, lacking high-quality facial images and corresponding editing instructions. Furthermore, current evaluation metrics exhibit systemic biases in this task, often favoring lazy editing or overfit editing. To bridge these gaps, we propose FED-Bench, a comprehensive benchmark featuring rigorous testing and an accurate evaluation suite. First, we carefully construct a benchmark of 747 triplets through a cascaded and scalable pipeline, each comprising an original image, an editing instruction, and a ground-truth image for precise evaluation. Second, we introduce FED-Score, a cross-granularity evaluation protocol that disentangles assessment into three dimensions: Alignment for verifying instruction following, Fidelity for testing image quality and identity preservation, and Relative Expression Gain for quantifying the magnitude of expression changes, effectively mitigating the aforementioned evaluation biases. Third, we benchmark 18 image editing models, revealing that current approaches struggle to simultaneously achieve high fidelity and accurate expression manipulation, with fine-grained instruction following identified as the primary bottleneck. Finally, leveraging the scalable characteristic of introduced benchmark engine, we provide a 20k+ in-the-wild facial training set and demonstrate its effectiveness by fine-tuning a baseline model that achieves significant performance gains. Our benchmark and related code will be made publicly open soon.

Abstract

Image Editing

Explore similar work

FED-Bench: A Cross-Granular Benchmark for Disentangled Evaluation of Facial Expression Editing

High-Fidelity 3D Facial Avatar Synthesis with Controllable Fine-Grained Expressions

ExpressEdit: Fast Editing of Stylized Facial Expressions with Diffusion Models in Photoshop