Motion Manipulation via Unsupervised Keypoint Positioning in Face Animation

Hong Li; Boyu Liu; Xuhui Liu; Baochang Zhang

cs.CVMar 4, 2026

Motion Manipulation via Unsupervised Keypoint Positioning in Face Animation

Authors: Hong Li, Boyu Liu, Xuhui Liu, Baochang Zhang

Organizations: BUAA · KAUST

Abstract

Face animation deals with controlling and generating facial features with a wide range of applications. The methods based on unsupervised keypoint positioning can produce realistic and detailed virtual portraits. However, they cannot achieve controllable face generation since the existing keypoint decomposition pipelines fail to fully decouple identity semantics and intertwined motion information (e.g., rotation, translation, and expression). To address these issues, we present a new method, Motion Manipulation via unsupervised keypoint positioning in Face Animation (MMFA). We first introduce self-supervised representation learning to encode and decode expressions in the latent feature space and decouple them from other motion information. Secondly, we propose a new way to compute keypoints aiming to achieve arbitrary motion control. Moreover, we design a variational autoencoder to map expression features to a continuous Gaussian distribution, allowing us for the first time to interpolate facial expressions in an unsupervised framework. We have conducted extensive experiments on publicly available datasets to validate the effectiveness of MMFA, which show that MMFA offers pronounced advantages over prior arts in creating realistic animation and manipulating face motion.

Figures & tables

Figure 1: An example of MMFA. MMFA can realize realistic motion attribute editing while achieving face animation.

Figure 2: Overview of MMFA, where $\copyright$ indicates channel-wise concatenation. Part (a) details the steps by which our model decomposes keypoints. Part (b) describes the expression encoder-decoder structure and self-supervised representation learning. Part (c) shows the structure of the multi-scale generator.

Figure 3: Visual comparison of attribute editing between MMFA and Face vid2vid. The second column displays the semantic images corresponding to the neutral keypoints $N$ defined as keypoints with no rotation, translation, and expression correction. It is important to note that in MMFA, the neutral keypoints have been scaled according to the canonical keypoints, while in Face vid2vid, the canonical keypoints are also the neutral keypoints. Subsequent columns show the semantic images obtained after applying translations $t$ , rotations $R$ , and expression deformations $\delta$ .

Figure 4: Illustration of the latent VAE training.

Figure 5: Visual comparisons with state-of-the-art methods.

Method	Same-identity							Cross-identity
Method	LPIPS $\downarrow$	CSIM $\uparrow$	PSNR $\uparrow$	APD $\downarrow$	AKD $\downarrow$	AED $\downarrow$	FID $\downarrow$	CSIM $\uparrow$	APD $\downarrow$	AED $\downarrow$	FID $\downarrow$
FOMM Siarohin et al. (2019b)	0.108	0.965	23.340	0.010	1.300	0.0223	22.755	0.903	0.027	0.0768	105.216
MRAA Siarohin et al. (2021)	0.101	0.970	24.641	0.011	1.209	0.0218	21.397	0.901	0.026	0.0766	87.276
Face-vid2vid Wang et al. (2021b)	0.107	0.972	22.620	0.014	1.486	0.0245	20.092	0.927	0.033	0.0764	86.054
DaGAN Hong et al. (2022)	0.094	0.974	24.168	0.011	1.198	0.0210	14.463	0.907	0.023	0.0770	86.012
LIA Wang et al. (2022)	0.106	0.967	23.670	0.012	1.391	0.0226	23.992	0.936	0.141	0.0774	83.065

Table 1: Quantitative comparison of face animation on VoxCeleb Nagrani et al. (2017) .

Figure 6: Comparison with DPE Pang et al. (2023) for independent pose and expression editing.

Figure 7: Visual comparisons with state-of-the-art methods.

Figure 8: Results of interpolation in the VAE latent space. The last seven columns represent the face of $S$ with the pose of $D$ , the generated results with the VAE interpolation ( $d=z_{D}-z_{S}$ ), and the face of $S$ with the pose and expression of $D$ , respectively.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Same-identity							Cross-identity
Method	LPIPS $\downarrow$	CSIM $\uparrow$	PSNR $\uparrow$	APD $\downarrow$	AKD $\downarrow$	AED $\downarrow$	FID $\downarrow$	CSIM $\uparrow$	APD $\downarrow$	AED $\downarrow$	FID $\downarrow$
Base model	0.110	0.971	22.694	0.020	1.621	0.0250	16.781	0.925	0.041	0.0764	88.680
w/ $\mathcal{L}_{Exp}$	0.111	0.971	22.390	0.021	1.684	0.0256	14.681	0.930	0.045	0.0760	84.108
w/ $\mathcal{L}_{Exp}\&\mathcal{L}_{C}$	0.112	0.970	22.480	0.022	1.742	0.0256	16.232	0.932	0.047	0.0760	79.960
w/ $\mathcal{L}_{Exp},\mathcal{L}_{C}\&\mathcal{L}_{M}$	0.106	0.974	22.898	0.017	1.476	0.0241	13.265	0.925	0.042	0.0759	77.445

Appendix

Table 2: Quantitative results of ablation study on VoxCeleb Nagrani et al. (2017) .

Figure 10: Visualization comparison of our MMFA with other state-of-the-art methods.

Figure 11: More visualization results of motion attribute editing.

Figure 12: Visualization results of active editing of yaw angle.

Figure 13: Visualization results of active editing of pitch angle.

Figure 14: Visualization results of active editing of roll angle.

Figure 15: Results of interpolation in the VAE latent space. The last seven columns represent the face of $S$ with the pose of $D$ , the generated results with the VAE interpolation ( $d=z_{D}-z_{S}$ ), and the face of $S$ with the pose and expression of $D$ , respectively.

Facial Expression Synthesis

See all 28 papers

SuperFace: Preference-Aligned Facial Expression Estimation Beyond Pseudo Supervision
May 7, 2026Zejian Kang, Xuanyang Xu, Wentao Yang +6Facial Expression Recognition Preference Learning
Zhejiang University · The Chinese University of Hong Kong, Shenzhen · Westlake University +3
KM-Speaker: Keypoint-Based Style Control for High-Quality Speech-Driven 3D Facial Animation and Dialogue Localization
Jun 26, 2026Arthur Josi, Emeline Got, Abdallah Dib +2Audio-Driven Facial Animation Conditional Generative Modeling
Ecole de Technologie Sup´erieure · Ubisoft LaForge
Instant Expressive Gaussian Head Avatars at Over 100 FPS
Dec 18, 2025Kaiwen Jiang, Xueting Li, Seonwook Park +33D Gaussian Splatting Human Avatar Animation
University of California, San Diego, United States · NVIDIA, United States

May 7, 2026J/K move · Enter open · S save

SuperFace: Preference-Aligned Facial Expression Estimation Beyond Pseudo Supervision

Zejian Kang, Xuanyang Xu, Wentao Yang, Kai Zheng+5

Zhejiang University · The Chinese University of Hong Kong, Shenzhen · Westlake University+3

Facial Expression Recognition Preference Learning

Open paper AskPDF arXiv

Accurate facial estimation is crucial for realistic digital human animation, and ARKit blendshape coefficients offer an interpretable representation by mapping facial motions to semantic animation controls. However, learning high-quality ARKit coefficient prediction remains limited by the absence of reliable ground-truth supervision. Existing methods typically rely on capture software such as Live Link Face to provide pseudo labels, which may contain noisy activations, biased coefficient magnitudes, and missing or inaccurate facial actions. Consequently, models trained with supervised learning tend to reproduce imperfect pseudo labels rather than optimize for perceptual expression fidelity. In this paper, we propose SuperFace, a preference-driven framework that moves ARKit facial expression estimation from pseudo-label imitation toward human-aligned perceptual optimization. Instead of treating software-estimated coefficients as fixed ground truth, SuperFace uses them only as an initialization and further improves coefficient prediction through human preference feedback on rendered facial expressions. By aligning the model with perceptual judgments rather than numerical pseudo labels, SuperFace enables more visually faithful and expressive facial animation. Experiments show that SuperFace improves expression fidelity over Live Link Face supervision, demonstrating the effectiveness of preference-driven optimization for semantic facial action prediction.

Abstract

Facial Expression Synthesis

Explore similar work

SuperFace: Preference-Aligned Facial Expression Estimation Beyond Pseudo Supervision

KM-Speaker: Keypoint-Based Style Control for High-Quality Speech-Driven 3D Facial Animation and Dialogue Localization

Instant Expressive Gaussian Head Avatars at Over 100 FPS