GSwap: Realistic Head Swapping with Dynamic Neural Gaussian Field

Jingtao Zhou; Xuan Gao; Dongyu Liu; Junhui Hou; Yudong Guo; Juyong Zhang

cs.CVMar 24, 2026

GSwap: Realistic Head Swapping with Dynamic Neural Gaussian Field

Authors: Jingtao Zhou, Xuan Gao, Dongyu Liu, Junhui Hou, Yudong Guo, Juyong Zhang

Organizations: School of Mathematical Science, University of Science and Technology of China · Department of Computer Science, City University of Hong Kong

Abstract

We present GSwap, a novel consistent and realistic video head-swapping system empowered by dynamic neural Gaussian portrait priors, which significantly advances the state of the art in face and head replacement. Unlike previous methods that rely primarily on 2D generative models or 3D Morphable Face Models (3DMM), our approach overcomes their inherent limitations, including poor 3D consistency, unnatural facial expressions, and restricted synthesis quality. Moreover, existing techniques struggle with full head-swapping tasks due to insufficient holistic head modeling and ineffective background blending, often resulting in visible artifacts and misalignments. To address these challenges, GSwap introduces an intrinsic 3D Gaussian feature field embedded within a full-body SMPL-X surface, effectively elevating 2D portrait videos into a dynamic neural Gaussian field. This innovation ensures high-fidelity, 3D-consistent portrait rendering while preserving natural head-torso relationships and seamless motion dynamics. To facilitate training, we adapt a pretrained 2D portrait generative model to the source head domain using only a few reference images, enabling efficient domain adaptation. Furthermore, we propose a neural re-rendering strategy that harmoniously integrates the synthesized foreground with the original background, eliminating blending artifacts and enhancing realism. Extensive experiments demonstrate that GSwap surpasses existing methods in multiple aspects, including visual quality, temporal coherence, identity preservation, and 3D consistency.

Figures & tables

Fig. 1: Given a target video and few-shot source images, our GSwap firstly adapts a pre-trained 2D portrait generative model to the source head domain. It then elevates the 2D video head swapping task into the dynamic neural gaussian field, significantly enhancing the temporal consistency and overall quality of the head swap.

Fig. 2: The overall pipeline of our GSwap. Firstly, we adapt a pretrained 2D portrait generation model to the source head domain with source images $\mathcal{I}_{\textrm{src}}$ as input, generating a batch of training data through inpainting (detailed in Fig. 3 ). Secondly, we introduce the neural Gaussian field as our portrait representation and use a neural rerenderer to handle the mismatch region between foreground and background, generating the output in photo-realistic quality.

Fig. 3: We begin by performing domain adaptation on a pre-trained 2D head generation model. Subsequently, we use this model to inpaint the head region in the target video, thus creating a dataset suitable for head swapping applications. We will mosaic some pictures when publishing.

Fig. 4: We first employ a coarse face reenactment method to obtain a head with the corresponding pose and expression. Then, we warp the head mask of the face reenactment result based on the five key points between the reenacted head and the target head.

Fig. 5: To mitigate the expression and pose misalignments, we retrack the SMPL-X parameters ( $\beta_{\textrm{gen}}$ , $\theta_{\textrm{gen}}$ , $\psi_{\textrm{gen}}$ ) of $\mathcal{I}_{\textrm{gen}}$ during the training phase. The parameters tracked from the original target video $\mathcal{I}_{\textrm{tgt}}$ are then used for inference.

Fig. 6: Our method leverages a dynamic Gaussian portrait framework integrated with adapted diffusion portrait priors, achieving better quality and remarkable consistency.

Method	DiffSwap	DeepLiveCam	BlendFace	InfoSwap	FaceAdapter	Heser	LivePortrait ∗	Ours
FPS $\uparrow$	0.2	60	0.9	1.0	0.3	0.1	30	76

TABLE I: Inference Speed. (Liveportrait here is the method of “single-image head swap + LivePortrait”)

Fig. 7: Qualitive Comparison with the combined method of “Single-Image Head Swap + LivePortrait”.

Method	CSIM $\uparrow$	AED $\downarrow$	APD $\downarrow$	CLIP Score $\uparrow$
InfoSwap [ 22 ]	0.494	0.314	0.008	99.30%
BlendFace [ 89 ]	0.501	0.249	0.006	99.46%
FaceAdapter [ 58 ]	0.533	0.276	0.008	99.34%
DiffSwap [ 7 ]	0.258	0.235	0.009	95.20%
DeepLiveCam [ 88 ]	0.712	0.251	0.005	99.27%
LivePortrait [ 90 ]	0.381	0.281	0.006	99.60%

TABLE II: Quantitive Results.

Method	ID $\uparrow$	Exp $\uparrow$	Skin Color $\uparrow$	Inpainting $\uparrow$	Holistic $\uparrow$
DiffSwap [ 7 ]	0.40	0.72	0.49	–	0.40
DeepLiveCam [ 88 ]	1.80	2.55	1.67	–	2.17
Heser [ 13 ]	1.08	0.70	1.25	0.60	0.85
Ours	2.72	2.03	2.59	2.40	2.57

TABLE III: Average score from the user study, rating from 0 to 3.

Fig. 8: To validate our 3D consistency, we provide rendered results under both novel view and novel pose conditions. Since the novel view synthesis lacks corresponding background input, only the head region is displayed in the multi-view results.

Fig. 9: The deformed mesh aligns with the geometry of the real human face, effectively mitigating potential error accumulation caused by the discrepancy between the SMPL-X facial representation and the ground-truth geometry.

Fig. 10: There exist some expression and pose misalignments between TARGET and Generated Dataset , and the Generated Dataset frames suffer from inconsistency.

Fig. 11: To address inconsistencies in head swap dataset frames, we retrack the data frames to mitigate these discrepancies. This approach effectively enhances the accuracy of facial expressions.

Fig. 12: Neural Rerendering(NR) can seamlessly handle background mismatch.

Fig. 13: Our domain adaptation strategy accepts an arbitrary numbers of reference source images. We can achieve reasonable results with just one source frame as input. Increasing the number of source frames could further enhance identity preservation and facial details.

Fig. 14: Lack of temporal conditioning features (TCF) leads to artifacts. Neural Gaussian Field (NF) performs better at representing high-frequency details

Fig. 15: Undress (UD) and pasting background (PBG) can generate better head-swapped images.

Fig. 16: Extreme cases with large head rotation angles.

Video Editing

Video editing uses AI to alter, compose, relight, or reorganize moving footage while controlling how changes persist across frames.

See all 179 papers

DirectSwap: Paired, Mask-Free Video Head Swapping with Full-Reference Evaluation
Dec 10, 2025Yanan Wang, Shengcai Liao, Panwen Hu +4Video Editing Video Generation Evaluation
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), UAE · United Arab Emirates University (UAEU), Al Ain, UAE · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, UAE +1
AHS: Adaptive Head Synthesis via Synthetic Data Augmentations
Apr 17, 2026Taewoong Kang, Hyojin Jang, Sohyun Jeong +4Synthetic Data Augmentation Unpaired Image-to-Image Translation
KAIST · Korea University · FLIPTION
DipGuava: Disentangling Personalized Gaussian Features for 3D Head Avatars from Monocular Video
Mar 30, 2026Jeonghaeng Lee, Seok Keun Choi, Zhixuan Li +23D Avatar Generation 3D Gaussian Splatting
Yonsei University, Korea · Nanyang Technological University, Singapore

Dec 10, 2025J/K move · Enter open · S save

DirectSwap: Paired, Mask-Free Video Head Swapping with Full-Reference Evaluation

Yanan Wang, Shengcai Liao, Panwen Hu, Xin Li+3

Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), UAE · United Arab Emirates University (UAEU), Al Ain, UAE · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, UAE+1

Video Editing Video Generation Evaluation

Open paper AskPDF arXiv

Head swapping replaces an entire head while preserving pose, expression, body motion, and scene. Progress is limited by the lack of cross-identity paired videos: real footage cannot provide different identities performing exactly the same motion, leaving the task without paired supervision or frame-aligned ground truth. Existing methods therefore rely on same-identity masked reconstruction, which restricts supervision to predefined editable regions. To address this, we introduce an identity-expression decoupled synthesis pipeline that constructs expression-synchronized cross-identity video pairs from real footage. Expression-bearing facial regions are retained under small clip-consistent geometric perturbations, while the surrounding head is regenerated with a new identity as synthesized swapping input. These pairs form a cross-identity benchmark with frame-aligned real-world video as ground-truth targets, enabling full-reference evaluation of identity, expression, pose, reconstruction fidelity, and temporal stability. This yields HeadSwapBench, the first cross-identity paired dataset for video head swapping, supporting both training (20,278 videos) and benchmarking (1,040 videos). With the cross-identity supervision enabled by this paired dataset, we propose DirectSwap, a mask-free video head swapping training paradigm. Under otherwise identical settings, this formulation outperforms head- and rectangle-masked same-identity reconstruction, particularly for bidirectional head-silhouette changes. At inference, driving-output divergence and emergent reference attention estimate the edit support, allowing unchanged non-head content to be restored from the driving video without external segmentation or additional training. On 1,040 clips from 104 unseen subjects, the resulting model demonstrates that the proposed paired supervision supports effective whole-head swapping.

Abstract

Video Editing

Explore similar work

DirectSwap: Paired, Mask-Free Video Head Swapping with Full-Reference Evaluation

AHS: Adaptive Head Synthesis via Synthetic Data Augmentations

DipGuava: Disentangling Personalized Gaussian Features for 3D Head Avatars from Monocular Video