PAGE: Parts-Aware Guidance for Co-Speech Gesture Portrait Video Generation

Yabiao Wang; Jiafu Wu; Yicheng Xu; Teng Hu; Shengzhe Zhou; Yanxiao Sun; Jiangning Zhang; Ying Tai

cs.CVOct 9, 2026

PAGE: Parts-Aware Guidance for Co-Speech Gesture Portrait Video Generation

Authors: Yabiao Wang, Jiafu Wu, Yicheng Xu, Teng Hu, Shengzhe Zhou, Yanxiao Sun, Jiangning Zhang, Ying Tai,

Organizations: Institute of Cyber-Systems and Control, Zhejiang University, Hangzhou 310027, China

Abstract

Co-speech gesture video generation aims to synthesize realistic, audio-driven portrait videos from a single reference image and a speech signal. Despite advances in visual quality, diffusion-based approaches still suffer from degraded local details and imperfect audio-visual alignment. We present PAGE (Parts-Aware GuidancE), a general, backbone-agnostic optimization framework for diffusion-based generation that enhances local spatial fidelity and global audio-visual consistency at both the training-objective and denoising-trajectory levels. During training, we propose Parts-Specific Reweighting (PSR), which adaptively increases supervision in semantically important regions such as lips, eyes, and hands, improving visual fidelity in perceptually critical areas. During inference, we introduce Consistency Classifier Guidance (CCG), an audio-conditioned latent classifier that identifies and corrects audio-visual inconsistencies in the generated video. We further design Differential Guidance (DG), an efficient inference strategy that refines facial and body motion separately while avoiding cross-region interference. To enable large-scale evaluation, we construct the Chinese Announcer Speech (CAS) dataset, the first comprehensive Chinese benchmark for co-speech gesture portrait video generation. Compared with the widely used PATS dataset, CAS includes substantially more identities, video clips, and scene diversity. Extensive experiments on both PATS and CAS demonstrate that PAGE achieves state-of-the-art performance in perceptual quality, gesture diversity, and audio-visual alignment. Project page: https://aigc-explorer.github.io/PAGE/.

Figures & tables

Fig. 1: (a) Limitations of existing methods include degraded local fidelity (e.g., hands, facial regions) and audio–visual misalignment (e.g., motion that does not match speech). (b) Qualitative ablation. Parts-Specific Reweighting (PSR) improves local visual quality in critical regions, while Consistency Classifier Guidance (CCG) refines the generated sequence at inference time to enhance audio–visual alignment. The full PAGE framework combines both, producing high-quality and temporally consistent co-speech gesture videos.

Fig. 2: Overview of the proposed PAGE with three components: (a) Backbone : a video generation model $\bm{G}$ (e.g., Wan2.1). Video frames are encoded with a VAE; reference-image latents are extracted using CLIP and the VAE. These features, together with noise from a conditional first frame or a noised video clip, are concatenated along the channel dimension. Audio embeddings from a pretrained audio encoder are injected into $G$ via cross-attention. (b) Parts-Specific Reweighting (PSR) : a dynamic, region-aware loss weighting scheme constructed from pose-keypoint confidences, which upweights supervision for critical regions during training. (c) Consistency Classifier Guidance (CCG) : at inference, consistency gradients derived from joint audio–video features guide denoising to improve temporal audio–visual coherence. The VAE decoder then maps denoised latents back to pixel space.

Fig. 3: Overview of Parts-Specific Reweighting (PSR). We obtain pose keypoints and their confidences, construct spatial masks for face, hands, and body, and then combine them into a single reweighting map. Higher weights are assigned to perceptually sensitive regions such as lips and finger gaps.

Fig. 4: Architecture of the proposed CCG classifier. We augment a pretrained DINOv2 backbone with audio cross-attention to fuse speech features into the visual latent sequence. A lightweight binary head then predicts whether the audio–video pair is aligned. Separate classifiers are trained for the face and non-face regions.

Fig. 5: Procedure for constructing negative samples for classifier training. Negative (misaligned) samples are generated by the trained $G$ , while positive samples are taken from real videos. Spatial masking enforces region-specific supervision for the face and non-face classifiers.

Fig. 6: Inference pipeline of PAGE with Differential Guidance (DG). The face and non-face classifiers alternately guide the denoising trajectory. Compensation terms reduce cross-region interference, resulting in improved temporal alignment and local detail compared to naive sequential guidance.

Dataset	#Clips	Total Duration	Total Frames	Frames/Clip	#Identities	Resolution	Full/Half Body	Language
PATS [ 42 ]	11395	145428s (40.40h)	4362840	382.873	4	1080 $\times$ 1080	Half	English
CAS (Ours)	19052	307841s (85.51h)	7684644	403.351	833	1920 $\times$ 1080	Full and Half	Chinese

TABLE I: Comparison between the PATS and CAS datasets. CAS is larger in scale, more diverse in speaking scenarios, and provides high-resolution clips in Mandarin.

Method	FVD $\downarrow$	BAS $\uparrow$	FGD $\downarrow$	Div. $\uparrow$
SDT [ 2 ]	294.71	0.283	11.46	13.08
S2G [ 10 ]	268.43	0.297	2.23	13.34
EchoMimicV2 [ 11 ]	257.91	0.300	3.50	13.25
EchoMimicV3 [ 48 ]	242.30	0.297	2.79	13.28
OmniAvatar-1.3B [ 49 ]	223.22	0.293	1.62	13.31
OmniAvatar-14B [ 49 ]	215.81	0.294	1.48	13.35

TABLE II: Quantitative results on PATS. PAGE-DG means the Differential Guidance (DG) strategy is used for CCG classifier during inference. Bold denotes the best.

Method	FVD $\downarrow$	BAS $\uparrow$	FGD $\downarrow$	Div. $\uparrow$
S2G [ 10 ]	506.68	0.2829	1.67	12.43
EchoMimicV2 [ 11 ]	1107.65	0.2823	1.84	12.17
EchoMimicV3 [ 48 ]	301.94	0.2837	1.46	12.98
OmniAvatar-1.3B [ 49 ]	384.21	0.2833	1.18	13.11
OmniAvatar-14B [ 49 ]	353.65	0.2854	0.48	13.05
Hallo3 [ 15 ]	296.48	0.2864	1.12	13.11

TABLE III: Quantitative results on CAS. PAGE-DG means the Differential Guidance (DG) strategy is used for CCG classifier during inference. Bold denotes the best.

Fig. 7: A qualitative comparison with previous work . We highlight the obvious flaws in earlier methods with red boxes. As shown, previous studies encounter issues like background inconsistencies, hand blurriness, finger distortion, poor face identity preservation and audio-visual misalignment. In contrast, our method can produce high-quality frames.

Method	Realism $\uparrow$	Diversity $\uparrow$	Synchrony $\uparrow$	Overall Quality $\uparrow$
S2G [ 10 ]	0.0%	0.0%	1.0%	0.0%
EchoMimicV2 [ 11 ]	4.5%	7.5%	20.5%	1.5%
EchoMimicV3 [ 48 ]	0.5%	3.5%	7.5%	3.5%
OmniAvatar-1.3B [ 49 ]	0.0%	0.0%	0.5%	0.0%
OmniAvatar-14B [ 49 ]	16.0%	3.0%	6.0%	17.0%
Hallo3 [ 15 ]	5.0%	9.5%	10.33%	5.5%

TABLE IV: User preferred percentage for each method. Test videos from PATS and CAS.

Name	FVD $\downarrow$	BAS $\uparrow$	FGD $\downarrow$	Div. $\uparrow$
Wan-1.3B-Baseline	221.87	0.289	1.51	13.28
w/ PSR	212.46	0.293	1.42	13.33
w/ SG	194.53	0.300	1.05	13.39
w/ DG	189.04	0.303	1.01	13.44
w/ PSR+SG	187.69	0.302	0.98	13.42
w/ PSR+DG	181.32	0.305	0.93	13.48

TABLE V: Ablation of core components.

Guidance Step Rate	FVD $\downarrow$	BAS $\uparrow$	FGD $\downarrow$	Div. $\uparrow$
0%	212.46/ 212.46	0.293/ 0.293	1.42/ 1.42	13.33/ 13.33
20%	203.74/ 198.53	0.295/ 0.299	1.22/ 1.16	13.35/ 13.36
40%	190.13/ 189.61	0.298/ 0.304	1.05/ 0.98	13.40/ 13.42
60%	189.76/ 185.98	0.297/ 0.302	1.02/ 0.98	13.39/ 13.45
80%	189.13/ 185.27	0.301/ 0.303	0.97 / 0.95	13.40/ 13.47
100%	187.69/ 181.32	0.302/ 0.305	0.98/ 0.93	13.42/ 13.48

TABLE VI: Ablation of guidance step rate . In each number pair, the former is PAGE-SG and the latter is PAGE-DG.

Name	#Param.	Accuracy $\uparrow$	FVD $\downarrow$
Vit-S-PAGE [ 50 ]	0.03B	0.731	210.05
Wan2.1(2 Blocks)	0.09B	0.975	192.38
Wan2.1(30 Blocks)	1.3B	0.987	189.54
DINO-v3-based [ 51 ]	0.05B	0.992	186.76
DINO-v2-based [ 40 ]	0.03B	0.979	181.32

TABLE VII: Ablation of CCG Classifier.

Name	FVD $\downarrow$	BAS $\uparrow$	FGD $\downarrow$	Div. $\uparrow$
Normal injection	214.73	0.292	1.45	13.29
Latent alignment(ours)	212.46	0.293	1.42	13.33

TABLE VIII: Ablation of audio injection.

$\lambda_{face}$	$\lambda_{non\_face}$	FVD $\downarrow$	BAS $\uparrow$	FGD $\downarrow$	Div. $\uparrow$
2	1	194.07	0.301	1.14	13.4
0.5	1	203.14	0.296	1.2	13.35
1	1	187.69	0.302	0.98	13.42
1	0.5	201.56	0.297	1.17	13.38
1	2	190.63	0.302	1.08	13.4

TABLE IX: Ablation of two classifiers’ guidance weights.

$\lambda_{diff}$	FVD $\downarrow$	BAS $\uparrow$	FGD $\downarrow$	Div. $\uparrow$
0.5	198.24	0.298	1.12	13.4
1	181.32	0.305	0.93	13.48
2	187.98	0.302	0.99	13.43

TABLE X: Ablation of differential guidance weight.

Method	Inference Time(s)	Method	Inference Time(s)
S2G [ 10 ]	23.8	Hallo3 [ 15 ]	3569.2
EchoMimicV3(1.3B) [ 48 ]	140.1	PAGE w/o CCG	139.4
OmniAvatar-1.3B [ 49 ]	138.4	PAGE-SG	142.2
OmniAvatar-14B [ 49 ]	787.0	PAGE-DG	140.6

TABLE XI: Inference time refers to generating a 10s video, all experiments conducted under identical conditions.

Wan2.1 (DiT+FM)	PSR	CCG+DG	FVD $\downarrow$	BAS $\uparrow$	FGD $\downarrow$	Div. $\uparrow$
✓	✗	✗	221.87	0.289	1.51	13.28
✓	✓	✗	212.46	0.293	1.42	13.33
✓	✗	✓	189.04	0.303	1.01	13.44
✓	✓	✓	181.32	0.305	0.93	13.48

TABLE XII: Ablation of the proposed PSR, CCG, and DG modules on Wan.

SVD (U-Net+DDPM)	PSR	CCG+DG	FVD $\downarrow$	BAS $\uparrow$	FGD $\downarrow$	Div. $\uparrow$
✓	✗	✗	260.14	0.281	4.07	13.16
✓	✓	✗	253.06	0.283	3.61	13.18
✓	✗	✓	244.23	0.288	3.32	13.23
✓	✓	✓	241.07	0.291	3.05	13.24

TABLE XIII: Ablation of the proposed PSR, CCG, and DG modules on SVD.

$(\omega_{1},\omega_{2},\omega_{3})$	FVD $\downarrow$	BAS $\uparrow$	FGD $\downarrow$	Div. $\uparrow$
(1,1,1)	221.87	0.289	1.51	13.28
(2.5,1.5,1)	213.14	0.291	1.45	13.36
(10,5,2)	212.46	0.293	1.42	13.33
(24,10,4)	210.96	0.293	1.43	13.29
Adaptive Learning	211.37	0.297	1.44	13.31

TABLE XIV: Ablation with different PSR weight settings.

Fig. 8: CAS dataset construction pipeline.

Fig. 9: Good Clip. The video clip consistently features the same clearly visible person throughout, with rich facial expressions and body motions, except that the subtitles need to be removed in a subsequent processing step.

Panel of Figure 24 — Fig. 10: Substandard Clip. Further data processing is required to remove invalid frames and obtain video clips that meet the desired standards.

Panel of Figure 25 — Fig. 11: Substandard Clip. Further data processing is required to remove invalid frames and obtain video clips that meet the desired standards.

Fig. 12: Identity reference image selection. After face clustering, the frames in each original video are grouped into multiple candidate identity clusters, each corresponding to a potential speaker identity. For each cluster, we select the frame with the highest Qwen-VL-derived human-quality score as the identity reference image, which serves as the anchor image (marked by a red solid line in the figure) for subsequent temporal segmentation.

Fig. 13: Example frames from CAS. CAS contains 19,052 curated clips spanning varied identities, poses, scene types, and camera viewpoints. This diversity supports robust evaluation of co-speech gesture generation, including facial articulation, upper-body dynamics, and global expressiveness.

Video Generation

Video generation studies models that create and edit moving images from text, images, audio, and structured controls.

See all 364 papers

PersonaGesture: Single-Reference Co-Speech Gesture Personalization for Unseen Speakers
May 7, 2026Xiangyue Zhang, Yiyi Cai, Kunhang Li +6Human Motion Generation Co-Speech Gesture Generation
The University of Tokyo · Shanda AI Research Tokyo · Renmin University +1
Puppeteer: Object-Grounded Posture-Aware Co-Speech Gesture Generation
Aug 31, 2026Vida Adeli, Soroush Mehraban, Jacob Rommann +3Motion Diffusion Models Human Motion Generation
Pickford AI · University of Toronto · Vector Institute
PersonaGest: Personalized Co-Speech Gesture Generation with Semantic-Guided Hierarchical Motion Representation
May 8, 2026Junchuan Zhao, Qifan Liang, Ye WangRepresentation Learning Human Motion Generation
School of Computing, National University of Singapore

May 7, 2026J/K move · Enter open · S save

PersonaGesture: Single-Reference Co-Speech Gesture Personalization for Unseen Speakers

Xiangyue Zhang, Yiyi Cai, Kunhang Li, Kaixing Yang+5

The University of Tokyo · Shanda AI Research Tokyo · Renmin University+1

Human Motion Generation Co-Speech Gesture Generation Diffusion Models for Motion Generation Persona-Conditioned Generation

Open paper AskPDF arXiv

We propose PersonaGesture, a diffusion-based pipeline for single-reference co-speech gesture personalization of unseen speakers. Given target speech and one motion clip from a new speaker, the model must synthesize gestures that follow the new utterance while retaining speaker-specific pose choices, without per-speaker optimization. This setting is useful for avatars and virtual agents, but it is hard because the reference mixes stable speaker habits with utterance-specific trajectories. PersonaGesture consists of two key components, Adaptive Style Infusion (ASI) and Implicit Distribution Rectification (IDR), to separate temporal identity evidence from residual statistic correction. A Style Perceiver first encodes the variable-length reference into compact speaker-memory tokens. ASI injects these tokens into denoising through zero-initialized residual cross-attention, enabling style evidence to affect motion formation without replacing the pretrained speech-to-motion prior. Building on this, IDR applies a length-aware diagonal affine map in latent space to correct residual channel-wise moments estimated from the same reference. Across BEAT2 and ZeroEGGS, we evaluate quantitative metrics, reference-identity controls, same-audio diagnostics, qualitative comparisons, and human preference. Experiments show that separating denoising-time speaker memory from conservative post-generation moment correction improves unseen-speaker personalization over collapsed style codes, full-reference attention, and one-clip finetuning. Project: https://xiangyue-zhang.github.io/PersonaGesture.

Abstract

Video Generation

Explore similar work

PersonaGesture: Single-Reference Co-Speech Gesture Personalization for Unseen Speakers

Puppeteer: Object-Grounded Posture-Aware Co-Speech Gesture Generation

PersonaGest: Personalized Co-Speech Gesture Generation with Semantic-Guided Hierarchical Motion Representation