Co-speech gesture video generation aims to synthesize realistic, audio-driven portrait videos from a single reference image and a speech signal. Despite advances in visual quality, diffusion-based approaches still suffer from degraded local details and imperfect audio-visual alignment. We present PAGE (Parts-Aware GuidancE), a general, backbone-agnostic optimization framework for diffusion-based generation that enhances local spatial fidelity and global audio-visual consistency at both the training-objective and denoising-trajectory levels. During training, we propose Parts-Specific Reweighting (PSR), which adaptively increases supervision in semantically important regions such as lips, eyes, and hands, improving visual fidelity in perceptually critical areas. During inference, we introduce Consistency Classifier Guidance (CCG), an audio-conditioned latent classifier that identifies and corrects audio-visual inconsistencies in the generated video. We further design Differential Guidance (DG), an efficient inference strategy that refines facial and body motion separately while avoiding cross-region interference. To enable large-scale evaluation, we construct the Chinese Announcer Speech (CAS) dataset, the first comprehensive Chinese benchmark for co-speech gesture portrait video generation. Compared with the widely used PATS dataset, CAS includes substantially more identities, video clips, and scene diversity. Extensive experiments on both PATS and CAS demonstrate that PAGE achieves state-of-the-art performance in perceptual quality, gesture diversity, and audio-visual alignment. Project page: https://aigc-explorer.github.io/PAGE/.
Figures & tables
Fig. 1: (a) Limitations of existing methods include degraded local fidelity (e.g., hands, facial regions) and audio–visual misalignment (e.g., motion that does not match speech). (b) Qualitative ablation. Parts-Specific Reweighting (PSR) improves local visual quality in critical regions, while Consistency Classifier Guidance (CCG) refines the generated sequence at inference time to enhance audio–visual alignment. The full PAGE framework combines both, producing high-quality and temporally consistent co-speech gesture videos.
Fig. 2: Overview of the proposed PAGE with three components: (a) Backbone : a video generation model G (e.g., Wan2.1). Video frames are encoded with a VAE; reference-image latents are extracted using CLIP and the VAE. These features, together with noise from a conditional first frame or a noised video clip, are concatenated along the channel dimension. Audio embeddings from a pretrained audio encoder are injected into G via cross-attention. (b) Parts-Specific Reweighting (PSR) : a dynamic, region-aware loss weighting scheme constructed from pose-keypoint confidences, which upweights supervision for critical regions during training. (c) Consistency Classifier Guidance (CCG) : at inference, consistency gradients derived from joint audio–video features guide denoising to improve temporal audio–visual coherence. The VAE decoder then maps denoised latents back to pixel space.
Fig. 3: Overview of Parts-Specific Reweighting (PSR). We obtain pose keypoints and their confidences, construct spatial masks for face, hands, and body, and then combine them into a single reweighting map. Higher weights are assigned to perceptually sensitive regions such as lips and finger gaps.
Fig. 4: Architecture of the proposed CCG classifier. We augment a pretrained DINOv2 backbone with audio cross-attention to fuse speech features into the visual latent sequence. A lightweight binary head then predicts whether the audio–video pair is aligned. Separate classifiers are trained for the face and non-face regions.
Fig. 5: Procedure for constructing negative samples for classifier training. Negative (misaligned) samples are generated by the trained G , while positive samples are taken from real videos. Spatial masking enforces region-specific supervision for the face and non-face classifiers.
Fig. 6: Inference pipeline of PAGE with Differential Guidance (DG). The face and non-face classifiers alternately guide the denoising trajectory. Compensation terms reduce cross-region interference, resulting in improved temporal alignment and local detail compared to naive sequential guidance.
Dataset
#Clips
Total Duration
Total Frames
Frames/Clip
#Identities
Resolution
Full/Half Body
Language
PATS [ 42 ]
11395
145428s (40.40h)
4362840
382.873
4
1080 × 1080
Half
English
CAS (Ours)
19052
307841s (85.51h)
7684644
403.351
833
1920 × 1080
Full and Half
Chinese
TABLE I: Comparison between the PATS and CAS datasets. CAS is larger in scale, more diverse in speaking scenarios, and provides high-resolution clips in Mandarin.
Method
FVD ↓
BAS ↑
FGD ↓
Div. ↑
SDT [ 2 ]
294.71
0.283
11.46
13.08
S2G [ 10 ]
268.43
0.297
2.23
13.34
EchoMimicV2 [ 11 ]
257.91
0.300
3.50
13.25
EchoMimicV3 [ 48 ]
242.30
0.297
2.79
13.28
OmniAvatar-1.3B [ 49 ]
223.22
0.293
1.62
13.31
OmniAvatar-14B [ 49 ]
215.81
0.294
1.48
13.35
TABLE II: Quantitative results on PATS. PAGE-DG means the Differential Guidance (DG) strategy is used for CCG classifier during inference. Bold denotes the best.
Method
FVD ↓
BAS ↑
FGD ↓
Div. ↑
S2G [ 10 ]
506.68
0.2829
1.67
12.43
EchoMimicV2 [ 11 ]
1107.65
0.2823
1.84
12.17
EchoMimicV3 [ 48 ]
301.94
0.2837
1.46
12.98
OmniAvatar-1.3B [ 49 ]
384.21
0.2833
1.18
13.11
OmniAvatar-14B [ 49 ]
353.65
0.2854
0.48
13.05
Hallo3 [ 15 ]
296.48
0.2864
1.12
13.11
TABLE III: Quantitative results on CAS. PAGE-DG means the Differential Guidance (DG) strategy is used for CCG classifier during inference. Bold denotes the best.
Fig. 7: A qualitative comparison with previous work . We highlight the obvious flaws in earlier methods with red boxes. As shown, previous studies encounter issues like background inconsistencies, hand blurriness, finger distortion, poor face identity preservation and audio-visual misalignment. In contrast, our method can produce high-quality frames.
Method
Realism ↑
Diversity ↑
Synchrony ↑
Overall Quality ↑
S2G [ 10 ]
0.0%
0.0%
1.0%
0.0%
EchoMimicV2 [ 11 ]
4.5%
7.5%
20.5%
1.5%
EchoMimicV3 [ 48 ]
0.5%
3.5%
7.5%
3.5%
OmniAvatar-1.3B [ 49 ]
0.0%
0.0%
0.5%
0.0%
OmniAvatar-14B [ 49 ]
16.0%
3.0%
6.0%
17.0%
Hallo3 [ 15 ]
5.0%
9.5%
10.33%
5.5%
TABLE IV: User preferred percentage for each method. Test videos from PATS and CAS.
Name
FVD ↓
BAS ↑
FGD ↓
Div. ↑
Wan-1.3B-Baseline
221.87
0.289
1.51
13.28
w/ PSR
212.46
0.293
1.42
13.33
w/ SG
194.53
0.300
1.05
13.39
w/ DG
189.04
0.303
1.01
13.44
w/ PSR+SG
187.69
0.302
0.98
13.42
w/ PSR+DG
181.32
0.305
0.93
13.48
TABLE V: Ablation of core components.
Guidance Step Rate
FVD ↓
BAS ↑
FGD ↓
Div. ↑
0%
212.46/ 212.46
0.293/ 0.293
1.42/ 1.42
13.33/ 13.33
20%
203.74/ 198.53
0.295/ 0.299
1.22/ 1.16
13.35/ 13.36
40%
190.13/ 189.61
0.298/ 0.304
1.05/ 0.98
13.40/ 13.42
60%
189.76/ 185.98
0.297/ 0.302
1.02/ 0.98
13.39/ 13.45
80%
189.13/ 185.27
0.301/ 0.303
0.97 / 0.95
13.40/ 13.47
100%
187.69/ 181.32
0.302/ 0.305
0.98/ 0.93
13.42/ 13.48
TABLE VI: Ablation of guidance step rate . In each number pair, the former is PAGE-SG and the latter is PAGE-DG.
Name
#Param.
Accuracy ↑
FVD ↓
Vit-S-PAGE [ 50 ]
0.03B
0.731
210.05
Wan2.1(2 Blocks)
0.09B
0.975
192.38
Wan2.1(30 Blocks)
1.3B
0.987
189.54
DINO-v3-based [ 51 ]
0.05B
0.992
186.76
DINO-v2-based [ 40 ]
0.03B
0.979
181.32
TABLE VII: Ablation of CCG Classifier.
Name
FVD ↓
BAS ↑
FGD ↓
Div. ↑
Normal injection
214.73
0.292
1.45
13.29
Latent alignment(ours)
212.46
0.293
1.42
13.33
TABLE VIII: Ablation of audio injection.
λface
λnon_face
FVD ↓
BAS ↑
FGD ↓
Div. ↑
2
1
194.07
0.301
1.14
13.4
0.5
1
203.14
0.296
1.2
13.35
1
1
187.69
0.302
0.98
13.42
1
0.5
201.56
0.297
1.17
13.38
1
2
190.63
0.302
1.08
13.4
TABLE IX: Ablation of two classifiers’ guidance weights.
λdiff
FVD ↓
BAS ↑
FGD ↓
Div. ↑
0.5
198.24
0.298
1.12
13.4
1
181.32
0.305
0.93
13.48
2
187.98
0.302
0.99
13.43
TABLE X: Ablation of differential guidance weight.
Method
Inference Time(s)
Method
Inference Time(s)
S2G [ 10 ]
23.8
Hallo3 [ 15 ]
3569.2
EchoMimicV3(1.3B) [ 48 ]
140.1
PAGE w/o CCG
139.4
OmniAvatar-1.3B [ 49 ]
138.4
PAGE-SG
142.2
OmniAvatar-14B [ 49 ]
787.0
PAGE-DG
140.6
TABLE XI: Inference time refers to generating a 10s video, all experiments conducted under identical conditions.
Wan2.1 (DiT+FM)
PSR
CCG+DG
FVD ↓
BAS ↑
FGD ↓
Div. ↑
✓
✗
✗
221.87
0.289
1.51
13.28
✓
✓
✗
212.46
0.293
1.42
13.33
✓
✗
✓
189.04
0.303
1.01
13.44
✓
✓
✓
181.32
0.305
0.93
13.48
TABLE XII: Ablation of the proposed PSR, CCG, and DG modules on Wan.
SVD (U-Net+DDPM)
PSR
CCG+DG
FVD ↓
BAS ↑
FGD ↓
Div. ↑
✓
✗
✗
260.14
0.281
4.07
13.16
✓
✓
✗
253.06
0.283
3.61
13.18
✓
✗
✓
244.23
0.288
3.32
13.23
✓
✓
✓
241.07
0.291
3.05
13.24
TABLE XIII: Ablation of the proposed PSR, CCG, and DG modules on SVD.
(ω1,ω2,ω3)
FVD ↓
BAS ↑
FGD ↓
Div. ↑
(1,1,1)
221.87
0.289
1.51
13.28
(2.5,1.5,1)
213.14
0.291
1.45
13.36
(10,5,2)
212.46
0.293
1.42
13.33
(24,10,4)
210.96
0.293
1.43
13.29
Adaptive Learning
211.37
0.297
1.44
13.31
TABLE XIV: Ablation with different PSR weight settings.
Fig. 8: CAS dataset construction pipeline.
Fig. 9: Good Clip. The video clip consistently features the same clearly visible person throughout, with rich facial expressions and body motions, except that the subtitles need to be removed in a subsequent processing step.
Fig. 10: Substandard Clip. Further data processing is required to remove invalid frames and obtain video clips that meet the desired standards.
Fig. 11: Substandard Clip. Further data processing is required to remove invalid frames and obtain video clips that meet the desired standards.
Fig. 12: Identity reference image selection. After face clustering, the frames in each original video are grouped into multiple candidate identity clusters, each corresponding to a potential speaker identity. For each cluster, we select the frame with the highest Qwen-VL-derived human-quality score as the identity reference image, which serves as the anchor image (marked by a red solid line in the figure) for subsequent temporal segmentation.
Fig. 13: Example frames from CAS. CAS contains 19,052 curated clips spanning varied identities, poses, scene types, and camera viewpoints. This diversity supports robust evaluation of co-speech gesture generation, including facial articulation, upper-body dynamics, and global expressiveness.
We propose PersonaGesture, a diffusion-based pipeline for single-reference co-speech gesture personalization of unseen speakers. Given target speech and one motion clip from a new speaker, the model must synthesize gestures that follow the new utterance while retaining speaker-specific pose choices, without per-speaker optimization. This setting is useful for avatars and virtual agents, but it is hard because the reference mixes stable speaker habits with utterance-specific trajectories. PersonaGesture consists of two key components, Adaptive Style Infusion (ASI) and Implicit Distribution Rectification (IDR), to separate temporal identity evidence from residual statistic correction. A Style Perceiver first encodes the variable-length reference into compact speaker-memory tokens. ASI injects these tokens into denoising through zero-initialized residual cross-attention, enabling style evidence to affect motion formation without replacing the pretrained speech-to-motion prior. Building on this, IDR applies a length-aware diagonal affine map in latent space to correct residual channel-wise moments estimated from the same reference. Across BEAT2 and ZeroEGGS, we evaluate quantitative metrics, reference-identity controls, same-audio diagnostics, qualitative comparisons, and human preference. Experiments show that separating denoising-time speaker memory from conservative post-generation moment correction improves unseen-speaker personalization over collapsed style codes, full-reference attention, and one-clip finetuning. Project: https://xiangyue-zhang.github.io/PersonaGesture.
Xiangyue Zhang, Yiyi Cai, Kunhang Li +6
The University of Tokyo · Shanda AI Research Tokyo · Renmin University +1
Generating co-speech gestures that are temporally coherent, semantically aligned with speech, and grounded with surrounding objects remains challenging. Prior speech-driven gesture models emphasize audio-gesture alignment but do not explicitly account for posture constraints or surrounding objects, failing to capture the inherent correlation between body gestures and the physical space. We present Puppeteer, a posture-aware, object-grounded co-speech gesture diffusion model operating in a causal latent space. We decompose long gestures into structured primitives and learn a causal variational autoencoder that encodes them into temporally ordered latent tokens, each depending only on the past. We then perform conditional diffusion directly in the causal latent space, conditioning on speech signals, motion history, an initial posture reference, and object geometry to synthesize physically consistent gestures. This temporally ordered latent formulation enables explicit temporal control and supports tasks such as gesture in-betweening and gesture completion. To better assess co-speech gesture synthesis beyond existing measures, we introduce new evaluation metrics tailored to this task. We also created SceneGes, the first curated synthetic 3D dataset of embodied co-speech gestures and corresponding 3D objects, enabling object-grounded gesture generation. Experiments show that Puppeteer generates more diverse and temporally synchronized gestures than prior methods, while enabling object-grounded gesture synthesis.
Vida Adeli, Soroush Mehraban, Jacob Rommann +3
Pickford AI · University of Toronto · Vector Institute
Co-speech gesture generation aims to synthesize realistic body movements that are semantically coherent with speech and faithful to a user-specified gestural style. Existing VQ-VAE based co-speech gesture generation methods improve generation quality but fail to encode semantic structure into the motion representation or explicitly disentangle content from style, limiting both semantic coherence and personalization fidelity. We present PersonaGest, a two-stage framework addressing both limitations. In the first stage, a semantic-guided RVQ-VAE disentangles motion content and gestural style within the residual quantization structure, where a Semantic-Aware Motion Codebook (SMoC) organizes the content codebook by gesture semantics and contrastive learning further enforces content-style separation. In the second stage, a Masked Generative Transformer generates content tokens via a semantic-aware re-masking strategy, followed by a cascade of Style Residual Transformers conditioned on a reference motion prompt for style control. Extensive experiments demonstrate state-of-the-art performance on objective metrics and perceptual user studies, with strong style consistency to the reference prompt. Our project page with demo videos is available at https://danny-nus.github.io/PersonaGest/
Junchuan Zhao, Qifan Liang, Ye Wang
School of Computing, National University of Singapore