Co-speech motion generation remains challenging over long horizons due to the need for temporal coherence and consistent style. Existing methods typically generate long motions segment by segment, either conditioning on a short seed of recent frames, which gradually drifts, or retaining ever-longer explicit histories, which are costly to maintain and hard to learn from limited data. We trace these limitations to a key asymmetry between past and present: past motion carries slowly varying style that should be compactly summarized, whereas the current segment exhibits fast-changing local dynamics that require fine-grained modeling. This asymmetry motivates a simple Past-As-State, Present-as-Attention principle for long-horizon co-speech motion generation. Following this principle, we propose PASPA, which summarizes past history with an inter-block persistent state and models the current segment with intra-block bidirectional self-attention. We instantiate PASPA within a blockwise flow matching framework, with multi-block supervision for effective training over variable histories, prefill-decode state caching for efficient rollout, and hybrid classifier-free guidance for enhanced history and speech conditioning. Extensive public-benchmark experiments show that PASPA achieves state-of-the-art FGD performance and preserves motion style across long-sequence generation.
Figures & tables
Figure 1: Long-horizon co-speech motion generation (52 s) from the same speech and speaker, with the GT style keeping both hands raised near the chest. (a) Short-Seed Truncation matches the style initially but monotonically drifts: hands gradually drop. (b) Long-Context Token Attention retains full history but struggles to maintain a consistent style, matching or deviating regardless of temporal distance. (c) PASPA (ours) preserves the raised-hands style throughout, closely matching the GT.
Figure 2: History propagation in co-speech motion generation. (a) Short-Seed Truncation keeps only recent frames. (b) Long-Context Token Attention retains history as tokens for global attention. (c) PASPA compresses history into a persistent state and applies full attention within the current block.
Figure 3: Autocorrelation r(τ) on BEAT2, averaged across speakers with 95% confidence intervals (shaded). Style features remain correlated beyond 100 frames, whereas local features decay rapidly. Inset: raw windowed pose statistics. See Appendix A for more details.
Figure 4: Network architecture of PASPA. (a) BGDN propagates history through a persistent state S[n]0 via a block-causal GDN kernel. (b) BMHSA captures local dynamics via efficient block-diagonal bidirectional attention. The overall network (bottom) sequentially applies BGDN and BMHSA.
Method
1 Speaker
All Speakers
FGD ↓
BC ↑
Div ↑
MSE ↓
FGD ↓
BC ↑
Div ↑
MSE ↓
CaMN [ 45 ]
6.64
0.677
10.86
–
5.12
0.200
5.58
–
DSG [ 80 ]
8.81
0.724
11.49
–
11.74
0.734
11.12
–
EMAGE [ 44 ]
5.51
0.772
13.06
7.680
6.92
0.284
6.06
6.908
DiffSHEG [ 11 ]
8.99
0.714
11.91
7.665
–
–
–
–
MambaTalk [ 79 ]
5.37
0.781
13.05
6.289
–
–
–
–
Table 1: Comparison with state-of-the-art (SOTA) methods on the BEAT2 dataset.
Figure 5: Qualitative comparison. PASPA consistently matches the GT in gesture direction (red arrows), amplitude, and speech alignment. Baselines exhibit various failure modes (red circles): unnatural hand positions, hesitant or mismatched gestures, and missing semantic response.
Figure 7Table 8Figure 9
Guidance
120 s (3,600 frames)
300 s (9,000 frames)
600 s (18,000 frames)
Lat. (ms)
Mem. (GB)
Lat. (ms)
Mem. (GB)
Lat. (ms)
Mem. (GB)
No CFG
11,411
0.36
28,562
0.40
55,978
0.47
Hybrid CFG
12,439
0.37
29,966
0.41
61,093
0.48
Table 4: Inference overhead of Hybrid CFG under the final PASPA configuration.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Category
Hyperparameter
Setting / Value
Dataset & Setup
Skeleton
55 joints (30 hands, 25 torso)
Translation
3D global root translation
Expression
100-dim FLAME blendshapes
Rotation Representation
axis-angle (raw) → rot6d [ 97 ]
Motion Dimension
55×6+3=333
Audio Features
WavLM [ 12 ] , 16 kHz, 1024-dim, pre-extracted
Appendix
Table 5: Implementation details of our best training and inference configurations on BEAT2.
Figure 11: User study interface. Participants rank four anonymized methods (A/B/C/D, randomized per clip) produced by SemTalk, GestureLSM, EMAGE and PASPA, against a non-rankable ground-truth reference (GT), along four criteria: realism, semantic consistency, motion-speech sync, and diversity.
Table 13
Table 8: CFG scale ablations for Condition CFG and Hybrid CFG. Cond CFG denotes condition CFG. Both panels use the same activation interval t∈[0.1,0.6] .
Table 9: Formulations of Hybrid CFG decomposition modes. x^0 , x^0c , x^0h , and x^0c,h denote predicted clean motions under neither condition nor history, condition only, history only, and both condition and history, respectively.
Table 16
Guidance
1 Speaker
All Speakers
FGD ↓
BC ↑
Div ↑
MSE ↓
FGD ↓
BC ↑
Div ↑
MSE ↓
w/o CFG
3.76
0.744
13.80
5.788
2.59
0.639
9.48
3.802
Hybrid CFG
3.71
0.746
13.38
5.768
2.33
0.645
9.42
3.793
Appendix
Table 12: Effect of inference-time CFG on the main BEAT2 results.
Figure 12: Long-rollout endpoint metrics across T∈{10,30,60} s, evaluated on the terminal 10 s. Shaded regions denote 95% Wald confidence intervals estimated from clip-level bootstrap resampling.
Metric
Main result
50 generation seeds
Mean
Sample std.
95% CI of mean
FGD ↓
2.33
2.237
0.074
[2.216,2.258]
BC ↑
0.645
0.6407
0.0015
[0.6402,0.6411]
Div ↑
9.42
9.273
0.079
[9.251,9.296]
MSE ↓
3.793
3.761
0.003
[3.760,3.762]
Appendix
Table 13: Generation-seed variability in the All Speakers setting. Only the generation seed varies across 50 runs; CIs are 95% Student- t intervals for the mean.
Method
Realism
Semantic
Sync
Diversity
Overall
EMAGE
2.63 ± 1.09
2.41 ± 1.13
2.49 ± 1.08
2.50 ± 1.09
2.51 ± 1.10
GestureLSM
2.60 ± 1.15
2.80 ± 1.12
2.82 ± 1.09
2.83 ± 1.12
2.76 ± 1.12
SemTalk
2.63 ± 1.04
2.59 ± 1.01
2.58 ± 1.08
2.57 ± 0.99
2.59 ± 1.03
PASPA
2.14 ± 1.12
2.20 ± 1.12
2.11 ± 1.11
2.10 ± 1.14
2.14 ± 1.12
Appendix
Table 14: User study results. Each cell reports the mean rank ± standard deviation across all participants and clips (lower is better; 1 = best, 4 = worst).
Figure 13: Failure case on sequence 16_jorge_1_3_3 . Time progresses from left to right. (a) Ground-truth. (b) Our generated sequence.
Real-time co-speech gesture generation must produce 3D motion clip by clip as speech arrives. Existing streaming methods are open-loop: each clip depends on past context, but the model cannot check or correct its trajectory. Small errors therefore accumulate and cause drift over long sequences. We observe that this failure is mainly caused by the lack of a forward constraint rather than poor short-clip quality. A plausible key pose at the end of each clip provides a destination anchor that limits drift. Based on this observation, we propose StreamTalk, a closed-loop framework with a periodic generate-retrieve-refine cycle. Streaming Pose-Guided Generation first predicts a coarse clip, retrieves a plausible tail pose from a speaker-specific motion database, and refines the clip using this pose before continuing to the next window. During training, Stochastic Anchor Masking randomly masks pose and translation frames, teaching the model to recover complete motion from sparse boundary conditions. A part-aware DiT separates hand, body, and translation streams to reduce interference between global displacement and local articulation. On BEAT2, StreamTalk achieves state-of-the-art FGD, reduces long-horizon drift relative to open-loop baselines, and runs in real time at 76 FPS. Project page: https://xiangyue-zhang.github.io/StreamTalk/.
Xiangyue Zhang, Jianfang Li, Jiaxu Zhang +2
The University of Tokyo · Alibaba Group · Nanyang Technological University +1
Generating natural co-speech gestures from streaming speech is essential for embodied conversational agents, where motion must be produced while a user is still speaking. Recent streaming gesture systems make online generation possible by autoregressing over discrete motion tokens, but this design compresses high-dimensional continuous motion into finite codebooks and can limit the realism and diversity of generated gestures. To preserve both causality and continuous expressiveness, we propose GestureFAR, a flow-autoregressive framework for streaming co-speech gesture generation. First, GestureFAR autoregresses over causal continuous motion latents, using a transformer to model streaming audio-motion context and a per-token flow-matching head to sample the next latent from a continuous distribution. Second, we introduce a head-only flow distillation strategy that freezes the causal backbone and distills the multi-step per-token flow head into a single network evaluation using consistency and distribution-matching objectives. This keeps the model token-causal while removing the main latency bottleneck for live interaction. Experiments on BEAT2 show that GestureFAR significantly improves the quality--latency trade-off among streaming-capable methods, preserving strong gesture quality while enabling real-time token-causal generation. Project Page: https://andypinxinliu.github.io/GestureFAR
Pinxin Liu, Haiyang Liu, Jiahao Luo +3
University of Rochester · University of Tokyo · University of California Santa Cruz +2
While the field of co-speech gesture generation has seen significant advances, producing holistic, semantically grounded gestures remains a challenge. Existing approaches rely on external semantic retrieval methods, which limit their generalisation capability due to dependency on predefined linguistic rules. Flow-matching-based methods produce promising results; however, the network is optimised using only semantically congruent samples without exposure to negative examples, leading to learning rhythmic gestures rather than sparse motion, such as iconic and metaphoric gestures. Furthermore, by modelling body parts in isolation, the majority of methods fail to maintain crossmodal consistency. We introduce a Contrastive Flow Matching-based co-speech gesture generation model that uses mismatched audio-text conditions as negatives, training the velocity field to follow the correct motion trajectory while repelling semantically incongruent trajectories. Our model ensures cross-modal coherence by embedding text, audio, and holistic motion into a composite latent space via cosine and contrastive objectives. Extensive experiments and a user study demonstrate that our proposed approach outperforms state-of-the-art methods on two datasets, BEAT2 and SHOW.
Lanmiao Liu, Esam Ghaleb, Aslı Özyürek +1
Max Planck Institute for Psycholinguistics · Donders Institute for Brain Cognition and Behaviour · Utrecht University