We propose LiveGesture, the first fully streamable, speech-driven full-body gesture generation framework that operates with zero look-ahead and supports arbitrary sequence length. Unlike existing co-speech gesture methods, which are designed for offline generation and either treat body regions independently or entangle all joints within a single model, LiveGesture is built from the ground up for causal, region-coordinated motion generation. LiveGesture consists of two main modules: the Streamable Vector Quantized Motion Tokenizer (SVQ) and the Hierarchical Autoregressive Transformer (HAR). The SVQ tokenizer converts the motion sequence of each body region into causal, discrete motion tokens, enabling real-time, streamable token decoding. On top of SVQ, HAR employs region-expert autoregressive (xAR) transformers to model expressive, fine-grained motion dynamics for each body region. A causal spatio-temporal fusion module (xAR Fusion) then captures and integrates correlated motion dynamics across regions. Both xAR and xAR Fusion are conditioned on live, continuously arriving audio signals encoded by a streamable causal audio encoder. To enhance robustness under streaming noise and prediction errors, we introduce autoregressive masking training, which leverages uncertainty-guided token masking and random region masking to expose the model to imperfect, partially erroneous histories during training. Experiments on the BEAT2 dataset demonstrate that LiveGesture produces coherent, diverse, and beat-synchronous full-body gestures in real time, matching or surpassing state-of-the-art offline methods under true zero look-ahead conditions.
Figures & tables
Figure 2: Overview of the Streamable Asymmetric Motion Tokenizer. (a) For each body region, a bidirectional encoder and causal stream decoder learn low-rate latent sequences. (b) Freezing this autoencoder, the Streamable VQ-VAE (SVQ) adds a region-specific codebook and projection head to quantize latents into discrete, time-synchronous motion tokens that remain compatible with strictly causal decoding for streaming.
Figure 3: Overview of the Hierarchical Autoregressive Model in LiveGesture . A streamable audio encoder and optional text encoder provide causal audio/text tokens to four local AR Region-eXperts (upper body, lower body, hands, face), each modeling its own SVQ motion token stream. Their frozen states are adapted by per-region Pre-Infusion Adapters (PILOR) and fused by xAR-Fuse, a causal spatial–temporal transformer with audio–motion cross-attention, global temporal attention, and inter-region spatial attention that predicts next-step SVQ tokens for zero–look-ahead streaming full-body gesture generation.
Figure 4: Qualitative comparison with state-of-the-art methods on BEAT. LiveGesture generates more diverse, expressive full-body gestures that better follow speech rhythm, while red circles mark representative failure cases of prior methods.
Methods
Venue
Streaming
FGD ( ↓ )
BC ( → )
Diversity ( ↑ )
MSE ( ↓ )
Offline Solutions
HA2G [ 24 ]
CVPR’22
✗
12.32
0.677
8.626
–
DisCo [ 16 ]
MM’22
✗
9.417
0.643
9.912
–
CaMN [ 19 ]
ECCV’22
✗
6.644
0.676
10.86
–
TalkShow [ 45 ]
CVPR’23
✗
6.209
0.695
13.47
7.791
DiffSHEG [ 4 ]
CVPR’24
✗
7.141
0.743
8.21
9.571
Table 1: State-of-the-art comparison on BEAT. Best results are shown in bold and second best are underlined . The Streaming column indicates whether the method supports zero-look-ahead streaming ( ✓ ) or is offline-only ( ✗ ); Our method is the only streaming model while remaining superior in most of important metrics.
Figure 5: Interactive human–avatar conversation enabled by LiveGesture . User speech is converted into a spoken reply by VITA-Audio, while our LiveGesture streaming gesture model simultaneously generates synchronized full-body SMPL-X motions from live audio, allowing the avatar to respond in real time.
Table 2: Ablation studies of LiveGesture design choices, including core components, architecture, tokenization, audio encoders, loss weights, and UGR.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Methods
Venue
Streaming
FGD ( ↓ )
BC ( → )
Diversity ( ↑ )
Offline Solutions
GestureLSM [ 22 ]
ICCV’25
✗
4.088
0.714
13.24
Streaming Solution (Ours)
LiveGesture (ours)
Ours
✓
4.51
0.783
13.31
Appendix
Table 1: Comparison with state-of-the-art methods on BEAT2 without the facial motion module. LiveGesture remains the only zero–look-ahead streaming model while achieving competitive or superior performance in BC and Diversity.
Method
Streaming
Realness ↑
Synchrony ↑
Smoothness ↑
CaMN [ 19 ]
✗
3.3
3.2
3.6
EMAGE [ 18 ]
✗
3.5
3.5
3.7
GestureLSM [ 22 ]
✗
4.2
4.1
4.3
LiveGesture (ours)
✓
4.1
4.3
3.9
Appendix
Table 2: Mean Opinion Scores (MOS, 1–5, higher is better) from the user study on BEAT2 test clips. The Streaming column indicates whether the method supports zero–look-ahead streaming ( ✓ ) or is offline-only ( ✗ ). LiveGesture is the only strictly streaming model and is preferred on speech–gesture synchrony while remaining competitive in realness and smoothness.
Codebook size
FGD ( ↓ )
BC ( → )
Diversity ( ↑ )
512 entries
6.63
0.734
12.14
1024 entries
5.13
0.774
13.21
2048 entries
4.51
0.794
13.91
Appendix
Table 3: Effect of codebook size in the SVQ motion tokenizer on full-body gesture generation. Larger codebooks increase representational capacity and yield consistent gains across all metrics.
Method
FGD ( ↓ )
BC ( → )
Diversity ( ↑ )
Causal self-attention
4.63
0.784
12.53
Causal cross-attention
4.57
0.794
13.91
Appendix
Table 4: Comparison of causal self-attention and causal audio–motion cross-attention in the Hierarchical Autoregressive Transformer (HAR). All models maintain strictly aligned tokenization rates between audio tokens {at} and motion tokens {xtr} for fair comparison.
γ
FGD ↓
BC →
Div. ↑
1.00
4.61
0.781
12.90
1.25
4.57
0.794
13.91
1.35
5.23
0.763
12.80
2.00
6.42
0.756
11.74
Appendix
Table 5: Effect of classifier-free guidance scale γ on streaming gesture generation.
Method
FGD ↓
BC →
Div. ↑
Full-body SVQ (1 tokenizer)
6.84
0.753
11.23
Per-region SVQ (4 tokenizers)
4.57
0.794
13.91
Appendix
Table 6: Comparison between a single full-body SVQ tokenizer and compositional per-region SVQ tokenizers. Both variants use the same total codebook capacity (2048 entries with 128-d embeddings).
pdrop range
FGD ↓
BC →
Div. ↑
U(0,0.2)
4.57
0.794
13.91
U(0,0.3)
4.85
0.753
13.14
U(0,0.5)
5.30
0.770
12.82
Appendix
Table 7: Effect of random region masking (RM) on fusion training. pdrop∼U(0,a) denotes the range of probabilities used to fully mask a region’s token trajectory.
pnoise range
FGD ↓
BC →
Div. ↑
U(0,0.2)
4.57
0.794
13.91
U(0,0.3)
4.67
0.773
13.01
U(0,0.5)
5.30
0.760
12.63
Appendix
Table 8: Effect of noise injection in local region-eXperts during Stage 1 training. pnoise∼U(0,a) determines the probability of adding Gaussian noise to embedded history tokens.
Cross-attn layers
Temporal-attn layers
FGD ↓
BC →
Div. ↑
2
2
4.60
0.791
13.42
3
3
4.57
0.794
13.91
Appendix
Table 9: Effect of causal audio–motion cross-attention depth and causal temporal self-attention depth in the Region-eXpert Autoregressive Transformer (xAR).
Cross-attn layers
Spatio–temporal layers
FGD ↓
BC →
Div. ↑
2
2
4.65
0.784
13.35
3
3
4.57
0.794
13.91
Appendix
Table 10: Effect of causal audio–motion cross-attention depth and causal spatial–temporal attention depth in the fusion transformer (xAR-Fuse).
Generating natural co-speech gestures from streaming speech is essential for embodied conversational agents, where motion must be produced while a user is still speaking. Recent streaming gesture systems make online generation possible by autoregressing over discrete motion tokens, but this design compresses high-dimensional continuous motion into finite codebooks and can limit the realism and diversity of generated gestures. To preserve both causality and continuous expressiveness, we propose GestureFAR, a flow-autoregressive framework for streaming co-speech gesture generation. First, GestureFAR autoregresses over causal continuous motion latents, using a transformer to model streaming audio-motion context and a per-token flow-matching head to sample the next latent from a continuous distribution. Second, we introduce a head-only flow distillation strategy that freezes the causal backbone and distills the multi-step per-token flow head into a single network evaluation using consistency and distribution-matching objectives. This keeps the model token-causal while removing the main latency bottleneck for live interaction. Experiments on BEAT2 show that GestureFAR significantly improves the quality--latency trade-off among streaming-capable methods, preserving strong gesture quality while enabling real-time token-causal generation. Project Page: https://andypinxinliu.github.io/GestureFAR
Pinxin Liu, Haiyang Liu, Jiahao Luo +3
University of Rochester · University of Tokyo · University of California Santa Cruz +2
Real-time co-speech gesture generation must produce 3D motion clip by clip as speech arrives. Existing streaming methods are open-loop: each clip depends on past context, but the model cannot check or correct its trajectory. Small errors therefore accumulate and cause drift over long sequences. We observe that this failure is mainly caused by the lack of a forward constraint rather than poor short-clip quality. A plausible key pose at the end of each clip provides a destination anchor that limits drift. Based on this observation, we propose StreamTalk, a closed-loop framework with a periodic generate-retrieve-refine cycle. Streaming Pose-Guided Generation first predicts a coarse clip, retrieves a plausible tail pose from a speaker-specific motion database, and refines the clip using this pose before continuing to the next window. During training, Stochastic Anchor Masking randomly masks pose and translation frames, teaching the model to recover complete motion from sparse boundary conditions. A part-aware DiT separates hand, body, and translation streams to reduce interference between global displacement and local articulation. On BEAT2, StreamTalk achieves state-of-the-art FGD, reduces long-horizon drift relative to open-loop baselines, and runs in real time at 76 FPS. Project page: https://xiangyue-zhang.github.io/StreamTalk/.
Xiangyue Zhang, Jianfang Li, Jiaxu Zhang +2
The University of Tokyo · Alibaba Group · Nanyang Technological University +1
Co-speech gesture generation has made significant progress toward realistic full-body motion from speaker audio, yet existing models lack fine-grained spatial controllability of individual joints. To address this, we introduce InteractGesture, a model-agnostic, inference-time method for spatially controllable gesture generation. InteractGesture guides target latent estimates of a diffusion sampler through a differentiable RVQ-VAE decoder, backpropagating spatial control gradients to adjust motion latents during sampling. A primary challenge in streaming co-speech generation is chunk-wise dependency: standard sequential inference freezes prior chunks, preventing spatial constraints in future chunks from adjusting preceding trajectories and causing boundary inconsistencies. To overcome this limitation, we propose Progressive Chunk Guidance, a chunk-window strategy that maintains an active set of editable chunk latents with staggered delays, enabling spatial constraints to propagate gradients backward across chunk boundaries during streaming generation. Experiments on the BEAT2 dataset show that InteractGesture improves multi-joint spatial control while preserving overall gesture quality. Furthermore, our approach supports diverse applications, including sparse joint positioning, dense joint trajectory control, and directional pointing. Our project page is available at https://exitudio.github.io/interactgesture-page .
Ekkasit Pinyoanuntapong, Ajinkya Deogade, Paul Streli +5