LiveGesture Streamable Co-Speech Gesture Generation Model

Muhammad Usama Saleem; Mayur Jagdishbhai Patel; Ekkasit Pinyoanuntapong; Zhongxing Qin; Li Yang; Hongfei Xue; Ahmed Helmy; Chen Chen

cs.CVApr 13, 2026

LiveGesture Streamable Co-Speech Gesture Generation Model

Authors: Muhammad Usama Saleem, Mayur Jagdishbhai Patel, Ekkasit Pinyoanuntapong, Zhongxing Qin, Li Yang, Hongfei Xue, Ahmed Helmy, Chen Chen,

Organizations: University of North Carolina · University of Central Florida

Abstract

We propose LiveGesture, the first fully streamable, speech-driven full-body gesture generation framework that operates with zero look-ahead and supports arbitrary sequence length. Unlike existing co-speech gesture methods, which are designed for offline generation and either treat body regions independently or entangle all joints within a single model, LiveGesture is built from the ground up for causal, region-coordinated motion generation. LiveGesture consists of two main modules: the Streamable Vector Quantized Motion Tokenizer (SVQ) and the Hierarchical Autoregressive Transformer (HAR). The SVQ tokenizer converts the motion sequence of each body region into causal, discrete motion tokens, enabling real-time, streamable token decoding. On top of SVQ, HAR employs region-expert autoregressive (xAR) transformers to model expressive, fine-grained motion dynamics for each body region. A causal spatio-temporal fusion module (xAR Fusion) then captures and integrates correlated motion dynamics across regions. Both xAR and xAR Fusion are conditioned on live, continuously arriving audio signals encoded by a streamable causal audio encoder. To enhance robustness under streaming noise and prediction errors, we introduce autoregressive masking training, which leverages uncertainty-guided token masking and random region masking to expose the model to imperfect, partially erroneous histories during training. Experiments on the BEAT2 dataset demonstrate that LiveGesture produces coherent, diverse, and beat-synchronous full-body gestures in real time, matching or surpassing state-of-the-art offline methods under true zero look-ahead conditions.

Figures & tables

Figure 2: Overview of the Streamable Asymmetric Motion Tokenizer. (a) For each body region, a bidirectional encoder and causal stream decoder learn low-rate latent sequences. (b) Freezing this autoencoder, the Streamable VQ-VAE (SVQ) adds a region-specific codebook and projection head to quantize latents into discrete, time-synchronous motion tokens that remain compatible with strictly causal decoding for streaming.

Figure 3: Overview of the Hierarchical Autoregressive Model in LiveGesture . A streamable audio encoder and optional text encoder provide causal audio/text tokens to four local AR Region-eXperts (upper body, lower body, hands, face), each modeling its own SVQ motion token stream. Their frozen states are adapted by per-region Pre-Infusion Adapters (PILOR) and fused by xAR-Fuse, a causal spatial–temporal transformer with audio–motion cross-attention, global temporal attention, and inter-region spatial attention that predicts next-step SVQ tokens for zero–look-ahead streaming full-body gesture generation.

Figure 4: Qualitative comparison with state-of-the-art methods on BEAT. LiveGesture generates more diverse, expressive full-body gestures that better follow speech rhythm, while red circles mark representative failure cases of prior methods.

Methods	Venue	Streaming	FGD ( $\downarrow$ )	BC ( $\rightarrow$ )	Diversity ( $\uparrow$ )	MSE ( $\downarrow$ )
Offline Solutions
HA2G [ 24 ]	CVPR’22	✗	12.32	0.677	8.626	–
DisCo [ 16 ]	MM’22	✗	9.417	0.643	9.912	–
CaMN [ 19 ]	ECCV’22	✗	6.644	0.676	10.86	–
TalkShow [ 45 ]	CVPR’23	✗	6.209	0.695	13.47	7.791
DiffSHEG [ 4 ]	CVPR’24	✗	7.141	0.743	8.21	9.571

Table 1: State-of-the-art comparison on BEAT. Best results are shown in bold and second best are underlined . The Streaming column indicates whether the method supports zero-look-ahead streaming ( ✓ ) or is offline-only ( ✗ ); Our method is the only streaming model while remaining superior in most of important metrics.

Figure 5: Interactive human–avatar conversation enabled by LiveGesture . User speech is converted into a spoken reply by VITA-Audio, while our LiveGesture streaming gesture model simultaneously generates synchronized full-body SMPL-X motions from live audio, allowing the avatar to respond in real time.

Table 2: Ablation studies of LiveGesture design choices, including core components, architecture, tokenization, audio encoders, loss weights, and UGR.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Methods	Venue	Streaming	FGD ( $\downarrow$ )	BC ( $\rightarrow$ )	Diversity ( $\uparrow$ )
Offline Solutions
GestureLSM [ 22 ]	ICCV’25	✗	4.088	0.714	13.24
Streaming Solution (Ours)
LiveGesture (ours)	Ours	✓	4.51	0.783	13.31

Appendix

Table 1: Comparison with state-of-the-art methods on BEAT2 without the facial motion module. LiveGesture remains the only zero–look-ahead streaming model while achieving competitive or superior performance in BC and Diversity.

Method	Streaming	Realness $\uparrow$	Synchrony $\uparrow$	Smoothness $\uparrow$
CaMN [ 19 ]	✗	3.3	3.2	3.6
EMAGE [ 18 ]	✗	3.5	3.5	3.7
GestureLSM [ 22 ]	✗	4.2	4.1	4.3
LiveGesture (ours)	✓	4.1	4.3	3.9

Appendix

Table 2: Mean Opinion Scores (MOS, 1–5, higher is better) from the user study on BEAT2 test clips. The Streaming column indicates whether the method supports zero–look-ahead streaming ( ✓ ) or is offline-only ( ✗ ). LiveGesture is the only strictly streaming model and is preferred on speech–gesture synchrony while remaining competitive in realness and smoothness.

Codebook size	FGD ( $\downarrow$ )	BC ( $\rightarrow$ )	Diversity ( $\uparrow$ )
512 entries	6.63	0.734	12.14
1024 entries	5.13	0.774	13.21
2048 entries	4.51	0.794	13.91

Appendix

Table 3: Effect of codebook size in the SVQ motion tokenizer on full-body gesture generation. Larger codebooks increase representational capacity and yield consistent gains across all metrics.

Method	FGD ( $\downarrow$ )	BC ( $\rightarrow$ )	Diversity ( $\uparrow$ )
Causal self-attention	4.63	0.784	12.53
Causal cross-attention	4.57	0.794	13.91

Appendix

Table 4: Comparison of causal self-attention and causal audio–motion cross-attention in the Hierarchical Autoregressive Transformer (HAR). All models maintain strictly aligned tokenization rates between audio tokens

\{a_{t}\}

and motion tokens

\{x^{r}_{t}\}

for fair comparison.

$\gamma$	FGD $\downarrow$	BC $\rightarrow$	Div. $\uparrow$
1.00	4.61	0.781	12.90
1.25	4.57	0.794	13.91
1.35	5.23	0.763	12.80
2.00	6.42	0.756	11.74

Appendix

Table 5: Effect of classifier-free guidance scale

\gamma

on streaming gesture generation.

Method	FGD $\downarrow$	BC $\rightarrow$	Div. $\uparrow$
Full-body SVQ (1 tokenizer)	6.84	0.753	11.23
Per-region SVQ (4 tokenizers)	4.57	0.794	13.91

Appendix

Table 6: Comparison between a single full-body SVQ tokenizer and compositional per-region SVQ tokenizers. Both variants use the same total codebook capacity (2048 entries with 128-d embeddings).

$p_{\text{drop}}$ range	FGD $\downarrow$	BC $\rightarrow$	Div. $\uparrow$
$\mathcal{U}(0,0.2)$	4.57	0.794	13.91
$\mathcal{U}(0,0.3)$	4.85	0.753	13.14
$\mathcal{U}(0,0.5)$	5.30	0.770	12.82

Appendix

Table 7: Effect of random region masking (RM) on fusion training.

p_{\text{drop}}\sim\mathcal{U}(0,a)

denotes the range of probabilities used to fully mask a region’s token trajectory.

$p_{\text{noise}}$ range	FGD $\downarrow$	BC $\rightarrow$	Div. $\uparrow$
$\mathcal{U}(0,0.2)$	4.57	0.794	13.91
$\mathcal{U}(0,0.3)$	4.67	0.773	13.01
$\mathcal{U}(0,0.5)$	5.30	0.760	12.63

Appendix

Table 8: Effect of noise injection in local region-eXperts during Stage 1 training.

p_{\text{noise}}\sim\mathcal{U}(0,a)

determines the probability of adding Gaussian noise to embedded history tokens.

Cross-attn layers	Temporal-attn layers	FGD $\downarrow$	BC $\rightarrow$	Div. $\uparrow$
2	2	4.60	0.791	13.42
3	3	4.57	0.794	13.91

Appendix

Table 9: Effect of causal audio–motion cross-attention depth and causal temporal self-attention depth in the Region-eXpert Autoregressive Transformer (xAR).

Cross-attn layers	Spatio–temporal layers	FGD $\downarrow$	BC $\rightarrow$	Div. $\uparrow$
2	2	4.65	0.784	13.35
3	3	4.57	0.794	13.91

Appendix

Table 10: Effect of causal audio–motion cross-attention depth and causal spatial–temporal attention depth in the fusion transformer (xAR-Fuse).

Human Motion Generation

Human motion generation creates articulated movement from language, audio, video, spatial constraints, or interaction goals.

See all 284 papers

GestureFAR: Streaming Co-Speech Gesture Generation with Flow Autoregression
Sep 18, 2026Pinxin Liu, Haiyang Liu, Jiahao Luo +3Co-Speech Gesture Generation Flow-Based Generative Modeling
University of Rochester · University of Tokyo · University of California Santa Cruz +2
StreamTalk: Streaming Co-Speech Gesture Generation with Key-Pose Anchoring
Aug 3, 2026Xiangyue Zhang, Jianfang Li, Jiaxu Zhang +2Human Motion Generation Co-Speech Gesture Generation
The University of Tokyo · Alibaba Group · Nanyang Technological University +1
InteractGesture: Progressive Chunk Guidance for Continuous Streaming Co-Speech Gesture Control
Aug 26, 2026Ekkasit Pinyoanuntapong, Ajinkya Deogade, Paul Streli +5Co-Speech Gesture Generation Human Motion Generation
Meta · University of North Carolina at Charlotte

Sep 18, 2026J/K move · Enter open · S save

GestureFAR: Streaming Co-Speech Gesture Generation with Flow Autoregression

Pinxin Liu, Haiyang Liu, Jiahao Luo, Junhua Huang+2

University of Rochester · University of Tokyo · University of California Santa Cruz+2

Co-Speech Gesture Generation Flow-Based Generative Modeling

Open paper AskPDF arXiv

Generating natural co-speech gestures from streaming speech is essential for embodied conversational agents, where motion must be produced while a user is still speaking. Recent streaming gesture systems make online generation possible by autoregressing over discrete motion tokens, but this design compresses high-dimensional continuous motion into finite codebooks and can limit the realism and diversity of generated gestures. To preserve both causality and continuous expressiveness, we propose GestureFAR, a flow-autoregressive framework for streaming co-speech gesture generation. First, GestureFAR autoregresses over causal continuous motion latents, using a transformer to model streaming audio-motion context and a per-token flow-matching head to sample the next latent from a continuous distribution. Second, we introduce a head-only flow distillation strategy that freezes the causal backbone and distills the multi-step per-token flow head into a single network evaluation using consistency and distribution-matching objectives. This keeps the model token-causal while removing the main latency bottleneck for live interaction. Experiments on BEAT2 show that GestureFAR significantly improves the quality--latency trade-off among streaming-capable methods, preserving strong gesture quality while enabling real-time token-causal generation. Project Page: https://andypinxinliu.github.io/GestureFAR

Abstract

Human Motion Generation

Explore similar work

GestureFAR: Streaming Co-Speech Gesture Generation with Flow Autoregression

StreamTalk: Streaming Co-Speech Gesture Generation with Key-Pose Anchoring

InteractGesture: Progressive Chunk Guidance for Continuous Streaming Co-Speech Gesture Control