Generating natural co-speech gestures from streaming speech is essential for embodied conversational agents, where motion must be produced while a user is still speaking. Recent streaming gesture systems make online generation possible by autoregressing over discrete motion tokens, but this design compresses high-dimensional continuous motion into finite codebooks and can limit the realism and diversity of generated gestures. To preserve both causality and continuous expressiveness, we propose GestureFAR, a flow-autoregressive framework for streaming co-speech gesture generation. First, GestureFAR autoregresses over causal continuous motion latents, using a transformer to model streaming audio-motion context and a per-token flow-matching head to sample the next latent from a continuous distribution. Second, we introduce a head-only flow distillation strategy that freezes the causal backbone and distills the multi-step per-token flow head into a single network evaluation using consistency and distribution-matching objectives. This keeps the model token-causal while removing the main latency bottleneck for live interaction. Experiments on BEAT2 show that GestureFAR significantly improves the quality--latency trade-off among streaming-capable methods, preserving strong gesture quality while enabling real-time token-causal generation. Project Page: https://andypinxinliu.github.io/GestureFAR
Figures & tables
Figure 2: GestureFAR student lies in the lower-left region, achieving the best FGD with the lowest generation cost while remaining streamable and token-causal.
Figure 3: Overview of GestureFAR. GestureFAR generates co-speech gestures from streaming audio by autoregressing over continuous motion latents. A causal motion VAE maps full-body motion into a streamable latent representation. A streaming audio-motion transformer predicts the condition for each next latent using past motion and currently available audio. A lightweight per-token flow head samples the next continuous latent from this condition. For real-time deployment, we distill only the flow head into a one-step student while keeping the transformer frozen.
Figure 4: Stateful streaming deployment. GestureFAR is deployed with persistent model caches, body anchors, a short motion buffer, and motion-space blending. The caches make chunked generation incremental, body anchors prevent translation drift and re-centering artifacts, and the buffer decouples bursty backend generation from fixed-rate frontend playback. Blending smooths transitions between speaking, listening, and idle states, enabling stable live avatar rendering.
Figure 5: Qualitative comparison on BEAT2. We visualize generated full-body gestures from different methods under the same speech inputs, with the semantically salient words highlighted in red. Compared with prior methods, our model produces gestures that are more natural, diverse, and better aligned with the speech semantics.
Method
Venue
Streaming
Token causal
FGD ↓
BC →
Diversity →
ms/token ↓
Offline / block-based solutions
CaMN ( Liu et al. 2022a )
ECCV’22
✗
✗
6.644
0.676
10.86
–
TalkSHOW ( Yi et al. 2023 )
CVPR’23
✗
✗
6.209
0.695
13.47
–
ProbTalk ( Liu et al. 2024b )
CVPR’24
✗
✗
5.040
0.771
13.27
–
EMAGE ( Liu et al. 2023 )
CVPR’24
✗
✗
5.512
0.772
13.06
–
MambaTalk ( Xu et al. 2024 )
NeurIPS’24
✗
✗
5.366
0.781
13.05
–
Table 1: State-of-the-art comparison on BEAT2. Offline and block-based baselines follow the LiveGesture comparison protocol. We additionally report whether each method supports streaming generation and strict token-causal emission. FGD is lower better. BC and Diversity are marked with → because values closer to the ground-truth statistics are preferred. Runtime is reported as per-token generation cost when available.
Table 2: Ablation studies of GestureFAR design choices. We isolate representation, generator factorization, distillation objectives, flow-head capacity, distillation scope, and teacher sampling cost.
Figure 6: Our GestureFAR have higher user ratings with a clear margin on Naturalness , Synchrony , and Smoothness .
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Methods
FGD ( ↓ )
BC ( → )
Diversity ( → )
Ground-Truth
–
0.477
7.29
CaMN ( Liu et al. 2022a )
0.512
0.200
5.58
EMAGE ( Liu et al. 2023 )
0.692
0.284
6.06
GestureLSM ( Liu et al. 2025b )
0.466
0.525
9.23
MIBURI ( Mughal et al. 2026 )
0.480
0.461
10.44
GestureFAR
0.289
0.484
7.63
Appendix
Table 3: The quantitative results on BEAT-2 all speaker setting. We bold the best results.
Methods
FGD ( ↓ )
Diversity ( → )
Ground-Truth
–
2.50
EMAGE ( Liu et al. 2023 )
4.43
2.13
Audio2PhotoReal ( Ng et al. 2024 )
2.94
2.36
GestureLSM ( Liu et al. 2025b )
2.64
2.34
MIBURI ( Mughal et al. 2026 )
5.12
2.01
GestureFAR
2.08
2.51
Appendix
Table 4: The quantitative results on Audio2PhotoReal. We bold the best results.
We propose LiveGesture, the first fully streamable, speech-driven full-body gesture generation framework that operates with zero look-ahead and supports arbitrary sequence length. Unlike existing co-speech gesture methods, which are designed for offline generation and either treat body regions independently or entangle all joints within a single model, LiveGesture is built from the ground up for causal, region-coordinated motion generation. LiveGesture consists of two main modules: the Streamable Vector Quantized Motion Tokenizer (SVQ) and the Hierarchical Autoregressive Transformer (HAR). The SVQ tokenizer converts the motion sequence of each body region into causal, discrete motion tokens, enabling real-time, streamable token decoding. On top of SVQ, HAR employs region-expert autoregressive (xAR) transformers to model expressive, fine-grained motion dynamics for each body region. A causal spatio-temporal fusion module (xAR Fusion) then captures and integrates correlated motion dynamics across regions. Both xAR and xAR Fusion are conditioned on live, continuously arriving audio signals encoded by a streamable causal audio encoder. To enhance robustness under streaming noise and prediction errors, we introduce autoregressive masking training, which leverages uncertainty-guided token masking and random region masking to expose the model to imperfect, partially erroneous histories during training. Experiments on the BEAT2 dataset demonstrate that LiveGesture produces coherent, diverse, and beat-synchronous full-body gestures in real time, matching or surpassing state-of-the-art offline methods under true zero look-ahead conditions.
Muhammad Usama Saleem, Mayur Jagdishbhai Patel, Ekkasit Pinyoanuntapong +6
University of North Carolina · University of Central Florida
Real-time co-speech gesture generation must produce 3D motion clip by clip as speech arrives. Existing streaming methods are open-loop: each clip depends on past context, but the model cannot check or correct its trajectory. Small errors therefore accumulate and cause drift over long sequences. We observe that this failure is mainly caused by the lack of a forward constraint rather than poor short-clip quality. A plausible key pose at the end of each clip provides a destination anchor that limits drift. Based on this observation, we propose StreamTalk, a closed-loop framework with a periodic generate-retrieve-refine cycle. Streaming Pose-Guided Generation first predicts a coarse clip, retrieves a plausible tail pose from a speaker-specific motion database, and refines the clip using this pose before continuing to the next window. During training, Stochastic Anchor Masking randomly masks pose and translation frames, teaching the model to recover complete motion from sparse boundary conditions. A part-aware DiT separates hand, body, and translation streams to reduce interference between global displacement and local articulation. On BEAT2, StreamTalk achieves state-of-the-art FGD, reduces long-horizon drift relative to open-loop baselines, and runs in real time at 76 FPS. Project page: https://xiangyue-zhang.github.io/StreamTalk/.
Xiangyue Zhang, Jianfang Li, Jiaxu Zhang +2
The University of Tokyo · Alibaba Group · Nanyang Technological University +1
Co-speech gesture generation has made significant progress toward realistic full-body motion from speaker audio, yet existing models lack fine-grained spatial controllability of individual joints. To address this, we introduce InteractGesture, a model-agnostic, inference-time method for spatially controllable gesture generation. InteractGesture guides target latent estimates of a diffusion sampler through a differentiable RVQ-VAE decoder, backpropagating spatial control gradients to adjust motion latents during sampling. A primary challenge in streaming co-speech generation is chunk-wise dependency: standard sequential inference freezes prior chunks, preventing spatial constraints in future chunks from adjusting preceding trajectories and causing boundary inconsistencies. To overcome this limitation, we propose Progressive Chunk Guidance, a chunk-window strategy that maintains an active set of editable chunk latents with staggered delays, enabling spatial constraints to propagate gradients backward across chunk boundaries during streaming generation. Experiments on the BEAT2 dataset show that InteractGesture improves multi-joint spatial control while preserving overall gesture quality. Furthermore, our approach supports diverse applications, including sparse joint positioning, dense joint trajectory control, and directional pointing. Our project page is available at https://exitudio.github.io/interactgesture-page .
Ekkasit Pinyoanuntapong, Ajinkya Deogade, Paul Streli +5