GestureFAR: Streaming Co-Speech Gesture Generation with Flow Autoregression

Pinxin Liu; Haiyang Liu; Jiahao Luo; Junhua Huang; Chunhao Zou; Luchuan Song

cs.CVSep 18, 2026

GestureFAR: Streaming Co-Speech Gesture Generation with Flow Autoregression

Authors: Pinxin Liu, Haiyang Liu, Jiahao Luo, Junhua Huang, Chunhao Zou, Luchuan Song

Organizations: University of Rochester · University of Tokyo · University of California Santa Cruz · University of California Los Angeles ·

Abstract

Generating natural co-speech gestures from streaming speech is essential for embodied conversational agents, where motion must be produced while a user is still speaking. Recent streaming gesture systems make online generation possible by autoregressing over discrete motion tokens, but this design compresses high-dimensional continuous motion into finite codebooks and can limit the realism and diversity of generated gestures. To preserve both causality and continuous expressiveness, we propose GestureFAR, a flow-autoregressive framework for streaming co-speech gesture generation. First, GestureFAR autoregresses over causal continuous motion latents, using a transformer to model streaming audio-motion context and a per-token flow-matching head to sample the next latent from a continuous distribution. Second, we introduce a head-only flow distillation strategy that freezes the causal backbone and distills the multi-step per-token flow head into a single network evaluation using consistency and distribution-matching objectives. This keeps the model token-causal while removing the main latency bottleneck for live interaction. Experiments on BEAT2 show that GestureFAR significantly improves the quality--latency trade-off among streaming-capable methods, preserving strong gesture quality while enabling real-time token-causal generation. Project Page: https://andypinxinliu.github.io/GestureFAR

Figures & tables

Figure 2: GestureFAR student lies in the lower-left region, achieving the best FGD with the lowest generation cost while remaining streamable and token-causal.

Figure 3: Overview of GestureFAR. GestureFAR generates co-speech gestures from streaming audio by autoregressing over continuous motion latents. A causal motion VAE maps full-body motion into a streamable latent representation. A streaming audio-motion transformer predicts the condition for each next latent using past motion and currently available audio. A lightweight per-token flow head samples the next continuous latent from this condition. For real-time deployment, we distill only the flow head into a one-step student while keeping the transformer frozen.

Figure 4: Stateful streaming deployment. GestureFAR is deployed with persistent model caches, body anchors, a short motion buffer, and motion-space blending. The caches make chunked generation incremental, body anchors prevent translation drift and re-centering artifacts, and the buffer decouples bursty backend generation from fixed-rate frontend playback. Blending smooths transitions between speaking, listening, and idle states, enabling stable live avatar rendering.

Figure 5: Qualitative comparison on BEAT2. We visualize generated full-body gestures from different methods under the same speech inputs, with the semantically salient words highlighted in red. Compared with prior methods, our model produces gestures that are more natural, diverse, and better aligned with the speech semantics.

Method	Venue	Streaming	Token causal	FGD $\downarrow$	BC $\rightarrow$	Diversity $\rightarrow$	ms/token $\downarrow$
Offline / block-based solutions
CaMN ( Liu et al. 2022a )	ECCV’22	✗	✗	6.644	0.676	10.86	–
TalkSHOW ( Yi et al. 2023 )	CVPR’23	✗	✗	6.209	0.695	13.47	–
ProbTalk ( Liu et al. 2024b )	CVPR’24	✗	✗	5.040	0.771	13.27	–
EMAGE ( Liu et al. 2023 )	CVPR’24	✗	✗	5.512	0.772	13.06	–
MambaTalk ( Xu et al. 2024 )	NeurIPS’24	✗	✗	5.366	0.781	13.05	–

Table 1: State-of-the-art comparison on BEAT2. Offline and block-based baselines follow the LiveGesture comparison protocol. We additionally report whether each method supports streaming generation and strict token-causal emission. FGD is lower better. BC and Diversity are marked with

\rightarrow

because values closer to the ground-truth statistics are preferred. Runtime is reported as per-token generation cost when available.

Table 2: Ablation studies of GestureFAR design choices. We isolate representation, generator factorization, distillation objectives, flow-head capacity, distillation scope, and teacher sampling cost.

Figure 6: Our GestureFAR have higher user ratings with a clear margin on Naturalness , Synchrony , and Smoothness .

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Methods	FGD ( $\downarrow$ )	BC ( $\rightarrow$ )	Diversity ( $\rightarrow$ )
Ground-Truth	–	0.477	7.29
CaMN ( Liu et al. 2022a )	0.512	0.200	5.58
EMAGE ( Liu et al. 2023 )	0.692	0.284	6.06
GestureLSM ( Liu et al. 2025b )	0.466	0.525	9.23
MIBURI ( Mughal et al. 2026 )	0.480	0.461	10.44
GestureFAR	0.289	0.484	7.63

Appendix

Table 3: The quantitative results on BEAT-2 all speaker setting. We bold the best results.

Methods	FGD ( $\downarrow$ )	Diversity ( $\rightarrow$ )
Ground-Truth	–	2.50
EMAGE ( Liu et al. 2023 )	4.43	2.13
Audio2PhotoReal ( Ng et al. 2024 )	2.94	2.36
GestureLSM ( Liu et al. 2025b )	2.64	2.34
MIBURI ( Mughal et al. 2026 )	5.12	2.01
GestureFAR	2.08	2.51

Appendix

Table 4: The quantitative results on Audio2PhotoReal. We bold the best results.

Co-Speech Gesture Generation

Co-speech gesture generation creates body movements that express and synchronize with spoken language.

See all 49 papers

LiveGesture Streamable Co-Speech Gesture Generation Model
Apr 13, 2026Muhammad Usama Saleem, Mayur Jagdishbhai Patel, Ekkasit Pinyoanuntapong +6Human Motion Generation Co-Speech Gesture Generation
University of North Carolina · University of Central Florida
StreamTalk: Streaming Co-Speech Gesture Generation with Key-Pose Anchoring
Aug 3, 2026Xiangyue Zhang, Jianfang Li, Jiaxu Zhang +2Human Motion Generation Co-Speech Gesture Generation
The University of Tokyo · Alibaba Group · Nanyang Technological University +1
InteractGesture: Progressive Chunk Guidance for Continuous Streaming Co-Speech Gesture Control
Aug 26, 2026Ekkasit Pinyoanuntapong, Ajinkya Deogade, Paul Streli +5Co-Speech Gesture Generation Human Motion Generation
Meta · University of North Carolina at Charlotte

Apr 13, 2026J/K move · Enter open · S save

LiveGesture Streamable Co-Speech Gesture Generation Model

Muhammad Usama Saleem, Mayur Jagdishbhai Patel, Ekkasit Pinyoanuntapong, Zhongxing Qin+5

University of North Carolina · University of Central Florida

Human Motion Generation Co-Speech Gesture Generation Motion Representation Learning

Open paper AskPDF arXiv

We propose LiveGesture, the first fully streamable, speech-driven full-body gesture generation framework that operates with zero look-ahead and supports arbitrary sequence length. Unlike existing co-speech gesture methods, which are designed for offline generation and either treat body regions independently or entangle all joints within a single model, LiveGesture is built from the ground up for causal, region-coordinated motion generation. LiveGesture consists of two main modules: the Streamable Vector Quantized Motion Tokenizer (SVQ) and the Hierarchical Autoregressive Transformer (HAR). The SVQ tokenizer converts the motion sequence of each body region into causal, discrete motion tokens, enabling real-time, streamable token decoding. On top of SVQ, HAR employs region-expert autoregressive (xAR) transformers to model expressive, fine-grained motion dynamics for each body region. A causal spatio-temporal fusion module (xAR Fusion) then captures and integrates correlated motion dynamics across regions. Both xAR and xAR Fusion are conditioned on live, continuously arriving audio signals encoded by a streamable causal audio encoder. To enhance robustness under streaming noise and prediction errors, we introduce autoregressive masking training, which leverages uncertainty-guided token masking and random region masking to expose the model to imperfect, partially erroneous histories during training. Experiments on the BEAT2 dataset demonstrate that LiveGesture produces coherent, diverse, and beat-synchronous full-body gestures in real time, matching or surpassing state-of-the-art offline methods under true zero look-ahead conditions.

Abstract

Co-Speech Gesture Generation

Explore similar work

LiveGesture Streamable Co-Speech Gesture Generation Model

StreamTalk: Streaming Co-Speech Gesture Generation with Key-Pose Anchoring

InteractGesture: Progressive Chunk Guidance for Continuous Streaming Co-Speech Gesture Control