InteractGesture: Progressive Chunk Guidance for Continuous Streaming Co-Speech Gesture Control

Ekkasit Pinyoanuntapong; Ajinkya Deogade; Paul Streli; Wenjing Zhang; Joanna Materzynska; Pu Wang; Vittorio Ferrari; Jie Shen

cs.CVAug 26, 2026

InteractGesture: Progressive Chunk Guidance for Continuous Streaming Co-Speech Gesture Control

Authors: Ekkasit Pinyoanuntapong, Ajinkya Deogade, Paul Streli, Wenjing Zhang, Joanna Materzynska, Pu Wang, Vittorio Ferrari, Jie Shen

Organizations: Meta · University of North Carolina at Charlotte

Abstract

Co-speech gesture generation has made significant progress toward realistic full-body motion from speaker audio, yet existing models lack fine-grained spatial controllability of individual joints. To address this, we introduce InteractGesture, a model-agnostic, inference-time method for spatially controllable gesture generation. InteractGesture guides target latent estimates of a diffusion sampler through a differentiable RVQ-VAE decoder, backpropagating spatial control gradients to adjust motion latents during sampling. A primary challenge in streaming co-speech generation is chunk-wise dependency: standard sequential inference freezes prior chunks, preventing spatial constraints in future chunks from adjusting preceding trajectories and causing boundary inconsistencies. To overcome this limitation, we propose Progressive Chunk Guidance, a chunk-window strategy that maintains an active set of editable chunk latents with staggered delays, enabling spatial constraints to propagate gradients backward across chunk boundaries during streaming generation. Experiments on the BEAT2 dataset show that InteractGesture improves multi-joint spatial control while preserving overall gesture quality. Furthermore, our approach supports diverse applications, including sparse joint positioning, dense joint trajectory control, and directional pointing. Our project page is available at https://exitudio.github.io/interactgesture-page .

Figures & tables

Figure 1: InteractGesture enables spatial gesture control for a speech-conditioned gesture generation model. (a) Speech Condition Only. (b) Joint Position Control: directs specified joints to point at described objects at designated frames. (c) Dense Joint Trajectory Control: guides joints along a continuous path for sophisticated gesture control. (d) Pointing Direction Control: orients the arm toward a target direction.

Figure 2: Comparison of chunk-guidance schedules. (a) Sequential Chunk Guidance: Fully completes and freezes chunk

k

before initializing chunk

k+1

. While causal, guidance from later chunks cannot adjust earlier motion. (b) Synchronous Chunk Guidance: Denoises all chunks simultaneously. Controls across the entire sequence influence overlapping context, but this requires full-sequence audio offline. (c) Progressive Chunk Guidance (Ours): Introduces chunks with a staggered delay and continuously refreshes context frames from intermediate target estimates

\hat{\mathbf{x}}_{0}

. This maintains streaming compatibility while propagating spatial control across chunk boundaries.

Figure 3: InteractGesture adds spatial control guidance to a pretrained co-speech gesture generation model at inference time. At every selected sampling step, the predicted target latent estimate is decoded into SMPL-X joints, a masked joint loss is minimized, and the edited target latent estimate is converted back into the sampler update used by the generator. The same mechanism supports absolute 3D targets, root-relative targets, and Progressive Chunk Guidance for continuous streaming generation from overlapping latent chunks.

Method	FGD ( $\downarrow$ )	BC ( $\rightarrow$ )	Diversity ( $\rightarrow$ )	Traj. Error ( $\downarrow$ )	Loc. Error ( $\downarrow$ )	Avg. Err. cm ( $\downarrow$ )
Ground-Truth	–	0.703	11.970	–	–	–
Inverse Kinematic	1.043	0.723	16.379	0.582	0.453	19.973
Chunk-Wise ControlNet	0.491	0.778	12.034	0.996	0.976	46.159
Sequential Chunk Guidance	0.690	0.816	13.845	0.574	0.470	11.701
Synchronous Chunk Guidance (Ours)	0.442	0.750	12.234	0.092	0.100	4.669
Progressive Chunk Guidance (Ours)	0.431	0.762	11.760	0.267	0.161	6.335

Table 1: Main spatial control comparison on BEAT2 evaluation. Lower is better for FGD, trajectory/location errors, and average control error. For metrics marked with

\rightarrow

, closer to Ground-Truth is better. Red bold marks the best generated method in each column, and blue underlined marks the second best.

Figure 4: Main qualitative result. InteractGesture uses spatial controls to steer co-speech gestures toward target hand and arm positions while preserving the speech-conditioned gesture prior. The visualization shows the guided generations alongside target positions, demonstrating whether the generated wrists and elbows adhere to the requested spatial constraints.

Setting	Scale	Sampling Guidance	Post Refine	FGD ( $\downarrow$ )	BC ( $\rightarrow$ )	Div ( $\rightarrow$ )	Traj10 ( $\downarrow$ )	Loc10 ( $\downarrow$ )	Avg. cm ( $\downarrow$ )
(a) Control Component Ablation
Delay 0: Synchronous Chunk Guidance
Uncontrolled Baseline	0.5	$\times$	$\times$	0.439	0.724	12.243	0.984	0.965	59.308
Sampling Guidance only	0.5	$\checkmark$	$\times$	0.438	0.744	12.175	0.426	0.427	11.471
Post-Refinement only	0.5	$\times$	$\checkmark$	0.438	0.743	12.195	0.338	0.338	9.428
Full Guidance (Ours)	0.5	$\checkmark$	$\checkmark$	0.442	0.750	12.234	0.092	0.100	4.669

Table 2: Ablation study on spatial guidance components and optimization scales on BEAT2. Panel (a) evaluates guidance modules at scale

0.5

. Panel (b) evaluates guidance scale sweeps. Delay

0

corresponds to Synchronous Chunk Guidance. Delay

1

corresponds to Progressive Chunk Guidance (

\Delta i=1

Co-Speech Gesture Generation

Co-speech gesture generation creates body movements that express and synchronize with spoken language.

See all 49 papers

GestureFAR: Streaming Co-Speech Gesture Generation with Flow Autoregression
Sep 18, 2026Pinxin Liu, Haiyang Liu, Jiahao Luo +3Co-Speech Gesture Generation Flow-Based Generative Modeling
University of Rochester · University of Tokyo · University of California Santa Cruz +2
StreamTalk: Streaming Co-Speech Gesture Generation with Key-Pose Anchoring
Aug 3, 2026Xiangyue Zhang, Jianfang Li, Jiaxu Zhang +2Human Motion Generation Co-Speech Gesture Generation
The University of Tokyo · Alibaba Group · Nanyang Technological University +1
LiveGesture Streamable Co-Speech Gesture Generation Model
Apr 13, 2026Muhammad Usama Saleem, Mayur Jagdishbhai Patel, Ekkasit Pinyoanuntapong +6Human Motion Generation Co-Speech Gesture Generation
University of North Carolina · University of Central Florida

Sep 18, 2026J/K move · Enter open · S save

GestureFAR: Streaming Co-Speech Gesture Generation with Flow Autoregression

Pinxin Liu, Haiyang Liu, Jiahao Luo, Junhua Huang+2

University of Rochester · University of Tokyo · University of California Santa Cruz+2

Co-Speech Gesture Generation Flow-Based Generative Modeling

Open paper AskPDF arXiv

Generating natural co-speech gestures from streaming speech is essential for embodied conversational agents, where motion must be produced while a user is still speaking. Recent streaming gesture systems make online generation possible by autoregressing over discrete motion tokens, but this design compresses high-dimensional continuous motion into finite codebooks and can limit the realism and diversity of generated gestures. To preserve both causality and continuous expressiveness, we propose GestureFAR, a flow-autoregressive framework for streaming co-speech gesture generation. First, GestureFAR autoregresses over causal continuous motion latents, using a transformer to model streaming audio-motion context and a per-token flow-matching head to sample the next latent from a continuous distribution. Second, we introduce a head-only flow distillation strategy that freezes the causal backbone and distills the multi-step per-token flow head into a single network evaluation using consistency and distribution-matching objectives. This keeps the model token-causal while removing the main latency bottleneck for live interaction. Experiments on BEAT2 show that GestureFAR significantly improves the quality--latency trade-off among streaming-capable methods, preserving strong gesture quality while enabling real-time token-causal generation. Project Page: https://andypinxinliu.github.io/GestureFAR

Abstract

Co-Speech Gesture Generation

Explore similar work

GestureFAR: Streaming Co-Speech Gesture Generation with Flow Autoregression

StreamTalk: Streaming Co-Speech Gesture Generation with Key-Pose Anchoring

LiveGesture Streamable Co-Speech Gesture Generation Model