MIBURI: Towards Expressive Interactive Gesture Synthesis

M. Hamza Mughal; Rishabh Dabral; Vera Demberg; Christian Theobalt

cs.CVMar 3, 2026

MIBURI: Towards Expressive Interactive Gesture Synthesis

Authors: M. Hamza Mughal, Rishabh Dabral, Vera Demberg, Christian Theobalt

Organizations: Max Planck Institute for Informatics, SIC · Saarland University

Abstract

Embodied Conversational Agents (ECAs) aim to emulate human face-to-face interaction through speech, gestures, and facial expressions. Current large language model (LLM)-based conversational agents lack embodiment and the expressive gestures essential for natural interaction. Existing solutions for ECAs often produce rigid, low-diversity motions, that are unsuitable for human-like interaction. Alternatively, generative methods for co-speech gesture synthesis yield natural body gestures but depend on future speech context and require long run-times. To bridge this gap, we present MIBURI, the first online, causal framework for generating expressive full-body gestures and facial expressions synchronized with real-time spoken dialogue. We employ body-part aware gesture codecs that encode hierarchical motion details into multi-level discrete tokens. These tokens are then autoregressively generated by a two-dimensional causal framework conditioned on LLM-based speech-text embeddings, modeling both temporal dynamics and part-level motion hierarchy in real time. Further, we introduce auxiliary objectives to encourage expressive and diverse gestures while preventing convergence to static poses. Comparative evaluations demonstrate that our causal and real-time approach produces natural and contextually aligned gestures against recent baselines. We urge the reader to explore demo videos on https://vcai.mpi-inf.mpg.de/projects/MIBURI/.

Figures & tables

Figure 2: Overview. Existing solutions [ 39 , 9 ] to animate ECAs involve a complex pipeline (above) of multiple components to generate gestures with speech. Miburi (below) generates full body co-speech gestures directly by utilizing internal semantic/acoustic tokens of speech-text foundation model [ 12 ] .

Method	Approach	Expressive	Causal	Real-time
Cassell et. al. [ 6 , 8 ]	Rule-based	✗	✓	✓
DigitalEinstein [ 52 ]	Rule-based	✗	✓	✓
Gesturebot [ 39 ]	Autoregressive	✗	✗	✓
EMAGE [ 28 ]	Masked Gesture Modelling	✓	✗	✗
ConvoFusion [ 38 ]	Diffusion	✓	✗	✗
Audio2Photoreal [ 41 ]	VQ+Diffusion	✓	✗	✗

Table 1: Approaches for offline Gesture Synthesis and ECAs

Figure 3: Miburi Architecture. Given Moshi’s speech/text tokens( Sec. 3.1 ), our approach generates a sequence of gesture tokens, which are obtained through Body-part aware Gesture Codecs( Sec. 3.2 ). This online framework takes in Moshi’s text/speech token as input and predict gesture tokens through autoregressive temporal and kinematic transformers( Sec. 3.3 ).

Figure 4: User Study for Perceptual Evaluation. Here, the red line indicates chance level (50%), * stands for

p<0.05

and *** for

p<0.001

	Multiple speakers (23)
	FGD $\downarrow$	BeatAlign $\rightarrow$	L1-Div $\rightarrow$	Facial-MSE $\downarrow$
GT		$0.446$	$8.45$
CaMN	$0.736$	$0.176$	$6.73$	–
EMAGE*	$0.850$	$0.236$	$6.58$	$4.6$
RAG-Gesture	$0.515$	$0.648$	$10.09$	–
GestureLSM	$0.537$	$0.481$	$8.41$	–

Table 2: Multi-speaker evaluation. Facial-MSE scaled by

10^{-8}

. * refers to retrained methods.

	Single-speaker ( Scott )
	FGD $\downarrow$	BeatAlign $\rightarrow$	L1-Div $\rightarrow$	Facial-MSE $\downarrow$
GT		$0.749$	$13.22$	–
CaMN	$0.969$	$0.698$	$10.61$	–
EMAGE	$0.552$	$0.795$	$13.06$	$7.68$
RAG-Gesture	$0.879$	$0.730$	$12.62$	–
GestureLSM	$0.410$	$0.719$	$13.42$	–

Table 3: Single-speaker evaluation. Facial-MSE scaled by

10^{-8}

	Causal	$\text{Latency}_{\text{A100}}\!\downarrow$	#Frames / Step
GestureLSM (8 steps)	✗	$0.1447\pm 0.0034$	$124$
EMAGE	✗	$0.0374\pm 0.0004$	$60$
MambaTalk	✗	$0.0529\pm 0.0039$	$60$
Miburi (ours)	✓	$0.0349\pm 0.0017$	$2$

Table 4: Latency and Causality Comparison. Wall-clock time is measured from the beginning of the forward pass to the conversion of outputs into SMPL-X parameters. Render times are excluded here. #Frames / Step indicates the number of frames generated per forward pass.

	$\text{FGD}\!\downarrow$	$\text{BeatAlign}\!\rightarrow$	$\text{L1-Div}\!\rightarrow$
GT	–	0.446	8.45
Miburi -L (+Face) w/ wav2vec	0.595	0.404	7.92
Miburi (+Face) w/ wav2vec	0.665	0.363	7.07
Miburi -L (+Face)	0.582	0.434	9.31
Miburi (+Face)	0.480	0.461	10.44

Table 5: Wav2vec ablation against moshi features.

	$\text{FGD}\!\downarrow$	$\text{BeatAlign}\!\rightarrow$	$\text{L1-Div}\!\rightarrow$	$\text{Step Time (s)}\!\downarrow$
GT		0.446	8.45
Single Transformer	1.256	0.731	5.48	0.096
Ours	0.480	0.461	10.44	0.035

Table 6: Comparison of Model Variants on Gesture Generation and Runtime.

	$\text{FGD}\!\downarrow$	$\text{BeatAlign}\!\rightarrow$	$\text{L1-Div}\!\rightarrow$
GT		0.446	8.45
$\mathcal{L}_{CE}+\mathcal{L}_{\text{va}}$	0.499	0.450	10.25
with MSE-loss	0.577	0.438	9.79
with $\mathcal{L}_{\text{con}}$	0.480	0.461	10.44

Table 7: Quantitative Effect of Losses on Generation.

	$\text{FGD}\!\downarrow$	$\text{MPJPE (m)}\!\downarrow$
$K{=}1$	0.55	0.043
$K{=}2$	0.42	0.032
$K{=}4$	0.135	0.022
$K{=}8$	0.059	0.016

Table 8: Effect of Number of Codebooks

K

on Motion Reconstruction. MPJPE is represented in meters.

Figure 5: System architecture of our real-time demo. The main inference process runs Moshi and Miburi in a continuous loop, while two parallel processes handle speech/text visualization and motion rendering. Data is streamed between processes at each timestep via websockets to support low-latency, full-duplex interaction. Right: the user-facing interface of the demo.

	$\text{FGD}\!\downarrow$	$\text{BeatAlign}\!\rightarrow$	$\text{L1-Div}\!\rightarrow$
GT	–	0.453	5.97
EMAGE [ 28 ]	3.786	0.022	1.79
GestureLSM [ 30 ]	3.744	0.776	13.75
Miburi	1.642	0.605	10.18

Table 9: Quantitative evaluation on the Embody3D dataset.

Figure 6: Kinematic Dependency Analysis. Here, “

\rightarrow

” means “attends to”.

Facial Expression Synthesis

See all 29 papers

GestureFAR: Streaming Co-Speech Gesture Generation with Flow Autoregression
Sep 18, 2026Pinxin Liu, Haiyang Liu, Jiahao Luo +3Co-Speech Gesture Generation Flow-Based Generative Modeling
University of Rochester · University of Tokyo · University of California Santa Cruz +2
LiveGesture Streamable Co-Speech Gesture Generation Model
Apr 13, 2026Muhammad Usama Saleem, Mayur Jagdishbhai Patel, Ekkasit Pinyoanuntapong +6Human Motion Generation Co-Speech Gesture Generation
University of North Carolina · University of Central Florida
Talk, Render, Act: Integrating Social Gesture and Digital Face with Synchronized Speech for Conversational Humanoid Robot
Oct 5, 2026Jin Jiang, Kun Li, Jiancong Ma +1Robot Motion Generation Co-Speech Gesture Generation
College of Computing and Artificial Intelligence (CCAI), United Arab Emirates University (UAEU), Al Ain, Abu Dhabi, United Arab Emirates

Sep 18, 2026J/K move · Enter open · S save

GestureFAR: Streaming Co-Speech Gesture Generation with Flow Autoregression

Pinxin Liu, Haiyang Liu, Jiahao Luo, Junhua Huang+2

University of Rochester · University of Tokyo · University of California Santa Cruz+2

Co-Speech Gesture Generation Flow-Based Generative Modeling

Open paper AskPDF arXiv

Generating natural co-speech gestures from streaming speech is essential for embodied conversational agents, where motion must be produced while a user is still speaking. Recent streaming gesture systems make online generation possible by autoregressing over discrete motion tokens, but this design compresses high-dimensional continuous motion into finite codebooks and can limit the realism and diversity of generated gestures. To preserve both causality and continuous expressiveness, we propose GestureFAR, a flow-autoregressive framework for streaming co-speech gesture generation. First, GestureFAR autoregresses over causal continuous motion latents, using a transformer to model streaming audio-motion context and a per-token flow-matching head to sample the next latent from a continuous distribution. Second, we introduce a head-only flow distillation strategy that freezes the causal backbone and distills the multi-step per-token flow head into a single network evaluation using consistency and distribution-matching objectives. This keeps the model token-causal while removing the main latency bottleneck for live interaction. Experiments on BEAT2 show that GestureFAR significantly improves the quality--latency trade-off among streaming-capable methods, preserving strong gesture quality while enabling real-time token-causal generation. Project Page: https://andypinxinliu.github.io/GestureFAR

Abstract

Facial Expression Synthesis

Explore similar work

GestureFAR: Streaming Co-Speech Gesture Generation with Flow Autoregression

LiveGesture Streamable Co-Speech Gesture Generation Model

Talk, Render, Act: Integrating Social Gesture and Digital Face with Synchronized Speech for Conversational Humanoid Robot