As embodied agents become central to VR, telepresence, and digital human applications, their motion must go beyond speech-aligned gestures: agents should turn toward users, respond to their movement, and maintain natural gaze. Current methods lack this spatial awareness. We close this gap with the first real-time, fully causal method for spatially-aware conversational motion, deployable on a streaming VR headset. Given a user's position and dyadic audio, our approach produces full-body motion that aligns gestures with speech while orienting the agent according to the user. Our architecture combines a causal transformer-based VAE with interleaved latent tokens for streaming inference and a flow matching model conditioned on user trajectory and audio. To support varying gaze preferences, we introduce a gaze scoring mechanism with classifier-free guidance to decouple learning from control: the model captures natural spatial alignment from data, while users can adjust eye contact intensity at inference time. On the Embody 3D dataset, our method achieves state-of-the-art motion quality at over 300 FPS -- 3x faster than non-causal baselines -- while capturing the subtle spatial dynamics of natural conversation. We validate our approach on a live VR system, bringing spatially-aware conversational agents to real-time deployment. Please see https://evonneng.github.io/sarah/ for details.
Figures & tables
Figure 1. Our method generates full-body 3D motion for a virtual agent that is spatially aware of the user while engaging in a conversation. Given the user’s floor-projected head trajectory and dyadic audio, we generate the agent’s complete 3D motion. Trajectory colors indicate time: blue → green (user) and yellow → red (agent). See project page for results.
Figure 2. Given the user’s 3D position and dyadic conversational audio, our model generates 3D motion that is conversationally and spatially aware (left). We use a fully causal transformer-based VAE with interleaved latent tokens at a fixed temporal stride; both encoder and decoder employ causal attention, where each μ/σ token attends only to preceding frames and earlier latents (center). These latents are passed to a transformer-based flow matching model that also uses causal masking and optionally accepts a gaze score for controlling the agent’s eye contact (right). Our lightweight architecture enables real-time, autoregressive streaming without distillation.
Figure 3. We represent each joint j as a 3D icosahedron. The centroid of the vertices yields the global position Πj , and we recover the global orientation Ωj via SVD against a reference icosahedron.
Figure 4. Our training data spans a wide range of gaze behaviors, from sustained eye contact to complete gaze aversion (left). To enable controllable gaze at inference, we compute a gaze score g , where dx is the agent’s facing direction and dy points toward the user (right). The score approaches 1 when facing the user directly and −1 when facing away.
FPS
FGD ↓ (m ×101 )
FGD acc↓ ( ×105 )
Foot Slide ↓∈[0,1]
Wrist Var ↑
Head Ang. ↑∈[−1,1]
C
R
Avg
NS/S
Avg
NS/S
Avg
NS/S
Avg
NS/S
Avg
NS/S
GT
–
–
–
–
–
–
–
0.01
0.01/0.01
137.6
122.3/179.7
0.81
0.80/0.84
Baselines
Random
✗
✓
4K
1.06
0.30/0.28
1.83
0.31/3.38
0.01
0.01/0.01
188.1
190.5/181.6
0.28
0.27/0.32
NN
✗
✓
1K
0.90
0.19/0.16
0.77
0.02/0.51
0.01
0.01/0.01
97.0
85.7/128.2
0.59
0.57/0.64
MDM ( Tevet et al., 2022 )
✗
✗
90
3.48
1.93/2.66
2.88
0.64/5.37
0.11
0.11/0.11
61.4
57.9/71.0
0.81
0.80/0.84
A2P ( Ng et al., 2024 )
✗
✗
90
2.01
0.54/0.80
2.31
0.43/4.95
0.02
0.02/0.02
69.4
59.0/98.2
0.71
0.70/0.73
Table 1. Comparison with baselines and ablations (abl.) on 2048 test sequences. C = causal, R = real-time. S = speaking (544 seq.), NS = non-speaking (1504 seq.). ↑ higher is better, ↓ lower is better. † Reducible to 600 fps without quality degradation.
g
FGD ↓
FGD acc↓
Foot Slide ↓
Wrist Var ↑
Head Ang. ↑
∅
1.28
2.19
0.01
105.0
0.83
0.0
0.99
2.18
0.01
111.1
0.56
0.8
0.92
2.19
0.01
110.8
0.76
1.0
1.49
2.20
0.01
106.6
0.96
Table 2. Effect of gaze control on motion. ∅ denotes that gaze control is disabled.
Figure 5. We visualize the agent’s facing direction via projected lines (agent: yellow → red; user: blue → green). With no alignment g=∅ , the agent’s gaze is more diverse; as we increase g , the agent increasingly turns towards the user.
Figure 6. Sequences from our real-time demo system, rendered with a photorealistic avatar. The top row visualizes the user’s headset location as a silver sphere. The bottom row shows the generated avatar from the user’s (headset) viewpoint. Our method generates realistic conversational motion that is responsive to the user’s spatial motion. Full videos are available on our project page .
We present EmbodiedHead, a speech-driven talking head framework that equips LLMs with real-time visual avatars for conversation. A practical embodied avatar must achieve real-time generation, unified listening-speaking behavior, and high rendered visual quality simultaneously. Our framework couples the first Rectified-Flow Diffusion Transformer (DiT) for this task with a differentiable renderer, enabling diverse, high-fidelity generation in as few as four sampling steps. Prior listening-speaking methods rely on dual-stream audio, introducing an interlocutor look-ahead dependency incompatible with causal user--LLM interaction. We instead adopt a single-stream interface with explicit per-frame listening-speaking state conditioning and a Streaming Audio Scheduler, suppressing spurious mouth motion during listening while enabling seamless turn-taking. A two-stage training scheme of coefficient-space pretraining and joint image-domain refinement further closes the gap between motion-level supervision and rendered quality. Extensive experiments demonstrate state-of-the-art visual quality and motion fidelity in both speaking and listening scenarios. Project page: https://vpx-ecnu.github.io/EmbodiedHead-Website/.
Yu Zhang, Kaiyuan Shen, Yang Li
School of Computer Science and Technology, East China Normal University, Shanghai, China · Garabido Shanghai Technology Co., Ltd., Shanghai, China
Talking-head and dyadic models now achieve real-time inference, yet fast motion generation alone does not produce an interactive conversation. A live system must synchronize the model's output with speech from an external voice agent, schedule video frames for playback, and handle interruptions. We introduce AVTR-1, an open stack for real-time interactive avatar conversations, built around a compact 153M-parameter autoregressive flow-matching motion generator conditioned on both participants' audio. We adapt its audio encoder for streaming through self-distillation. The stack turns the model's chunk-based generation into a continuous, synchronized audio-video stream driven by an external voice agent, and we analytically derive its contribution to the user-facing latencies and validate the resulting bounds with two commercial voice agents. Further experiments demonstrate that AVTR-1 leads the compared dyadic systems on all reported visual-quality metrics and most conventional listening-motion metrics while remaining competitive in lip synchronization. Its inference runtime operates in real time on data-center and consumer GPUs. However, conventional listening metrics do not establish whether the paired speaker's speech contributes to generated motion. We therefore introduce the Reference-Based Directed Granger Gain (R-DGG), which measures the additional predictive information carried by speaker speech after accounting for listener history and speaker motion. R-DGG finds statistically supported predictive dependence for recorded listeners and all evaluated dyadic systems, but not for talking-head generators without paired audio or mismatched speaker-listener pairs. We release the model weights, renderer, and serving backend under component-specific licenses.
Generating natural co-speech gestures from streaming speech is essential for embodied conversational agents, where motion must be produced while a user is still speaking. Recent streaming gesture systems make online generation possible by autoregressing over discrete motion tokens, but this design compresses high-dimensional continuous motion into finite codebooks and can limit the realism and diversity of generated gestures. To preserve both causality and continuous expressiveness, we propose GestureFAR, a flow-autoregressive framework for streaming co-speech gesture generation. First, GestureFAR autoregresses over causal continuous motion latents, using a transformer to model streaming audio-motion context and a per-token flow-matching head to sample the next latent from a continuous distribution. Second, we introduce a head-only flow distillation strategy that freezes the causal backbone and distills the multi-step per-token flow head into a single network evaluation using consistency and distribution-matching objectives. This keeps the model token-causal while removing the main latency bottleneck for live interaction. Experiments on BEAT2 show that GestureFAR significantly improves the quality--latency trade-off among streaming-capable methods, preserving strong gesture quality while enabling real-time token-causal generation. Project Page: https://andypinxinliu.github.io/GestureFAR
Pinxin Liu, Haiyang Liu, Jiahao Luo +3
University of Rochester · University of Tokyo · University of California Santa Cruz +2