SARAH: Spatially Aware Real-time Agentic Humans

Evonne Ng; Siwei Zhang; Zhang Chen; Michael Zollhoefer; Alexander Richard

cs.CVFeb 20, 2026

SARAH: Spatially Aware Real-time Agentic Humans

Authors: Evonne Ng, Siwei Zhang, Zhang Chen, Michael Zollhoefer, Alexander Richard

Organizations: Meta Reality Labs, Redmond, WA, USA

Abstract

As embodied agents become central to VR, telepresence, and digital human applications, their motion must go beyond speech-aligned gestures: agents should turn toward users, respond to their movement, and maintain natural gaze. Current methods lack this spatial awareness. We close this gap with the first real-time, fully causal method for spatially-aware conversational motion, deployable on a streaming VR headset. Given a user's position and dyadic audio, our approach produces full-body motion that aligns gestures with speech while orienting the agent according to the user. Our architecture combines a causal transformer-based VAE with interleaved latent tokens for streaming inference and a flow matching model conditioned on user trajectory and audio. To support varying gaze preferences, we introduce a gaze scoring mechanism with classifier-free guidance to decouple learning from control: the model captures natural spatial alignment from data, while users can adjust eye contact intensity at inference time. On the Embody 3D dataset, our method achieves state-of-the-art motion quality at over 300 FPS -- 3x faster than non-causal baselines -- while capturing the subtle spatial dynamics of natural conversation. We validate our approach on a live VR system, bringing spatially-aware conversational agents to real-time deployment. Please see https://evonneng.github.io/sarah/ for details.

Figures & tables

Figure 1. Our method generates full-body 3D motion for a virtual agent that is spatially aware of the user while engaging in a conversation. Given the user’s floor-projected head trajectory and dyadic audio, we generate the agent’s complete 3D motion. Trajectory colors indicate time: blue $\rightarrow$ green (user) and yellow $\rightarrow$ red (agent). See project page for results.

Figure 2. Given the user’s 3D position and dyadic conversational audio, our model generates 3D motion that is conversationally and spatially aware (left). We use a fully causal transformer-based VAE with interleaved latent tokens at a fixed temporal stride; both encoder and decoder employ causal attention, where each $\mu/\sigma$ token attends only to preceding frames and earlier latents (center). These latents are passed to a transformer-based flow matching model that also uses causal masking and optionally accepts a gaze score for controlling the agent’s eye contact (right). Our lightweight architecture enables real-time, autoregressive streaming without distillation.

Figure 3. We represent each joint $j$ as a 3D icosahedron. The centroid of the vertices yields the global position $\boldsymbol{\Pi}_{j}$ , and we recover the global orientation $\boldsymbol{\Omega}_{j}$ via SVD against a reference icosahedron.

Figure 4. Our training data spans a wide range of gaze behaviors, from sustained eye contact to complete gaze aversion (left). To enable controllable gaze at inference, we compute a gaze score $g$ , where $\mathbf{d}_{x}$ is the agent’s facing direction and $\mathbf{d}_{y}$ points toward the user (right). The score approaches $1$ when facing the user directly and $-1$ when facing away.

				FPS	FGD $\downarrow$ (m $\times 10^{1}$ )		FGD ${}_{\textbf{acc}}$ $\downarrow$ ( $\times 10^{5}$ )		Foot Slide $\downarrow$ $\in[0,1]$		Wrist Var $\uparrow$		Head Ang. $\uparrow$ $\in[-1,1]$
		C	R		Avg	NS/S	Avg	NS/S	Avg	NS/S	Avg	NS/S	Avg	NS/S
	GT	–	–	–	–	–	–	–	0.01	0.01/0.01	137.6	122.3/179.7	0.81	0.80/0.84
Baselines	Random	✗	✓	4K	1.06	0.30/0.28	1.83	0.31/3.38	0.01	0.01/0.01	188.1	190.5/181.6	0.28	0.27/0.32
	NN	✗	✓	1K	0.90	0.19/0.16	0.77	0.02/0.51	0.01	0.01/0.01	97.0	85.7/128.2	0.59	0.57/0.64
	MDM ( Tevet et al., 2022 )	✗	✗	90	3.48	1.93/2.66	2.88	0.64/5.37	0.11	0.11/0.11	61.4	57.9/71.0	0.81	0.80/0.84
	A2P ( Ng et al., 2024 )	✗	✗	90	2.01	0.54/0.80	2.31	0.43/4.95	0.02	0.02/0.02	69.4	59.0/98.2	0.71	0.70/0.73

Table 1. Comparison with baselines and ablations (abl.) on 2048 test sequences. C = causal, R = real-time. S = speaking (544 seq.), NS = non-speaking (1504 seq.).

\uparrow

higher is better,

\downarrow

lower is better. † Reducible to 600 fps without quality degradation.

$g$	FGD $\downarrow$	FGD ${}_{\textbf{acc}}$ $\downarrow$	Foot Slide $\downarrow$	Wrist Var $\uparrow$	Head Ang. $\uparrow$
$\emptyset$	1.28	2.19	0.01	105.0	0.83
0.0	0.99	2.18	0.01	111.1	0.56
0.8	0.92	2.19	0.01	110.8	0.76
1.0	1.49	2.20	0.01	106.6	0.96

Table 2. Effect of gaze control on motion.

\emptyset

denotes that gaze control is disabled.

Figure 5. We visualize the agent’s facing direction via projected lines (agent: yellow $\rightarrow$ red; user: blue $\rightarrow$ green). With no alignment $g=\emptyset$ , the agent’s gaze is more diverse; as we increase $g$ , the agent increasingly turns towards the user.

Figure 6. Sequences from our real-time demo system, rendered with a photorealistic avatar. The top row visualizes the user’s headset location as a silver sphere. The bottom row shows the generated avatar from the user’s (headset) viewpoint. Our method generates realistic conversational motion that is responsive to the user’s spatial motion. Full videos are available on our project page .

Human Motion Generation

Human motion generation creates articulated movement from language, audio, video, spatial constraints, or interaction goals.

See all 286 papers

EmbodiedHead: Real-Time Listening and Speaking Avatar for Conversational Agents
Apr 19, 2026Yu Zhang, Kaiyuan Shen, Yang LiStreaming Video Generation Talking Head Generation
School of Computer Science and Technology, East China Normal University, Shanghai, China · Garabido Shanghai Technology Co., Ltd., Shanghai, China
AVTR-1: Open Stack for Real-Time Interactive Avatars
Sep 19, 2026Artem Kravtsov, Dmitrii Ziganshin, Vsevolod Poletaev +4Real-Time Interactive Video Generation Audio-Video Generation
Avaturn Live
GestureFAR: Streaming Co-Speech Gesture Generation with Flow Autoregression
Sep 18, 2026Pinxin Liu, Haiyang Liu, Jiahao Luo +3Co-Speech Gesture Generation Flow-Based Generative Modeling
University of Rochester · University of Tokyo · University of California Santa Cruz +2

Apr 19, 2026J/K move · Enter open · S save

EmbodiedHead: Real-Time Listening and Speaking Avatar for Conversational Agents

Yu Zhang, Kaiyuan Shen, Yang Li

School of Computer Science and Technology, East China Normal University, Shanghai, China · Garabido Shanghai Technology Co., Ltd., Shanghai, China

Streaming Video Generation Talking Head Generation Human Avatar Animation

Open paper AskPDF arXiv

We present EmbodiedHead, a speech-driven talking head framework that equips LLMs with real-time visual avatars for conversation. A practical embodied avatar must achieve real-time generation, unified listening-speaking behavior, and high rendered visual quality simultaneously. Our framework couples the first Rectified-Flow Diffusion Transformer (DiT) for this task with a differentiable renderer, enabling diverse, high-fidelity generation in as few as four sampling steps. Prior listening-speaking methods rely on dual-stream audio, introducing an interlocutor look-ahead dependency incompatible with causal user--LLM interaction. We instead adopt a single-stream interface with explicit per-frame listening-speaking state conditioning and a Streaming Audio Scheduler, suppressing spurious mouth motion during listening while enabling seamless turn-taking. A two-stage training scheme of coefficient-space pretraining and joint image-domain refinement further closes the gap between motion-level supervision and rendered quality. Extensive experiments demonstrate state-of-the-art visual quality and motion fidelity in both speaking and listening scenarios. Project page: https://vpx-ecnu.github.io/EmbodiedHead-Website/.

Abstract

Human Motion Generation

Explore similar work

EmbodiedHead: Real-Time Listening and Speaking Avatar for Conversational Agents

AVTR-1: Open Stack for Real-Time Interactive Avatars

GestureFAR: Streaming Co-Speech Gesture Generation with Flow Autoregression