Sonic-AMG: Streaming Human Motion Generation from Spatial Audio

Jingcheng Ni; Zekun Li; Zhiyang Dou; Xiaoyan Cong; Ziyao Zeng; Hongyang Du; Randall Balestriero; Taku Komura

cs.CVOct 9, 2026

Sonic-AMG: Streaming Human Motion Generation from Spatial Audio

Authors: Jingcheng Ni, Zekun Li, Zhiyang Dou, Xiaoyan Cong, Ziyao Zeng, Hongyang Du, Randall Balestriero, Taku Komura,

Organizations: Brown University · MIT · Yale University · The University of Hong Kong

Abstract

Modeling human reactions to spatial audio requires understanding both what is heard and where it comes from. Prior work generates motion offline from complete acoustic and spatial input sequences. Humans, however, react to sounds without knowing what they will hear next. We present Sonic-AMG, to our knowledge the first framework for streaming human motion generation from spatial audio. The model encodes acoustic content and body-relative source geometry as separate tokens. Conditioned on these tokens, a causal transformer with a flow head generates continuous motion latents, from which a causal decoder reconstructs motion segments. Standard teacher-forced training derives motion histories and body-relative source locations from ground-truth motion, whereas streaming inference relies on generated motion for both, resulting in a training-inference mismatch. To reduce this history discrepancy, we introduce rehearsal training, which combines full-sequence teacher-forced supervision with supervised prediction along short autoregressive self-rollouts. To keep spatial conditioning consistent with generated motion, we propose spatial feedback, which recomputes body-relative source locations from generated root poses during both rehearsal training and inference. On SAM, Sonic-AMG achieves state-of-the-art FID and R-precision. It further supports real-time online generation of coherent reactions across sound events and source locations. Project page: https://jingchengni.com/sonic-amg/.

Figures & tables

Figure 1: Streaming motion generation from spatial audio. Sonic-AMG generates continuous reactions to successive sounds, updating each source’s position relative to the generated body. Colors link sound sources and audio segments to the corresponding motions. The unified scene shows successive motion snapshots from purple to blue to yellow. Translucent bands connect audio segments to their motion responses, while faded poses highlight motion transitions between audio segments.

Method	Spatial Audio	Causal	Online SSL
EDGE ( Tseng et al., 2023 )	$\times$	$\times$	—
POPDG ( Luo et al., 2024 )	$\times$	$\times$	—
LODGE ( Li et al., 2024a )	$\times$	$\times$	—
MOSPA ( Xu et al., 2025 )	$\checkmark$	$\times$	$\times$
Ours	$\checkmark$	$\checkmark$	$\checkmark$

Table 1: Method comparison. Online SSL denotes spatial conditioning updated from generated motion.

Figure 2: Sonic-AMG architecture. A causal transformer and flow head generate motion latents from interleaved acoustic ( what ), spatial ( where ), and motion tokens. Spatial feedback updates body-relative SSL from decoded root poses. $X_{k}$ , $z_{k}$ , and $m_{k}$ denote a four-frame motion block, its latent, and its projected token, respectively.

Method	Causal	Eval. SSL	FID $\downarrow$	R@1 $\uparrow$	R@2 $\uparrow$	R@3 $\uparrow$	Diversity $\rightarrow$
GT motion †	—	—	0.001	1.000 $\pm$ 0.000	1.000 $\pm$ 0.000	1.000 $\pm$ 0.000	23.616 $\pm$ 0.188
EDGE †	$\times$	GT	13.993	0.886 $\pm$ 0.005	0.960 $\pm$ 0.003	0.977 $\pm$ 0.002	23.099 $\pm$ 0.196
POPDG †	$\times$	GT	20.967	0.762 $\pm$ 0.006	0.886 $\pm$ 0.005	0.934 $\pm$ 0.003	22.536 $\pm$ 0.170
LODGE †	$\times$	GT	102.289	0.444 $\pm$ 0.006	0.594 $\pm$ 0.005	0.679 $\pm$ 0.004	21.101 $\pm$ 0.141
Bailando †	$\times$	GT	168.396	0.077 $\pm$ 0.003	0.134 $\pm$ 0.003	0.182 $\pm$ 0.004	17.347 $\pm$ 0.247
MOSPA	$\times$	GT	8.242	0.942 $\pm$ 0.004	0.984 $\pm$ 0.002	0.994 $\pm$ 0.001	23.412 $\pm$ 0.169

Table 2: Quantitative comparison on SAM. Causal denotes support for causal generation.

\dagger

denotes results from Xu et al. (2025) .

Figure 3: Qualitative comparison on SAM. EDGE, POPDG, and LODGE panels are reproduced from Xu et al. (2025) ; purple markers indicate sound sources.

Figure 4: Reactions over the full audio duration. MOSPA (top) generates 8 s of motion; Sonic-AMG (bottom) covers the full 10–12 s audio clips. Six poses are sampled uniformly over each sequence’s own duration. Purple markers denote sound sources; red dots trace root trajectories.

Panel of Figure 5 — Figure 5: Physical simulation of generated reactions. See Appendix B.3 for details.

GPU	Method	Eval. SSL	Mean $\downarrow$	Median $\downarrow$	P95 $\downarrow$	RTF $\downarrow$	Real-time
RTX A6000	MOSPA	GT	—	—	—	0.357	$\times$
	Ours	GT	41.2	41.1	42.5	0.309	$\checkmark$
	Ours	Online	50.2	50.9	53.2	0.376	$\checkmark$
TITAN RTX	MOSPA	GT	—	—	—	1.427	$\times$
	Ours	GT	102.5	105.8	108.1	0.769	$\checkmark$
	Ours	Online	118.0	117.9	125.6	0.885	$\checkmark$

Table 3: Streaming computation. All measurements use batch size one. For Sonic-AMG, step times (ms) are measured from precomputed acoustic tokens. Online includes motion decoding and spatial feedback; GT measures latent generation with precomputed SSL. RTF is computation time divided by generated motion duration. Real-time denotes causal streaming with

\mathrm{RTF}<1

. MOSPA has no four-frame step time.

Training strategy	Eval. SSL	FID $\downarrow$	R@1 $\uparrow$	R@2 $\uparrow$	R@3 $\uparrow$	Diversity $\rightarrow$
GT motion †		0.001	1.000 $\pm$ 0.000	1.000 $\pm$ 0.000	1.000 $\pm$ 0.000	23.616 $\pm$ 0.188
Two-forward ( $0\!\to\!1$ )	GT	10.797	0.839 $\pm$ 0.006	0.924 $\pm$ 0.004	0.956 $\pm$ 0.003	22.856 $\pm$ 0.160
Two-forward ( $1\!\to\!0$ )	GT	8.001	0.926 $\pm$ 0.004	0.972 $\pm$ 0.002	0.983 $\pm$ 0.001	23.106 $\pm$ 0.136
Two-forward ( $p=1$ )	GT	8.033	0.934 $\pm$ 0.004	0.981 $\pm$ 0.002	0.992 $\pm$ 0.001	23.223 $\pm$ 0.172
Rollout Only	GT	14.854	0.860 $\pm$ 0.005	0.937 $\pm$ 0.004	0.966 $\pm$ 0.003	22.550 $\pm$ 0.161
Rollout Only with History	GT	8.585	0.915 $\pm$ 0.004	0.968 $\pm$ 0.003	0.984 $\pm$ 0.003	22.883 $\pm$ 0.142

Table 4: Training strategies. Each group uses the same SSL protocol for predicted training histories and evaluation. Two-forward schedules increase (

0\!\to\!1

), decrease (

1\!\to\!0

), or fix (

p=1

) the fraction of replaced motion inputs.

Table 10

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Human reactions and humanoid tracking in physics simulation. Aligned snapshots of the continuous Music $\rightarrow$ Explosion $\rightarrow$ Birdsong sequence show human motion (top) and G1 simulation (bottom). Robot tracking is described in Appendix B.3 .

Panel of Figure 7 — Figure 7: Additional reactions under online spatial feedback. Original generated motions for six sounds, with poses ordered chronologically. Purple markers denote sound sources; clip identifiers match the supplementary videos.

Panel of Figure 8 — Figure 8: Additional reactions under online spatial feedback (continued).

Sound	SAM clip	Style	Interval (s)
Music	0541212	2	$[0,10.00)$
Explosion	0330202	2	$[10.00,20.27)$
Birdsong	0113311	1	$[20.27,30.13)$

Appendix

Table 7: Continuous demonstration setup. Source locations change at event boundaries. Intervals refer to the generated 30 Hz sequence; style identifiers follow SAM.

Figure 9: Sonic-AMG with audio-derived spatial conditioning. The semantic and spatial branches derive what and where tokens from log-mel and ILD/IPD features, respectively. These tokens use the same interface to the autoregressive motion generator. The training-only SSL head predicts the normalized 11-dimensional geometric descriptor from pooled spatial features, supervised by ground-truth geometry through an MSE loss. Inference uses the learned spatial features directly, without the SSL head or spatial feedback. Motion poses are schematic.

Method	FID $\downarrow$	R@1 $\uparrow$	R@2 $\uparrow$	R@3 $\uparrow$	Diversity $\rightarrow$
GT motion †	0.001	1.000	1.000	1.000	23.616
EDGE †	13.993	0.886	0.960	0.977	23.099
POPDG †	20.967	0.762	0.886	0.934	22.536
LODGE †	102.289	0.444	0.594	0.679	21.101
Bailando †	168.396	0.077	0.134	0.182	17.347
MOSPA	8.242	0.942	0.984	0.994	23.412

Appendix

Table 8: No-SSL comparison on SAM. Baselines are reused from Table 2 .

\dagger

denotes results from Xu et al. (2025) .

Human Motion Generation

Human motion generation creates articulated movement from language, audio, video, spatial constraints, or interaction goals.

See all 291 papers

Sep 20, 2026

MoSAT: Human Motion Generation from Spatial Audio and Textual Description

Human motion is shaped by both external acoustic events and behavioral intent: spatial audio conveys environmental cues that elicit or guide a response, while text specifies the desired action and how it should be performed. In this paper, we study the novel task of human motion synthesis jointly conditioned on spatial audio and natural language, a problem that has been largely overlooked in previous research. To support this task, We introduce STAM, a dataset of motion sequences paired with spatial audio and detailed textual annotations whose rich vocabulary affords precise and nuanced specification of human motions. We further introduce MoSAT, a latent flow-matching framework for full-body motion generation jointly conditioned on natural-language intent and directional spatial-audio cues through hierarchical cross-attention before generating motion. Such a hierarchical design enhances temporally coherent and semantically aligned motion sequences. We also develop tri-modal evaluators for comprehensive evaluation on this novel task. Extensive experiments show that MoSAT achieves the SOTA performance by leveraging spatial audio's intrinsic motion-shaping properties alongside textual semantics, enabling precise and diverse motion in various scenarios.

Shuyang Xu, Zhiyang Dou, Yiduo Hao +8

The University of Hong Kong · Massachusetts Institute of Technology · University of Pennsylvania +5

Text-to-Motion Generation Human Motion Generation

Ask PDF

Oct 17, 2026

MOMO: Human Motion from Moving Sounds

Human responses to sound depend not only on its semantic content, but also on the evolving spatial relationship between source and listener, conveyed through spatial audio and its temporal dynamics. Modeling human reactions to such dynamic spatial audio is important for character animation, human behavior analysis, and robotics. As of yet, existing audio-driven motion generation methods primarily focus on non-spatial or static acoustic conditions, leaving this important problem largely underexplored. To bridge this gap, we formulate the novel task of dynamic spatial-audio-driven human motion generation and introduce two motion-captured datasets. We introduce the Dynamic Spatial Audio and Motion dataset (DySAM) as the first dataset and benchmark for this task, providing synchronized binaural audio, 3D human motions, and moving sound source trajectories. Given the importance of dynamic spatial cues of sound sources, we further provide the Dynamic Spatial Audio Localization dataset (DySAL), an auxiliary binaural localization dataset that pairs diverse acoustic events with head poses and moving source trajectories to support spatial audio perception. For benchmarking, we propose MOMO for human Motion generation from Moving Sounds, a diffusion-based framework guided by three coupled principles: localization, understanding, and generation. MOMO performs localization by estimating head-centric spatial direction cues of the moving source directly from raw binaural audio, avoiding reliance on ground-truth source coordinates. It performs understanding by extracting frame-wise audio-content representations, and performs generation by conditioning motion diffusion on both semantic features and dynamic spatial cues. Experiments show that MOMO outperforms existing baselines in motion fidelity, audio-motion alignment, and physical plausibility.

Deli Cai, Shuyang Xu, Zhiyang Dou +4

South China University of Technology · The University of Hong Kong · MIT +1

Human Motion Generation Spatial Audio Diffusion Models for Motion Generation

Ask PDF

Oct 12, 2026

SonicWorld: Source-Centric Latent Dynamics for Spatial Audio Reasoning

Dynamic spatial audio reasoning requires a model to associate changing acoustic observations with the correct source and recover its spatial state at a queried time or interval, a challenge intensified by overlapping speech and source motion. To study this problem, we introduce Sonic-Bench, comprising 2.8K first-order Ambisonics clips and 28.8K target-specific question-answer pairs across 480 simulated indoor scenes. The benchmark covers localization, distance estimation, and motion reasoning in single- and dual-source settings. We further propose SonicWorld, which integrates physical, learned acoustic, and semantic features into persistent source states through learned transitions and recurrent updates, and projects them into an Omni LLM for spatial reasoning. Compared with Spatial-Omni on the Sonic-Bench test split, SonicWorld improves direction accuracy from 31.98% to 34.69% and reduces azimuth error from 57.22° to 48.01°. In zero-shot transfer to Spatial LibriSpeech, SonicWorld retains strong localization performance, achieving 73.98% direction accuracy and 19.29° azimuth error, compared with 27.14% and 70.74° for Spatial-Omni.

Chenlong Wei, Hanling Wang, Mingjin Lou +6

Xi’an Jiaotong-Liverpool University · University of California, San Diego · Imperial College London +2

Spatial Audio Spatial Reasoning Benchmarks Audio-Language Models Direction-of-Arrival Estimation World Model Learning

Ask PDF

Abstract

Human Motion Generation

Explore similar work

MoSAT: Human Motion Generation from Spatial Audio and Textual Description

MOMO: Human Motion from Moving Sounds

SonicWorld: Source-Centric Latent Dynamics for Spatial Audio Reasoning