Sonic-AMG: Streaming Human Motion Generation from Spatial Audio

Jingcheng Ni; Zekun Li; Zhiyang Dou; Xiaoyan Cong; Ziyao Zeng; Hongyang Du; Randall Balestriero; Taku Komura

cs.CVOct 9, 2026

Sonic-AMG: Streaming Human Motion Generation from Spatial Audio

Authors: Jingcheng Ni, Zekun Li, Zhiyang Dou, Xiaoyan Cong, Ziyao Zeng, Hongyang Du, Randall Balestriero, Taku Komura,

Organizations: Brown University · MIT · Yale University · The University of Hong Kong

Abstract

Modeling human reactions to spatial audio requires understanding both what is heard and where it comes from. Prior work generates motion offline from complete acoustic and spatial input sequences. Humans, however, react to sounds without knowing what they will hear next. We present Sonic-AMG, to our knowledge the first framework for streaming human motion generation from spatial audio. The model encodes acoustic content and body-relative source geometry as separate tokens. Conditioned on these tokens, a causal transformer with a flow head generates continuous motion latents, from which a causal decoder reconstructs motion segments. Standard teacher-forced training derives motion histories and body-relative source locations from ground-truth motion, whereas streaming inference relies on generated motion for both, resulting in a training-inference mismatch. To reduce this history discrepancy, we introduce rehearsal training, which combines full-sequence teacher-forced supervision with supervised prediction along short autoregressive self-rollouts. To keep spatial conditioning consistent with generated motion, we propose spatial feedback, which recomputes body-relative source locations from generated root poses during both rehearsal training and inference. On SAM, Sonic-AMG achieves state-of-the-art FID and R-precision. It further supports real-time online generation of coherent reactions across sound events and source locations. Project page: https://jingchengni.com/sonic-amg/.

Figures & tables

Figure 1: Streaming motion generation from spatial audio. Sonic-AMG generates continuous reactions to successive sounds, updating each source’s position relative to the generated body. Colors link sound sources and audio segments to the corresponding motions. The unified scene shows successive motion snapshots from purple to blue to yellow. Translucent bands connect audio segments to their motion responses, while faded poses highlight motion transitions between audio segments.

Method	Spatial Audio	Causal	Online SSL
EDGE ( Tseng et al., 2023 )	$\times$	$\times$	—
POPDG ( Luo et al., 2024 )	$\times$	$\times$	—
LODGE ( Li et al., 2024a )	$\times$	$\times$	—
MOSPA ( Xu et al., 2025 )	$\checkmark$	$\times$	$\times$
Ours	$\checkmark$	$\checkmark$	$\checkmark$

Table 1: Method comparison. Online SSL denotes spatial conditioning updated from generated motion.

Figure 2: Sonic-AMG architecture. A causal transformer and flow head generate motion latents from interleaved acoustic ( what ), spatial ( where ), and motion tokens. Spatial feedback updates body-relative SSL from decoded root poses. $X_{k}$ , $z_{k}$ , and $m_{k}$ denote a four-frame motion block, its latent, and its projected token, respectively.

Method	Causal	Eval. SSL	FID $\downarrow$	R@1 $\uparrow$	R@2 $\uparrow$	R@3 $\uparrow$	Diversity $\rightarrow$
GT motion †	—	—	0.001	1.000 $\pm$ 0.000	1.000 $\pm$ 0.000	1.000 $\pm$ 0.000	23.616 $\pm$ 0.188
EDGE †	$\times$	GT	13.993	0.886 $\pm$ 0.005	0.960 $\pm$ 0.003	0.977 $\pm$ 0.002	23.099 $\pm$ 0.196
POPDG †	$\times$	GT	20.967	0.762 $\pm$ 0.006	0.886 $\pm$ 0.005	0.934 $\pm$ 0.003	22.536 $\pm$ 0.170
LODGE †	$\times$	GT	102.289	0.444 $\pm$ 0.006	0.594 $\pm$ 0.005	0.679 $\pm$ 0.004	21.101 $\pm$ 0.141
Bailando †	$\times$	GT	168.396	0.077 $\pm$ 0.003	0.134 $\pm$ 0.003	0.182 $\pm$ 0.004	17.347 $\pm$ 0.247
MOSPA	$\times$	GT	8.242	0.942 $\pm$ 0.004	0.984 $\pm$ 0.002	0.994 $\pm$ 0.001	23.412 $\pm$ 0.169

Table 2: Quantitative comparison on SAM. Causal denotes support for causal generation.

\dagger

denotes results from Xu et al. (2025) .

Figure 3: Qualitative comparison on SAM. EDGE, POPDG, and LODGE panels are reproduced from Xu et al. (2025) ; purple markers indicate sound sources.

Figure 4: Reactions over the full audio duration. MOSPA (top) generates 8 s of motion; Sonic-AMG (bottom) covers the full 10–12 s audio clips. Six poses are sampled uniformly over each sequence’s own duration. Purple markers denote sound sources; red dots trace root trajectories.

Panel of Figure 5 — Figure 5: Physical simulation of generated reactions. See Appendix B.3 for details.

GPU	Method	Eval. SSL	Mean $\downarrow$	Median $\downarrow$	P95 $\downarrow$	RTF $\downarrow$	Real-time
RTX A6000	MOSPA	GT	—	—	—	0.357	$\times$
	Ours	GT	41.2	41.1	42.5	0.309	$\checkmark$
	Ours	Online	50.2	50.9	53.2	0.376	$\checkmark$
TITAN RTX	MOSPA	GT	—	—	—	1.427	$\times$
	Ours	GT	102.5	105.8	108.1	0.769	$\checkmark$
	Ours	Online	118.0	117.9	125.6	0.885	$\checkmark$

Table 3: Streaming computation. All measurements use batch size one. For Sonic-AMG, step times (ms) are measured from precomputed acoustic tokens. Online includes motion decoding and spatial feedback; GT measures latent generation with precomputed SSL. RTF is computation time divided by generated motion duration. Real-time denotes causal streaming with

\mathrm{RTF}<1

. MOSPA has no four-frame step time.

Training strategy	Eval. SSL	FID $\downarrow$	R@1 $\uparrow$	R@2 $\uparrow$	R@3 $\uparrow$	Diversity $\rightarrow$
GT motion †		0.001	1.000 $\pm$ 0.000	1.000 $\pm$ 0.000	1.000 $\pm$ 0.000	23.616 $\pm$ 0.188
Two-forward ( $0\!\to\!1$ )	GT	10.797	0.839 $\pm$ 0.006	0.924 $\pm$ 0.004	0.956 $\pm$ 0.003	22.856 $\pm$ 0.160
Two-forward ( $1\!\to\!0$ )	GT	8.001	0.926 $\pm$ 0.004	0.972 $\pm$ 0.002	0.983 $\pm$ 0.001	23.106 $\pm$ 0.136
Two-forward ( $p=1$ )	GT	8.033	0.934 $\pm$ 0.004	0.981 $\pm$ 0.002	0.992 $\pm$ 0.001	23.223 $\pm$ 0.172
Rollout Only	GT	14.854	0.860 $\pm$ 0.005	0.937 $\pm$ 0.004	0.966 $\pm$ 0.003	22.550 $\pm$ 0.161
Rollout Only with History	GT	8.585	0.915 $\pm$ 0.004	0.968 $\pm$ 0.003	0.984 $\pm$ 0.003	22.883 $\pm$ 0.142

Table 4: Training strategies. Each group uses the same SSL protocol for predicted training histories and evaluation. Two-forward schedules increase (

0\!\to\!1

), decrease (

1\!\to\!0

), or fix (

p=1

) the fraction of replaced motion inputs.

Table 10

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Human reactions and humanoid tracking in physics simulation. Aligned snapshots of the continuous Music $\rightarrow$ Explosion $\rightarrow$ Birdsong sequence show human motion (top) and G1 simulation (bottom). Robot tracking is described in Appendix B.3 .

Panel of Figure 7 — Figure 7: Additional reactions under online spatial feedback. Original generated motions for six sounds, with poses ordered chronologically. Purple markers denote sound sources; clip identifiers match the supplementary videos.

Panel of Figure 8 — Figure 8: Additional reactions under online spatial feedback (continued).

Sound	SAM clip	Style	Interval (s)
Music	0541212	2	$[0,10.00)$
Explosion	0330202	2	$[10.00,20.27)$
Birdsong	0113311	1	$[20.27,30.13)$

Appendix

Table 7: Continuous demonstration setup. Source locations change at event boundaries. Intervals refer to the generated 30 Hz sequence; style identifiers follow SAM.

Figure 9: Sonic-AMG with audio-derived spatial conditioning. The semantic and spatial branches derive what and where tokens from log-mel and ILD/IPD features, respectively. These tokens use the same interface to the autoregressive motion generator. The training-only SSL head predicts the normalized 11-dimensional geometric descriptor from pooled spatial features, supervised by ground-truth geometry through an MSE loss. Inference uses the learned spatial features directly, without the SSL head or spatial feedback. Motion poses are schematic.

Method	FID $\downarrow$	R@1 $\uparrow$	R@2 $\uparrow$	R@3 $\uparrow$	Diversity $\rightarrow$
GT motion †	0.001	1.000	1.000	1.000	23.616
EDGE †	13.993	0.886	0.960	0.977	23.099
POPDG †	20.967	0.762	0.886	0.934	22.536
LODGE †	102.289	0.444	0.594	0.679	21.101
Bailando †	168.396	0.077	0.134	0.182	17.347
MOSPA	8.242	0.942	0.984	0.994	23.412

Appendix

Table 8: No-SSL comparison on SAM. Baselines are reused from Table 2 .

\dagger

denotes results from Xu et al. (2025) .

Human Motion Generation

Human motion generation creates articulated movement from language, audio, video, spatial constraints, or interaction goals.

See all 291 papers

MoSAT: Human Motion Generation from Spatial Audio and Textual Description
Sep 20, 2026Shuyang Xu, Zhiyang Dou, Yiduo Hao +8Text-to-Motion Generation Human Motion Generation
The University of Hong Kong · Massachusetts Institute of Technology · University of Pennsylvania +5
MOMO: Human Motion from Moving Sounds
Oct 17, 2026Deli Cai, Shuyang Xu, Zhiyang Dou +4Human Motion Generation Spatial Audio
South China University of Technology · The University of Hong Kong · MIT +1
SonicWorld: Source-Centric Latent Dynamics for Spatial Audio Reasoning
Oct 12, 2026Chenlong Wei, Hanling Wang, Mingjin Lou +6Spatial Audio Spatial Reasoning Benchmarks
Xi’an Jiaotong-Liverpool University · University of California, San Diego · Imperial College London +2

Sep 20, 2026J/K move · Enter open · S save

MoSAT: Human Motion Generation from Spatial Audio and Textual Description

Shuyang Xu, Zhiyang Dou, Yiduo Hao, Zekun Li+7

The University of Hong Kong · Massachusetts Institute of Technology · University of Pennsylvania+5

Text-to-Motion Generation Human Motion Generation

Open paper AskPDF arXiv

Human motion is shaped by both external acoustic events and behavioral intent: spatial audio conveys environmental cues that elicit or guide a response, while text specifies the desired action and how it should be performed. In this paper, we study the novel task of human motion synthesis jointly conditioned on spatial audio and natural language, a problem that has been largely overlooked in previous research. To support this task, We introduce STAM, a dataset of motion sequences paired with spatial audio and detailed textual annotations whose rich vocabulary affords precise and nuanced specification of human motions. We further introduce MoSAT, a latent flow-matching framework for full-body motion generation jointly conditioned on natural-language intent and directional spatial-audio cues through hierarchical cross-attention before generating motion. Such a hierarchical design enhances temporally coherent and semantically aligned motion sequences. We also develop tri-modal evaluators for comprehensive evaluation on this novel task. Extensive experiments show that MoSAT achieves the SOTA performance by leveraging spatial audio's intrinsic motion-shaping properties alongside textual semantics, enabling precise and diverse motion in various scenarios.

Abstract

Human Motion Generation

Explore similar work

MoSAT: Human Motion Generation from Spatial Audio and Textual Description

MOMO: Human Motion from Moving Sounds

SonicWorld: Source-Centric Latent Dynamics for Spatial Audio Reasoning