cs.CVOct 17, 2026

MOMO: Human Motion from Moving Sounds

Authors: Deli Cai, Shuyang Xu, Zhiyang Dou, Zitong Lan, Mingmin Zhao, Changxing Ding, Taku Komura

Organizations: South China University of Technology · The University of Hong Kong · MIT · University of Pennsylvania

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Human responses to sound depend not only on its semantic content, but also on the evolving spatial relationship between source and listener, conveyed through spatial audio and its temporal dynamics. Modeling human reactions to such dynamic spatial audio is important for character animation, human behavior analysis, and robotics. As of yet, existing audio-driven motion generation methods primarily focus on non-spatial or static acoustic conditions, leaving this important problem largely underexplored. To bridge this gap, we formulate the novel task of dynamic spatial-audio-driven human motion generation and introduce two motion-captured datasets. We introduce the Dynamic Spatial Audio and Motion dataset (DySAM) as the first dataset and benchmark for this task, providing synchronized binaural audio, 3D human motions, and moving sound source trajectories. Given the importance of dynamic spatial cues of sound sources, we further provide the Dynamic Spatial Audio Localization dataset (DySAL), an auxiliary binaural localization dataset that pairs diverse acoustic events with head poses and moving source trajectories to support spatial audio perception. For benchmarking, we propose MOMO for human Motion generation from Moving Sounds, a diffusion-based framework guided by three coupled principles: localization, understanding, and generation. MOMO performs localization by estimating head-centric spatial direction cues of the moving source directly from raw binaural audio, avoiding reliance on ground-truth source coordinates. It performs understanding by extracting frame-wise audio-content representations, and performs generation by conditioning motion diffusion on both semantic features and dynamic spatial cues. Experiments show that MOMO outperforms existing baselines in motion fidelity, audio-motion alignment, and physical plausibility.

Figures & tables

Figure 1: This paper proposes a novel task for synthesizing 3D human motions in response to dynamic spatial audio . To support this new task, we collect DySAM, the first dataset comprising synchronized dynamic spatial audio, corresponding sound source trajectories, and high-quality 3D human motion. In this figure, the model generates diverse and realistic human reactions to various moving acoustic events.
Figure 2: Diverse Reactions to Moving Sounds.
Figure 3: Data Collection Setup.
DatasetDyn. SSSS3D Jointpos/rot\mathrm{Joint_{pos/rot}}ModelJointsSubjectsHours
Dance with Melody ( Tang et al., 2018 )×\times×\times✓/ ×\times-21-1.6
DanceNet ( Zhuang et al., 2022 )×\times×\times✓/ ×\times-5520.9
AIST++ ( Li et al., 2021 )×\times×\times✓/✓COCO/SMPL17/24305.2
PopDanceSet ( Luo et al., 2024 )×\times×\times✓/✓COCO/SMPL17/241323.5
FineDance ( Li et al., 2023 )×\times×\times✓/✓SMPL & hand joints522714.6
SAM ( Xu et al., 2025 )×\times✓✓/✓SMPL-X55129.5
Table 1: Statistics of the DySAM dataset. DySAM is the first dataset for dynamic spatial-audio-driven human motion generation, containing synchronized binaural audio, 3D human motion, and dynamic sound source trajectories (Dyn. SS). We also provide DySAL, an auxiliary sound source localization dataset containing dynamic binaural recordings of diverse acoustic events with frame-wise source annotations.
Figure 4: Snapshots from the our DySAM dataset, spanning human sounds, animal sounds, and everyday environmental sounds. In each panel, the colors of the actor and the sound source transition from light to dark over time to indicate temporal evolution. Red dots mark the movement trajectory of the actor, while blue dots mark the trajectory of the moving sound source.
Figure 5: Overview of MOMO framework. MOMO comprises three coupled functional components: a sound source localization (SSL) network predicts a frame-wise head-centric DoA sequence D^\hat{\bm{D}} from binaural audio ( localization ); a CLAP audio encoder extracts frame-wise audio content features A\bm{A} from mono audio ( understanding ); and a conditional diffusion model that iteratively denoise the Gaussian noise to generated motion, integrating Frame-wise Motion Correction to adaptively correct each motion according to frame-wise conditions ( generation ).
Figure 6: Qualitative comparison of state-of-the-art methods for the dynamic spatial audio-to-motion task. We visualize motion results from five cases.
MethodR-precision↑\text{R-precision}\uparrowFID↓\text{FID}\downarrowDiversity→\text{Diversity}\rightarrowAPD→\text{APD}\rightarrowSkating Ratio (%)↓\text{Skating Ratio (\%)}\downarrow
Top1Top2Top3
Real Motion0.720±0.000{0.720}^{\pm{0.000}}0.886±0.000{0.886}^{\pm{0.000}}0.976±0.000{0.976}^{\pm{0.000}}0.0010.00129.66±0.512{29.66}^{\pm{0.512}}45.531.261.26
EDGE ( Tseng et al., 2023 )0.457±0.006{0.457}^{\pm{0.006}}0.730±0.010{0.730}^{\pm{0.010}}0.819±0.009{0.819}^{\pm{0.009}}7.498±0.485{7.498}^{\pm{0.485}}27.49±0.119{27.49}^{\pm{0.119}}35.449.23
POPDG ( Luo et al., 2024 )0.462±0.006{0.462}^{\pm{0.006}}0.681±0.005{0.681}^{\pm{0.005}}0.782±0.004{0.782}^{\pm{0.004}}5.727±0.485{5.727}^{\pm{0.485}}28.15±0.141{28.15}^{\pm{0.141}}34.992.77
MOSPA ( Xu et al., 2025 )0.585±0.013{0.585}^{\pm{0.013}}0.788±0.009{0.788}^{\pm{0.009}}0.885±0.006{0.885}^{\pm{0.006}}5.160±0.875{5.160}^{\pm{0.875}}33.50±0.583{33.50}^{\pm{0.583}}50.293.69
MoMask++ ( Guo et al., 2025 )0.415±0.006{0.415}^{\pm{0.006}}0.656±0.008{0.656}^{\pm{0.008}}0.775±0.009{0.775}^{\pm{0.009}}3.909±0.331{3.909}^{\pm{0.331}}28.39±0.787{28.39}^{\pm{0.787}}38.2414.11
Table 2: Quantitative Comparisons on the DySAM Dataset. ± indicates a 95% confidence interval. Bold face indicates the best result. →\rightarrow means the closer to Real Motion the better.
Table 3: Ablation on the DySAM Dataset, including the types of sound source and audio features, Frame-wise Correction (Frame Corr.) , and Correction Strength Predictor (Strength Pred.).
SSL Training DataDySAL Val. Angular Error ( ∘ ) ↓\downarrow
DySAM48.92
DySAM + DySAL36.62
Table 4: Effect of the auxiliary DySAL training data. We report the angular error on the DySAL validation set.
Table 5: Ablation of the sound source localization network. We evaluate the DoA representation and temporal architecture on the DySAM validation set. All variants are trained using the same training data. Reported numbers are the full-frame angular errors ( ∘ ) between the predicted and ground-truth source directions.
Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Audio Type
alarmbarkbicycle_bellbird
car_bidirectcar_fastcar_slowcat
coughdrilldrill_laughengine_slow
fireworkgun_handgun_machinehelicopter
horn_carhorn_trainhorn_truckinsect_bees
insect_flyinginsect_msqtmonster_roarmonster_step
Appendix
Table A1: The 42 distinct audio types covered in DySAM, corresponding to 109 unique audio sources. Each audio source is accompanied by 8 to 32 ten-second motion sequences, and the dataset contains 2,336 (binaural audio, motion) pairs in total.
Figure A1: Four examples of DySAM dataset. In each image, the left column shows the original C3D body markers and speaker markers, SMPL-X motion (gray body) , and calculated speaker location; the right column shows the canonicalized SMPL-X motion (purple body) and the head coordinate system. The red, green, and blue axes denote the x, y, and z axes, respectively.
Figure C2: The evaluator framework consists of a motion autoencoder co-trained with Bi-GRU feature extractors. A reconstruction loss Lrecon\mathcal{L}_{\mathrm{recon}} is applied between the ground-truth motion x\mathbf{x} and the decoded motion x^\hat{\mathbf{x}} produced by the motion autoencoder. Two Bi-GRU encoders extract the audio feature and the sound source feature from the binaural audio and the sound source trajectory, respectively, and the concatenation of the two features forms the condition latent. Additionally, an InfoNCE loss LInfoNCE\mathcal{L}_{\mathrm{InfoNCE}} operates on the extracted condition features and motion features.
Metric DefinitionSkating Ratio on GT ↓\downarrow
GMD formulation ( Karunratanakul et al., 2023 )13.90%
Ours1.26%
Appendix
Table D2: Skating ratio computed on the ground-truth motions of DySAM using different metric definitions.
Feature NameDimension
MFCC20
MFCC delta20
constant-Q chromagram12
STFT chromagram12
onset strength1
tempogram267
Appendix
Table E3: The extracted audio features for a single ear comprise a 335335 -dimensional vector. The total dimension of the binaural audio feature vector a\mathbf{a} is 670.
Figure F3: Qualitative comparison of reaction genre control on three audio examples. Given the same binaural audio, label 1, i.e., sensitive, produces stronger and larger-amplitude reactions than label 0, i.e., insensitive, demonstrating effective genre-based motion control.
Figure G4: Simulated robot demonstration in MuJoCo. The generated SMPL-X motion is retargeted to a humanoid robot using GMR and tracked by a pretrained TWIST controller.
Figure H5: User study results.
Figure H6: Screenshot of the user study interface.
More in this topic

Human Motion Generation

Human motion generation creates articulated movement from language, audio, video, spatial constraints, or interaction goals.

See all 291 papers

Explore similar work

CardsList
Sep 20, 2026

MoSAT: Human Motion Generation from Spatial Audio and Textual Description

Human motion is shaped by both external acoustic events and behavioral intent: spatial audio conveys environmental cues that elicit or guide a response, while text specifies the desired action and how it should be performed. In this paper, we study the novel task of human motion synthesis jointly conditioned on spatial audio and natural language, a problem that has been largely overlooked in previous research. To support this task, We introduce STAM, a dataset of motion sequences paired with spatial audio and detailed textual annotations whose rich vocabulary affords precise and nuanced specification of human motions. We further introduce MoSAT, a latent flow-matching framework for full-body motion generation jointly conditioned on natural-language intent and directional spatial-audio cues through hierarchical cross-attention before generating motion. Such a hierarchical design enhances temporally coherent and semantically aligned motion sequences. We also develop tri-modal evaluators for comprehensive evaluation on this novel task. Extensive experiments show that MoSAT achieves the SOTA performance by leveraging spatial audio's intrinsic motion-shaping properties alongside textual semantics, enabling precise and diverse motion in various scenarios.

Shuyang Xu, Zhiyang Dou, Yiduo Hao +8

The University of Hong Kong · Massachusetts Institute of Technology · University of Pennsylvania +5

Text-to-Motion GenerationHuman Motion Generation
AskPDF
Oct 9, 2026

Sonic-AMG: Streaming Human Motion Generation from Spatial Audio

Modeling human reactions to spatial audio requires understanding both what is heard and where it comes from. Prior work generates motion offline from complete acoustic and spatial input sequences. Humans, however, react to sounds without knowing what they will hear next. We present Sonic-AMG, to our knowledge the first framework for streaming human motion generation from spatial audio. The model encodes acoustic content and body-relative source geometry as separate tokens. Conditioned on these tokens, a causal transformer with a flow head generates continuous motion latents, from which a causal decoder reconstructs motion segments. Standard teacher-forced training derives motion histories and body-relative source locations from ground-truth motion, whereas streaming inference relies on generated motion for both, resulting in a training-inference mismatch. To reduce this history discrepancy, we introduce rehearsal training, which combines full-sequence teacher-forced supervision with supervised prediction along short autoregressive self-rollouts. To keep spatial conditioning consistent with generated motion, we propose spatial feedback, which recomputes body-relative source locations from generated root poses during both rehearsal training and inference. On SAM, Sonic-AMG achieves state-of-the-art FID and R-precision. It further supports real-time online generation of coherent reactions across sound events and source locations. Project page: https://jingchengni.com/sonic-amg/.

Jingcheng Ni, Zekun Li, Zhiyang Dou +6

Brown University · MIT · Yale University +1

Human Motion GenerationAutoregressive Generation
AskPDF
Jun 2, 2026

Mamba-Enhanced Implicit Motion Learning for Audio-Driven Portrait Animation

Audio-driven human motion video generation aims to synthesize realistic and temporally coherent human animations from a single static image, with applications in talking-head synthesis, co-speech gesture generation, and dynamic presentations. Moving beyond conventional keypoint-based methods that often struggle to capture subtle motion dynamics, We propose a novel implicit-motion framework for generating realistic and temporally coherent human motion videos from a single static image and audio. Our approach uses a two-stage pipeline that decouples motion prediction from rendering. The first stage integrates appearance priors and hierarchical depth cues into a region-aware attention mechanism to model latent motion features. The second stage employs a Mamba-enhanced diffusion model to directly predict these features from audio and the source image, enabling unsupervised learning of fine-grained motion patterns. This decoupled architecture enhances flexibility and efficiency. Trained on a new 380-hour high-quality dataset, our method outperforms prior work across multiple public benchmarks and our collected data in accuracy, naturalness, and temporal coherence, setting a new state-of-the-art.

Xuan Wei, Jiahui Chen, Kaiheng Li +2

Department of Digital Media Technology, School of Film, Xiamen University, Xiamen, China · 2National Institute for Data Science in Health and Medicine, Xiamen University, Xiamen, China · Institute of Artificial Intelligence, Xiamen University, Xiamen, China

Video Diffusion ModelsHuman Motion GenerationAudio-Driven Facial Animation
AskPDF