EMODY Flow: Emotion-Aware Audio-Driven Full-Body Motion Generation

Harsh Kumar Agarwal; Xavier Alameda-Pineda; Olivier Perrotin

cs.GRAug 19, 2026

EMODY Flow: Emotion-Aware Audio-Driven Full-Body Motion Generation

Authors: Harsh Kumar Agarwal, Xavier Alameda-Pineda, Olivier Perrotin

Organizations: INRIA, Univ. Grenoble Alpes Grenoble, France · Univ. Grenoble Alpes, CNRS, Grenoble INP, GIPSA-lab Grenoble, France

Abstract

Embodied conversational agents require synchronized full-body motion (body gestures and facial expressions) that aligns with speech and emotional state. Omni-modal large language models excel at multimodal understanding but produce only linguistic outputs, leaving a critical gap in embodied response generation. We identify and address a failure of emotion conditioning: like other conditional generators that under-use weak conditioning signals, a flow-matching model given both a rich audio embedding and a discrete emotion label suppresses the emotion, generating near-identical motion regardless of the specified emotion. We present EMODY Flow, a lightweight (around 35M parameters) flow-matching framework that attaches to a frozen Qwen-3 Omni model and reuses its internal Mimi audio-codecs to condition two parallel DiT generators - one for SMPL-X body pose, one for FLAME facial expressions. A training-time auxiliary emotion classifier restores emotion sensitivity by forcing generated motion to be emotion-identifiable. EMODY Flow sets a new state of the art on BEAT2 gesture quality, with FGD 0.302, Beat Correlation 0.853, and Diversity 24.62 - improving over the best prior results by 26%, 5%, and 62% respectively - and transfers to zero-shot facial animation on TFHP without domain-specific fine-tuning. Beyond these quantitative gains, the classifier yields clearly emotion-separated motion, which we demonstrate qualitatively through a multidimensional-scaling analysis of the generated gestures.

Figures & tables

Figure 1. EMODY Flow attaches two lightweight flow-matching generators to the frozen Talker component of Qwen-3 Omni. Mimi audio codec embeddings produced by the Talker simultaneously drive speech synthesis (via Code2Wav) and, together with an emotion label, condition separate body and face flow models. Architecture overview of EMODY Flow showing how Qwen-3 Omni's Thinker-Talker-Code2Wav pipeline is augmented with a Body Flow Model and a Face Flow Model, each conditioned on Mimi codec embeddings and an emotion class.

Figure 2. Flow-matching generator DiT module. Each block applies self-attention across time, cross-attention to Mimi audio embeddings, and an MLP - all conditioned via AdaLN on the flow timestep and emotion label. Nine conditioning parameters per block (vs. six in standard DiT) enable independent modulation of each sub-layer. Diagram of the DiT block used in EMODY Flow, showing self-attention, cross-attention, and MLP sub-layers with AdaLN conditioning from the timestep and emotion embedding.

Figure 3. MDS projections (from pairwise FGD) of body (top) and face (bottom) motion, before (left) and after (right) auxiliary-classifier training. Crosses (

\times

) are ground-truth emotion centroids; small markers are generated samples (colour = target emotion, shape = source-audio emotion). Before: generated samples of every emotion collapse into a single central cloud far from the class centroids - the emotion label is ignored. After: body samples (b) spread out and gravitate toward their target-emotion centroids, showing clear emotion separation; face samples (d) spread but remain substantially entangled, reflecting the greater difficulty of emotion conditioning for facial motion. A two-by-two grid of MDS scatter plots. Panels (a) and (c) show generated body and face motion clustering at the center before classifier training. Panel (b) shows body motion separating into distinct emotion clusters after training, while panel (d) shows face motion only partially separating.

Method	FGD $\downarrow$	BC $\uparrow$	Diversity $\uparrow$
DisCo ( 17 )	0.942	0.644	9.91
DiffStyleGesture ( 35 )	0.881	0.724	11.49
TalkSHOW ( 36 )	0.621	0.695	13.47
EMAGE ( 18 )	0.551	0.772	13.06
LOM ( 2 )	0.530	0.778	15.17
ViBES ( 37 )	0.526	0.810	13.03

Table 1. Co-speech gesture generation on BEAT2. Best in bold , second best underlined .

Method	LVE $\downarrow$	MOD $\downarrow$	FDD $\downarrow$
CodeTalker* ( 32 )	13.17	4.74	24.45
SelfTalk* ( 24 )	13.93	3.04	19.81
MultiTalk* ( 29 )	11.66	2.35	15.68
DiffPoseTalk ( 28 )	11.01	2.55	37.68
ARTalk ( 5 )	11.67	2.46	20.77
ViBES ( 37 )	10.87	2.29	6.05

Table 2. Talking-head synthesis on TFHP. Lower is better for all metrics.

*

indicates zero-shot (not trained on TFHP).

Panel of Figure 4 — Figure 4. Representative frames from motion sequences generated by EMODY Flow (with classifier) conditioned on four emotion classes. Posture, gesture amplitude, and facial configuration differ visibly across emotions. Four rows of rendered avatar frames showing neutral, happy, anger, and sad emotion conditions with distinct body postures and facial expressions.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Body	Face
Output dimension	168 ( $\theta{+}t$ )	100 ( $\psi_{f}$ )
Sequence length	300	300
Hidden dimension	512	512
Layers	6	6
Attention heads	16	16
MLP expansion ratio	2.0	2.0

Appendix

Table 3. DiT flow-generator configuration.

Hyperparameter	Value
Optimizer	AdamW ( $\beta_{1}{=}0.9$ , $\beta_{2}{=}0.999$ )
Base learning rate	$1\times 10^{-4}$ (scaled by batch size)
Batch size	4
LR schedule	Cosine annealing, 2-epoch warmup
$\lambda_{\mathrm{FM}}$	1.0
$\lambda_{\mathrm{clf}}$ (body)	0.005

Appendix

Table 4. Training, loss, and solver hyperparameters.

Stage	Body	Face
1. Generator only ( $\mathcal{L}_{\mathrm{FM}}$ )	5000	2500
2. Classifier fit (generator updates discarded)	150	150
3. Generator vs. frozen classifier ( $\mathcal{L}_{\mathrm{total}}$ )	775	1300

Appendix

Table 5. Three-stage training schedule (number of epochs). In stage 2 the generator is co-optimized but its updates are discarded; only the classifier is kept.

Layer	Specification
Input	$(B,\,T,\,168\text{ or }100)$ motion
Conv1D stem	Conv( $168$ or $100\!\to\!256$ , $k{=}15$ )
ResBlock 1	Conv( $256\!\to\!256$ , $k{=}15$ )
ResBlock 2	Conv( $256\!\to\!128$ , stride 2)
ResBlock 3	Conv( $128\!\to\!256$ , stride 2)
Global pooling	AdaptiveAvgPool1D( $1$ )

Appendix

Table 6. Auxiliary 1D-ResNet classifier architecture.

Emotion	Count	%	Weight
Neutral	559	51.1	0.019
Happiness	79	7.2	0.134
Anger	78	7.1	0.136
Sadness	61	5.5	0.173
Contempt	79	7.2	0.134
Surprise	79	7.2	0.134

Appendix

Table 7. Emotion-class distribution in the BEAT2 training split and the inverse-frequency class weights used for the classifier loss.

Human Motion Generation

Human motion generation creates articulated movement from language, audio, video, spatial constraints, or interaction goals.

See all 291 papers

Motion-Omni: End-to-End Joint Speech and Full-Body Motion for Spoken Dialogue
Aug 28, 2026Chengqian Ma, Wei Tao, Haoyu Zhang +1Multimodal Generation Co-Speech Gesture Generation
Peking University · LIGHTSPEED · The Chinese University of Hong Kong, Shenzhen +1
GestureFAR: Streaming Co-Speech Gesture Generation with Flow Autoregression
Sep 18, 2026Pinxin Liu, Haiyang Liu, Jiahao Luo +3Co-Speech Gesture Generation Flow-Based Generative Modeling
University of Rochester · University of Tokyo · University of California Santa Cruz +2
Generative Learning as a Tool to Improve Perception of Emotional Body Motion Expressions
Jun 27, 2026Huakun Liu, Miao Cheng, Xin Wei +6Human Motion Generation Conditional Generative Modeling
Nara Institute of Science and Technology · Tohoku University

Aug 28, 2026J/K move · Enter open · S save

Motion-Omni: End-to-End Joint Speech and Full-Body Motion for Spoken Dialogue

Chengqian Ma, Wei Tao, Haoyu Zhang, Yiwen Guo

Peking University · LIGHTSPEED · The Chinese University of Hong Kong, Shenzhen+1

Multimodal Generation Co-Speech Gesture Generation Spoken Dialogue Systems Human Motion Generation

Open paper AskPDF arXiv

An avatar that holds a conversation should decide what to say and to move while saying it, yet these abilities live in separate model families: spoken dialogue models produce speech without motion, and co-speech motion models produce motion only from audio handed to them. The standard remedy is a cascade that first generates the spoken response and then runs a motion model over the finished audio, which requires a second full inference pass and precludes any joint optimisation between the two. We present Motion-Omni, an end-to-end framework in which a spoken dialogue model natively outputs explicit facial expression together with hand, upper-body and lower-body motion, generated directly from the hidden states that produce the speech. Joint training is not optional here: with the speech pathway frozen, motion remains misaligned with the audio, and co-adapting the LLM, Speech Generator and Motion Generator under both objectives is what recovers alignment while retaining spoken-dialogue ability. Supervision comes from a scalable, model-agnostic pipeline that pseudo-labels consistent-voice speech responses with a replaceable motion teacher, yielding 422,856 quality-ranked pairs (1,402 hours). We further release SwDA-500 and, to our knowledge, the first public evaluation protocol for stochastic open-ended full-body spoken dialogue, matching audio across motion systems while unifying rendering, automatic metrics, human evaluation, and latency measurement. Instantiated with a Qwen2.5-7B-Instruct backbone, Motion-Omni-Q7 matches the same-audio teacher cascade to within 2% on reference-free motion metrics while responding 5.4 x faster (RTF=0.78, faster than real time), surpasses all non-teacher cascades on beat correlation and diversity, and reaches a 2.62% word error rate, the lowest among the omni-modal systems compared.

Abstract

Human Motion Generation

Explore similar work

Motion-Omni: End-to-End Joint Speech and Full-Body Motion for Spoken Dialogue

GestureFAR: Streaming Co-Speech Gesture Generation with Flow Autoregression

Generative Learning as a Tool to Improve Perception of Emotional Body Motion Expressions