Motion-Omni: End-to-End Joint Speech and Full-Body Motion for Spoken Dialogue

Chengqian Ma; Wei Tao; Haoyu Zhang; Yiwen Guo

cs.SDAug 28, 2026

Motion-Omni: End-to-End Joint Speech and Full-Body Motion for Spoken Dialogue

Authors: Chengqian Ma, Wei Tao, Haoyu Zhang, Yiwen Guo

Organizations: Peking University · LIGHTSPEED · The Chinese University of Hong Kong, Shenzhen · Independent Researcher

Abstract

An avatar that holds a conversation should decide what to say and to move while saying it, yet these abilities live in separate model families: spoken dialogue models produce speech without motion, and co-speech motion models produce motion only from audio handed to them. The standard remedy is a cascade that first generates the spoken response and then runs a motion model over the finished audio, which requires a second full inference pass and precludes any joint optimisation between the two. We present Motion-Omni, an end-to-end framework in which a spoken dialogue model natively outputs explicit facial expression together with hand, upper-body and lower-body motion, generated directly from the hidden states that produce the speech. Joint training is not optional here: with the speech pathway frozen, motion remains misaligned with the audio, and co-adapting the LLM, Speech Generator and Motion Generator under both objectives is what recovers alignment while retaining spoken-dialogue ability. Supervision comes from a scalable, model-agnostic pipeline that pseudo-labels consistent-voice speech responses with a replaceable motion teacher, yielding 422,856 quality-ranked pairs (1,402 hours). We further release SwDA-500 and, to our knowledge, the first public evaluation protocol for stochastic open-ended full-body spoken dialogue, matching audio across motion systems while unifying rendering, automatic metrics, human evaluation, and latency measurement. Instantiated with a Qwen2.5-7B-Instruct backbone, Motion-Omni-Q7 matches the same-audio teacher cascade to within 2% on reference-free motion metrics while responding 5.4 x faster (RTF=0.78, faster than real time), surpasses all non-teacher cascades on beat correlation and diversity, and reaches a 2.62% word error rate, the lowest among the omni-modal systems compared.

Figures & tables

Figure 1: Motion-Omni jointly generates speech and full-body motion. The dialogue is illustrative, with unpaired LOM-generated poses. The timing comparison uses the matched-audio, offline SwDA-500 benchmark (Table 5 ).

Figure 2: Motion-Omni framework with four components and their conditioning topology.

Figure 3: Speech-data curation and motion supervision. Insets schematically show the paired modalities: audio–text in Stage 1, text–speech units in Stage 2, and speech units–motion codes in Stage 3.

Stage	Samples	Hours
Stage 1: ASR	1,572,119	4,295.8
Stage 2: TTS	1,642,715	4,517.0
Stage 3: TTSM	422,856	1,402.3
Stage 4: Mixture	724,508	1,929.4
Total	4,362,198	12,144.6

Table 1: Training data by stage.

Model	test-en $\downarrow$
Human Reference
Human	2.14
Dedicated TTS
CosyVoice Du et al. [2024a]	4.29
CosyVoice 2 Du et al. [2024b]	2.57
FireRedTTS Guo et al. [2024]	3.82

Table 2: WER (%) on Seed-TTS-Eval.

	Reference-free					Teacher-referenced
System	Div. $\uparrow$	BC $\uparrow$	$\mathbf{BC_{\text{bi}}}\uparrow$	LSE-C $\uparrow$	LSE-D $\downarrow$	FGD $\downarrow$	MSE $\downarrow$	LVD $\downarrow$
Qwen2.5-Omni + MambaTalk	11.92	7.17	8.56	6.853	7.920	4.08	7.092	6.851
MO-audio + MambaTalk	12.07	7.03	8.60	6.797	7.955	4.15	7.128	7.019
Qwen2.5-Omni + GestureLSM	13.61	7.44	9.04	–	–	3.29	–	–
MO-audio + GestureLSM	13.55	7.49	9.11	–	–	3.17	–	–
Qwen2.5-Omni + EMAGE	11.67	7.31	8.66	6.578	8.151	3.75	7.553	7.205

Table 3: Motion, facial geometry and lip-sync metrics on SwDA-500. Teacher-referenced metrics compare against LOM-generated motion and thus favour LOM-based cascades; FGD, BC and

\mathrm{BC}_{\text{bi}}

are in

{\times}10^{-1}

units. Bold / underline mark the best/second-best; dashes denote values not computed.

Comparison	R1 Rhythm	R2 Semantics	R3 Naturalness	Pooled W/T/L (W–L)
Qwen2.5-Omni + LOM	12 3 10	13 12	10 3 12	35/6/34 ( $+1$ )
Qwen2.5-Omni + EMAGE	7 2 16	10 3 12	16 4 5	33/9/33 ( $0$ )
Same MO speech audio:
MO-audio + LOM	5 12 8	9 8 8	11 7 7	25/27/23 ( $+2$ )
MO-audio + EMAGE	13 7 5	15 3 7	17 8	45/10/20 ( $\bm{+25}$ )

Table 4: Human preference on motion-related dimensions. Bold / underline mark the largest/second-largest pooled wins-minus-losses margin.

Model	$T_{\text{resp}}$ (s) $\downarrow$	RTF $\downarrow$	Slowdown
Qwen2.5-Omni + LOM	99.08	18.34	$22.9{\times}$
Qwen2.5-Omni + EMAGE	48.65	8.79	$11.3{\times}$
MO-audio + LOM	23.35	4.39	$5.4{\times}$
MO-audio + EMAGE	4.63	0.84	$1.1{\times}$
Motion-Omni-Q7 (ours)	4.32	0.78	$1.0{\times}$

Table 5: Response latency on SwDA-500. Bold / underline mark the best/second-best value; Slowdown is relative to our model.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Part	Dim	Composition
Face	$106$	jaw ( $1\times 6$ ) $+$ FLAME expression ( $100$ )
Hand	$180$	$30$ hand joints $\times\,6$
Upper	$78$	$13$ upper-body joints $\times\,6$
Lower	$61$	$9$ lower-body joints $\times\,6$ $+$ translation ( $3$ ) $+$ foot contact ( $4$ )

Appendix

Table 6: SMPL-X + FLAME feature decomposition used by the four LOM VQ-VAEs. The face branch combines the SMPL-X jaw rotation with FLAME Li et al. [2017] expression coefficients; the remaining three branches are SMPL-X joint rotations (6D rotation representation).

Parameter	Value
Working dimension $d_{m}$	512
Number of attention heads $h$	8
Per-head dimension $d_{h}$	64
TQGF depth $L_{\text{TQGF}}$	2
Self-attention depth $L_{\text{self}}$	6
TQGF FFN multiplier	4

Appendix

Table 7: Motion Generator architectural hyperparameters.

Setting	Stage 1	Stage 2	Stage 3 (a–d)	Stage 4 †
Trainable modules	SP	SG	SG + MG	LLM + SP + SG + MG
LR (SP)	$1\!\times\!10^{-3}$	–	–	$1\!\times\!10^{-6}$
LR (SG)	–	$1\!\times\!10^{-4}$	$1\!\times\!10^{-5}$	$5\!\times\!10^{-7}$
LR (MG)	–	–	$2\!\times\!10^{-4}$	$2\!\times\!10^{-6}$
LR (LLM)	–	–	–	$2\!\times\!10^{-7}$
LR schedule	cosine	cosine	const + warmup	const + warmup

Appendix

Table 8: Training configuration of the four stages (rows are settings; columns are stages).

\dagger

: Stage 4 starts from the best Stage 3d checkpoint. “MG”: Motion Generator; “SG”: Speech Generator; “SP”: Speech Projector; “LR” is the peak learning rate. Warmup is given as a fraction of total steps unless noted.

Model	Alpaca	Common	Wild	SD-QA	MMSU	OBQA	BBH	IFEval	AdvBench	Overall
Closed-weight (API only)
GPT-4o-Audio OpenAI [2024]	4.78	4.49	4.58	75.50	80.25	89.23	84.10	76.02	98.65	86.75
Open-weight
Kimi-Audio KimiTeam et al. [2025]	4.46	3.97	4.20	63.12	62.17	83.52	69.70	61.10	100.00	76.91
Qwen2.5-Omni Xu et al. [2025]	4.49	3.93	–	55.71	61.32	81.10	–	52.87	99.42	–
VITA-1.5 Fu et al. [2025]	4.21	3.66	3.48	38.88	52.15	71.65	55.30	38.14	97.69	64.53

Appendix

Table 9: Spoken dialogue quality on VoiceBench Chen et al. [2026c] . Higher is better throughout. GPT-4o-Audio is served through an API and its weights are not released; every other system releases its weights. Dashes denote scores not reported.

Model / System	UTMOSv2 en mean $\uparrow$
Qwen3-TTS Team [2026]	4.05
CosyVoice 3 Du et al. [2025]	3.92
GLM-TTS Cui et al. [2025]	3.31
VoxCPM1.5 Zhou et al. [2025]	3.17
F5-TTS Chen et al. [2025b]	3.15
CosyVoice Du et al. [2024a]	3.06

Appendix

Table 10: Automatic speech naturalness proxy on SwDA-500 generated audio. In each column, bold marks the best value and underline the second best.

No.	Fusion / context source	BC $\uparrow$
1	TQGF / LLM K/V	0.827
2	Concatenation / Speech Generator K/V	0.809
3	GLU / Speech Generator K/V	0.799
4	Cross-attention / Speech Generator K/V	0.905
5	FiLM / Speech Generator K/V	0.852
6	TQGF / Speech Generator K/V	0.912

Appendix

Table 11: Motion-conditioning ablation under the Stage 3a setting.

Checkpoint	FGD $\downarrow$
Stage 3a	0.3974
Stage 3b	0.3258
Stage 3c	0.3079
Stage 3d	0.3075
Stage 4	0.3040

Appendix

Table 12: Teacher-reference FGD across training stages.

Panel of Figure 4 — Figure 4: Motion quality against response latency on SwDA-500. Circles are two-stage cascades; the star is our end-to-end model. Points closer to the top-left are better. The dashed line marks $\mathrm{RTF}=1$ , below which a system responds faster than real time.

Panel of Figure 5 — Figure 5: Motion-Omni-Q7 outputs on three released clips. Each row shows seven frames of a single response in temporal order, together with the text the model spoke in that response; the speech and the motion are produced in the same autoregressive pass. Frames are the seven mutually most distinct upper-body poses subject to a minimum temporal gap, not uniform samples.

Panel of Figure 6 — Figure 6: The same response audio driving five different motion sources. All rows use the speech Motion-Omni-Q7 generated for “Here are five positive character traits: honesty, empathy, kindness, confidence, and curiosity.”, sampled at the same seven timestamps ( $t=1.1$ to $6.9$ s in steps of about $0.98$ s) and rendered with an identical pipeline, so the only variable across rows is the motion stream.

Multimodal Generation

Multimodal generation creates coordinated text, images, audio, video, 3D, and other outputs from mixed inputs while preserving meaning and consistency.

See all 243 papers

Ex-Omni-2D: Expressive Omni-Modal Dialogue Models with Native Visual Presence
Aug 11, 2026Haoyu Zhang, Zhipeng Li, Xiaoying Tang +2Audio-Video Generation Streaming Video Generation
The Chinese University of Hong Kong, Shenzhen · LIGHTSPEED · Independent Researcher
EMODY Flow: Emotion-Aware Audio-Driven Full-Body Motion Generation
Aug 19, 2026Harsh Kumar Agarwal, Xavier Alameda-Pineda, Olivier PerrotinHuman Motion Generation Co-Speech Gesture Generation
INRIA, Univ. Grenoble Alpes Grenoble, France · Univ. Grenoble Alpes, CNRS, Grenoble INP, GIPSA-lab Grenoble, France
Ex-Omni: Enabling 3D Facial Animation Generation for Omni-modal Large Language Models
Feb 6, 2026Haoyu Zhang, Zhipeng Li, Yiwen Guo +1Multimodal Large Language Models Audio-Driven Facial Animation
The Chinese University of Hong Kong, Shenzhen · LIGHTSPEED · Independent Researcher

Aug 11, 2026J/K move · Enter open · S save

Ex-Omni-2D: Expressive Omni-Modal Dialogue Models with Native Visual Presence

Haoyu Zhang, Zhipeng Li, Xiaoying Tang, Tianshu Yu+1

The Chinese University of Hong Kong, Shenzhen · LIGHTSPEED · Independent Researcher

Audio-Video Generation Streaming Video Generation Multimodal Generation Multimodal Video Generation

Open paper AskPDF arXiv

Omni-modal dialogue models can understand multimodal inputs and synthesize spoken replies, but a spoken answer still leaves the agent visually absent. We introduce Ex-Omni-2D, a framework that answers a multimodal query with coordinated text, personalized speech, and reference-conditioned video. The dialogue model first writes a structured Visual Thought Plan (VTP) for scene, emotion, and motion, then generates the response text and multi-codebook speech units. These speech units are decoded into audio and aligned with video frames, giving the speech and avatar modules a common timing signal while allowing them to learn from different data sources. The video module is trained as a full-sequence Teacher conditioned on reference appearance, VTP semantics, and frame-aligned speech units. We further explore to distill it into a few-step block-causal Streaming Student; its Prefix Streaming mechanism carries the previous clean latent into the next chunk and is analyzed as a partial mitigation for late-chunk subject drift. At

400\times720

720\times400

, the four-step four-GPU Student provides incremental output with lower startup latency than the full-sequence Teacher.

Abstract

Multimodal Generation

Explore similar work

Ex-Omni-2D: Expressive Omni-Modal Dialogue Models with Native Visual Presence

EMODY Flow: Emotion-Aware Audio-Driven Full-Body Motion Generation

Ex-Omni: Enabling 3D Facial Animation Generation for Omni-modal Large Language Models