cs.CVNov 16, 2026

DuoFlow: Towards Synchronized Speech and Motion Generation with Fine-Grained Control

Authors: Jun Dong, Ronghui Li, Zhongyuan Hu, Yunqi Zhao, Lijie Guo, Ruizhang Xu, Di Kang, Linchao Bao, 

Organizations: Tsinghua University · Tencent

Abstract

An embodied speaker's expressiveness depends not only on what is said, but also on how the voice sounds and how the body moves. However, existing joint speech-motion generation models are largely driven implicitly by transcript semantics or intermediate speech representations, offering little explicit control over motion hierarchy, body parts, or vocal style. To bridge this gap, we construct MotionSpeech, a structured benchmark for controllable joint speech-motion generation that includes transcripts, voice-style descriptions, and motion conditions at two hierarchical levels. Building on this benchmark, we propose DuoFlow, a dual-stream latent flow transformer that jointly denoises speech and motion under a shared noise level and a unified second-based temporal coordinate. Modality-specific residual connectors inject structured conditions into the corresponding streams, layer-wise bidirectional cross-modal attention coordinates their generation, and a monotonic phoneme pathway combines normalized progress anchors with CTC supervision to strengthen speech temporal modeling. DuoFlow provides a unified interface for fine-grained motion and vocal-style control and demonstrates the ability to synthesize high-quality, synchronized, and instruction-consistent speech-motion sequences. Project page, https://jundong-dev.github.io/DuoFlow-project-page.

Figures & tables

Explore similar work

CardsList
  1. TalkFlow: Structured Latent Flow Matching for Speech Driven Holistic Motion Generation

    Oct 29, 2026Xinmu Wang, Fumi Honda, Xiang Gao +3Human Motion GenerationCo-Speech Gesture Generation

  2. Motion-Omni: End-to-End Joint Speech and Full-Body Motion for Spoken Dialogue

    Aug 28, 2026Chengqian Ma, Wei Tao, Haoyu Zhang +1Multimodal GenerationCo-Speech Gesture Generation

  3. EMODY Flow: Emotion-Aware Audio-Driven Full-Body Motion Generation

    Aug 19, 2026Harsh Kumar Agarwal, Xavier Alameda-Pineda, Olivier PerrotinHuman Motion GenerationCo-Speech Gesture Generation