BiTDiff: Fine-Grained 3D Conducting Motion Generation via BiMamba-Transformer Diffusion

Tianzhi Jia; Kaixing Yang; Xiaole Yang; Xulong Tang; Ke Qiu; Shikui Wei; Yao Zhao

cs.CVApr 6, 2026

BiTDiff: Fine-Grained 3D Conducting Motion Generation via BiMamba-Transformer Diffusion

Authors: Tianzhi Jia, Kaixing Yang, Xiaole Yang, Xulong Tang, Ke Qiu, Shikui Wei, Yao Zhao

Organizations: Institute of Information Science, Beijing Jiaotong University, Beijing, China · Renmin University of China, Beijing, China · ADVANCE.AI, Beijing, China · Malou Tech Inc, Plano, Texas, USA

Abstract

3D conducting motion generation aims to synthesize fine-grained conductor motions from music, with broad potential in music education, virtual performance, digital human animation, and human-AI co-creation. However, this task remains underexplored due to two major challenges: (1) the lack of large-scale fine-grained 3D conducting datasets and (2) the absence of effective methods that can jointly support long-sequence generation with high quality and efficiency. To address the data limitation, we develop a quality-oriented 3D conducting motion collection pipeline and construct CM-Data, a fine-grained SMPL-X dataset with about 10 hours of conducting motion data. To the best of our knowledge, CM-Data is the first and largest public dataset for 3D conducting motion generation. To address the methodological limitation, we propose BiTDiff, a novel framework for 3D conducting motion generation, built upon a BiMamba-Transformer hybrid model architecture for efficient long-sequence modeling and a Diffusion-based generative strategy with human-kinematic decomposition for high-quality motion synthesis. Specifically, BiTDiff introduces auxiliary physical-consistency losses and a hand-/body-specific forward-kinematics design for better fine-grained motion modeling, while leveraging BiMamba for memory-efficient long-sequence temporal modeling and Transformer for cross-modal semantic alignment. In addition, BiTDiff supports training-free joint-level motion editing, enabling downstream human-AI interaction design. Extensive quantitative and qualitative experiments demonstrate that BiTDiff achieves state-of-the-art (SOTA) performance for 3D conducting motion generation on the CM-Data dataset. Code will be available upon acceptance.

Figures & tables

Figure 1. 3D conducting motion generation is a promising research direction with broad applications (up), such as Music Education , Virtual Performance , and Co-Creation . Moreover, the conducting motions generated by BiTDiff (low), trained on our proposed dataset CM-Data , are not only temporally coherent and rhythmically aligned with music, but also finely detailed and artistically expressive.

Figure 2. Examples from our pipeline. Our reconstruction pipeline captures not only coarse body-level motions but also fine-grained details such as hand articulation and facial expressions. Moreover, multi-view visualizations demonstrate the accuracy and temporal stability of the recovered 3D conducting motions.

Figure 3. Overview of BiTDiff. The left panel presents the detailed model architecture, while the right panel illustrates the inference strategy. Here, $0<t_{2}<t_{1}<T$ indicate two intermediate timesteps in the diffusion process.

	Hand		Body		Face		Alignment	Efficiency
	FID ${}_{\text{h}}$ $\downarrow$	DIV ${}_{\text{h}}$ $\uparrow$	FID ${}_{\text{b}}$ $\downarrow$	DIV ${}_{\text{b}}$ $\uparrow$	FID ${}_{\text{f}}$ $\downarrow$	DIV ${}_{\text{f}}$ $\uparrow$	BAS $\uparrow$	L@1024 $\downarrow$	L@4096 $\downarrow$
Ground Truth	–	10.69	–	3.01	–	5.60	0.272	–	–
VirtualConductor ( Liu et al., 2022 )	101.57	6.42	86.83	2.31	48.64	1.41	0.211	3.84s	10.92s
DiffusionConductor ( Zhao et al., 2023 )	39.42	8.91	31.76	2.84	42.18	3.08	0.286	18.73s	74.12s
EDGE ( Tseng et al., 2023 )	36.28	9.14	27.95	2.96	39.87	3.26	0.289	5.41	20.13
Lodge ( Li et al., 2024b )	37.11	10.02	20.96	3.54	30.85	3.35	0.296	3.62s	9.51s

Table 1. Quantitative comparison of generation quality and efficiency on the CM-Data dataset.

Method	MS $\uparrow$	MF $\uparrow$	MC $\uparrow$
Ground Truth	4.21	4.47	4.09
EDGE ( Tseng et al., 2023 )	3.41	3.35	3.28
DiffSHEG ( Chen et al., 2024 )	3.46	3.38	3.31
Lodge ( Li et al., 2024b )	3.49	3.43	3.36
BiTDiff ( Ours )	3.96	3.88	4.17

Table 2. User study on the CM-Data dataset.

Figure 4. Qualitative comparison with SOTAs.

Method	FID h $\downarrow$	DIV h $\uparrow$	FID b $\downarrow$	DIV b $\uparrow$	BAS $\uparrow$	L@4096 $\downarrow$
w/o Vel.	30.47	9.88	22.63	3.93	0.247	2.56s
Naive FK	69.38	6.72	17.58	4.15	0.265	2.56s
Mamba (uni)	47.96	8.68	31.41	3.74	0.281	1.91s
Transformer	27.33	10.02	18.87	3.71	0.298	13.84s
BiTDiff (Full)	25.81	10.34	19.14	3.78	0.302	2.56s

Table 3. Ablation study on CM-Data .

Figure 5. Motion editing visualization of BiTDiff.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 1. Dataset statistics of CM-Data. (a) Distribution of musical styles. (b) Distribution of performance scenarios. (c) Word cloud of keywords collected from video titles and captions. These statistics illustrate the diversity of CM-Data in terms of musical content, performance setting, and textual metadata.

Figure 2. Questionnaire interface used in our user study.

Human Motion Generation

Human motion generation creates articulated movement from language, audio, video, spatial constraints, or interaction goals.

See all 291 papers

MG-Former: A Transformer-Based Framework for Music-Driven 3D Conducting Gesture Generation
May 2, 2026Ke Qiu, Yawen Qin, Tianzhi Jia +3Cross-Modal Alignment Human Motion Generation
Malou Tech Inc · South-Central Minzu University · Beijing Jiaotong University +2
Image Diffusion Models Are Good 3D Human Motion Generators
Oct 9, 2026Siyuan Yang, Chong Wang, Hao Cheng +4Text-to-Motion Generation Diffusion Models for Motion Generation
Division of Robotics, Perception and Learning, KTH Royal Institute of Technology, Sweden · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · School of Artificial Intelligence, Hebei University of Technology, China +3
Kimodo: Scaling Controllable Human Motion Generation
Mar 16, 2026Davis Rempe, Mathis Petrovich, Ye Yuan +21Human Motion Generation Diffusion Models for Motion Generation
NVIDIA

May 2, 2026J/K move · Enter open · S save

MG-Former: A Transformer-Based Framework for Music-Driven 3D Conducting Gesture Generation

Ke Qiu, Yawen Qin, Tianzhi Jia, Xiaole Yang+2

Malou Tech Inc · South-Central Minzu University · Beijing Jiaotong University+2

Cross-Modal Alignment Human Motion Generation

Open paper AskPDF arXiv

Generating expressive conducting gestures from music is a challenging cross-modal motion synthesis problem: the output must follow long-range musical structure, preserve beat-level synchronization, and remain plausible as a fine-grained 3D human performance. Existing conducting-motion studies are often limited by sparse pose representations, small-scale data, or evaluation protocols that do not directly measure whether music and gesture are mutually aligned. This paper presents TransConductor, a Transformer-based framework for music-driven conducting gesture generation. We introduce ConductorMotion, a SMPL-parameter data construction pipeline that recovers detailed body motion from conducting videos and forms a dataset targeted at professional conducting gestures. Given acoustic descriptors extracted from audio and an initial pose, TransConductor uses a Trans-Temporal Music Encoder and a Trans-Temporal Conducting Gesture Decoder to autoregressively predict SMPL pose parameters. To better assess artistic correspondence, we further build a retrieval-based evaluation model that embeds music and gestures into a shared space and yields FID, modality distance, multi-modality distance, and diversity metrics. Experiments show that TransConductor outperforms dance-generation and conducting-generation baselines, while ablations verify the benefits of the Transformer backbone and the proposed alignment loss.

Abstract

Human Motion Generation

Explore similar work

MG-Former: A Transformer-Based Framework for Music-Driven 3D Conducting Gesture Generation

Image Diffusion Models Are Good 3D Human Motion Generators

Kimodo: Scaling Controllable Human Motion Generation