BiTDiff: Fine-Grained 3D Conducting Motion Generation via BiMamba-Transformer Diffusion
Authors: Tianzhi Jia, Kaixing Yang, Xiaole Yang, Xulong Tang, Ke Qiu, Shikui Wei, Yao Zhao
Organizations: Institute of Information Science, Beijing Jiaotong University, Beijing, China · Renmin University of China, Beijing, China · ADVANCE.AI, Beijing, China · Malou Tech Inc, Plano, Texas, USA
3D conducting motion generation aims to synthesize fine-grained conductor motions from music, with broad potential in music education, virtual performance, digital human animation, and human-AI co-creation. However, this task remains underexplored due to two major challenges: (1) the lack of large-scale fine-grained 3D conducting datasets and (2) the absence of effective methods that can jointly support long-sequence generation with high quality and efficiency. To address the data limitation, we develop a quality-oriented 3D conducting motion collection pipeline and construct CM-Data, a fine-grained SMPL-X dataset with about 10 hours of conducting motion data. To the best of our knowledge, CM-Data is the first and largest public dataset for 3D conducting motion generation. To address the methodological limitation, we propose BiTDiff, a novel framework for 3D conducting motion generation, built upon a BiMamba-Transformer hybrid model architecture for efficient long-sequence modeling and a Diffusion-based generative strategy with human-kinematic decomposition for high-quality motion synthesis. Specifically, BiTDiff introduces auxiliary physical-consistency losses and a hand-/body-specific forward-kinematics design for better fine-grained motion modeling, while leveraging BiMamba for memory-efficient long-sequence temporal modeling and Transformer for cross-modal semantic alignment. In addition, BiTDiff supports training-free joint-level motion editing, enabling downstream human-AI interaction design. Extensive quantitative and qualitative experiments demonstrate that BiTDiff achieves state-of-the-art (SOTA) performance for 3D conducting motion generation on the CM-Data dataset. Code will be available upon acceptance.
Figures & tables
Figure 1. 3D conducting motion generation is a promising research direction with broad applications (up), such as Music Education , Virtual Performance , and Co-Creation . Moreover, the conducting motions generated by BiTDiff (low), trained on our proposed dataset CM-Data , are not only temporally coherent and rhythmically aligned with music, but also finely detailed and artistically expressive.
Figure 2. Examples from our pipeline. Our reconstruction pipeline captures not only coarse body-level motions but also fine-grained details such as hand articulation and facial expressions. Moreover, multi-view visualizations demonstrate the accuracy and temporal stability of the recovered 3D conducting motions.
Figure 3. Overview of BiTDiff. The left panel presents the detailed model architecture, while the right panel illustrates the inference strategy. Here, 0<t2<t1<T indicate two intermediate timesteps in the diffusion process.
Hand
Body
Face
Alignment
Efficiency
FID h↓
DIV h↑
FID b↓
DIV b↑
FID f↓
DIV f↑
BAS ↑
L@1024 ↓
L@4096 ↓
Ground Truth
–
10.69
–
3.01
–
5.60
0.272
–
–
VirtualConductor ( Liu et al., 2022 )
101.57
6.42
86.83
2.31
48.64
1.41
0.211
3.84s
10.92s
DiffusionConductor ( Zhao et al., 2023 )
39.42
8.91
31.76
2.84
42.18
3.08
0.286
18.73s
74.12s
EDGE ( Tseng et al., 2023 )
36.28
9.14
27.95
2.96
39.87
3.26
0.289
5.41
20.13
Lodge ( Li et al., 2024b )
37.11
10.02
20.96
3.54
30.85
3.35
0.296
3.62s
9.51s
Table 1. Quantitative comparison of generation quality and efficiency on the CM-Data dataset.
Method
MS ↑
MF ↑
MC ↑
Ground Truth
4.21
4.47
4.09
EDGE ( Tseng et al., 2023 )
3.41
3.35
3.28
DiffSHEG ( Chen et al., 2024 )
3.46
3.38
3.31
Lodge ( Li et al., 2024b )
3.49
3.43
3.36
BiTDiff ( Ours )
3.96
3.88
4.17
Table 2. User study on the CM-Data dataset.
Figure 4. Qualitative comparison with SOTAs.
Method
FID h ↓
DIV h ↑
FID b ↓
DIV b ↑
BAS ↑
L@4096 ↓
w/o Vel.
30.47
9.88
22.63
3.93
0.247
2.56s
Naive FK
69.38
6.72
17.58
4.15
0.265
2.56s
Mamba (uni)
47.96
8.68
31.41
3.74
0.281
1.91s
Transformer
27.33
10.02
18.87
3.71
0.298
13.84s
BiTDiff (Full)
25.81
10.34
19.14
3.78
0.302
2.56s
Table 3. Ablation study on CM-Data .
Figure 5. Motion editing visualization of BiTDiff.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 1. Dataset statistics of CM-Data. (a) Distribution of musical styles. (b) Distribution of performance scenarios. (c) Word cloud of keywords collected from video titles and captions. These statistics illustrate the diversity of CM-Data in terms of musical content, performance setting, and textual metadata.
Figure 2. Questionnaire interface used in our user study.
Generating expressive conducting gestures from music is a challenging cross-modal motion synthesis problem: the output must follow long-range musical structure, preserve beat-level synchronization, and remain plausible as a fine-grained 3D human performance. Existing conducting-motion studies are often limited by sparse pose representations, small-scale data, or evaluation protocols that do not directly measure whether music and gesture are mutually aligned. This paper presents TransConductor, a Transformer-based framework for music-driven conducting gesture generation. We introduce ConductorMotion, a SMPL-parameter data construction pipeline that recovers detailed body motion from conducting videos and forms a dataset targeted at professional conducting gestures. Given acoustic descriptors extracted from audio and an initial pose, TransConductor uses a Trans-Temporal Music Encoder and a Trans-Temporal Conducting Gesture Decoder to autoregressively predict SMPL pose parameters. To better assess artistic correspondence, we further build a retrieval-based evaluation model that embeds music and gestures into a shared space and yields FID, modality distance, multi-modality distance, and diversity metrics. Experiments show that TransConductor outperforms dance-generation and conducting-generation baselines, while ablations verify the benefits of the Transformer backbone and the proposed alignment loss.
Ke Qiu, Yawen Qin, Tianzhi Jia +3
Malou Tech Inc · South-Central Minzu University · Beijing Jiaotong University +2
Text-to-motion generation typically relies on dataset-specific skeletal representations, with models trained separately for benchmarks with different joint layouts. We investigate whether image diffusion architectures can model 3D human motion through a common image-compatible representation that accommodates heterogeneous skeletal layouts while preserving motion-specific kinematic structure. To this end, we organize root-centered joint trajectories into topology-informed joint--time images and model global root dynamics through a complementary latent stream. A motion autoencoder conditions local-motion decoding on the root latent, while a text-conditioned Scalable Interpolant Transformer (SiT) jointly generates both latent streams, with supervision on decoded 3D joint positions further constraining generation in motion space. In addition to dataset-specific models, we train a single motion autoencoder on HumanML3D and KIT-ML, followed by a single text-conditioned SiT shared across both datasets, preserving their respective skeletal definitions without skeleton retargeting to a common joint layout. Experiments on HumanML3D and KIT-ML show that our dataset-specific models achieve state-of-the-art performance, while shared training yields further gains on both benchmarks, demonstrating the benefits of shared modeling across heterogeneous skeletal layouts.
Siyuan Yang, Chong Wang, Hao Cheng +4
Division of Robotics, Perception and Learning, KTH Royal Institute of Technology, Sweden · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · School of Artificial Intelligence, Hebei University of Technology, China +3
High-quality human motion data is becoming increasingly important for applications in robotics, simulation, and entertainment. Recent generative models offer a potential data source, enabling human motion synthesis through intuitive inputs like text prompts or kinematic constraints on poses. However, the small scale of public mocap datasets has limited the motion quality, control accuracy, and generalization of these models. In this work, we introduce Kimodo, an expressive and controllable kinematic motion diffusion model trained on 700 hours of optical motion capture data. Our model generates high-quality motions while being easily controlled through text and a comprehensive suite of kinematic constraints including full-body keyframes, sparse joint positions/rotations, 2D waypoints, and dense 2D paths. This is enabled through a carefully designed motion representation and two-stage denoiser architecture that decomposes root and body prediction to minimize motion artifacts while allowing for flexible constraint conditioning. Experiments on the large-scale mocap dataset justify key design decisions and analyze how the scaling of dataset size and model size affect performance.