Not Like Transformers: Drop the Beat Representation for Dance Generation with Mamba-Based Diffusion Model

Sangjune Park; Inhyeok Choi; Donghyeon Soon; Youngwoo Jeon; Kyungdon Joo

cs.CVMar 9, 2026

Not Like Transformers: Drop the Beat Representation for Dance Generation with Mamba-Based Diffusion Model

Authors: Sangjune Park, Inhyeok Choi, Donghyeon Soon, Youngwoo Jeon, Kyungdon Joo

Organizations: Ulsan National Institute of Science and Technology, South Korea · Daegu Gyeongbuk Institute of Science and Technology, South Korea

Abstract

Dance is a form of human motion characterized by emotional expression and communication, playing a role in various fields such as music, virtual reality, and content creation. Existing methods for dance generation often fail to adequately capture the inherently sequential, rhythmical, and music-synchronized characteristics of dance. In this paper, we propose MambaDance, a new dance generation approach that leverages a Mamba-based diffusion model. Mamba, well-suited to handling long and autoregressive sequences, is integrated into our two-stage diffusion architecture, substituting off-the-shelf Transformer. Additionally, considering the critical role of musical beats in dance choreography, we propose a Gaussian-based beat representation to explicitly guide the decoding of dance sequences. Experiments on AIST++ and FineDance datasets for each sequence length show that our proposed method effectively generates plausible dance movements while reflecting essential characteristics, consistently from short to long dances, compared to the previous methods. Additional qualitative results and demo videos are available at \small{https://vision3d-lab.github.io/mambadance}.

Figures & tables

Figure 1: We propose MambaDance , a Mamba-based two-stage diffusion framework with an informative Gaussian beat representation. The result is coherent, beat-synchronized motion across variable lengths on AIST++ [ 15 ] and FineDance [ 14 ] .

Figure 2: The overall architecture of MambaDance . We extract music feature $m$ , and a novel beat representation $b$ from the binary mask of beat of the feature (blue box). Two-stage diffusion architecture makes our approach enable length-agnostic generation in a single inference (green box). Decoder of the diffusion consists of the proposed Mamba [ 6 , 3 ] -based modules, e.g., Single-Modal Mamba (SMM), Cross-Modal Mamba (CMM), and Adaptive Linear Modulation (AdaLM) (gray box).

Figure 3: Single-Modal Mamba (left) and Cross-Modal Mamba (right). For the input sequences to the Cross-Modal Mamba, Light blue, dark blue, and pink blocks correspond to motion, condition, and timestep tokens, respectively.

Figure 4: Visualizations of raw beat (a), Nearest Beat Distance (NBD) (b), and our Gaussian beat representation (c). Horizontal axis denotes frame indices (time step) of a sequence and vertical axis indicates signal. The signal range of NBD and the proposed representation are $[0,11]$ and $[0,1]$ , respectively.

Dataset	Model	Fidelity			Beat	Diversity		Wins ( $\uparrow$ )
		$\operatorname{FID}_{k}$ ( $\downarrow$ )	$\operatorname{FID}_{g}$ ( $\downarrow$ )	$\operatorname{PFC}$ ( $\downarrow$ )	$\operatorname{BAS}$ ( $\uparrow$ )	$\operatorname{Div}_{k}$ ( $\rightarrow$ )	$\operatorname{Div}_{g}$ ( $\rightarrow$ )
FineDance	GT	-	-	0.1852	-	10.9924	7.7424	-
	EDGE	179.01 ±3.10	1234.28 ±236.95	0.3994 ±0.0177	0.2261 ±0.0013	10.34 ±0.36	31.74 ±2.76	3.5%
	POPDG	190.23 ±1.18	1479.14 ±6.44	0.3765 ±0.0124	0.2361 ±0.0033	7.22 ±0.14	14.53 ±0.12	5.5%
	Lodge	84.99 ±2.07	64.57 ±10.74	0.0585 ±0.014	0.2410 ±0.0063	7.98 ±0.20	7.67 ±0.68	35.0%
	Ours	51.36 ±0.67	43.11 ±10.54	0.0119 ±0.0008	0.2441 ±0.0044	6.38 ±0.17	6.44 ±0.88	56.0%

Table 1: Quantitative results on the FineDance and AIST++ datasets. GT motion is used as the reference.

\downarrow

indicates lower is better, and

\rightarrow

indicates closer to the real motion is better. The best and second-best results are highlighted in bold and underline, respectively.

Model	Fidelity			Beat	Diversity
Model	$\operatorname{FID}_{k}$ ( $\downarrow$ )	$\operatorname{FID}_{g}$ ( $\downarrow$ )	$\operatorname{PFC}$ ( $\downarrow$ )	$\operatorname{BAS}$ ( $\uparrow$ )	$\operatorname{Div}_{k}$ ( $\rightarrow$ )	$\operatorname{Div}_{g}$ ( $\rightarrow$ )
GT	-	-	0.1852	-	10.9924	7.7424
Lodge	84.99 ±2.07	64.57 ±10.74	0.0585 ±0.014	0.2410 ±0.0063	7.98 ±0.20	7.67 ±0.68
Mamba-only	60.94 ±2.01	43.46 ±7.29	0.0180 ±0.0017	0.2402 ±0.0044	6.48 ±0.40	7.21 ±0.79
Mamba+NBD	83.55 ±1.35	38.43 ±0.71	0.0687 ±0.0046	0.2478 ±0.0054	5.3958 ±0.17	5.78 ±0.30
Ours	51.36 ±0.67	43.11 ±10.54	0.0119 ±0.0008	0.2441 ±0.0044	6.38 ±0.17	6.44 ±0.88

Table 2: Ablation on Mamba and beat representation (FineDance). In contrast to the Transformer-based Lodge [ 13 ] , “Mamba-only” denotes a fully Mamba-based model without an explicit beat prior. “Mamba+NBD” uses Nearest Beat Distance [ 10 ] as a beat representation.

Figure 5: Qualitative comparison on the FineDance (top) and AIST++ (bottom) dataset. Each row shows a set of sampled frames captured at consistent intervals from the full sequence.

Ablations		Metrics
AdaLM	CMM	$\operatorname{FID}_{k}$ ( $\downarrow$ )	$\operatorname{PFC}$ ( $\downarrow$ )	$\operatorname{BAS}$ ( $\uparrow$ )	$\operatorname{Div}_{k}$ ( $\rightarrow$ )	$\operatorname{Div}_{g}$ ( $\rightarrow$ )
GT		-	-	0.1852	10.99	7.74
✗	✗	61.28 ±1.72	0.6316 ±0.0066	0.2465 ±0.0044	8.59 ±0.16	5.77 ±0.41
✗	✓	62.93 ±1.80	0.7878 ±0.0070	0.2479 ±0.0057	9.93 ±0.18	6.11 ±0.60
✓	✓	51.36 ±0.67	0.0119 ±0.0008	0.2441 ±0.0044	6.38 ±0.17	6.44 ±0.88

Table 3: Ablation study on different decoder structures. The last row corresponds to our MambaDance .

Human Motion Generation

Human motion generation creates articulated movement from language, audio, video, spatial constraints, or interaction goals.

See all 291 papers

TokenDance: Token-to-Token Music-to-Dance Generation with Bidirectional Mamba
Mar 28, 2026Ziyue Yang, Kaixing Yang, Xulong TangHuman Motion Generation
Brown University · Renmin University of China · The University of Texas at Dallas
FlowerDance: MeanFlow for Efficient and Refined 3D Dance Generation
Nov 26, 2025Kaixing Yang, Xulong Tang, Ziqiao Peng +4MeanFlow Human Motion Generation
Renmin University of China · Malou Tech Inc · Wuhan University +1
BeatDance: Generating Beat-Consistent 3D Dance with Hierarchical Spatial-Temporal Modeling
Sep 29, 2026Xiaojian Shen, Dahu Shi, Jianrong Zhang +7Human Motion Generation Audio-Visual Synchronization
College of Software, Jilin University, Changchun, 130012, China · College of Computer Science and Technology, Zhejiang University, Hangzhou, 310007, China · ReLER, AAII, University of Technology Sydney, Sydney, Australia +5

Mar 28, 2026J/K move · Enter open · S save

TokenDance: Token-to-Token Music-to-Dance Generation with Bidirectional Mamba

Ziyue Yang, Kaixing Yang, Xulong Tang

Brown University · Renmin University of China · The University of Texas at Dallas

Human Motion Generation

Open paper AskPDF arXiv

Music-to-dance generation has broad applications in virtual reality, dance education, and digital character animation. However, the limited coverage of existing 3D dance datasets confines current models to a narrow subset of music styles and choreographic patterns, resulting in poor generalization to real-world music. Consequently, generated dances often become overly simplistic and repetitive, substantially degrading expressiveness and realism. To tackle this problem, we present TokenDance, a two-stage music-to-dance generation framework that explicitly addresses this limitation through dual-modality tokenization and efficient token-level generation. In the first stage, we discretize both dance and music using Finite Scalar Quantization, where dance motions are factorized into upper and lower-body components with kinematic-dynamic constraints, and music is decomposed into semantic and acoustic features with dedicated codebooks to capture choreography-specific structures. In the second stage, we introduce a Local-Global-Local token-to-token generator built on a Bidirectional Mamba backbone, enabling coherent motion synthesis, strong music-dance alignment, and efficient non-autoregressive inference. Extensive experiments demonstrate that TokenDance achieves overall state-of-the-art (SOTA) performance in both generation quality and inference speed, highlighting its effectiveness and practical value for real-world music-to-dance applications.

Abstract

Human Motion Generation

Explore similar work

TokenDance: Token-to-Token Music-to-Dance Generation with Bidirectional Mamba

FlowerDance: MeanFlow for Efficient and Refined 3D Dance Generation

BeatDance: Generating Beat-Consistent 3D Dance with Hierarchical Spatial-Temporal Modeling