Music-driven character animation enables and enhances transformative applications in entertainment and interactive education. However, synthesizing realistic drumming motion from audio remains challenging due to the inherent tension between high-acceleration dynamics and the need for extreme spatial-temporal precision. Existing approaches, often reliant on motion matching or MIDI input, struggle with generalizing to diverse real-world audio. Moreover, the field lacks standardized evaluation metrics capable of distinguishing precise drumming from noisy motion. In this paper, we introduce a generative diffusion framework featuring a dual-objective loss function that decouples skeletal integrity from drumstick precision, thus enabling centimeter-level stick precision without sacrificing natural body dynamics. Additionally, leveraging our own dataset and data augmentation strategy, the model generalizes to non-curated, in-the-wild audio. To rigorously evaluate performance, we propose two novel metrics: an impact-to-target distance to quantify spatial precision and an audio-motion correlation score to assess temporal alignment. Our quantitative analysis and user studies demonstrate that our system generates high-quality motion that is often indistinguishable from ground-truth performances.
Figures & tables
Figure 1: Overview. We augment the audio in our dataset resulting in motion sequences aligned with multiple audio variations to ensure generalization. Raw audio is then processed into explainable features to condition a diffusion-based Transformer Decoder. We employ a hybrid pose representation, using joint rotations for the body and Cartesian positions for drumsticks. The model is trained via a dual-objective loss to balance natural body dynamics with high-precision stick impacts. Finally, performance is evaluated using Impact Point Deviations and Percussive Alignment Scores to assess spatial and temporal fidelity.
Figure 2: Standard 10-component drum kit used in our data collection. (1) Snare, (2) bass/kick drum, (3) hi-hat, (4) hi-hat pedal, (5-6) high toms, (7) floor tom, (8) ride cymbal, (9-10) crash cymbals.
Figure 3: Dual-objective loss achieves superior spatial precision. Stick-tip trajectories during a snare roll. Ground truth (left), rotations-only model (center), and our dual-objective model (right). Tight clustering on the drum surface demonstrates consistent impact points.
Component
Rot. Only (cm) ↓
Ours (cm) ↓
Snare
14.5
0.5
High tom (L)
5.0
1.2
High tom (R)
8.2
3.0
Floor tom
6.4
3.3
Mean (drums)
8.5
2.0
Hi-hat
2.5
1.3
Table 1: Impact point deviation per drum component. Values are Euclidean distances (cm) between the impact point centroids of our generated motion and ground-truth over the test set. Lower values are indicative of higher-quality motion. All standard errors are below 0.01 cm for the rotations-only model and below 0.006 cm for the dual-objective model (ours).
Figure 4: Our model achieves temporal alignment comparable to ground-truth data. PAS distributions as violin plots for GT (leftmost, black), noisy GT (second and fourth, red), rotations-only model (middle, purple), and our model (rightmost, orange), when applied to our test dataset. Each sample is one-second long. Noisy GT is obtained by adding Gaussian noise with standard deviation σ =50 ms and σ=25 ms to every motion onset time. When noise is added to the data, PAS decreases, indicating that the metric successfully discriminates audio-motion alignment quality.
Figure 5: Users show comparable preference for motion-capture data and for motion generated by our model. Our generated motions (orange) strongly outperform the rotations-only model (RO, purple) with a 92.8% preference rate ( p -value <0.001 ). When compared to ground truth (GT, blue), our method achieves a 41.3% preference rate, demonstrating no statistically significant difference from a random guess ( p -value 0.08 ).
Figure 6: Data augmentation ensures synthesis of high-quality motion from wild audio. PAS distributions as violin plots for motion generated from 100 random audio samples from the Groove dataset. We compare our dual-objective model trained without (left) and with (right) data augmentation.
Figure 7: Emergent gain modulation. Motion generated by our dual-objective model with input audio at a normal volume (0 dB, pink) and a low volume (-15 dB, green). Loud audio produces with larger arcs, while softer audio produces gentle trajectories with smaller arcs. This behavior is learnt from our dataset’s intensity diversity.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Our clustering algorithm filters out noise to find the impact points for each drum component. Candidate impact points (top) and filtered impact points (bottom). Cluster centroids indicated as a black marker. Data corresponding to our ground-truth test set.
Figure 9: Impact-point clusters of generated motion and ground truth overlap. Clusters for the different drum components projected onto the XY (top-down), XZ (front-facing), and YZ (side-profile) planes. Blue points represent ground truth data, while orange points indicate generated impact distributions. The central black dot and green arrow indicate the drummer’s seat position and facing orientation, respectively. All units are in centimeters (cm). The set of impact points shown here was chosen at random from our test set.
Figure 10: Peaks in stick acceleration are more clearly correlated with audio events than peaks in velocity. Magnitude of the velocity (orange) and the acceleration (blue) of the tip of the right stick, in one of the ground-truth motion sequences. Note activations from the MIDI signal are indicated as colored bars (left vertical axis). The right stick hits the hi-hat at ∼3.6 s and the crash at ∼5 s. The hi-hat pedal is activated with the foot, while the first hi-hat, the first crash, and the first snare hits are triggered with the left stick.
Figure 11: Qualitative comparison of drum transcription performance . The visualization displays the ground-truth (GT) audio waveform and the corresponding GT MIDI events. Each drum component is played twice, sequentially: kick (black), hi-hat pedal (light olive), hi-hat (dark olive), ride (blue purple), snare (yellow), tom 1 (blue), tom 2 (green), crash 1 (purple), floor tom (red), and crash 2 (aquamarine). The SOTA drum transcription [ ZAB23 ] exhibits some classification errors, such as identifying the incorrect drum and failing to differentiate between various toms. In contrast, our transcription, generated via the audio-to-motion and motion-to-MIDI pipeline, demonstrates high alignment with the ground truth across the entire instrument set.
Current methods for creating drum loop audio in digital music production, such as using one-shot samples or resampling, often demand non-trivial efforts of creators. While recent generative models achieve high fidelity and adhere to text, they lack the specific control needed for such a task. Existing symbolic-to-audio research often focuses on single, tonal instruments, leaving the challenge of polyphonic, percussive drum synthesis unaddressed. We address this gap by introducing ``Break-the-Beat!,'' a model capable of rendering a drum MIDI with the timbre of a reference audio. It is built by fine-tuning a pre-trained text-to-audio model with our proposed content encoder and a effective hybrid conditioning mechanism. To enable this, we construct a new dataset of paired target-reference drum audio from existing drum audio datasets. Experiments demonstrate that our model generates high-quality drum audio that follows high-resolution drum MIDI, achieving strong performance across metrics of audio quality, rhythmic alignment, and beat continuity. This offer producers a new, controllable tool for creative production. Demo page: https://ik4sumii.github.io/break-the-beat/
We study real-time audio-responsive character control as a deployment-faithful problem: strictly causal, bounded-latency streaming that must generate coherent full-body motion at interactive frame rates while the audio condition can change abruptly, including tempo shifts, drops, or user edits. Prior music-to-motion systems are largely optimized for offline generation with global context, and degrade in streaming rollouts where conditioning history becomes stale or unreliable. We introduce DiscoForcing, a streaming audio-driven diffusion framework that combines a causal music encoder that captures rhythmic structure and phase dynamics with a diffusion-forcing sequence model trained under heterogeneous noise levels across the temporal horizon. Building on this, we design a hybrid temporal schedule and a history-guided streaming sampler to explicitly trade off responsiveness against long-horizon consistency under non-stationary audio. Implemented in an end-to-end real-time interactive system with online avatar playback and humanoid deployment workflows, DiscoForcing delivers more stable long-horizon rollouts and sharper audio-motion alignment than prior baselines under matched causality and latency constraints while maintaining real-time throughput.
Generating realistic drum audio directly from symbolic representations is a challenging task at the intersection of music perception and machine learning. We propose a system that transforms an expressive drum grid, a time-aligned MIDI representation with microtiming and velocity information, into drum audio by predicting discrete codes of a neural audio codec. Our approach uses a Transformer-based model to map the drum grid input to a sequence of codec tokens, which are then converted to waveform audio via a pre-trained codec decoder. We experiment with multiple state-of-the-art neural codecs, namely EnCodec, DAC, and X-Codec, to assess how the choice of audio representation impacts the quality of the generated drums. The system is trained and evaluated on the Expanded Groove MIDI Dataset, E-GMD, a large collection of human drum performances with paired MIDI and audio. We evaluate the fidelity and musical alignment of the generated audio using objective metrics. Overall, our results establish codec-token prediction as an effective route for drum grid-to-audio generation and provide practical insights into selecting audio tokenizers for percussive synthesis.