Can Music Generation Models Tap Along? Evaluating Generative Approaches to Beat Tracking
Authors: Kun Fang, Ziyu Wang, Ichiro Fujinaga
Organizations: Schulich School of Music, McGill University, Montréal, QC, Canada · Centre for Interdisciplinary Research in Music Media and Technology (CIRMMT), Montréal, Canada · Courant Institute, New York University, New York, USA · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, UAE
Existing studies have shown that music generative models can generate synchronized tracks that play alongside an input song. Can we extend this capability to develop a generative approach to beat tracking? To answer this question, we equip a pretrained MusicGen model with an audio-conditioning adapter to generate percussion tracks synchronized with the input music, from which we extract the beat positions. We train separate models with percussion targets of varying rhythmic complexity and compare them with two other generative models and conventional beat trackers. On GTZAN, our model achieves performance close to state-of-the-art beat trackers. Our model also demonstrates greater robustness: it shows the smallest drop in F-measure when drums are removed from the input at inference time. In certain genres, such as metal and disco, our model is also less likely to be misled by strong onsets or to miss beats with weak onsets. These results suggest that percussion generation has the potential to become a viable approach to beat tracking, and that its output is not merely a sequence of beat positions but an audible percussion track.
Figures & tables
Figure 1: Overall pipeline. The music generative model generates a percussion track from the song; we compare three targets and train one model per target. A lightweight beat tracker then extracts the beat positions.
Figure 2: Our percussion generation model. N and N′ are the numbers of layers of the two MusicGen models; T is the number of frames. Gray modules are frozen, and orange modules are trained.
Method
Training songs
Params
F 70
F 100
F 150
CMLt
AMLt
CMLc
AMLc
PQ
Ours (Click)
1319 (49 h)
10.4 M
0.851
0.871
0.889
0.756
0.907
0.731
0.874
6.27
Ours (Groove synth)
1319 (49 h)
10.4 M
0.850
0.866
0.882
0.736
0.912
0.713
0.881
6.88
Ours (Groove sampled)
1319 (49 h)
10.4 M
0.849
0.863
0.878
0.727
0.907
0.704
0.877
5.93
Stable Audio ControlNet
1319 (49 h)
561 M
0.691
0.728
0.772
0.553
0.753
0.512
0.687
7.54
STAGE
214 (13 h)
420 M
0.335
0.443
0.609
0.244
0.482
0.201
0.387
7.09
Beat This!
4556 (167 h)
20.3 M
0.888
0.901
0.913
0.793
0.895
0.751
0.834
–
Table 1: Results on GTZAN (full-mix input). Top: generative methods; bottom: traditional beat tracking models. F x : F-measure with a ±x ms tolerance. Params: trainable parameters, excluding the TCN beat tracker. PQ: Production Quality of Audiobox Aesthetics (1–10). Bold: best; underline: second best.
Figure 3: F-measure (70 ms, top) and AMLt (bottom) of each method before and after the drum track is removed. Filled markers are the original song and open markers the drum-removed audio; a shorter connecting line means a smaller drop and greater robustness. Numbers below the lines are the drops. Ours and Ours (w/o DR aug) both use the Groove (sampled) target; w/o DR aug denotes training without drum-removal augmentation.
Figure 4: Miss rate (top) and false-positive rate (bottom) as a function of onset strength for each genre of GTZAN on drum-removed audio; lower is better for both. Only the songs for which all methods have the correct tempo are used (458 in total).
Figure 5: Onset strength s(t) and the beats of each method on gtzan_disco_00036 (11–16 s), with the full mix on top and drum-removed audio below. Numbers give the onset strength group (1 = weakest, 5 = strongest).
Current methods for creating drum loop audio in digital music production, such as using one-shot samples or resampling, often demand non-trivial efforts of creators. While recent generative models achieve high fidelity and adhere to text, they lack the specific control needed for such a task. Existing symbolic-to-audio research often focuses on single, tonal instruments, leaving the challenge of polyphonic, percussive drum synthesis unaddressed. We address this gap by introducing ``Break-the-Beat!,'' a model capable of rendering a drum MIDI with the timbre of a reference audio. It is built by fine-tuning a pre-trained text-to-audio model with our proposed content encoder and a effective hybrid conditioning mechanism. To enable this, we construct a new dataset of paired target-reference drum audio from existing drum audio datasets. Experiments demonstrate that our model generates high-quality drum audio that follows high-resolution drum MIDI, achieving strong performance across metrics of audio quality, rhythmic alignment, and beat continuity. This offer producers a new, controllable tool for creative production. Demo page: https://ik4sumii.github.io/break-the-beat/
Over the past two decades, the task of musical beat tracking has transitioned from heuristic onset detection algorithms to highly capable deep neural networks (DNN). Although DNN-based beat tracking models achieve near-perfect performance on mainstream, percussive datasets, the SMC dataset has stubbornly yielded low F-measure scores. By testing how well state-of-the-art models detect beats on individual tracks in the SMC dataset, we identify three distinct failure modes: octave errors, continuity errors, and complete tracking failure where all metrics fall below 0.3. We reveal that state-of-the-art models tend to generate "confident-but-wrong" activations. Furthermore, we show that the standard DBN's default minimum tempo of 55 BPM prevents it from inferring the correct tempo for 21% of SMC tracks, forcing double-tempo predictions on slow music. By exposing such fundamental oversights, we provide concrete directions for improving beat and downbeat detection, specifically emphasizing training data diversification and multi-hypothesis tempo estimation.
Current neural networks for beat tracking generate invalid outputs, such as consecutive downbeats and erratic tempo changes, even when these are not present in the training data. Heavy post-processing techniques can alleviate these problems, but the original cause of this inconsistent behaviour remains unknown. We hypothesise that it stems from inadequate modelling of multiple plausible output beat grids, resulting in an invalid mixture of competing interpretations. We propose a masked diffusion approach that properly models multiple outputs and enables the model to build coherent predictions through iterative inference. We devise three modifications to standard masked diffusion that enable its application to beat tracking: independent masking of beats and downbeats during training and inference, a balanced masking scheduler for inference, and peak-picking across inference steps. Our approach reduces erratic behaviours and improves beat-tracking performance.
Francesco Foscarin, Filip Korzeniowski, Richard Vogl