Transfer Learning for Rhythm Game Chart Generation from Music Transcription and Across Games
Authors: Sojeong An, Nathan Pruyne, Chris Donahue, Dasaem Jeong
Organizations: Music & Arts Learning (MALer) Lab, Sogang University, South Korea · Computer Science Department, Carnegie Mellon University, Pittsburgh, USA
Rhythm games ask the player to act in time with a recording, following a chart: a sequence of actions, each with a time and, depending on the game, a lane, type, or playfield position. Charts are hand-written per recording and difficulty. Automatic generation from audio has been studied with pipeline and end-to-end approaches, and recent work reuses pre-trained audio encoders, but never compares them with one trained from scratch. None of these targets automatic music transcription (AMT), yet most chart times are note onsets, which AMT models locate. We freeze a pre-trained audio encoder, an AMT model in particular, and train only a linear projection and a Transformer decoder on charts. With MuScriptor, this fits human charts more closely than encoders trained from scratch and places more accurate note types than MERT, a music encoder pre-trained without transcription labels. Joint training on three osu! modes raises event F1 on the mode with the fewest charts from 0.492 to 0.506, while charts from Clone Hero, similar in layout, lower it to 0.482. Transfer between games thus appears to depend on charting style and the music covered.
Figures & tables
Figure 1: Overview of the system.
Figure 2: Generated charts in play, each with the token sequence of one segment; the Clone Hero panel shows a chart from the corpus. Standard places notes freely on a 2D playfield, while taiko , mania and Clone Hero assign each note a lane or a column. Mania prepends the set of columns already held before BOS.
Mode
Songs
Charts
Charts/song
Hours
Stars
standard
2,111
8,687
4.12
117.9
1.03–13.15
taiko
1,845
7,734
4.19
92.6
0.63–10.36
mania
1,955
6,268
3.21
99.5
0.59–10.07
Clone Hero
2,782
2,782
1.00
210.6
—
Table 1: Dataset statistics.
standard
taiko
Onset
Event
Onset
Event
Encoder
Trained
NLL ↓
P
R
F1
F1
Type
Trained
NLL ↓
P
R
F1
F1
Type
C-LSTM [ 1 ]
0.7M
1.388
0.628
0.688
0.654
0.420
0.637
0.7M
0.894
0.753
0.741
0.743
0.391
0.524
Transformer [ 7 ]
240.2M
1.447
0.507
0.587
0.541
0.323
0.591
239.9M
0.948
0.706
0.715
0.707
0.367
0.516
YourMT3+ [ 9 ]
1.6M
1.429
0.600
0.642
0.617
0.388
0.622
0.8M
0.944
0.713
0.728
0.716
0.370
0.514
MuScriptor [ 7 ]
0.5M
1.359
0.646
0.675
0.657
0.433
0.653
0.3M
0.846
0.754
0.747
0.746
0.403
0.538
Table 2: Onsets are matched at 25 ms; Event F1 also requires the note type, and Type is the type accuracy over matched onsets. NLL is the teacher-forced cross-entropy per token; other scores are averaged over charts. Trained is what is learned between the audio and the decoder — encoder and projection when trained from scratch, projection alone when frozen — and excludes the decoder (25M standard , 4M taiko ).
Hold
Trained on
NLL ↓
Onset
Event
Tap
Start
End
mania only
0.473
0.728
0.492
0.599
0.122
0.072
mania + standard + taiko
0.501
0.738
0.506
0.612
0.099
0.056
mania + Clone Hero
0.506
0.707
0.482
0.584
0.096
0.054
Table 3: The MuScriptor encoder and the decoder are fixed across rows; only the training data varies. Onset F1 at 25 ms; the last three columns are per-class F1.
We present STRUM (Spectral Transcription and Rhythm Understanding Model), an audio-to-chart pipeline that converts raw recordings into playable Clone Hero / YARG charts for drums, guitar, bass, vocals, and keys without any oracle metadata. STRUM is a multi-stage hybrid: a two-stage CRNN onset detector and a six-model ensemble classifier for drums; neural onset detectors with monophonic pitch tracking for guitar and bass; word-aligned ASR for vocals; and spectral keyboard detection for keys. We evaluate on a 30-song in-envelope benchmark constructed by screening candidate songs on a single audio-quality criterion -- the median 1-second drum-stem RMS after htdemucs_6s source separation. On this benchmark STRUM achieves drums onset F1 = 0.838, bass F1 = 0.694, guitar F1 = 0.651, and vocals F1 = 0.539 at a +/- 100 ms tolerance with per-song global offset search. We report a complete ablation of seven drum-pipeline components with paired per-song Wilcoxon tests, an analysis of ground-truth-to-audio timing distributions in community Clone Hero charts, and a per-class confusion matrix for the drum classifier. Code, model weights, and the full benchmark manifest are released.
Existing methods for automatic music transcription are often limited to single-instrument recordings or fail on complex, real music mixes. Although previous work utilizes synthetic training data, the resulting models generalize poorly, leading to largely unusable transcription output in realistic, multi-instrument settings. In this work, we analyze the effectiveness of synthetic data for pre-training while combining it with fine-tuning on real music audio and post-training using reinforcement learning. We further introduce conditioning on instrument presence to customize transcriptions. Finally, we release MuScriptor, an open-weight multi-instrument music transcription model that works on real-world music recordings from across a diverse range of musical genres.
Simon Rouard, Michael Krause, Axel Roebel +2
Kyutai · Mirelo AI · UMR STMS, IRCAM-CNRS Sorbonne Univ.
Transcribing music into a human-readable score requires a coherent understanding of rhythm, harmony, melody, and form. Two obstacles limit this goal: annotated recordings are scarce, and accurate local predictions can still produce inconsistent musical sequences. We present SheetSage2, a unified music transcription framework that combines synthetic data, task-specific structured decoding, and autoregressive distillation. Automatically annotated MIDI, rendered into audio, provides scalable supervision across music understanding tasks. Task-specific structured decoders integrate complementary musical cues and their temporal dependencies to produce musically coherent scores. Autoregressive distillation further retains transcription accuracy without task-specific dynamic programming at inference. Across eight benchmark collections, a single SheetSage2-AR model exceeds the listed prior systems on 12 of 15 benchmark--metric pairs in our evaluation, substantially improving over SheetSage1 and surpassing task-specific models on several benchmarks. Model weights and inference code are publicly available.