Transfer Learning for Rhythm Game Chart Generation from Music Transcription and Across Games

Sojeong An; Nathan Pruyne; Chris Donahue; Dasaem Jeong

cs.SDOct 10, 2026

Transfer Learning for Rhythm Game Chart Generation from Music Transcription and Across Games

Authors: Sojeong An, Nathan Pruyne, Chris Donahue, Dasaem Jeong

Organizations: Music & Arts Learning (MALer) Lab, Sogang University, South Korea · Computer Science Department, Carnegie Mellon University, Pittsburgh, USA

Abstract

Rhythm games ask the player to act in time with a recording, following a chart: a sequence of actions, each with a time and, depending on the game, a lane, type, or playfield position. Charts are hand-written per recording and difficulty. Automatic generation from audio has been studied with pipeline and end-to-end approaches, and recent work reuses pre-trained audio encoders, but never compares them with one trained from scratch. None of these targets automatic music transcription (AMT), yet most chart times are note onsets, which AMT models locate. We freeze a pre-trained audio encoder, an AMT model in particular, and train only a linear projection and a Transformer decoder on charts. With MuScriptor, this fits human charts more closely than encoders trained from scratch and places more accurate note types than MERT, a music encoder pre-trained without transcription labels. Joint training on three osu! modes raises event F1 on the mode with the fewest charts from 0.492 to 0.506, while charts from Clone Hero, similar in layout, lower it to 0.482. Transfer between games thus appears to depend on charting style and the music covered.

Figures & tables

Panel of Figure 2 — Figure 2: Generated charts in play, each with the token sequence of one segment; the Clone Hero panel shows a chart from the corpus. Standard places notes freely on a 2D playfield, while taiko , mania and Clone Hero assign each note a lane or a column. Mania prepends the set of columns already held before BOS.

Mode	Songs	Charts	Charts/song	Hours	Stars
standard	2,111	8,687	4.12	117.9	1.03–13.15
taiko	1,845	7,734	4.19	92.6	0.63–10.36
mania	1,955	6,268	3.21	99.5	0.59–10.07
Clone Hero	2,782	2,782	1.00	210.6	—

Table 1: Dataset statistics.

	standard							taiko
			Onset			Event				Onset			Event
Encoder	Trained	NLL $\downarrow$	P	R	F1	F1	Type	Trained	NLL $\downarrow$	P	R	F1	F1	Type
C-LSTM [ 1 ]	0.7M	1.388	0.628	0.688	0.654	0.420	0.637	0.7M	0.894	0.753	0.741	0.743	0.391	0.524
Transformer [ 7 ]	240.2M	1.447	0.507	0.587	0.541	0.323	0.591	239.9M	0.948	0.706	0.715	0.707	0.367	0.516
YourMT3+ [ 9 ]	1.6M	1.429	0.600	0.642	0.617	0.388	0.622	0.8M	0.944	0.713	0.728	0.716	0.370	0.514
MuScriptor [ 7 ]	0.5M	1.359	0.646	0.675	0.657	0.433	0.653	0.3M	0.846	0.754	0.747	0.746	0.403	0.538

Table 2: Onsets are matched at 25 ms; Event F1 also requires the note type, and Type is the type accuracy over matched onsets. NLL is the teacher-forced cross-entropy per token; other scores are averaged over charts. Trained is what is learned between the audio and the decoder — encoder and projection when trained from scratch, projection alone when frozen — and excludes the decoder (25M standard , 4M taiko ).

					Hold
Trained on	NLL $\downarrow$	Onset	Event	Tap	Start	End
mania only	0.473	0.728	0.492	0.599	0.122	0.072
mania + standard + taiko	0.501	0.738	0.506	0.612	0.099	0.056
mania + Clone Hero	0.506	0.707	0.482	0.584	0.096	0.054

Table 3: The MuScriptor encoder and the decoder are fixed across rows; only the training data varies. Onset F1 at 25 ms; the last three columns are per-class F1.

Transfer Learning

Transfer learning studies how models reuse knowledge across tasks, domains, or modalities when target data or computation are limited.

See all 422 papers

STRUM: A Spectral Transcription and Rhythm Understanding Model for End-to-End Generation of Playable Rhythm-Game Charts
May 12, 2026Joshua OpriaAudio Understanding Automatic Music Transcription
Independent Researcher
MuScriptor: An Open Model for Multi-Instrument Music Transcription
Jul 9, 2026Simon Rouard, Michael Krause, Axel Roebel +2Automatic Music Transcription
Kyutai · Mirelo AI · UMR STMS, IRCAM-CNRS Sorbonne Univ.
SheetSage2: Coherent Lead-Sheet Transcription with Synthetic Supervision
Oct 4, 2026Junyan Jiang, Ruibin Yuan, Jiahao Pan +4Automatic Music Transcription Music Understanding
New York University · MBZUAI · ACE Studio +1

May 12, 2026J/K move · Enter open · S save

STRUM: A Spectral Transcription and Rhythm Understanding Model for End-to-End Generation of Playable Rhythm-Game Charts

Joshua Opria

Independent Researcher

Audio Understanding Automatic Music Transcription Sound Event Detection Music Information Retrieval Audio Classification

Open paper AskPDF arXiv

We present STRUM (Spectral Transcription and Rhythm Understanding Model), an audio-to-chart pipeline that converts raw recordings into playable Clone Hero / YARG charts for drums, guitar, bass, vocals, and keys without any oracle metadata. STRUM is a multi-stage hybrid: a two-stage CRNN onset detector and a six-model ensemble classifier for drums; neural onset detectors with monophonic pitch tracking for guitar and bass; word-aligned ASR for vocals; and spectral keyboard detection for keys. We evaluate on a 30-song in-envelope benchmark constructed by screening candidate songs on a single audio-quality criterion -- the median 1-second drum-stem RMS after htdemucs_6s source separation. On this benchmark STRUM achieves drums onset F1 = 0.838, bass F1 = 0.694, guitar F1 = 0.651, and vocals F1 = 0.539 at a +/- 100 ms tolerance with per-song global offset search. We report a complete ablation of seven drum-pipeline components with paired per-song Wilcoxon tests, an analysis of ground-truth-to-audio timing distributions in community Clone Hero charts, and a per-class confusion matrix for the drum classifier. Code, model weights, and the full benchmark manifest are released.

Abstract

Transfer Learning

Explore similar work

STRUM: A Spectral Transcription and Rhythm Understanding Model for End-to-End Generation of Playable Rhythm-Game Charts

MuScriptor: An Open Model for Multi-Instrument Music Transcription

SheetSage2: Coherent Lead-Sheet Transcription with Synthetic Supervision