MIDI-REPA: Improving Text-to-Music Generation with Symbolic Representation Alignment

Song-Ze Yu; Fang-Duo Tsai; Yi-Hsuan Yang

cs.SDOct 14, 2026

MIDI-REPA: Improving Text-to-Music Generation with Symbolic Representation Alignment

Authors: Song-Ze Yu, Fang-Duo Tsai, Yi-Hsuan Yang

Organizations: National Taiwan University · University of California, San Diego

Abstract

State-of-the-art text-to-music models increasingly rely on scaling model capacity, data, and compute, while still learning musical structure directly from dense audio. Yet melody, rhythm, harmony, and structure can be represented compactly in symbolic form. We introduce MIDI-REPA, a representation-alignment framework that derives MIDI supervision from automatic music transcription (AMT) to align intermediate representations of a flow-matching DiT. Building on the 122M-parameter FluxAudio-S baseline from the 2026 ICME Academic Text-To-Music Challenge, we investigate what, where, and when to align. Aligning a MuScriptor-derived piano roll with an early DiT layer throughout the full flow trajectory performs the best. Without additional training data or techniques, MIDI-REPA remains competitive on the ATTM objective metrics while remaining trainable on a single RTX 3090 in about 31 hours. Our results highlight symbolic supervision as a simple and inexpensive learning signal for improving text-to-music generation. We release our transcription pipeline, JamendoMIDI, training code, and audio samples at https://midi-repa.vaclis.net.

Figures & tables

Figure 1: MIDI-REPA framework. Symbolic targets extracted by an AMT system supervise intermediate audio representations through a lightweight projection head and cosine-similarity alignment loss, jointly with the denoising objective. Our design study examines (a) what symbolic target to align, (b) where to apply alignment in the backbone, and (c) when to activate it along the flow-matching trajectory.

Teacher	FAD $\downarrow$	CLAP $\uparrow$
MuScriptor cube	$\underline{0.5117\pm 0.0042}$	$\mathbf{0.2665\pm 0.0030}$
MuScriptor roll	$\mathbf{0.5040\pm 0.0030}$	$\underline{0.2603\pm 0.0026}$
Basic Pitch roll	$0.5363\pm 0.0036$	$0.2396\pm 0.0029$
Baseline	$0.5275\pm 0.0021$	$0.2220\pm 0.0021$

Table 1: What to align: Symbolic targets at L4 on SDD. Bold and underline indicate the best and second-best results; gray indicates performance worse than the baseline.

Layer	FoG-A	Rank	Layer	FoG-A	Rank
L0	1.1206	2	L6	0.0917	4
L1	0.0869	5	L7	0.0552	7
L2	0.0412	8	L8	0.0365	10
L3	0.0244	12	L9	2.4628	1
L4	0.0779	6	L10	0.0280	11
L5	0.0982	3	L11	0.0389	9

Table 2: Baseline FoG-A over 512 training windows and five noise levels. Bold: top three contributions; these are not alignment-performance ranks.

Layer	FAD $\downarrow$	CLAP $\uparrow$
L0	$0.5289\pm 0.0032$	$0.2431\pm 0.0016$
L2	$\mathbf{0.4947\pm 0.0037}$	$\mathbf{0.2686\pm 0.0015}$
L4	$0.5040\pm 0.0030$	$\underline{0.2603\pm 0.0026}$
L5	$0.5202\pm 0.0036$	$0.2588\pm 0.0014$
L8	$\underline{0.4980\pm 0.0033}$	$0.2587\pm 0.0015$
L9	$0.5211\pm 0.0045$	$0.2457\pm 0.0023$

Table 3: Where to align: Alignment layer with MuScriptor roll on SDD. L0/L2 are joint blocks; the remaining layers are fused blocks.

System	Params	A100-eq GPU-H	FAD $\downarrow$	CLAP $\uparrow$	CCS $\uparrow$	Rank
Submission e07	402M	148.9	$0.417$	$0.261$	$0.867$	$2$
Submission e01	189M	25.1	$0.577$	$0.338$	$0.863$	$3$
Submission e05	499M	20.0	$0.487$	$0.305$	$0.800$	$4$
Submission e08	450M	190.3	$0.495$	$0.295$	$0.804$	$4$
Submission p05	2.4B	60.0	$0.514$	$0.306$	$0.800$	$6$
Submission p00	502M	48.0	$0.557$	$0.311$	$0.796$	$7$

Table 4: ATTM objective comparison on 80 in-domain prompts. Borda ranks are recomputed over the 17 official systems plus Ours; only the six official finalists and the FluxAudio-S baseline are shown. Training cost follows ATTM’s peak-FP16-throughput normalization.

Figure 2: Extrapolated final outputs from different timesteps along the baseline generation trajectory for the prompt “Atmospheric music featuring a soothing vibraphone, enhanced with gentle piano and ambient synth pads, evokes a calm, reflective mood with soft, resonant tones and a spacious, dreamy texture.” Musical structure becomes largely established around $t=0.88$ – $0.80$ .

Alignment interval $t$	FAD $\downarrow$	CLAP $\uparrow$
$t\in[0,1]$	$\mathbf{0.4947\pm 0.0037}$	$\underline{0.2686\pm 0.0015}$
$t\in[0.88,1]$	$0.5440\pm 0.0042$	$\mathbf{0.2858\pm 0.0018}$
$t\in[0.84,1]$	$0.5201\pm 0.0038$	$0.2686\pm 0.0036$
$t\in[0.80,1]$	$\underline{0.5200\pm 0.0033}$	$0.2641\pm 0.0031$
$t\in[0,0.8]$	$0.5205\pm 0.0029$	$0.2683\pm 0.0016$
$t\in[0,0.2]$	$0.5828\pm 0.0035$	$0.2581\pm 0.0024$

Table 5: When to align: timestep interval with MuScriptor roll at L2 on SDD. Generation proceeds from

t=1

(noise) to

t=0

(music).

Text-to-Music Generation

Text-to-music generation turns natural-language descriptions into audio or symbolic scores, with control over style, instruments, mood, lyrics, and structure.

See all 48 papers

MI-MIDI: Mechanistic Interpretability of Text-to-MIDI Generation Models via Probing, Lenses and Steering
Aug 6, 2026Jakub Poćwiardowski, Mateusz ModrzejewskiControllable Music Generation Symbolic Music Generation
Institute of Computer Science Warsaw University of Technology Warsaw, Poland
Making the Most of Limited Data: Score-Aware Training for Text-to-Music Generation
Jun 5, 2026Yun-Chen Cheng, Tzu-Hung Huang, Chih-Pin TanText-to-Music Generation Audio-Text Alignment
Graduate Institute of Communication Engineering, National Taiwan University, Taipei, Taiwan
Instrumental Text-to-Music Generation with Auxiliary Conditioning Branches
May 20, 2026Junyoung KohText-to-Music Generation Diffusion Transformer
Department of Artificial Intelligence, Yonsei University MAAP KRAFTON Seoul, Republic of Korea

Aug 6, 2026J/K move · Enter open · S save

MI-MIDI: Mechanistic Interpretability of Text-to-MIDI Generation Models via Probing, Lenses and Steering

Jakub Poćwiardowski, Mateusz Modrzejewski

Institute of Computer Science Warsaw University of Technology Warsaw, Poland

Controllable Music Generation Symbolic Music Generation Mechanistic Interpretability Activation Steering

Open paper AskPDF arXiv

Mechanistic interpretability of music generation has concentrated on audio models, leaving symbolic models largely unexplored. We analyze two public text-to-MIDI systems of contrasting design: the purpose-built encoder--decoder text2midi and MIDI-LLM, a Llama~~3.2~~1B model extended with MIDI tokens using linear probing, the logit and tuned lenses, activation patching and difference-in-means steering. Across these methods, we recover musically meaningful structure and show how architecture shapes its formation and control. Pitch, instrumentation, harmony and texture are linearly decodable in both models. text2midi refines predictions gradually across depth, whereas MIDI-LLM works largely in its inherited textual basis before a sharp late rotation into the musical vocabulary; patching identifies a matching late attenuation of prompt-driven instrument transfer. Steering produces bidirectional changes in register and polyphony in both systems, and in tempo/energy in MIDI-LLM. Our two-orientation protocol isolates directional control and shows that all-layer interventions are robust in text2midi but accumulate disruptively in MIDI-LLM. Together, the results provide a practical toolkit for tracing and controlling musical concepts in symbolic generators. Audio examples are available on a demo website.

Abstract

Text-to-Music Generation

Explore similar work

MI-MIDI: Mechanistic Interpretability of Text-to-MIDI Generation Models via Probing, Lenses and Steering

Making the Most of Limited Data: Score-Aware Training for Text-to-Music Generation

Instrumental Text-to-Music Generation with Auxiliary Conditioning Branches