MI-MIDI 2: From Observation to Control with Sparse Autoencoders in Text-to-MIDI Generation

Jakub Poćwiardowski; Mateusz Modrzejewski

cs.SDOct 23, 2026

MI-MIDI 2: From Observation to Control with Sparse Autoencoders in Text-to-MIDI Generation

Authors: Jakub Poćwiardowski, Mateusz Modrzejewski

Organizations: Institute of Computer Science, Warsaw University of Technology, Warsaw, Poland

Abstract

Sparse autoencoders (SAEs) can expose features of a generative model without predefined labels, but in language models the evidence used to interpret these features has proven an unreliable guide to which of them steer the output. Whether interpretations of SAE features predict control in music generators has not, to our knowledge, been tested. We develop an end-to-end protocol for discovering and evaluating SAE features in text-to-MIDI generation and apply it to text2midi and MIDI-LLM. The protocol characterizes features through correlations with musical metrics, shared instrument programs, and automated descriptions, each with its own control: note-density adjustment, matched inactive excerpts, and permutation calibration. It then tests candidates by intervention, measured directly in the generated MIDI against unmodified generation and a matched random direction, ranks them in screens, and confirms selected features on new prompts. Evidence that survives these controls points to features that steer the output. Descriptions exceed permutation controls for instrumentation, and a feature whose description names a synth-lead program introduces that program in a prespecified confirmation and in a replication on disjoint prompts and reappears in independently trained dictionaries. Features selected by density-adjusted correlations show a small positive shift of their target metric as a family, whereas features selected by raw correlations show none. An atlas provides MIDI, piano-roll, and audio examples for every feature of the targeted screen.

Figures & tables

Figure 1: Overview of the experiments. Each observational route has its own control; interventions compare the feature direction with baseline generation and a matched random direction, with endpoints measured in MIDI. Screens rank candidates and give family-level estimates; selected features are confirmed on new prompts.

Model	Layer	Features	FVU	Dead (%)	Frequent (%)	Retained
text2midi	2	6144	0.0311	2.18	53.22	2369
text2midi	8	6144	0.0747	0.63	55.79	2143
text2midi	13	6144	0.1750	0.42	63.17	1746
text2midi	16	6144	0.1568	0.16	61.12	2082
MIDI-LLM	2	8192	0.2493	2.42	22.66	3742
MIDI-LLM	7	8192	0.3672	0.31	21.02	4422

Table 1: The final dictionaries retain large candidate sets after frequency filtering. FVU: on the extracted activation corpus; frequent: firing in more than 25% of windows; retained: firing in 1–25%. One training run per row; the models differ in architecture, tokenization, segmentation, and dictionary size.

Model	Layer	Eligible	Raw	Adj.	Raw	Adj.	Kept (%)
			$\tau=0.10$		$\tau=0.20$
text2midi	2	5092	3295	1146	1715	149	8.7
text2midi	8	4724	3321	1356	1773	221	12.5
text2midi	13	4854	3469	1358	1954	199	10.2
text2midi	16	5208	3565	1396	1730	150	8.7
MIDI-LLM	2	4057	1773	534	527	38	7.2

Table 2: Adjusting for note and token counts removes most replicated metric associations. Counts: features whose strongest discovery metric reaches

|r|\geq\tau

with the same sign on discovery, validation, and test segments, before (raw) and after (adj.) residualizing on log counts; eligible: firing on 1–50% of discovery segments; kept: adj./raw at

\tau=0.20

. Earlier checkpoints: Appendix C .

Figure 2: Only instrumentation descriptions exceed the permutation calibration. After describing a feature, the LLM reviewer picks four of eight held-out excerpts, four of which activate the feature. Points: mean correct picks, 95% intervals resampling cards (group sizes in parentheses); line: random choice (2 of 4); band: 95% interval of the 30-card calibration with shuffled activations; form and unresolved scopes (24 cards) omitted. Values: Table 15 .

Feature	Description	Recognition
text2midi L13 f3340	Synth-lead (Lead 2 sawtooth) / electronic-patch instrumentation	4/4
MIDI-LLM L7 f499	Solo acoustic-piano melody	4/4
MIDI-LLM L2 f5880	Short, detached notes with thin overlap	4/4
MIDI-LLM L2 f1732	Dense, drum-heavy texture with chromatic/out-of-scale notes	4/4
MIDI-LLM L2 f793	Wide, harmonically static chord hits with weak tonal clarity	4/4

Table 3: Selected audit-passing descriptions identify all four active held-out excerpts. Labels are the LLM reviewer’s short labels from the archived responses.

Pool	Selection	Features	Hypotheses	Evaluation
MusPy	The strongest discovery/validation association for each metric and model.	15	24	The metric changes in the direction predicted by its correlation.
MIDI programs	The program is frequent in active examples and absent from matched inactive examples.	27	27	The target program occurs; note and duration shares describe its contribution.
Descriptions	The description identifies all four active test excerpts; twelve cards pass the audit.	48	48	Blinded ratings measure agreement with the fixed description.

Table 4: Three observational routes define the intervention pools. The pools overlap and contain 80 distinct features.

Study	Prompts	Baseline	Feature	Random	Lift (pp) [95% CI]	$p$
Confirmation	20	2	14	1	65 [45, 85]	0.000122
Replication	10	0	6	0	60 [30, 90]	0.015625

Table 5: The program-81 effect of the synth-lead feature (text2midi L13 f3340) recurs on two disjoint prompt sets at

P_{90}

. Counts: outputs with at least one non-drum program-81 note; lift: feature minus random presence in percentage points, paired-prompt bootstrap 95% interval;

p

: exact one-sided test on discordant pairs. The confirmation was prespecified.

Hypothesis source	Endpoint	Hyp.	Prompts	Test	Family-level result
Synth-lead feature, confirmation	Program presence	1	20	$p=0.000122$	Prespecified; passes all four criteria
Synth-lead feature, replication	Program presence	1	10	$p=0.0156$	Disjoint prompts and seed
Density-adjusted correlation, 21 SAEs	MusPy metric	42	8	–	$+0.18$ SD $[-0.04,+0.37]$ vs. baseline; $+0.17$ SD $[+0.02,+0.33]$ vs. random
Program frequency	Program presence	27	10	–	Absent programs introduced for 6 of 10 features (random: 1)
Strongest raw metric correlation	MusPy metric	24	10	–	$-0.23$ SD $[-1.23,+0.91]$ vs. random
Description, audit-passing	Blinded LLM rating	12	5	$p=0.3125$	Synth-lead feature rated highest; pooled interval includes 0

Table 6: Intervention outcomes by hypothesis source. Test:

p

of a prespecified test; per-feature screen results are in Appendix E . Family level: mean oriented effect at

P_{90}

in baseline SD, 95% interval over shared prompts (Table 7 ); program row: features whose program is absent from all baselines; description row: pooled LLM rating test of the primary subset.

Family (hypotheses, SAEs)	Target	Feature $-$ random	Positive	Feature $-$ baseline	Random $-$ baseline
Density-adjusted (42, 21 SAEs)	$P_{50}$	$+0.02$ $[-0.09,+0.12]$	22/42	$-0.01$ $[-0.16,+0.12]$	$-0.04$ $[-0.16,+0.10]$
	$P_{90}$	$+0.17$ $[+0.02,+0.33]$	27/42	$+0.18$ $[-0.04,+0.37]$	$+0.01$ $[-0.15,+0.19]$
Raw correlation (24, 7 SAEs)	$P_{50}$	$-0.04$ $[-0.30,+0.31]$	9/24	$+0.14$ $[-0.18,+0.53]$	$+0.18$ $[-0.11,+0.48]$
	$P_{75}$	$+0.15$ $[-0.55,+1.01]$	15/24	$+0.06$ $[-0.26,+0.44]$	$-0.09$ $[-0.70,+0.56]$
	$P_{90}$	$-0.23$ $[-1.23,+0.91]$	8/24	$-0.24$ $[-1.20,+0.51]$	$-0.02$ $[-0.72,+0.54]$

Table 7: Family-level effects. Mean oriented effect in baseline SD with 95% interval resampling the shared prompts; positive values agree with the correlation. Positive: hypotheses with feature

-

random

>0

; no

P_{75}

condition in the density-adjusted screen.

Figure 3: Dose responses of selected text2midi features relative to the baseline and the matched random direction. Points: paired mean differences over ten prompts, 95% bootstrap intervals; (a, b) target-program share of non-drum notes, (c) pitch entropy. In (a), the replication for the synth-lead feature, baseline and random outputs contain no program-81 notes, so the two comparisons coincide; (b) and (c) are the largest exploratory effects of their families.

P_{90}

is the prespecified dose.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Layer	EF	$k$	FVU	Dead (%)	Frequent (%)	Retained
text2midi	2	4	32	0.0370	0.65	80.57	496
text2midi	2	8	32	0.0316	1.56	55.76	2174
text2midi	13	4	32	0.2038	0.13	83.98	288
text2midi	13	8	32	0.1885	0.13	67.04	1489
MIDI-LLM	12	4	32	0.3693	0.43	18.88	4968
MIDI-LLM	12	32	32	0.3923	83.72	1.84	2822

Appendix

Table 8: Pilot comparisons for the expansion factor (EF) and TopK budget

k

. Dead and frequent: window-frequency definitions of Table 1 ; one pilot run per row.

Model and layer	$\Delta\mathrm{CE}_{\rm SAE}$	$\Delta\mathrm{CE}_{\rm mean}$	Recovery (%)
text2midi L13	0.0324 [0.0203, 0.0443]	4.1488 [4.0473, 4.2532]	99.22 [98.94, 99.51]
MIDI-LLM L12	0.0852 [0.0569, 0.1159]	1.9234 [1.7918, 2.0575]	95.57 [93.95, 97.09]

Appendix

Table 9: Functional fidelity of the two evaluated dictionaries.

\Delta\mathrm{CE}

: increase in next-token cross-entropy over the unmodified model; recovery:

R_{\mathrm{CE}}

from file-balanced means on 200 excerpts from 146 files; 95% file-bootstrap intervals.

Metric	Definition
Pitch entropy	Shannon entropy of the MIDI-pitch distribution.
Pitch-class entropy	Shannon entropy after reducing pitch modulo twelve.
Pitch count	Number of distinct MIDI pitches.
Pitch-class count	Number of distinct pitch classes.
Pitch range	Difference between the highest and lowest pitched notes.
Polyphony	Average number of simultaneous pitches over nonempty time steps.

Appendix

Table 10: The twelve MusPy measurements. Scale consistency measures pitch-class membership in a diatonic scale and does not assess a tonal center.

Model	Layer	Eligible	D,V selected	Test	D,V selected	Test	D,V selected	Test
			$\|r\|\geq 0.10$		$\|r\|\geq 0.20$		$\|r\|\geq 0.30$
text2midi	2	2252	963/2252 (42.8%)	755/963	137/2252 (6.1%)	111/137	8/2252 (0.4%)	6/8
text2midi	8	1963	992/1963 (50.5%)	820/992	193/1963 (9.8%)	150/193	13/1963 (0.7%)	8/13
text2midi	13	1652	745/1652 (45.1%)	608/745	126/1652 (7.6%)	97/126	7/1652 (0.4%)	6/7
text2midi	16	2008	949/2008 (47.3%)	765/949	140/2008 (7.0%)	107/140	9/2008 (0.4%)	4/9
MIDI-LLM	2	2393	734/2393 (30.7%)	543/734	32/2393 (1.3%)	23/32	3/2393 (0.1%)	3/3

Appendix

Table 11: Replication of metric associations across characterization splits. D,V: features with the same sign and

|r|\geq\tau

in discovery and validation; next column: of these, also on test; percentages: of eligible features (window-frequency filter, firing on 1–50% of discovery excerpts).

	text2midi			MIDI-LLM
Metric	$0.10$	$0.20$	$0.30$	$0.10$	$0.20$	$0.30$
Pitch entropy	2093	102	0	1297	78	8
Pitch-class entropy	834	10	0	443	10	1
Pitch count	3100	439	29	2454	272	30
Pitch-class count	1731	79	0	1216	57	2
Pitch range	1969	115	7	1232	79	11

Appendix

Table 12: Associations per metric, summed over the four final dictionaries of each model (same sign and

|r|\geq\tau

in discovery and validation; a feature can count under several metrics). Eligible features per metric: 7,876 (text2midi) and 8,423 (MIDI-LLM); drum metrics: 7,871 and 8,366.

Metric	Feature	$r_{\min}$	$r_{T}$	Feature	$r_{\min}$	$r_{T}$
	text2midi			MIDI-LLM
Pitch entropy	L8 f1015	0.279	+0.258	L14 f893	0.418	+0.387
Pitch-class entropy	L8 f5166	0.231	+0.211	L14 f893	0.321	+0.302
Pitch count	L8 f2187	0.372	+0.295	L14 f893	0.486	+0.435
Pitch-class count	L8 f5166	0.286	+0.259	L14 f893	0.389	+0.363
Pitch range	L2 f4317	0.402	+0.387	L7 f6043	0.488	+0.437

Appendix

Table 13: Strongest feature per metric and model, which defines the metric pool (24 hypotheses, 15 features).

r_{\min}

: smaller absolute discovery or validation correlation, same sign;

r_{T}

: test correlation, reported after selection.

Model	Dictionary	Eligible	Raw	Adj.	Raw	Adj.	Raw	Adj.
			$\tau=0.10$		$\tau=0.20$		$\tau=0.30$
text2midi	L2 EF4 k32 (historical)	2426	1790	680	1144	118	403	18
text2midi	L2 EF8 k32 (historical)	5041	3316	1093	1713	151	480	24
text2midi	L8 EF8 k32 (historical)	4510	3283	1343	1840	234	692	40
text2midi	L13 EF4 k32 (historical)	2032	1727	743	1175	135	506	28
text2midi	L13 EF8 k32 (historical)	4763	3571	1365	2024	217	684	41

Appendix

Table 14: Density reanalysis of the 13 earlier checkpoints. Counts as in Table 2 ; historical: earlier runs with the stated layer, expansion factor, and

k

Group	Cards	$H$ [95% CI]	Difference to calibration [95% CI]	$H=4$
All real features	786	$2.221$ $[2.162,2.282]$	$+0.121$ $[-0.131,+0.388]$	48 (11.2)
Instrumentation	72	$2.792$ $[2.569,3.014]$	$+0.692$ $[+0.364,+1.033]$	21 (1.0)
Rhythm and timing	37	$2.405$ $[2.135,2.649]$	$+0.305$ $[-0.057,+0.659]$	1 (0.5)
Percussion	81	$2.358$ $[2.173,2.543]$	$+0.258$ $[-0.048,+0.577]$	5 (1.2)
Pitch and register	29	$2.241$ $[1.931,2.552]$	$+0.141$ $[-0.239,+0.549]$	2 (0.4)
Harmony and tonality	225	$2.156$ $[2.044,2.267]$	$+0.056$ $[-0.211,+0.338]$	11 (3.2)

Appendix

Table 15: Description recognition by group.

H

: mean correct selections out of four (random choice: 2); difference: to the 30-card permutation calibration;

H=4

: perfect cards, with the random-choice expectation in parentheses.

Screen	Hypotheses	Max. prompts	Smallest possible $p$	Smallest observed $q$
Program presence	27	10	0.000977	1.000
Raw-correlation metric	24	10	0.000977	0.727
Density-adjusted metric	42	8	0.003906	0.246

Appendix

Table 16: Exact-test resolution and observed Benjamini–Hochberg results. The smallest possible

p

2^{-n}

for the maximum number

n

of prompts; undefined pairs or ties can raise this bound. The final column reports the smallest observed

q

in each primary family.

Family	Target	Re-estimated scale	Retained	Dictionary
Density-adjusted (42, 21 SAEs)	$P_{50}$	$+0.02$ $[-0.10,+0.19]$	19,945/20,000	$+0.02$ $[-0.13,+0.18]$
Density-adjusted (42, 21 SAEs)	$P_{90}$	$+0.17$ $[-0.01,+0.46]$	19,937/20,000	$+0.17$ $[-0.01,+0.34]$
Raw correlation (24, 7 SAEs)	$P_{50}$	$-0.04$ $[-0.27,+0.36]$	12,117/20,000	$-0.04$ $[-0.13,+0.09]$
Raw correlation (24, 7 SAEs)	$P_{75}$	$+0.15$ $[-1.00,+0.84]$	10,991/20,000	$+0.15$ $[-0.19,+0.69]$
Raw correlation (24, 7 SAEs)	$P_{90}$	$-0.23$ $[-1.09,+0.50]$	4,849/20,000	$-0.23$ $[-0.61,+0.11]$

Appendix

Table 17: Sensitivity of family-level feature-minus-random effects, in baseline SD. Re-estimated scale: mean effect and 95% prompt-bootstrap interval after excluding undefined draws; retained: usable draws out of 20,000; dictionary: 95% interval resampling dictionaries conditional on the observed prompts.

Model	Feature	Metric	Pairs	Oriented difference	$p$	$q$
MIDI-LLM	L2 f5050	Drum grid (duple)	5	-0.019	0.6250	0.948
MIDI-LLM	L2 f5050	Drum consistency	5	-0.019	0.6250	0.948
MIDI-LLM	L2 f6981	Drum grid (triple)	2	+0.042	0.5000	0.948
MIDI-LLM	L7 f6043	Pitch range	9	-2.111	0.6250	0.948
MIDI-LLM	L7 f6043	Polyphony	9	-0.260	0.7695	0.948
MIDI-LLM	L7 f6633	Scale consistency	9	+0.036	0.1309	0.785

Appendix

Table 18: All 24 primary metric hypotheses of the targeted screen at

P_{90}

. Oriented difference: feature minus random, signed by the observational correlation (positive agrees with the hypothesis), in metric units; pairs exclude undefined values.

Model	Feature	Program	Baseline	Feature	Random	$p$	$q$
MIDI-LLM	L2 f920	28	0	0	0	1.0000	1.000
MIDI-LLM	L2 f5165	49	0	0	0	1.0000	1.000
MIDI-LLM	L7 f499	0	8	7	8	1.0000	1.000
MIDI-LLM	L7 f6424	24	0	2	0	0.2500	1.000
text2midi	L2 f571	1	7	7	6	0.5000	1.000
text2midi	L2 f896	30	2	2	0	0.2500	1.000

Appendix

Table 19: All 27 program hypotheses of the targeted screen beyond the synth-lead feature. Zero-based programs; counts: program-positive outputs among ten prompts at

P_{90}

Feature	Metric	Feature $-$ baseline	Feature $-$ random	$p$	$q$
L8 f1015	Pitch entropy	0.271 [-0.102, 0.661]	0.418 [0.078, 0.783]	0.0303	0.727
L13 f3827	Polyphony rate	-0.003 [-0.123, 0.112]	0.132 [-0.004, 0.282]	0.0684	0.750
L8 f5166	Pitch-class count	0.100 [-1.000, 1.200]	1.100 [0.000, 2.400]	0.0938	0.750

Appendix

Table 20: text2midi metric effects that differ between the two references (

P_{90}

). Raw metric differences, paired-prompt bootstrap 95% intervals over ten prompts; all three hypotheses predict an increase;

p

q

: feature minus random within the 24 metric hypotheses.

Figure 4: Paired prompts at

P_{90}

for selected effects. Lines join the random and feature outputs of one prompt; top: target-program note share; bottom: hypothesized metric; synth lead: replication of the confirmed effect, the others exploratory.

Figure 5: Effects of the 15 metric features on all twelve metrics at $P_{90}$ . Color: absolute feature-minus-random effect in baseline SD; outline: hypothesized metric; asterisk: drum metric, prompts with defined values only.

Model	Dictionary	Feature	Metric	Pairs	Oriented difference [95% CI]	$p$	$q$
text2midi	L2 EF4 k32 (historical)	f1247	Pitch range	8	$+0.12$ $[-9.00,+9.50]$	0.5000	0.778
text2midi	L2 EF4 k32 (historical)	f1596	Polyphony	8	$-0.81$ $[-1.56,-0.01]$	0.9453	0.984
text2midi	L2 EF8 k32 (historical)	f3526	Pitch range	8	$+0.38$ $[-15.00,+11.38]$	0.4922	0.778
text2midi	L2 EF8 k32 (historical)	f5056	Polyphony	8	$+1.63$ $[+0.68,+2.58]$	0.0117	0.246
text2midi	L2 EF8 k32 (final)	f1712	Pitch range	8	$+0.62$ $[-21.90,+17.63]$	0.4844	0.778
text2midi	L2 EF8 k32 (final)	f2208	Polyphony	8	$-0.16$ $[-2.20,+1.68]$	0.5469	0.792

Appendix

Table 21: All 42 features of the density-adjusted screen at

P_{90}

. Oriented difference: feature minus random, signed by the adjusted correlation, in metric units, with paired bootstrap 95% interval;

q

: Benjamini–Hochberg across the 42 features; final: dictionaries of Table 1 ; historical: earlier checkpoints (Appendix A ).

Model	Condition	Notes	Relative norm (%)	Re-encoded firing (%)
text2midi	Feature	107.4 $\pm$ 5.8	21.4 $\pm$ 16.2	55.84 $\pm$ 42.88
text2midi	Random	108.7 $\pm$ 5.6	21.4 $\pm$ 16.3	0.29 $\pm$ 0.60
text2midi	Ablation	105.4 $\pm$ 1.1	0.0 $\pm$ 0.1	0.02 $\pm$ 0.05
MIDI-LLM	Feature	255.4 $\pm$ 0.5	12.9 $\pm$ 10.8	35.89 $\pm$ 40.07
MIDI-LLM	Random	255.5 $\pm$ 0.4	13.4 $\pm$ 12.3	0.07 $\pm$ 0.18
MIDI-LLM	Ablation	255.6 $\pm$ 0.0	0.0 $\pm$ 0.0	0.00 $\pm$ 0.01

Appendix

Table 22: Intervention diagnostics of the targeted screen. Mean

\pm

SD across 52 text2midi or 28 MIDI-LLM feature means; relative norm:

100\|h^{\prime}-h\|_{2}/\|h\|_{2}

; firing: percentage of edited states whose target coordinate is positive after re-encoding; induction at

P_{90}

; ablation removes the naturally active contribution.

Subset	Features	Correct choices	Rating difference [95% CI]	$p$
Audit-passing subset	12	4/8	0.167 [-0.300, 0.700]	0.3125
All descriptions	48	16/34	0.108 [-0.112, 0.338]	0.1885
Audit-passing, without synth lead	11	3/7	-0.036 [-0.400, 0.291]	0.6133

Appendix

Table 23: Pooled blinded LLM ratings show no advantage of the feature direction. Rating differences: averaged over five prompts per feature, then over features; intervals resample features; correct choices: decisions that pick a variant; last row: primary subset without the synth-lead feature.

Feature	Description	$\Delta\pm\mathrm{SD}$	Favored	Flags
text2midi L13 f3340	Synth-lead (Lead 2 sawtooth) / electronic-patch instrumentation	$+2.40\pm 0.55$	5/5	None
text2midi L2 f552	Dense, near-continuous drum/percussion texture	$+2.00\pm 1.87$	4/5	T, S
text2midi L2 f4021	Rich, multi-timbral instrumentation (many melodic instruments)	$+1.60\pm 1.14$	4/5	T, S
text2midi L16 f4606	Sparse texture with recurring timpani-type low percussion	$+1.40\pm 1.52$	3/5	S
MIDI-LLM L2 f6671	Prominent, continuous drum-kit layer relative to melodic notes	$+1.20\pm 2.39$	3/5	T, S, N
MIDI-LLM L12 f3013	Slow harmonic rhythm with long sustained notes over a static backdrop	$+1.00\pm 0.71$	4/5	N

Appendix

Table 24: The six largest feature-level rating differences.

\Delta

: mean feature-minus-random rating, SD over five prompts; favored: prompts with the feature output rated higher; flags: token concentration (T), token syntax (S), note-count imbalance (N).

Figure 6: Rating differences for all 48 description candidates. Bars: mean over five paired prompts at

P_{90}

on the 1–5 scale; error bars: SD; color: audit status, which defines the primary subset.

Mechanistic Interpretability

Mechanistic interpretability studies how neural networks implement behaviors through internal features, circuits, and causal information flow.

See all 388 papers

Aug 6, 2026

MI-MIDI: Mechanistic Interpretability of Text-to-MIDI Generation Models via Probing, Lenses and Steering

Mechanistic interpretability of music generation has concentrated on audio models, leaving symbolic models largely unexplored. We analyze two public text-to-MIDI systems of contrasting design: the purpose-built encoder--decoder text2midi and MIDI-LLM, a Llama~~3.2~~1B model extended with MIDI tokens using linear probing, the logit and tuned lenses, activation patching and difference-in-means steering. Across these methods, we recover musically meaningful structure and show how architecture shapes its formation and control. Pitch, instrumentation, harmony and texture are linearly decodable in both models. text2midi refines predictions gradually across depth, whereas MIDI-LLM works largely in its inherited textual basis before a sharp late rotation into the musical vocabulary; patching identifies a matching late attenuation of prompt-driven instrument transfer. Steering produces bidirectional changes in register and polyphony in both systems, and in tempo/energy in MIDI-LLM. Our two-orientation protocol isolates directional control and shows that all-layer interventions are robust in text2midi but accumulate disruptively in MIDI-LLM. Together, the results provide a practical toolkit for tracing and controlling musical concepts in symbolic generators. Audio examples are available on a demo website.

Jakub Poćwiardowski, Mateusz Modrzejewski

Institute of Computer Science Warsaw University of Technology Warsaw, Poland

Controllable Music Generation Symbolic Music Generation Mechanistic Interpretability Activation Steering

Ask PDF

Oct 14, 2026

MIDI-REPA: Improving Text-to-Music Generation with Symbolic Representation Alignment

State-of-the-art text-to-music models increasingly rely on scaling model capacity, data, and compute, while still learning musical structure directly from dense audio. Yet melody, rhythm, harmony, and structure can be represented compactly in symbolic form. We introduce MIDI-REPA, a representation-alignment framework that derives MIDI supervision from automatic music transcription (AMT) to align intermediate representations of a flow-matching DiT. Building on the 122M-parameter FluxAudio-S baseline from the 2026 ICME Academic Text-To-Music Challenge, we investigate what, where, and when to align. Aligning a MuScriptor-derived piano roll with an early DiT layer throughout the full flow trajectory performs the best. Without additional training data or techniques, MIDI-REPA remains competitive on the ATTM objective metrics while remaining trainable on a single RTX 3090 in about 31 hours. Our results highlight symbolic supervision as a simple and inexpensive learning signal for improving text-to-music generation. We release our transcription pipeline, JamendoMIDI, training code, and audio samples at https://midi-repa.vaclis.net.

Song-Ze Yu, Fang-Duo Tsai, Yi-Hsuan Yang

National Taiwan University · University of California, San Diego

Text-to-Music Generation Diffusion Model Alignment Audio Diffusion Models

Ask PDF

Aug 10, 2026

MusicLayout: Explicit Structural Planning for Controllable Text-to-Music Generation

Text-to-music generation has advanced rapidly, but current systems still rely primarily on global text prompts, leaving the structural organization of generated music implicit and difficult to inspect, control, or revise before audio generation. To address this issue, we introduce MusicLayout, an explicit intermediate representation for controlling musical structure in text-to-music generation. MusicLayout describes a musical piece as a time-aligned layout of sections, textures, repetitions, variations, and instrument-level arrangements, serving as an interpretable planning layer between textual intent and the generated music. We integrate MusicLayout into a text-to-music framework built on a unified autoregressive formulation, where the model first generates a MusicLayout representation and subsequently predicts audio tokens conditioned on this representation within a single sequence. The resulting MusicLayout can be inspected and modified prior to audio generation, providing a mechanism for layout-level structural control. We evaluate MusicLayout through layout-conditioned generation, layout manipulation experiments, and matched-data ablations, providing evidence that explicit layout planning can improve long-range structural organization and support layout-level control. We have released the implementation as open source on GitHub at https://github.com/XaryLee/MusicLayout.

Shuyu Li, Kejun Zhang, Jiahe Lei +5

College of Artificial Intelligence, Zhejiang University · 3Innovation Center of Yangtze River Delta, Zhejiang University · 4The Chinese University of Hong Kong +4

Text-to-Music Generation

Ask PDF

Abstract

Mechanistic Interpretability

Explore similar work

MI-MIDI: Mechanistic Interpretability of Text-to-MIDI Generation Models via Probing, Lenses and Steering

MIDI-REPA: Improving Text-to-Music Generation with Symbolic Representation Alignment

MusicLayout: Explicit Structural Planning for Controllable Text-to-Music Generation