Descriptor-Injected Cross-Modal Learning: A Systematic Exploration of Audio-MIDI Alignment via Spectral and Melodic Features

Mariano Fernández Méndez

cs.SDApr 11, 2026

Descriptor-Injected Cross-Modal Learning: A Systematic Exploration of Audio-MIDI Alignment via Spectral and Melodic Features

Authors: Mariano Fernández Méndez

Organizations: Asociación Civil AlterMundi, Córdoba, Argentina

Abstract

Cross-modal retrieval between audio recordings and symbolic music representations (MIDI) remains challenging because continuous waveforms and discrete event sequences encode different aspects of the same performance. We study descriptor injection, the augmentation of modality-specific encoders with hand-crafted domain features, as a bridge across this gap. In a three-phase campaign covering 13 descriptor-mechanism combinations, 6 architectural families, and 3 training schedules, the best configuration reaches a mean S of 84.0 percent across five independent seeds, improving the descriptor-free baseline by 8.8 percentage points. Causal ablation shows that the audio descriptor A4, based on octave-band energy dynamics, drives the gain in the top dual models, while the MIDI descriptor D4 has only a weak inference-time effect despite improving training dynamics. We also introduce reverse cross-attention, where descriptor tokens query encoder features, reducing attention operations relative to the standard formulation while remaining competitive. CKA analysis shows that descriptors substantially increase audio-MIDI transformer layer alignment, indicating representational convergence rather than simple feature concatenation. Perturbation analysis identifies high-frequency octave bands as the dominant discriminative signal. All experiments use MAESTRO v3.0.0 with an evaluation protocol controlling for composer and piece similarity.

Figures & tables

Panel of Figure 1 — Figure 1: Model architecture for the canonical audio–MIDI baseline. The audio branch uses a 4-layer CNN plus a 4-layer Transformer; the MIDI branch uses event embeddings plus a 4-layer pre-norm Transformer. Both project to a shared 256-D space and train with VICReg. Dashed arrows mark optional descriptor injection points (A4 on audio, D4 on MIDI).

Band	Frequency range	STFT bins	Width (Hz)	Musical content
0	47 – 94 Hz	4 – 8	47	Bass fundamentals
1	94 – 188 Hz	8 – 16	94	Bass harmonics
2	188 – 375 Hz	16 – 32	187	Low-mid register
3	375 – 750 Hz	32 – 64	375	Mid register
4	750 – 1500 Hz	64 – 128	750	Upper-mid/presence
5	1500 – 3000 Hz	128 – 256	1500	Presence/harmonics

Table 1: A4 octave-band definitions. STFT bins correspond to

n_{\text{FFT}}=2048

f_{s}=24

kHz.

Panel of Figure 2 — Figure 2: Gate 4.3 screening results: 13 descriptor $\times$ mechanism combinations ranked by S at 5 training epochs. Dashed line marks D0 baseline ( $S{=}60.2\%$ ). The top 7 arms all inject descriptors into their own modality’s encoder. Cross-modal injection (d4a4cm) is worst.

Rank	Arm	Descriptor	Mechanism	S (%)	Hard neg (%)	$\Delta$ (pp)
1	d4a4	D4+A4	Dual concat	69.8	91.6	$+9.6$
2	A4r	A4	Reverse cross-att	68.6	91.6	$+8.4$
3	D4r	D4	Reverse cross-att	64.2	93.2	$+4.0$
4	D4	D4	Concat	63.6	91.2	$+3.4$
4	A4	A4	Concat	63.6	92.4	$+3.4$
6	A4x	A4	Cross-attention	62.6	92.4	$+2.4$

Table 2: Gate 4.3 screening results (5 epochs). S = min(R@10 a→m , R@10 m→a ).

\Delta

is relative to the D0 baseline. Arms above the horizontal line outperform D0.

Rank	Arm	Family	S (%)	$\Delta$ vs D0 (pp)
1	t3-wt	Third Tower	67.6	$+7.4$
2	t3-tri	Third Tower	65.0	$+4.8$
3	moe-a4-v2	MoE	60.2	$\pm 0.0$
4	film-dual	FiLM	59.4	$-0.8$
4	moe-a4-v4	MoE	59.4	$-0.8$
6	film-a4	FiLM	59.2	$-1.0$

Table 3: Gate 4.4 architecture family screening (5 epochs, from d4a4 foundation). Only Third Tower with weighted loss outperforms the D0 baseline substantially.

Arm	Schedule	Epochs	Best S (%)	Best epoch	Params (M)	$\Delta$ (pp)
d4a4	Cosine-60	60	83.8	50	75.5	$+10.4$
a4r	Cosine-30	30	82.0	29	78.6	$+8.6$
t3-wt	Trapez-50	50	81.2	50	—	$+7.8$
a4r	Cosine-tail	60	80.6	60	78.6	$+7.2$
d4-a4r	Cosine-30	30	79.8	30	78.9	$+6.4$
D0	Cosine-tail	60	73.4	50	74.2	—

Table 4: Long-horizon results (30–60 epochs) from the seed-42 training run.

\Delta

: improvement over D0 at convergence. Multi-seed training replication is summarized in table 12 .

Arm	S (%)	R@1 a→m	R@10 a→m	MRR a→m	R@10 m→a	MRR m→a	Hard neg (%)
D0	73.4	22.2	74.8	0.380	73.4	0.372	98.4
d4a4	83.8	28.0	84.4	0.458	83.8	0.442	99.0
a4r	82.0	27.6	82.6	0.438	82.0	0.424	98.8
d4-a4r	79.8	25.2	81.4	0.426	79.8	0.420	99.4

Table 5: Canonical evaluation results (Test 12) from the seed-42 checkpoint. Pool size 256, 500 queries, seed 42. Multi-seed training replication is summarized in table 12 .

Panel of Figure 3 — Figure 3: Causal ablation results (Test 01) for the canonical seed-42 checkpoint. A4 ablation (left group) collapses S by 32–78 pp across all models. D4 ablation (right group) has near-zero effect in dual models ( $-0.4$ to $+0.6\,\text{pp}$ ). Note: a4r has no D4 descriptor, so its D4 bars equal its Normal bar for visual alignment.

	Ablation mode	d4a4 ( $\Delta$ )	a4r ( $\Delta$ )	d4-a4r ( $\Delta$ )
Audio (A4)	Zero	$-76.0\,\text{pp}$	$-77.6\,\text{pp}$	$-75.4\,\text{pp}$
	Noise	$-61.8\,\text{pp}$	$-53.0\,\text{pp}$	$-53.0\,\text{pp}$
	Shuffle	$-37.2\,\text{pp}$	$-32.2\,\text{pp}$	$-32.4\,\text{pp}$
MIDI (D4)	Zero	$+0.6\,\text{pp}$	—	$-0.4\,\text{pp}$
	Noise	$+0.6\,\text{pp}$	—	$\pm 0.0\,\text{pp}$
	Shuffle	$\pm 0.0\,\text{pp}$	—	$\pm 0.0\,\text{pp}$

Table 6: Causal ablation results (Test 01).

\Delta

: change in S relative to normal inference. A4 is causally essential; D4 has weak effect in this model family.

Panel of Figure 4 — Figure 4: Transposition invariance (Test 04) for the canonical seed-42 checkpoint. All models degrade with pitch shift, but descriptor-augmented models retain more performance. At $\pm 3$ semitones, a4r shows $+23.6\,\text{pp}$ advantage over D0. The reverse cross-attention models (a4r, d4-a4r) are the most invariant.

Arm	$-6$	$-3$	$-1$	$0$	$+1$	$+3$	$+6$	Ret. $\|\pm 3\|$
D0	13.8	26.6	65.6	73.4	64.0	27.4	13.4	37%
d4a4	24.2	41.4	75.2	83.8	75.6	44.6	25.6	51%
a4r	27.0	46.2	76.6	82.0	76.8	51.0	27.6	59%
d4-a4r	27.0	45.0	73.2	79.8	75.2	49.2	27.2	59%

Table 7: Transposition invariance (Test 04) for the seed-42 checkpoint. S (%) at various semitone shifts. Retention = S |±3| / S 0 . Multi-seed training replication is summarized in table 12 .

Arm	CKA mean	$\Delta$ vs D0	RSA mean
D0	0.435	—	0.446
d4a4	0.659	$+51\%$	0.646
a4r	0.766	$+76\%$	0.721
d4-a4r	0.794	$\mathbf{+82\%}$	0.761

Table 8: Cross-encoder CKA alignment (Test 06). Mean CKA across all 16 audio

\times

MIDI layer pairs. Higher = more aligned internal representations between encoders.

Panel of Figure 6 — Figure 6: Perturbation sensitivity by A4 octave band (Test 08) for the canonical seed-42 checkpoint. X-axis labels show full band ranges in kHz/Hz. Each group shows the mean $L_{2}$ change in output embedding when perturbing one band by $\epsilon{=}0.1$ . High-frequency bands (750+ Hz) carry 2–10 $\times$ more sensitivity. Each injection mechanism attends to different spectral regions: d4a4 peaks at 750–3000 Hz, a4r at 3–12 kHz, d4-a4r at 3–6 kHz.

Band	Hz range	d4a4	a4r	d4-a4r	Zone
0	47 – 94	0.073	0.238	0.303	Low
1	94 – 188	0.224	0.335	0.313	Low
2	188 – 375	0.375	0.381	0.514	Mid
3	375 – 750	0.546	0.423	0.526	Mid
4	750 – 1500	0.664	0.478	0.773	High
5	1500 – 3000	0.662	0.476	0.599	High

Table 9: Perturbation sensitivity by A4 octave band (Test 08). Values are mean

\|\Delta z\|_{2}

when perturbing each band by

\epsilon=0.1

. Higher = more sensitive. Bands 4–7 (750+ Hz) dominate.

Perturbation	Level	D0	d4a4	a4r	d4-a4r
Temporal shift	$\pm$ 8000 samp ( $\pm$ 0.33 s)	68.2	76.6	75.0	76.2
Velocity scale	0.5 $\times$	5.2	8.8	6.8	6.4
Velocity scale	1.5 $\times$	18.4	12.8	12.4	11.6
Octave transp.	$-$ 12 st	12.0	16.0	17.4	17.0
Octave transp.	$+$ 12 st	10.0	13.8	16.0	15.8

Table 10: Perturbation robustness (Test 09). Each cell shows worst-case

S

(%) across perturbation directions at the given level. Temporal shift values are in samples (24 kHz sample rate).

Panel of Figure 7 — Figure 7: Audio noise degradation (Test 09) for the canonical seed-42 checkpoint. At moderate noise ( $\geq 20$ dB), D0 is most robust because it has no corruptible A4 pathway. At 5 dB the ranking reverses: descriptor models retain higher absolute $S$ , likely due to their stronger cross-modal alignment providing a noise-invariant floor via the MIDI encoder.

Type	Probe	D0	d4a4	a4r	d4-a4r
Cross	Audio $\to$ pitch histogram	0.181	0.174	0.167	0.186
	Audio $\to$ interval hist.	0.094	0.112	0.095	0.115
	MIDI $\to$ chroma	0.330	0.245	0.255	0.251
	MIDI $\to$ centroid	0.616	0.637	0.662	0.652
Self	Audio $\to$ chroma	0.310	0.235	0.249	0.231
Self	MIDI $\to$ pitch histogram	0.239	0.236	0.233	0.233

Table 11: Linear probe

R^{2}

from frozen 256-dim embeddings (Test 03). Cross-modal probes decode features of one modality from the other modality’s embedding.

Panel of Figure 8 — Figure 8: Seed-42 summary dashboard: retrieval $S$ , collapse under A4 ablation, CKA alignment, transposition retention at $\pm 3$ semitones, peak perturbation sensitivity, and bidirectional Recall@1.

Arm	Mean S (%)	SD (pp)	Range	$\Delta$ vs D0 (pp)
d4a4	84.0	2.7	81.4–87.6	+8.8
d4-a4r	81.2	2.5	78.4–83.4	+6.0
a4r	80.7	1.9	79.4–84.0	+5.5
D0	75.2	2.3	71.8–77.4	—

Table 12: Multi-seed training replication across the four canonical arms. Means, standard deviations, and ranges are computed over five independent training seeds.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Arm	Descriptor	Domain	Dims	Mechanism	S ${}_{5\text{ep}}$ (%)	Description
D0	—	—	—	—	60.2	VICReg baseline (no descriptors)
D4	D4	MIDI	4	C	63.6	Local pitch intervals
A4	A4	Audio	8	C	63.6	Octave-band energy dynamics
A7	A7	Audio	12	C	58.8	Rational attractor (just intonation)
A8	A8	Audio	12	C	57.4	Onset-weighted chroma
A9	A9	Audio	12	C	58.8	IDF-weighted rational attractor

Appendix

Table 13: Complete descriptor catalog. Domain: A = audio, M = MIDI, D = dual. Mechanism: C = concatenation, X = standard cross-attention, R = reverse cross-attention, CM = cross-modal injection.

Index	Ratio	log 2	Cents	Musical interval
0	1/1	0.000	0	Unison / Octave
1	16/15	0.093	112	Minor second
2	9/8	0.170	204	Major second
3	6/5	0.263	316	Minor third
4	5/4	0.322	386	Major third
5	4/3	0.415	498	Perfect fourth

Appendix

Table 14: A7 just-intonation attractors. Log 2 values are octave-folded to

[0,1)

Layer	In ch.	Out ch.	Kernel	Stride	Padding	Norm	Activation
1	1	512	10	5	0	GN(32)	GELU
2	512	512	3	2	1	GN(32)	GELU
3	512	512	3	2	1	GN(32)	GELU
4	512	1024	3	2	1	GN(32)	GELU

Appendix

Table 15: MERTEncoderLite CNN feature extractor.

Hyperparameter	Value
Number of layers	4
Model dimension ( $d_{\text{model}}$ )	1024
Number of heads	8
Head dimension ( $d_{k}$ )	128
Feed-forward dimension ( $d_{\text{ff}}$ )	4096
Activation	GELU

Appendix

Table 16: Audio Transformer encoder specification.

Feature	Vocabulary	Embedding dim	Fraction of $d_{\text{model}}$
Pitch	128	256	$d/2$
Velocity	128	128	$d/4$
Duration	32	128	$d/4$
Concatenated		512	$d$
+ Linear proj. + LayerNorm		512

Appendix

Table 17: MIDI event embedding dimensions.

Hyperparameter	Value
Number of layers	4
Model dimension ( $d_{\text{model}}$ )	512
Number of heads	8
Head dimension ( $d_{k}$ )	64
Feed-forward dimension ( $d_{\text{ff}}$ )	2048
Activation	GELU

Appendix

Table 18: MIDI Transformer encoder specification.

Component	D0	d4a4	a4r	d4-a4r
Audio CNN	3.2M	3.2M	3.2M	3.2M
Audio Transformer	54.5M	54.5M	54.5M	54.5M
Audio pos. embeddings	6.1M	6.1M	6.1M	6.1M
MIDI embeddings	1.2M	1.2M	1.2M	1.2M
MIDI Transformer	10.5M	10.5M	10.5M	10.5M
Audio projection	0.8M	0.8M	0.8M	0.8M

Appendix

Table 19: Parameter count breakdown for each Gate 5B model.

Parameter	D0	d4a4	a4r	d4-a4r
Optimizer	AdamW ( $\beta_{1}{=}0.9$ , $\beta_{2}{=}0.999$ )
Weight decay	0.01
Batch size	16
Total epochs	60	60	30	30
Best epoch	50	50	29	30
Warmup steps	500

Appendix

Table 20: Complete training hyperparameters for Gate 5B canonical models.

Panel of Figure 9 — Figure 9: t-SNE embeddings for the 4 canonical models from the seed-42 checkpoint. Blue: audio; red: MIDI. Lines connect matched pairs. Figures are rendered from Test 10 exported PNGs.

Panel of Figure 10 — Figure 10: UMAP embeddings for the 4 canonical models from the seed-42 checkpoint (same split and sample count as t-SNE).

Figure 11: Distribution of cosine similarity for matched audio–MIDI pairs across canonical models from the seed-42 checkpoint (Test 10).

Cross-Modal Retrieval

Cross-modal retrieval matches queries and items across different data types, such as text, images, video, audio, and sensor signals.

See all 184 papers

MERIT: Learning Disentangled Music Representations for Audio Similarity
May 26, 2026Abhinaba Roy, Junyi Liang, Dorien HerremansDisentangled Representation Learning Audio Representation Learning
Singapore University of Technology and Design
Semantic Refinement of Universal Audio Representations through Audio-Description Alignment
Sep 8, 2026Lejun Min, Junyu Dai, Ruichen Zheng +7Audio-Text Alignment Audio Representation Learning
1Alibaba Token Foundry
MADS: A Multiview Acoustic Descriptor Set Beyond Standard Spectral Summaries
Sep 1, 2026Utsab Ghosh, Roshni ChakrabortyAudio Classification
ABV-Indian Institute of Information Technology and Management, Gwalior, India.

May 26, 2026J/K move · Enter open · S save

MERIT: Learning Disentangled Music Representations for Audio Similarity

Abhinaba Roy, Junyi Liang, Dorien Herremans

Singapore University of Technology and Design

Disentangled Representation Learning Audio Representation Learning Audio Understanding Music Information Retrieval Music Understanding

Open paper AskPDF arXiv

Current music similarity models typically compute a single, monolithic score, entangling distinct musical dimensions like melody, rhythm, and timbre. This limits user control and interpretability, making it impossible to execute nuanced queries. We introduce MERIT, a framework for learning disentangled, factor-specific music representations tailored to these three core dimensions. To overcome the lack of isolated musical variations in real-world audio, we use a novel training strategy that uses conditional audio generation and source-separated stems to strongly encourage single-factor variation in training data. Our evaluations demonstrate strong factor-wise disentanglement. Each head responds strongly to its intended perceptual dimension while remaining near chance on the others, a representational property that holds across both the synthetic training domain and independent real-world audio.

Abstract

Cross-Modal Retrieval

Explore similar work

MERIT: Learning Disentangled Music Representations for Audio Similarity

Semantic Refinement of Universal Audio Representations through Audio-Description Alignment

MADS: A Multiview Acoustic Descriptor Set Beyond Standard Spectral Summaries