MiCLA: Mixture-to-Constituent Latent Alignment for Polyphonic Audio Representation Learning

Kohei Yamamoto; Kosuke Okusa

cs.SDOct 20, 2026

MiCLA: Mixture-to-Constituent Latent Alignment for Polyphonic Audio Representation Learning

Authors: Kohei Yamamoto, Kosuke Okusa

Organizations: Research & Development Center, Technology Division, Oki Electric Industry Co., Ltd., Saitama, Japan. · Department of Data Science for Business Innovation, Chuo University, Tokyo, Japan.

Abstract

Real-world audio typically contains concurrent sound events, yet standard self-supervised methods treat each clip as an isolated training unit without explicitly modeling mixture--constituent relationships. We propose Mixture-to-Constituent Latent Alignment (MiCLA), a two-stage self-supervised pre-training framework that uses synthetic log-mel mixing to align mixture representations with their constituent clips. Stage 1 structures single-clip representations via masked self-distillation and contrastive regularization. Stage 2 introduces bidirectional alignment: for a synthetic mixture, MiCLA suppresses the component aligned with one constituent and aligns the residual with the other, using projective residualization on patch tokens and sample-adaptive soft residualization on the global class token. In downstream evaluations, MiCLA achieves 43.2% mean average precision (mAP) on AS-20K and 50.3% mAP on AS-2M under fine-tuning, and achieves the highest mAP among evaluated models on both AudioSet settings under frozen Convex Gated Probing (CGP). Furthermore, a Joint Semantic Retrieval diagnostic shows that MiCLA improves simultaneous retrieval of both constituent classes over target-averaging baselines. These results indicate that directional mixture-to-constituent alignment provides an effective pre-training signal for polyphonic audio representation learning.

Figures & tables

Fig. 1: Constituent-side pre-training in Stage 1. Masked self-distillation optimizes

\ell_{\mathrm{base}}

. Instance-level contrastive regularization

\ell_{\mathrm{ctr}}

structures representations across masked views. Both objectives are retained in Stage 2. Boxes labeled “C” denote CLS tokens.

Fig. 2: Mixture-to-constituent latent alignment introduced in Stage 2. In each target–counterpart direction, MiCLA suppresses the mixture component aligned with the counterpart. The resulting residual is aligned with the target constituent’s EMA-teacher representation. Patch tokens use hard projective residualization. CLS tokens use sample-adaptive soft residualization. Boxes labeled “C” denote CLS tokens.

Hyperparameters	Pre-Training (Stage 1)	Pre-Training (Stage 2)	Fine-Tuning
Hyperparameters	AS-2M	AS-2M	AS-2M	AS-20K	ESC-50	SPC-2
Optimizer	AdamW [ 39 ] with $\beta_{1}=0.9,\beta_{2}=0.95$
LR schedule	Cosine Annealing [ 40 ] with linear warm-up
Peak LR	$5.0\times 10^{-4}$	$5.0\times 10^{-5}$	$5.0\times 10^{-5}$
Minimum LR	0.0	0.0	$1.0\times 10^{-6}$	$1.0\times 10^{-6}$	$1.0\times 10^{-7}$	$1.0\times 10^{-7}$
Layer decay [ 41 ]	N/A	N/A	0.65	0.80	0.75	0.75

TABLE I: Hyperparameters for Pre-training and Fine-tuning.

Hyperparameters	Convex Gated Probing
Hyperparameters	AS-2M	AS-20K	FSD-50K	ESC-50	US8K	SPC-2	VoxCeleb-1	VoxForge	CREMA-D	GTZAN	NSynth	Surge
Optimizer	AdamW [ 39 ] with $\beta_{1}=0.9,\beta_{2}=0.999$
LR schedule	Cosine Annealing [ 40 ] with linear warm-up
Peak LR	$1.0\times 10^{-3}$
Minimum LR	$1.0\times 10^{-6}$
Weight decay	0.05	0.05	0.05	0.05	0.05	0.05	0.05	0.05	0.05	0.05	0.05	0.05

TABLE II: Hyperparameters for Convex Gated Probing (CGP). The loss function is Asymmetric Binary Cross-Entropy (A-BCE) [ 46 ] for multi-label datasets and Cross-Entropy (CE) for single-label datasets.

Model	#Params.	AS-2M	AS-20K	ESC-50	SPC-2
Model	#Params.	mAP (%)	mAP (%)	Acc (%)	Acc (%)
MaskSpec [ 4 ]	86M	47.1	32.3	89.6	97.7
data2vec [ 6 ]	94M	-	34.5	-	-
Audio-MAE [ 3 ]	86M	47.3	37.1	94.1	98.3
M2D/0.7 [ 21 ]	86M	47.9	38.6	96.0	98.4
ASiT [ 22 ]	86M	48.0	38.3	95.6	98.9

TABLE III: Fine-tuning results. MiCLA results are reported as the mean

\pm

standard deviation over three random seeds, while the results of the compared methods are cited as reported in the corresponding papers.

Model	AS-2M	AS-20K	FSD-50K
Model	mAP (%)	mAP (%)	mAP (%)
EAT [ 9 ]	45.0 ${\scriptstyle\pm 0.0}$	36.1 ${\scriptstyle\pm 0.0}$	59.5 ${\scriptstyle\pm 0.1}$
SSLAM [ 13 ]	43.8 ${\scriptstyle\pm 0.1}$	34.5 ${\scriptstyle\pm 0.0}$	57.8 ${\scriptstyle\pm 0.1}$
AudioMosaic [ 12 ]	44.1 ${\scriptstyle\pm 0.0}$	35.0 ${\scriptstyle\pm 0.0}$	57.0 ${\scriptstyle\pm 0.3}$
BAT [ 17 ]	45.7 ${\scriptstyle\pm 0.0}$	37.3 ${\scriptstyle\pm 0.1}$	62.5 ${\scriptstyle\pm 0.1}$
MiCLA (ours)	46.3 ${\scriptstyle\pm 0.0}$	37.8 ${\scriptstyle\pm 0.0}$	62.1 ${\scriptstyle\pm 0.1}$

TABLE IV: CGP results on multi-label evaluation settings. Values are reported as the mean

\pm

standard deviation over three random seeds.

Model	ESC-50	US8K	SPC-2	VoxCeleb-1	VoxForge	CREMA-D	GTZAN	NSynth	Surge
Model	Acc (%)	Acc (%)	Acc (%)	Acc (%)	Acc (%)	Acc (%)	Acc (%)	Acc (%)	Acc (%)
EAT [ 9 ]	93.2 ${\scriptstyle\pm 0.1}$	89.9 ${\scriptstyle\pm 0.1}$	95.7 ${\scriptstyle\pm 0.0}$	66.3 ${\scriptstyle\pm 0.1}$	98.4 ${\scriptstyle\pm 0.2}$	74.6 ${\scriptstyle\pm 0.1}$	83.8 ${\scriptstyle\pm 0.2}$	83.5 ${\scriptstyle\pm 0.2}$	49.4 ${\scriptstyle\pm 0.2}$
SSLAM [ 13 ]	92.6 ${\scriptstyle\pm 0.1}$	88.7 ${\scriptstyle\pm 0.3}$	95.9 ${\scriptstyle\pm 0.1}$	64.0 ${\scriptstyle\pm 0.2}$	98.3 ${\scriptstyle\pm 0.2}$	74.9 ${\scriptstyle\pm 0.2}$	84.6 ${\scriptstyle\pm 0.2}$	82.3 ${\scriptstyle\pm 0.3}$	48.7 ${\scriptstyle\pm 0.0}$
AudioMosaic [ 12 ]	91.0 ${\scriptstyle\pm 0.1}$	89.9 ${\scriptstyle\pm 0.1}$	95.9 ${\scriptstyle\pm 0.1}$	76.2 ${\scriptstyle\pm 0.2}$	97.7 ${\scriptstyle\pm 0.1}$	74.1 ${\scriptstyle\pm 0.3}$	83.4 ${\scriptstyle\pm 0.3}$	82.7 ${\scriptstyle\pm 0.1}$	47.9 ${\scriptstyle\pm 0.1}$
BAT [ 17 ]	94.2 ${\scriptstyle\pm 0.1}$	91.2 ${\scriptstyle\pm 0.1}$	97.3 ${\scriptstyle\pm 0.0}$	81.7 ${\scriptstyle\pm 0.2}$	99.0 ${\scriptstyle\pm 0.1}$	76.0 ${\scriptstyle\pm 0.1}$	87.9 ${\scriptstyle\pm 0.0}$	84.2 ${\scriptstyle\pm 0.6}$	50.7 ${\scriptstyle\pm 0.1}$
MiCLA (ours)	92.5 ${\scriptstyle\pm 0.1}$	92.5 ${\scriptstyle\pm 0.1}$	96.6 ${\scriptstyle\pm 0.1}$	70.0 ${\scriptstyle\pm 0.1}$	98.1 ${\scriptstyle\pm 0.0}$	73.5 ${\scriptstyle\pm 0.2}$	85.6 ${\scriptstyle\pm 0.7}$	86.4 ${\scriptstyle\pm 0.1}$	49.6 ${\scriptstyle\pm 0.1}$

TABLE V: CGP results on single-label classification tasks. Values are reported as the mean

\pm

standard deviation over three random seeds.

Fig. 3: Joint Semantic Retrieval on ESC-50. The panels show JTA@5 and JTA@10 for waveform and log-mel mixtures across mixing ratios

\mu

. A query is correct only when both constituent classes appear among the Top-

K

retrieved classes.

Ablation setting	ICR	RoPE	mAP	$\Delta$
No ICR	✗	✓	41.3 ${\scriptstyle\pm 0.1}$	-1.3
Absolute PosEmb.	✓	✗	41.4 ${\scriptstyle\pm 0.2}$	-1.2
Full Stage 1	✓	✓	42.6 ${\scriptstyle\pm 0.2}$	0.0

TABLE VI: Stage-1 ablation on constituent-clip representation structuring. All rows use masked self-distillation and are evaluated by AS-20K fine-tuning mAP. Values are the mean

\pm

standard deviation over three random seeds.

Comparison of Stage-2 training and mixture supervision
Ablation setting	S2	Mix	P	C	Soft	ICR	mAP	$\Delta$
Stage-1 training	✗	✗	✗	✗	–	✓	42.6 ${\scriptstyle\pm 0.2}$	-0.6
Stage-2 no mixtures	✓	✗	✗	✗	–	✓	42.7 ${\scriptstyle\pm 0.1}$	-0.5
Avg. constituent targets	✓	✓	✓	✓	–	✓	42.9 ${\scriptstyle\pm 0.1}$	-0.3
Comparison of alignment granularities
Patch align. only	✓	✓	✓	✗	–	✓	42.9 ${\scriptstyle\pm 0.0}$	-0.3

TABLE VII: Stage-2 ablation on synthetic-mixture supervision and mixture-to-constituent alignment. All models are evaluated by AS-20K fine-tuning mAP. Values are the mean

\pm

standard deviation over three random seeds.

Setting	Waveform	Linear-mel	Log-mel
mAP (%)	42.9 ${\scriptstyle\pm 0.0}$	42.8 ${\scriptstyle\pm 0.1}$	43.2 ${\scriptstyle\pm 0.1}$

TABLE VIII: Ablation on the mixture construction domain. Values are three-seed AS-20K fine-tuning mAP reported as mean

\pm

standard deviation.

Setting	Fixed	$\eta=1$	$\eta=2$	$\eta=4$	$\eta=6$	$\eta=8$
mAP (%)	42.9	43.0	43.1	43.3	43.2	43.1

TABLE IX: Sensitivity to the mixing-ratio distribution. Values are AS-20K fine-tuning mAP from single representative runs. “Fixed” denotes deterministic mixing with

\mu=0.5

, whereas

\eta

denotes the beta distribution parameter in

\mu\sim\operatorname{Beta}(\eta,\eta)

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Model	AS-2M	AS-20K	FSD-50K
Model	mAP (%)	mAP (%)	mAP (%)
EAT [ 9 ]	35.2	29.1	50.3
SSLAM [ 13 ]	33.2	27.4	47.2
AudioMosaic [ 12 ]	36.2	30.9	47.6
BAT [ 17 ]	37.4	32.5	55.8
MiCLA (ours)	34.7	29.5	52.1

Appendix

TABLE X: Supplementary linear probing results on multi-label settings.

Audio Representation Learning

Audio representation learning builds reusable features that capture speech, music, and sound for recognition, retrieval, generation, and analysis.

See all 211 papers

Do Audio Representations Compose Additively?
Sep 23, 2026Chenhao Xue, Zhijin Guo, Joyraj Chakraborty +2Audio Representation Learning Compositional Representation Learning
Department of Engineering Science, University of Oxford, Oxford, UK · University of Essex, Colchester, UK
MixProLAP: Mixture-Induced Uncertainty Modeling for Probabilistic Language-Audio Pretraining
Jun 18, 2026Yu Nakagome, Jaesong Lee, Soo-Whan ChungAudio-Text Alignment Multimodal Pretraining
LINE WORKS Corporation, Japan · NAVER Cloud Corporation, South Korea
FineLAP: Taming Heterogeneous Supervision for Fine-grained Language-Audio Pretraining
Apr 1, 2026Xiquan Li, Xuenan Xu, Ziyang Ma +4Audio-Language Models Audio Representation Learning
X-LANCE Lab, Shanghai Jiao Tong University, China · SJTU Paris Elite Institute of Technology, Shanghai Jiao Tong University, China · Shanghai Innovation Institute, China +1

Sep 23, 2026J/K move · Enter open · S save

Do Audio Representations Compose Additively?

Chenhao Xue, Zhijin Guo, Joyraj Chakraborty, Martin Reed+1

Department of Engineering Science, University of Oxford, Oxford, UK · University of Essex, Colchester, UK

Audio Representation Learning Compositional Representation Learning

Open paper AskPDF arXiv

Compositionality, the ability to represent complex acoustic scenes as combinations of simpler sound sources, is central to auditory perception and classical additive signal models. Still, it remains unclear whether modern pre-trained audio representations internalize additive structure without compositional supervision. Existing evaluation frameworks of audio compositional reasoning largely focus on cross-modal audio-text alignment, leaving open whether audio representations themselves exhibit additive compositional structure independent of text grounding, analogous to vector arithmetic in word representations. To investigate this, we adopt a two-step diagnostic for frozen audio representations. First, we quantify linear alignment between representations and sound source labels using canonical correlation analysis. Second, we test additive compositional generalization via leave-one-combination-out reconstruction, grouping clips by exact source-label set, averaging their representations, and predicting held-out means from per-source contributions fitted only on training combinations. With larger combination holdouts, CLAP outperforms the permuted and label-overlap baselines on FSD50K, while the speech models do not outperform the label-overlap baseline. We examine representations generated by Wav2Vec2, HuBERT, and CLAP on FSD50K and CHiME-Home datasets. All three models show consistently higher linear correlation and more accurate leave-one-combination-out reconstructions than the permuted baselines. However, only CLAP shows large cosine similarity gains, which could be associated with its training on many kinds of audio and text. Finally, we note that all three models exhibit reconstruction residuals, revealing limits of additive compositionality such as nonlinear or non-compositional audio structure.

Abstract

Audio Representation Learning

Explore similar work

Do Audio Representations Compose Additively?

MixProLAP: Mixture-Induced Uncertainty Modeling for Probabilistic Language-Audio Pretraining

FineLAP: Taming Heterogeneous Supervision for Fine-grained Language-Audio Pretraining