Exploring Post-Hoc Disentanglement of Audio Foundation Models for Music Similarity
Organizations: Universidade de São Paulo
Abstract
Audio foundation models like CLAP, MuQ, and MERT excel at capturing general musical similarity. On the other hand, their entangled latent spaces limit interpretability and fine-grained attribute control. Conditional Similarity Networks (CSN) address this by pairing a backbone representation model with learned masks that gate the embedding space into attribute-specific subspaces. However, existing methods train the backbone from scratch, rather than leveraging pretrained foundation models. We instead present a systematic study of post-hoc disentanglement, adapting CSN to gate frozen pretrained embeddings. We evaluate adding trainable parameters to adapt the embedding under different masking and pooling regimes across two datasets. Our findings show that minimal adaptation capacity is sufficient for post-hoc gating to improve both attribute-specific disentanglement and overall retrieval performance, while pooling, masking, and depth choices yield limited additional gains.
Figures & tables
| CLAP | MERT | MuQ | ||||
|---|---|---|---|---|---|---|
| Configuration | Jamendo | MTT | Jamendo | MTT | Jamendo | MTT |
| Baseline | 69.3 0.0 | 74.8 0.0 | 58.1 0.0 | 60.3 0.0 | 64.3 0.0 | 70.0 0.0 |
| Identity + Mean + Learned Mask | 71.6 0.3 | 76.5 0.1 | 57.8 0.2 | 60.2 0.3 | 64.6 1.1 | 69.9 0.8 |
| Identity + Attention + Learned Mask | 70.0 0.3 | 74.2 0.4 | 58.1 0.3 | 60.5 0.3 | 64.0 0.4 | 69.9 0.6 |
| One Layer + Mean + Fixed Mask | 71.5 0.3 | 74.1 0.4 | 68.4 0.7 | 70.0 2.1 | 71.0 0.3 | 70.8 0.6 |
| One Layer + Mean + Learned Mask | 71.2 0.4 | 74.2 0.2 | 68.4 0.7 | 70.1 1.7 | 70.9 0.3 | 71.5 0.6 |
| CLAP | MERT | MuQ | ||||
|---|---|---|---|---|---|---|
| Configuration | Jamendo | MTT | Jamendo | MTT | Jamendo | MTT |
| Identity + Mean + Learned Mask | 2.14 0.30 | 2.71 0.42 | -0.03 0.15 | 0.07 0.37 | 0.49 0.81 | 0.37 1.07 |
| Identity + Attention + Learned Mask | 0.35 0.31 | 0.24 0.36 | -0.03 0.21 | 0.24 0.47 | -0.13 0.34 | 0.57 0.84 |
| One Layer + Mean + Fixed Mask | 7.37 0.31 | 4.84 0.40 | 4.83 0.70 | 5.74 0.76 | 7.69 0.30 | 3.51 0.78 |
| One Layer + Mean + Learned Mask | 7.02 0.32 | 4.60 0.20 | 5.07 0.45 | 5.71 0.41 | 7.12 0.41 | 3.81 0.40 |
| One Layer + Attention + Fixed Mask | 7.41 0.21 | 5.30 0.49 | 4.13 1.14 | 4.07 0.29 | 6.97 0.33 | 3.59 0.43 |
| Model | Mask | Jamendo | MTT | ||||
|---|---|---|---|---|---|---|---|
| Inst. | Genre | Mood | Inst. | Genre | Mood | ||
| CLAP | None | 73.2 | 75.5 | 66.4 | 73.7 | 86.2 | 64.8 |
| Inst. | 74.1 | 71.3 | 63.7 | 73.9 | 81.0 | 58.2 | |
| Genre | 71.6 | 76.8 | 65.2 | 71.3 | 86.5 | 63.5 | |
| Mood | 65.6 | 69.6 | 67.2 | 70.1 | 81.3 | 70.3 | |
| MERT | None | 69.2 | 71.7 | 64.2 | 65.8 | 79.6 | 63.6 |
Representation Disentanglement
Representation disentanglement studies how to separate distinct factors of variation in learned features so models can interpret, control, and transfer them.
See all 104 papers