MusicSem: A Semantically Rich Language--Audio Dataset of Natural Music Descriptions

Rebecca Salganik; Teng Tu; Fei-Yueh Chen; Xiaohao Liu; Keifeng Lu; Ethan Luvisia; Zhiyao Duan; Guillaume Salha-Galvan

cs.MMFeb 19, 2026

MusicSem: A Semantically Rich Language--Audio Dataset of Natural Music Descriptions

Authors: Rebecca Salganik, Teng Tu, Fei-Yueh Chen, Xiaohao Liu, Keifeng Lu, Ethan Luvisia, Zhiyao Duan, Guillaume Salha-Galvan,

Organizations: University of Rochester, NY, USA · National University of Singapore, Singapore · SJTU Paris Elite Institute of Technology, Shanghai, China · Singapore Management University, Singapore ·

Abstract

Music representation learning is central to music information retrieval and generation. While recent advances in multimodal learning have improved alignment between text and audio for tasks such as cross-modal music retrieval, text-to-music generation, and music-to-text generation, existing models often struggle to capture users' expressed intent in natural language descriptions of music. This observation suggests that the datasets used to train and evaluate these models do not fully reflect the broader and more natural forms of human discourse through which music is described. In this paper, we introduce MusicSem, a dataset of 32,493 language-audio pairs derived from organic music-related discussions on the social media platform Reddit. Compared to existing datasets, MusicSem captures a broader spectrum of musical semantics, reflecting how listeners naturally describe music in nuanced and human-centered ways. To structure these expressions, we propose a taxonomy of five semantic categories: descriptive, atmospheric, situational, metadata-related, and contextual. In addition to the construction, analysis, and release of MusicSem, we use the dataset to evaluate a wide range of multimodal models for retrieval and generation, highlighting the importance of modeling fine-grained semantics. Overall, MusicSem serves as a novel semantics-aware resource to support future research on human-aligned multimodal music representation learning.

Figures & tables

Figure 1: The MusicSem website provides access to the full dataset, detailed documentation, visualizations, and source code for data construction and experiments at: https://music-sem-web.vercel.app/ .

Dataset Name	Year	L-A Pairs	Annotation	Base Dataset
MusicNet ( Thickstun et al., 2018 )	2018	330	Human	-
Song Describer ( Manco et al., 2023 )	2023	1,106	Human	-
YouTube8M-MusicTextClips ( McKee et al., 2023 )	2023	4,169	Human	-
MusicCaps ( Agostinelli et al., 2023 )	2023	5,521	Human	-
MuLaMCap ( Huang et al., 2023 )	2023	6,800,000	LLM	AudioSet
LP-MusicCaps ( Doh et al., 2023 )	2023	2,000,000	LLM	MusicCaps, Magnatagtune, & Million Song Dataset

Table 1: Statistics of existing language–audio music datasets that are most comparable to our setting. L–A Pairs denotes the number of language–audio pairs, while Annotation indicates the origin of the textual annotations describing the music.

Category	Example of textual description of music
Descriptive	“I like the high pass filter on the vocals in the chorus, really makes harmonies pop.”
Contextual	“Sabrina Carpenter’s Espresso is just a mix of old Ariana Grande and 2018 Dua Lipa.”
Situational	“I listened to this song on the way to quitting my exhausting corporate job.”
Atmospheric	“This song makes me feel like a manic pixie dream girl in a bougie coffeeshop.”
Metadata	“This deluxe edition of this song was released in 2013 and it has three bonus hiphop tracks.”

Table 2: Examples of caption elements illustrating the five different categories in our proposed taxonomy of musical semantics.

Text-to-Audio Generative Model	$G^{d}$	$G^{a}$	$G^{s}$	$G^{m}$	$G^{c}$
AudioLDM2 ( Liu et al., 2024a )	0.68	0.37	0.35	0.40	0.34
MusicLM ( Agostinelli et al., 2023 )	0.50	0.36	0.42	0.39	0.35
Mustango ( Melechovsky et al., 2024 )	0.62	0.27	0.25	0.26	0.32
MusicGen ( Copet et al., 2023 )	0.57	0.47	0.39	0.47	0.52
Stable Audio ( Evans et al., 2024 )	0.72	0.67	0.68	0.70	0.74
Text-to-Audio Retrieval Model ( $k=10)$	$R^{d}$	$R^{a}$	$R^{s}$	$R^{m}$	$R^{c}$

Table 3: Semantic sensitivity analysis of generative (top) and retrieval (bottom) models. Best performance is in bold. Superscripts d , a , s , m , and c denote descriptive, atmospheric, situational, metadata, and contextual, respectively.

Figure 2: Example of semantic content extracted from a Reddit post in MusicSem. The figure highlights how a single description can express a variety of different musical semantics, corresponding to the five categories defined in our taxonomy.

Figure 3: Overview of the extraction and verification pipeline used to construct MusicSem. After selecting the source Reddit threads, the dataset construction proceeds in two main stages: an extraction step that identifies candidate semantic content from the textual elements of each thread, and a summarization and verification step that reformulates the extracted content into sentence-like semantic annotations, verifies song–artist associations, and checks the plausibility of the extracted semantic information.

Statistics	MusicCaps	Song Describer	MusicSem
Statistics	( Agostinelli et al., 2023 )	( Manco et al., 2023 )	(ours)
Number of entries	5,521	1,100	32,493
Number of distinct words	6,245	2,824	22,738
Number of distinct music genres	267	152	493
Category of Musical Semantics	MusicCaps	Song Describer	MusicSem
Category of Musical Semantics	( Agostinelli et al., 2023 )	( Manco et al., 2023 )	(ours)

Table 4: General statistics (top) and coverage by music semantic category (bottom) for MusicSem and two other canonical language–audio music datasets.

Figure 4: An example of personalization and contextualization on Reddit.

Panel of Figure 5 — Figure 5: Visualizations for MusicSem. (a) Music genre distribution visualized as a word cloud, where larger font size indicates higher frequency. (b) Popularity distribution of songs. (c) Distribution of the number of words per language–audio pair.

Dataset	Model	$\mathbf{Recall@1\uparrow}$	$\mathbf{Recall@5\uparrow}$	$\mathbf{Recall@10\uparrow}$	$\mathbf{NDCG@5\uparrow}$	$\mathbf{NDCG@10\uparrow}$	$\mathbf{MRR\uparrow}$
MusicCaps	Random	0.04	0.18	0.36	0.10	0.16	0.31
	LARP	0.14	0.49	0.98	0.30	0.45	0.62
	CLAP	5.84	15.57	22.60	10.73	12.99	11.60
	ImageBind	3.15	10.18	14.91	6.72	8.25	7.23
	CLaMP3	2.73	8.82	13.65	5.81	7.32	9.07
Song Describer	Random	0.14	0.71	1.41	0.41	0.64	1.01

Table 5: Evaluation results on the text-to-music retrieval task. Best performance for each metric within a dataset is shown in bold, and second-best results are underlined. All scores are reported as percentages. In this retrieval setting, NDCG@1 is equivalent to Recall@1 and is therefore not reported in the table.

Dataset	Model	$\mathbf{B}_{1}\uparrow$	$\mathbf{B}_{2}\uparrow$	$\mathbf{B}_{3}\uparrow$	$\mathbf{M}$ ↑	$\mathbf{R}\uparrow$	CIDEr $\mathbf{\uparrow}$	BERT-S $\mathbf{\uparrow}$
MusicCaps	LP-MusicCaps	53.21	47.28	44.60	51.90	3.35	384.72	90.47
	MU-LLaMA	1.35	0.55	0.22	40.22	11.27	0.09	80.47
	FUTGA	8.80	3.07	1.19	44.77	11.90	2.63e-17	81.67
Song Describer	LP-MusicCaps	9.51	3.07	0.94	8.90	10.45	1.03	84.40
	MU-LLaMA	12.03	4.73	1.73	8.72	13.00	3.59	83.51
	FUTGA	3.39	1.28	0.43	8.72	6.30	3.58e-30	82.55

Table 6: Evaluation results on the music-to-text generation task. Best performance for each metric within a dataset is shown in bold.

Category of Semantics	LP-MusicCaps	MU-LLaMA	FUTGA	Ground Truth in MusicSem
Descriptive	100%	99%	100%	83%
Contextual	2%	1%	0%	17%
Situational	42%	0%	1%	38%
Atmospheric	78%	3%	91%	62%
Metadata	32%	2%	34%	15%

Table 7: Semantic analysis of music-to-text generation on MusicSem. We report the proportion of captions containing at least one element from each semantic category, for both ground-truth annotations in the MusicSem test set and model-generated captions.

Figure 6: Case study of music-to-text generation evaluation using MusicSem. The reference song is While Others Cry by The Future Sound of London. All models produce captions containing factual inaccuracies and primarily focus on descriptive attributes.

Dataset	Model	FAD ${}_{\text{V}}^{\text{MC}}$ ↓	FAD ${}_{\text{V}}^{\text{FMA}}$ ↓	FAD ${}_{\text{M}}^{\text{FMA}}$ ↓	FAD ${}_{\text{E}}^{\text{FMA}}$ ↓	KLD ↓	Vendi ↑	CS ↑
MusicCaps	MusicLM	5.70	21.57	87.39	249.72	1.79	1.55	0.28
	Stable Audio	6.97	15.60	82.21	377.02	1.90	1.31	0.31
	MusicGen	7.03	16.29	73.22	354.07	0.90	1.57	0.29
	AudioLDM2	3.29	19.31	60.02	202.11	0.61	1.57	0.36
	Mustango	1.27	22.96	55.84	161.47	1.51	1.48	0.27
	Mureka	-	-	-	-	-	-	-

Table 8: Evaluation results on the text-to-music generative task. Best performance for each metric within a dataset is shown in bold, and second-best results are underlined. Note: Experiments for Mureka on MusicCaps could not be completed due to unresolved issues (at the time of writing) with the Mureka API, which prevent automated access to audio data from their website. For completeness, we nonetheless report the results obtained for Mureka on MusicSem and Song Describer.

Category of Semantics	Metric	Score
Descriptive	$G^{d}$	0.55
Atmospheric	$G^{a}$	0.36
Situational	$G^{s}$	0.32
Contextual	$G^{c}$	0.29
Metadata	$G^{m}$	0.36

Table 9: Sensitivity of the CLAP score on MusicSem. The superscripts d , a , s , c , and m refer to descriptive, atmospheric, situational, contextual, and metadata, respectively.

Retrieval Performance	Recall@1	Recall@5	Recall@10	NDCG@5	NDCG@10	MRR
No Fine-Tuning	0.82	5.74	9.84	3.54	4.74	4.65
Fine-Tuning on MusicSem	4.51	11.48	17.42	8.06	9.95	9.16
Relative Improvement	+450.00%	+100.00%	+77.03%	+127.68%	+109.92%	+96.77%
Semantic Sensitivity	$G^{d}$	$G^{a}$	$G^{s}$	$G^{c}$	$G^{m}$
No Fine-Tuning	0.55	0.36	0.32	0.29	0.36
Fine-Tuning on MusicSem	0.64	0.38	0.41	0.43	0.38

Table 10: Effects of fine-tuning CLAP ( Wu et al., 2023 ) on the MusicSem training set, evaluated in terms of cross-modal text-to-music retrieval performance and semantic sensitivity on the MusicSem test set. Best performance for each metric is shown in bold. Improvements are reported relative to the non–fine-tuned baseline. In this retrieval setting, NDCG@1 is equivalent to Recall@1 and is therefore not reported in the table.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Name	Date	Architecture	Text Conditioner	Length	Sample Rate	Proprietary
Text-to-Music	MusicLM ( Agostinelli et al., 2023 )	2023	Hier. Trans. + SoundStream	w2v-BERT ( Chung et al., 2021 )	variable	24kHz
	AudioLDM 2 ( Liu et al., 2024a )	2023	VAE + 2D U-Net	CLAP ( Wu et al., 2023 )	variable	16kHz
	Stable Audio ( Evans et al., 2024 )	2023	VAE + 2D U-Net	CLAP ( Wu et al., 2023 )	up to 95s	48kHz
	MusicGen ( Copet et al., 2023 )	2024	AE + 1D U-Net	FLAN-T5 ( Chung et al., 2024 )	10s	48kHz
	Mustango ( Melechovsky et al., 2024 )	2024	VAE + 2D U-Net	FLAN-T5 ( Chung et al., 2024 )	10s	16kHz
	Mureka	2024	-	-	-	-	✓

Appendix

Table 11: Overview of all models evaluated in this work. Hier. , Trans. , Diff. , and Co-List. denote Hierarchical, Transformer, Diffusion, and Co-Listing, respectively.

Model	Inference Time (in seconds)	Generation Size (in seconds)	Tradeoff ↓
MusicLM	102	5	20.40
AudioLDM2	13	10	1.30
Mustango	50	10	5.00
MusicGen	40	20	2.00
Stable Audio	18	45	0.40
Mureka	120	150	0.80

Appendix

Table 12: Inference time of text-to-music generation models on MusicSem. Trade-off is defined as inference time divided by generation size.

Model	Inference Time (in seconds)	Generation Size (in characters)	Tradeoff ↓
LP-MusicCaps	8	2000	0.004
MU-LLaMA	4	70	0.057
FUTGA	15	1138	0.013

Appendix

Table 13: Inference time of music-to-text generation models on MusicSem. Trade-off is defined as inference time divided by generation size.

Model	Inference Time (in seconds)
LARP	0.26
CLAP	0.23
ImageBind	0.21
CLAMP3	0.28

Appendix

Table 14: Inference time of cross-modal retrieval models on MusicSem.

Figure 7: Comparison between Vendi scores for cover songs and random collection.

Music Information Retrieval

Music Information Retrieval develops computational methods for finding, classifying, describing, and analyzing music from audio, scores, and metadata.

See all 153 papers

Multimodal Music Recommendation System using LLMs
May 28, 2026Srikar Prabhas Kandagatla, Sreehitha R. Narayana, Chandana Magapu +6Sequential Recommendation Recommender Systems
University of Massachusetts Amherst, Massachusetts, USA · Dolby Laboratories Atlanta, Georgia, USA · Adobe Research San Jose, California, USA +2
From Prompting to Describing: A Cross-Cultural Study of Language for AI-Generated Music
Aug 6, 2026Sangheon Park, Claire ArthurText-to-Music Generation Audio Understanding
Georgia Institute of Technology School of Music
FIGMA: Towards FIne-Grained Music retrievAl
Jun 4, 2026Nishit Anand, Ashish Seth, Sreyan Ghosh +2Audio Understanding Audio-Text Retrieval
University of Maryland, College Park, USA

May 28, 2026J/K move · Enter open · S save

Multimodal Music Recommendation System using LLMs

Srikar Prabhas Kandagatla, Sreehitha R. Narayana, Chandana Magapu, Swetha Mohan+5

University of Massachusetts Amherst, Massachusetts, USA · Dolby Laboratories Atlanta, Georgia, USA · Adobe Research San Jose, California, USA+2

Sequential Recommendation Recommender Systems Music Recommendation Large Language Model-Based Recommendation Multimodal Recommendation Multimodal Fusion

Open paper AskPDF arXiv

Music recommendation systems typically treat songs as opaque tokens, relying on collaborative interaction histories which overlooks semantic or acoustic content. Prior work has explored LLM-augmented, multimodal, and text-enhanced approaches to sequential recommendation, and while some methods partially combine semantic, acoustic, or engagement signals, none jointly model all three within a unified LLM-based sequential reasoning framework that grounds recommendations in actual song content. In this work, we propose a multimodal framework for session-based music recommendation that enriches the LastFM-1K dataset with three complementary signals: (1) audio and lyric embeddings extracted using pretrained music and text representation models, (2) LLM-generated semantic metadata using the MGPHot annotation schema, and (3) listening completion ratios. We adopt the E4SRec framework by extending it with multimodal features and different item ID encoder backbones, including SASRec, BERT4Rec, and GRU4Rec. We further extend the LLM backbone option with LLaMa-2-13B, Qwen2.5-7B-Instruct, and LLaMa-3-70B in both zero-shot and fine-tuned settings. Our experiments show that integrating content-based features improves over ID-only baselines up to 95% in terms of Recall and 79% in terms of NDCG. Moreover, our experiments show that naive multimodal fusion does not always yield additive improvements, highlighting challenges in cross-modal integration. We release a large-scale multimodal benchmark for music recommendation.

Abstract

Music Information Retrieval

Explore similar work

Multimodal Music Recommendation System using LLMs

From Prompting to Describing: A Cross-Cultural Study of Language for AI-Generated Music

FIGMA: Towards FIne-Grained Music retrievAl