LARA-Gen: Enabling Continuous Emotion Control for Music Generation Models via Latent Affective Representation Alignment

Jiahao Mei; Shimeng Li; Xuenan Xu; Leyuan Jin; Yixuan Yuan; Zeyu Xie; Zihao Zheng; Ye Tao

cs.SDOct 7, 2025

LARA-Gen: Enabling Continuous Emotion Control for Music Generation Models via Latent Affective Representation Alignment

Authors: Jiahao Mei, Shimeng Li, Xuenan Xu, Leyuan Jin, Yixuan Yuan, Zeyu Xie, Zihao Zheng, Ye Tao,

Organizations: X-LANCE Lab, Shanghai Jiao Tong University, China · Shanghai AI Lab, China · Shanghai Mental Health Center, Shanghai Jiao Tong University School of Medicine, China

Abstract

Recent advances in text-to-music models have enabled coherent music generation from text prompts, yet fine-grained emotional control remains unresolved. We introduce LARA-Gen, a framework for continuous emotion control that aligns the internal hidden states with an external music understanding model through Latent Affective Representation Alignment (LARA), enabling effective training. In addition, we design an emotion control module based on a continuous valence-arousal space, disentangling emotional attributes from textual content and bypassing the bottlenecks of text-based prompting. Furthermore, we establish a benchmark with a curated test set and a robust Emotion Predictor, facilitating objective evaluation of emotional controllability in music generation. Extensive experiments demonstrate that LARA-Gen achieves continuous, fine-grained control of emotion and significantly outperforms baselines in both emotion adherence and music quality. Generated samples are available at https://anonymous2232330.github.io/laragen-web/.

Figures & tables

Figure 1: (a) LARA-Gen framework. A Proxy Network $\mathcal{P}_{\theta}$ aligns the internal hidden states $\mathbf{H}$ of the backbone model with target features $\bar{\mathbf{M}}$ from a frozen MERT encoder. (b) The architecture of the Emotion Predictor. It uses a sliding window over MERT features and an Emotion Regression Head $\mathcal{R}_{\phi}$ to produce a final valence-arousal prediction from an input music clip.

Figure 2: Emotion values predicted by our Emotion Predictor vs. ground-truth emotion values on the MUSE-VA test set. The LARA-Gen system outperforms the InspireMusic baseline in both error and correlation.

Generation System	Prompt	Loss	FAD $\downarrow$	Pairwise L2 $\uparrow$	Arousal			Valence
Generation System	Prompt	Loss	FAD $\downarrow$	Pairwise L2 $\uparrow$	CCC $\uparrow$	PCC $\uparrow$	RMSE $\downarrow$	CCC $\uparrow$	PCC $\uparrow$	RMSE $\downarrow$
Ground Truth	-	-	0.000	2.963	0.868	0.875	0.283	0.829	0.842	0.309
MusicGen	Text	-	3.060	2.743	0.092	0.153	0.676	0.129	0.190	0.623
InspireMusic	Text	-	3.325	3.454	0.060	0.083	0.673	0.044	0.058	0.659
ACE-Step	Text	-	5.198	1.858	0.088	0.214	0.622	0.064	0.133	0.656
Emotion Text Fine-tuning	Text	CE	2.153	2.947	0.795	0.816	0.340	0.700	0.763	0.401

Table 1: Emotional music generation performance on the MUSE-VA test set. Emotion consistency is evaluated by our proposed Emotion Predictor. Best and second-best results among generated systems are bold and underlined , respectively.

Generation System	OVL $\uparrow$	PCC $\uparrow$
Generation System	OVL $\uparrow$	Arousal	Valence
Ground Truth	3.90 $\pm$ 0.97	0.882 ∗∗∗	0.549 ∗
Text Fine-tuning	3.16 $\pm$ 1.07	0.709 ∗	0.613 ∗
LARA-Gen	3.47 $\pm$ 1.14	0.881 ∗∗∗	0.648 ∗

Table 2: Subjective evaluation results. Significance levels for PCC are denoted by asterisks (

{}^{*}p<0.05

{}^{***}p<0.001

). Best results among generated systems are highlighted in bold .

Controllable Music Generation

Controllable music generation steers composition and audio synthesis through prompts, musical structure, references, and interactive controls.

See all 91 papers

How Well Do Generative Music Models Follow Emotion Conditioning?
Aug 30, 2026Morteza HeydariText-to-Music Generation Music Generation Evaluation
Istinye University, Graduate School, Istanbul, Turkey
Aligning MusicLLM with Emotion using Instruction Tuning and Feedback-Driven Alignment
Jun 23, 2026Takuya Hasumi, Welly NaptaliLLM Alignment AI Alignment
LY Corporation
DiffSynth-Music: Audio-Conditioned KV-Cache Adapters for Controllable Music Generation
Sep 11, 2026Zhongjie Duan, Shengchuan Gao, Hong Zhang +1Controllable Music Generation Audio Diffusion Models
Alibaba Group · Shanghai Jiao Tong University

Aug 30, 2026J/K move · Enter open · S save

How Well Do Generative Music Models Follow Emotion Conditioning?

Morteza Heydari

Istinye University, Graduate School, Istanbul, Turkey

Text-to-Music Generation Music Generation Evaluation Controllable Music Generation Music Generation Timbre-Conditioned Music Generation

Open paper AskPDF arXiv

Recent generative music models offer increasingly fine-grained control through text and audio conditioning, yet how faithfully they follow intended emotional cues remains an open question. We address this gap with a unified evaluation pipeline for emotion-following in generated music. Using all 1000 tracks in GTZAN, we extract semantic audio descriptions with DashengLM, an audio captioning model, and estimate source-track valence and arousal with Music2Emotion, a music emotion recognition model. We construct affect-aware text prompts by combining descriptions with top-ranked emotion tags and generate 30-second outputs with three systems, Stable Audio Open, MusicGen, and InspireMusic, evaluating both text- and audio-conditioned generation. To measure emotion-following, we compute valence and arousal on generated audio and compare them with the source tracks using absolute error and Euclidean distance in valence-arousal space. Text-conditioned generation consistently outperforms audio conditioning, with MusicGen (text) and InspireMusic (text) achieving the best performance, while audio-conditioned variants prove less stable. We further find that valence is preserved more reliably than arousal and that emotion-following varies substantially across genres. These findings underscore the importance of evaluating affective controllability directly rather than relying solely on general quality or prompt-relevance metrics.

Abstract

Controllable Music Generation

Explore similar work

How Well Do Generative Music Models Follow Emotion Conditioning?

Aligning MusicLLM with Emotion using Instruction Tuning and Feedback-Driven Alignment

DiffSynth-Music: Audio-Conditioned KV-Cache Adapters for Controllable Music Generation