cs.SDNov 6, 2026

Concordance-Optimal Continuous Music Emotion Estimation: Variance Decomposition, Dual-Branch Temporal Fusion, and Conformal Prediction Intervals

Authors: Mohammad Faraz Akhtar, Aditya Parashar, Ayush Jain, Madhur Gupta, Ritesh Kumar, Soubhik Chakraborty

Organizations: Department of Computer Science and Engineering, Birla Institute of Technology, Mesra, Ranchi–835215, India · Department of Mathematics, Birla Institute of Technology, Mesra, Ranchi–835215, India

Abstract

Continuous music emotion recognition (MER) estimates how Arousal and Valence evolve within a song, yet the field's standard metric, pooled frame-level correlation, is dominated by between-song differences and cannot distinguish a model that tracks emotional dynamics from one that only ranks songs by their average level. We decompose annotation variance into between- and within-song components and show that mean squared error (MSE) targets the former, while windowed Lin's concordance correlation coefficient (CCC = r⋅Cbr \cdot C_b) targets the latter by penalising the variance compression MSE induces. A single weight λλ controls the trade-off: on 1,764 DEAM songs, raising λλ from 0 to 1 lowers pooled Arousal rr from 0.843 to 0.810 while raising mean within-song rr from 0.204 to 0.368. Building on this objective, we present MMAS (Multimodal Music Analysis System), an open-source model that predicts Arousal and Valence every 0.5 s from raw audio. It fuses frozen MERT-v1-95M embeddings with ten symbolic features from automatic transcription through a dual-branch temporal convolutional network with cross-modal attention. Because both branches are audio-derived, one model trains jointly on DEAM and PMEmo without domain conditioning. Under song-grouped 5-fold cross-validation (2,531 songs), pooled rr reaches 0.801/0.685 (Arousal/Valence) on DEAM and 0.742/0.746 on PMEmo. Against a ridge baseline on identical features, within-song Arousal rr roughly triples (0.344 vs. 0.125; n=353n = 353 songs, Wilcoxon p=1.7×10−21p = 1.7 \times 10^{-21}, d=0.56d = 0.56). Complete transcription failure costs at most 0.04 pooled rr. Split-conformal intervals attain 93.7% (Arousal) and 90.0% (Valence) marginal coverage at a 95% target, with the shortfall traced to within-song dependence. MMAS is released as a pip-installable package.

Figures & tables

Explore similar work

CardsList
  1. Predicting Music Valence and Arousal via Textual and Multimodal In-Context Learning in Large Language Models

    Oct 22, 2026Chi-En Dai, Chih-Pin Tan, Yi-Hsuan YangValence-Arousal ModelingEmotion Recognition

  2. Known Music, Unknown Response: Leakage-Audited Dynamic Arousal Forecasting With Scheduled Future Music

    Oct 16, 2026Bin Wen, Dai-Qiao Zhang, Tien-Ping TanEmotion RecognitionTime Series Forecasting

  3. MERIT: Learning Disentangled Music Representations for Audio Similarity

    May 26, 2026Abhinaba Roy, Junyi Liang, Dorien HerremansDisentangled Representation LearningAudio Representation Learning