Organizations: Department of Computer Science and Engineering, Birla Institute of Technology, Mesra, Ranchi–835215, India · Department of Mathematics, Birla Institute of Technology, Mesra, Ranchi–835215, India
Continuous music emotion recognition (MER) estimates how Arousal and Valence evolve within a song, yet the field's standard metric, pooled frame-level correlation, is dominated by between-song differences and cannot distinguish a model that tracks emotional dynamics from one that only ranks songs by their average level. We decompose annotation variance into between- and within-song components and show that mean squared error (MSE) targets the former, while windowed Lin's concordance correlation coefficient (CCC = r⋅Cb) targets the latter by penalising the variance compression MSE induces. A single weight λ controls the trade-off: on 1,764 DEAM songs, raising λ from 0 to 1 lowers pooled Arousal r from 0.843 to 0.810 while raising mean within-song r from 0.204 to 0.368. Building on this objective, we present MMAS (Multimodal Music Analysis System), an open-source model that predicts Arousal and Valence every 0.5 s from raw audio. It fuses frozen MERT-v1-95M embeddings with ten symbolic features from automatic transcription through a dual-branch temporal convolutional network with cross-modal attention. Because both branches are audio-derived, one model trains jointly on DEAM and PMEmo without domain conditioning. Under song-grouped 5-fold cross-validation (2,531 songs), pooled r reaches 0.801/0.685 (Arousal/Valence) on DEAM and 0.742/0.746 on PMEmo. Against a ridge baseline on identical features, within-song Arousal r roughly triples (0.344 vs. 0.125; n=353 songs, Wilcoxon p=1.7×10−21, d=0.56). Complete transcription failure costs at most 0.04 pooled r. Split-conformal intervals attain 93.7% (Arousal) and 90.0% (Valence) marginal coverage at a 95% target, with the shortfall traced to within-song dependence. MMAS is released as a pip-installable package.
Figures & tables
Property
Value
Total songs
1,802 (1,764 with usable dynamic annotations + features)
acoustic (768-d) and symbolic (10-d) feature vectors at frame t
t
frame index on the 0.5 s grid
n
number of frames in a track
Table 2: Nomenclature.
Figure 2: MERT layer probe (linear model, training songs only): pooled r per hidden layer; layer 7 is valence-optimal and near the arousal optimum.
#
Feature
Definition
Unit
1
Note density
Count of note onsets in the frame context, divided by the context duration
onsets per second
2
Mode ratio
Proportion of sounding pitch classes belonging to the major-key diatonic set relative to the minor set
dimensionless, [0,1]
3
Dissonance
Proportion of simultaneously sounding pitch pairs forming dissonant intervals (minor second, tritone, major seventh, and their compounds)
dimensionless, [0,1]
4
Beat entropy
Shannon entropy of the histogram of inter-onset intervals in the context, normalised by the log of the bin count; low values indicate regular rhythm
bits (normalised), [0,1]
5
Tempo variance
Variance of the local inter-onset-interval sequence, normalised by its mean (coefficient of variation)
dimensionless
6
Pitch mean
Mean MIDI note number over all notes sounding in the context
MIDI note number (0–127)
Table 3: The ten Branch B symbolic features. Each is computed from the transcribed MIDI within a 2 s context centred on grid frame t . “Dimensionless” indicates a ratio or normalised quantity on [0,1] unless stated otherwise.
Figure 3: Symbolic (Branch B) feature correlations over all DEAM frames: an activity cluster ( r≈0.6 – 0.9 ) and a near-orthogonal tonal/contour group.
Branch
Feature group
Dim
A
MERT-v1-95M layer-7 embedding (mean-pooled to 0.5 s)
768
B
Symbolic (note density, mode ratio, dissonance, beat entropy, tempo variance, pitch mean, contour slope, range, major weight, key clarity)
Table 5: Architecture and training hyperparameters (as run).
Figure 4: Joint-model training and validation loss; best validation at epoch 8, early stopping at 16.
Evaluation set
Arousal r / CCC
Arousal per-song r ( %>0.35 )
Valence r / CCC
Valence per-song r ( %>0.35 )
DEAM (1,764 songs)
0.801 / 0.799
0.362 (59.9%)
0.685 / 0.681
0.220 (45.6%)
PMEmo (523 songs)
0.742 / 0.725
0.291 (50.7%)
0.746 / 0.736
0.259 (50.7%)
Overall (2,287 songs)
0.790 / 0.789
0.345 (57.8%)
0.701 / 0.698
0.229 (46.7%)
Table 6: Joint-model results, Group 5-Fold CV, every song out-of-fold (shared MERT-768 + 10-symbolic space; 140,120 OOF frames).
Figure 5: Per-dimension out-of-fold accuracy of the joint DEAM+PMEmo model (pooled Pearson r ; 1,764 DEAM + 523 PMEmo songs, 5-fold), against the accuracy ceiling reported by the field’s meta-analysis [ 5 ] .
Figure 6: Predicted vs ground truth density over all 140,120 out-of-fold frames (hexbin): arousal tighter than valence.
Figure 7: Within-song arousal tracking on two rule-chosen out-of-fold DEAM songs (80th-percentile and median per-song r among songs with moving ground truth; predictions carry the tool’s 2.5 s smoothing).
Figure 8: Distribution of per-song dynamic r over all 2,287 out-of-fold songs, arousal and valence, with the 0.35 reference threshold.
Configuration
Arousal r / CCC
Arousal per-song r ( %>0.35 )
Valence r / CCC
Valence per-song r ( %>0.35 )
Full MMAS ( λ=0.5 )
0.819 / 0.818
0.363 (59.4%)
0.681 / 0.675
0.206 (44.4%)
− cross-attention (concat)
0.797 / 0.790
0.382 (60.3%)
0.672 / 0.652
0.214 (43.8%)
reverse attention ( Q = symbolic)
0.811 / 0.808
0.376 (59.9%)
0.680 / 0.673
0.218 (45.2%)
− symbolic branch (acoustic only)
0.767 / 0.760
0.371 (58.6%)
0.635 / 0.624
0.205 (43.2%)
symbolic only
0.367 / 0.323
0.123 (40.2%)
0.274 / 0.216
0.042 (31.7%)
− MERT-768 (legacy 32-d subsample)
0.743 / 0.736
0.344 (58.7%)
0.554 / 0.539
0.164 (40.9%)
Table 7: DEAM 5-fold CV ablation (1,764 songs, 112,220 frames, all out-of-fold). Full MMAS = windowed CrossModalTCN + both branches + shape loss at λ=0.5 .
Figure 9: DEAM 5-fold ablation (all out-of-fold): pooled and per-song r per configuration, arousal and valence.
Figure 10: The shape-loss trade-off (arousal): (a) pooled and per-song r against λ ; (b) the resulting frontier, with λ annotated at each point.
Dimension
n songs
Deep per-song r (SD)
Ridge per-song r (SD)
Δ ( 95% bootstrap CI)
Wilcoxon V
p
% songs improved
Cohen’s d
Arousal
353
0.344 (0.462)
0.125 (0.349)
+0.219 [0.178, 0.260]
12,974
1.7×10−21
74.5%
0.56
Valence
353
0.235 (0.433)
0.051 (0.322)
+0.183 [0.138, 0.228]
17,042
1.4×10−13
68.0%
0.42
Table 8: Per-song tracking, deep model vs Ridge (paired, held-out songs).
Figure 11: Within-song tracking, MMAS vs matched Ridge baseline: (a) mean per-song r for both systems; (b) the paired difference with its 95% song-level bootstrap confidence interval, Wilcoxon statistic, p -value, effect size, and percentage of songs improved (353 held-out songs).
Condition
Arousal r (pooled / per-song)
Valence r (pooled / per-song)
Intact
0.797 / 0.314
0.678 / 0.238
Noise σ=0.25
0.797 / 0.313
0.678 / 0.239
Noise σ=0.5
0.797 / 0.314
0.678 / 0.236
Noise σ=1.0
0.798 / 0.314
0.674 / 0.236
Mean-substitution
0.770 / 0.314
0.641 / 0.245
Raw-zero (hard failure)
0.776 / 0.308
0.640 / 0.236
Table 9: Branch B corruption at inference (released model, held-out test songs).
Dimension
q^ (half-width)
Interval width
Empirical frame coverage
Song-level coverage (mean)
Song-level coverage (10th pct.)
Arousal
0.219
0.437
93.7%
94.0%
78.3%
Valence
0.211
0.423
90.0%
91.6%
68.1%
Table 10: Split-conformal prediction intervals, α=0.05 (target coverage 95% ). Calibration on 13,832 validation frames; evaluation on 28,796 test frames from 456 unseen songs.
System
Arousal r
CCC
Dataset
Protocol
OpenSMILE baseline [ 1 ]
0.36 (per-song)
Not reported
DEAM
Official per-song
MCAN [ 23 ]
Not reported
0.50
DEAM subset
Random split
Meta-analysis best models [ 5 ]
0.81 (arousal) / 0.67 (valence)
Not reported
Across studies
Varies
Ridge baseline (ours)
0.829 (pooled)
0.823
DEAM
Group 5-Fold OOF
MMAS (ours, DEAM-only)
0.819 (pooled)
0.818
DEAM
Group 5-Fold OOF
MMAS (ours, joint DEAM+PMEmo)
0.801 (pooled)
0.799
DEAM
Joint Group 5-Fold OOF
Table 11: Comparison with published MER systems. Protocols differ; numbers are not directly comparable across rows.
Figure 12: The released model’s output on a PMEmo test song unseen by its weights, in the form the tool reports it: per-0.5 s Arousal and Valence with the split-conformal prediction interval of Section 5.9 , shown against the song’s annotation. The song is selected by rule (closest to the 60th percentile of per-song arousal r among test songs whose annotation moves), and the panel is regenerated from the committed prediction artifact rather than screen-captured.
Emotion recognition uses language, voice, face, body movement, and physiological signals to infer emotional categories or dimensions such as valence and arousal.
While arousal can be reliably predicted from musical audio, emotional valence remains difficult to model due to its subjective and context-dependent nature. In this work, we investigate the potential of in-context learning (ICL) with large language models to advance continuous valence and arousal prediction. Rather than predicting emotion scores directly, our framework leverages contextual demonstrations from an auxiliary dataset to refine the initial predictions produced by a small, task-specific base model. We systematically evaluate a text-based ICL variant using lyrics and generated audio captions, alongside a multimodal variant that incorporates raw audio signals. Using the entire MER60 dataset strictly as a held-out test set, our experiments demonstrate substantial performance gains, particularly for valence. Text-based ICL improves the valence R2 from 0.110 to 0.427 with 80 demonstration shots, while the multimodal variant achieves 0.346 with only 10 shots. These findings suggest that training-free ICL offers an effective inference-time strategy for current language models to predict continuous music emotion.
Chi-En Dai, Chih-Pin Tan, Yi-Hsuan Yang
Graduate Institute of Communication Engineering, National Taiwan University, Taiwan
Dynamic music emotion recognition usually estimates concurrent or lag-aligned labels from audio that has already been observed. We instead forecast the change in consensus arousal when the upcoming music is scheduled but all later affect labels remain hidden. Evaluation holds out complete songs, encodes each eligible 4-s chunk independently, predicts arousal innovation, and compares the correct future with displaced segments from the same song. On PMEmo and a 1,494-song DEAM evaluation pool excluded from engineering development, a complete past/future MuQ representation raises concordance correlation coefficient (CCC) by 0.389--0.507 over recent affect and remains 0.268--0.382 above a calibrated nonlinear state model. A future-only trajectory contains no past or current audio and still exceeds that state model by 0.150--0.249. The earliest legal future chunk, which ends 5.5--11 s before the target, also remains informative. Correct placement exceeds an overlapping 5-s shift by 0.100--0.209 and the acoustic-nearest eligible same-song segment by 0.078--0.105, with positive song-cluster lower bounds. These results show that both scheduled content and its within-song position carry forecast information under the stated observation rules. The study provides a leakage-controlled reference for known-future music forecasting without treating predictive substitutions as causal interventions.
Bin Wen, Dai-Qiao Zhang, Tien-Ping Tan
School of Computer Sciences, Universiti Sains Malaysia, 11800, Pulau Pinang, Malaysia · School of Mathematical Sciences, Universiti Sains Malaysia, 11800, Pulau Pinang, Malaysia
Current music similarity models typically compute a single, monolithic score, entangling distinct musical dimensions like melody, rhythm, and timbre. This limits user control and interpretability, making it impossible to execute nuanced queries. We introduce MERIT, a framework for learning disentangled, factor-specific music representations tailored to these three core dimensions. To overcome the lack of isolated musical variations in real-world audio, we use a novel training strategy that uses conditional audio generation and source-separated stems to strongly encourage single-factor variation in training data. Our evaluations demonstrate strong factor-wise disentanglement. Each head responds strongly to its intended perceptual dimension while remaining near chance on the others, a representational property that holds across both the synthetic training domain and independent real-world audio.