Organizations: Department of Computer Science and Engineering, Birla Institute of Technology, Mesra, Ranchi–835215, India · Department of Mathematics, Birla Institute of Technology, Mesra, Ranchi–835215, India
Continuous music emotion recognition (MER) estimates how Arousal and Valence evolve within a song, yet the field's standard metric, pooled frame-level correlation, is dominated by between-song differences and cannot distinguish a model that tracks emotional dynamics from one that only ranks songs by their average level. We decompose annotation variance into between- and within-song components and show that mean squared error (MSE) targets the former, while windowed Lin's concordance correlation coefficient (CCC = r⋅Cb) targets the latter by penalising the variance compression MSE induces. A single weight λ controls the trade-off: on 1,764 DEAM songs, raising λ from 0 to 1 lowers pooled Arousal r from 0.843 to 0.810 while raising mean within-song r from 0.204 to 0.368. Building on this objective, we present MMAS (Multimodal Music Analysis System), an open-source model that predicts Arousal and Valence every 0.5 s from raw audio. It fuses frozen MERT-v1-95M embeddings with ten symbolic features from automatic transcription through a dual-branch temporal convolutional network with cross-modal attention. Because both branches are audio-derived, one model trains jointly on DEAM and PMEmo without domain conditioning. Under song-grouped 5-fold cross-validation (2,531 songs), pooled r reaches 0.801/0.685 (Arousal/Valence) on DEAM and 0.742/0.746 on PMEmo. Against a ridge baseline on identical features, within-song Arousal r roughly triples (0.344 vs. 0.125; n=353 songs, Wilcoxon p=1.7×10−21, d=0.56). Complete transcription failure costs at most 0.04 pooled r. Split-conformal intervals attain 93.7% (Arousal) and 90.0% (Valence) marginal coverage at a 95% target, with the shortfall traced to within-song dependence. MMAS is released as a pip-installable package.
Figures & tables
Property
Value
Total songs
1,802 (1,764 with usable dynamic annotations + features)
acoustic (768-d) and symbolic (10-d) feature vectors at frame t
t
frame index on the 0.5 s grid
n
number of frames in a track
Table 2: Nomenclature.
Figure 2: MERT layer probe (linear model, training songs only): pooled r per hidden layer; layer 7 is valence-optimal and near the arousal optimum.
#
Feature
Definition
Unit
1
Note density
Count of note onsets in the frame context, divided by the context duration
onsets per second
2
Mode ratio
Proportion of sounding pitch classes belonging to the major-key diatonic set relative to the minor set
dimensionless, [0,1]
3
Dissonance
Proportion of simultaneously sounding pitch pairs forming dissonant intervals (minor second, tritone, major seventh, and their compounds)
dimensionless, [0,1]
4
Beat entropy
Shannon entropy of the histogram of inter-onset intervals in the context, normalised by the log of the bin count; low values indicate regular rhythm
bits (normalised), [0,1]
5
Tempo variance
Variance of the local inter-onset-interval sequence, normalised by its mean (coefficient of variation)
dimensionless
6
Pitch mean
Mean MIDI note number over all notes sounding in the context
MIDI note number (0–127)
Table 3: The ten Branch B symbolic features. Each is computed from the transcribed MIDI within a 2 s context centred on grid frame t . “Dimensionless” indicates a ratio or normalised quantity on [0,1] unless stated otherwise.
Figure 3: Symbolic (Branch B) feature correlations over all DEAM frames: an activity cluster ( r≈0.6 – 0.9 ) and a near-orthogonal tonal/contour group.
Branch
Feature group
Dim
A
MERT-v1-95M layer-7 embedding (mean-pooled to 0.5 s)
768
B
Symbolic (note density, mode ratio, dissonance, beat entropy, tempo variance, pitch mean, contour slope, range, major weight, key clarity)
Table 5: Architecture and training hyperparameters (as run).
Figure 4: Joint-model training and validation loss; best validation at epoch 8, early stopping at 16.
Evaluation set
Arousal r / CCC
Arousal per-song r ( %>0.35 )
Valence r / CCC
Valence per-song r ( %>0.35 )
DEAM (1,764 songs)
0.801 / 0.799
0.362 (59.9%)
0.685 / 0.681
0.220 (45.6%)
PMEmo (523 songs)
0.742 / 0.725
0.291 (50.7%)
0.746 / 0.736
0.259 (50.7%)
Overall (2,287 songs)
0.790 / 0.789
0.345 (57.8%)
0.701 / 0.698
0.229 (46.7%)
Table 6: Joint-model results, Group 5-Fold CV, every song out-of-fold (shared MERT-768 + 10-symbolic space; 140,120 OOF frames).
Figure 5: Per-dimension out-of-fold accuracy of the joint DEAM+PMEmo model (pooled Pearson r ; 1,764 DEAM + 523 PMEmo songs, 5-fold), against the accuracy ceiling reported by the field’s meta-analysis [ 5 ] .
Figure 6: Predicted vs ground truth density over all 140,120 out-of-fold frames (hexbin): arousal tighter than valence.
Figure 7: Within-song arousal tracking on two rule-chosen out-of-fold DEAM songs (80th-percentile and median per-song r among songs with moving ground truth; predictions carry the tool’s 2.5 s smoothing).
Figure 8: Distribution of per-song dynamic r over all 2,287 out-of-fold songs, arousal and valence, with the 0.35 reference threshold.
Configuration
Arousal r / CCC
Arousal per-song r ( %>0.35 )
Valence r / CCC
Valence per-song r ( %>0.35 )
Full MMAS ( λ=0.5 )
0.819 / 0.818
0.363 (59.4%)
0.681 / 0.675
0.206 (44.4%)
− cross-attention (concat)
0.797 / 0.790
0.382 (60.3%)
0.672 / 0.652
0.214 (43.8%)
reverse attention ( Q = symbolic)
0.811 / 0.808
0.376 (59.9%)
0.680 / 0.673
0.218 (45.2%)
− symbolic branch (acoustic only)
0.767 / 0.760
0.371 (58.6%)
0.635 / 0.624
0.205 (43.2%)
symbolic only
0.367 / 0.323
0.123 (40.2%)
0.274 / 0.216
0.042 (31.7%)
− MERT-768 (legacy 32-d subsample)
0.743 / 0.736
0.344 (58.7%)
0.554 / 0.539
0.164 (40.9%)
Table 7: DEAM 5-fold CV ablation (1,764 songs, 112,220 frames, all out-of-fold). Full MMAS = windowed CrossModalTCN + both branches + shape loss at λ=0.5 .
Figure 9: DEAM 5-fold ablation (all out-of-fold): pooled and per-song r per configuration, arousal and valence.
Figure 10: The shape-loss trade-off (arousal): (a) pooled and per-song r against λ ; (b) the resulting frontier, with λ annotated at each point.
Dimension
n songs
Deep per-song r (SD)
Ridge per-song r (SD)
Δ ( 95% bootstrap CI)
Wilcoxon V
p
% songs improved
Cohen’s d
Arousal
353
0.344 (0.462)
0.125 (0.349)
+0.219 [0.178, 0.260]
12,974
1.7×10−21
74.5%
0.56
Valence
353
0.235 (0.433)
0.051 (0.322)
+0.183 [0.138, 0.228]
17,042
1.4×10−13
68.0%
0.42
Table 8: Per-song tracking, deep model vs Ridge (paired, held-out songs).
Figure 11: Within-song tracking, MMAS vs matched Ridge baseline: (a) mean per-song r for both systems; (b) the paired difference with its 95% song-level bootstrap confidence interval, Wilcoxon statistic, p -value, effect size, and percentage of songs improved (353 held-out songs).
Condition
Arousal r (pooled / per-song)
Valence r (pooled / per-song)
Intact
0.797 / 0.314
0.678 / 0.238
Noise σ=0.25
0.797 / 0.313
0.678 / 0.239
Noise σ=0.5
0.797 / 0.314
0.678 / 0.236
Noise σ=1.0
0.798 / 0.314
0.674 / 0.236
Mean-substitution
0.770 / 0.314
0.641 / 0.245
Raw-zero (hard failure)
0.776 / 0.308
0.640 / 0.236
Table 9: Branch B corruption at inference (released model, held-out test songs).
Dimension
q^ (half-width)
Interval width
Empirical frame coverage
Song-level coverage (mean)
Song-level coverage (10th pct.)
Arousal
0.219
0.437
93.7%
94.0%
78.3%
Valence
0.211
0.423
90.0%
91.6%
68.1%
Table 10: Split-conformal prediction intervals, α=0.05 (target coverage 95% ). Calibration on 13,832 validation frames; evaluation on 28,796 test frames from 456 unseen songs.
System
Arousal r
CCC
Dataset
Protocol
OpenSMILE baseline [ 1 ]
0.36 (per-song)
Not reported
DEAM
Official per-song
MCAN [ 23 ]
Not reported
0.50
DEAM subset
Random split
Meta-analysis best models [ 5 ]
0.81 (arousal) / 0.67 (valence)
Not reported
Across studies
Varies
Ridge baseline (ours)
0.829 (pooled)
0.823
DEAM
Group 5-Fold OOF
MMAS (ours, DEAM-only)
0.819 (pooled)
0.818
DEAM
Group 5-Fold OOF
MMAS (ours, joint DEAM+PMEmo)
0.801 (pooled)
0.799
DEAM
Joint Group 5-Fold OOF
Table 11: Comparison with published MER systems. Protocols differ; numbers are not directly comparable across rows.
Figure 12: The released model’s output on a PMEmo test song unseen by its weights, in the form the tool reports it: per-0.5 s Arousal and Valence with the split-conformal prediction interval of Section 5.9 , shown against the song’s annotation. The song is selected by rule (closest to the 60th percentile of per-song arousal r among test songs whose annotation moves), and the panel is regenerated from the committed prediction artifact rather than screen-captured.
Emotion recognition uses language, voice, face, body movement, and physiological signals to infer emotional categories or dimensions such as valence and arousal.
School of Computer Sciences, Universiti Sains Malaysia, 11800, Pulau Pinang, Malaysia · School of Mathematical Sciences, Universiti Sains Malaysia, 11800, Pulau Pinang, Malaysia