Conversational speech emotion recognition must reconcile acoustic evidence across temporal scales with two interaction processes: cross-speaker contextual influence and within-speaker emotion evolution. We propose DSSM-CRF, an audio-only architecture that explicitly separates these processes. Bidirectional state-space models encode fused self-supervised speech representations at frame and dialogue scales, so each utterance representation captures local prosody and context from all speakers. The decoder then orders each speaker's utterances into an independent dynamic conditional random field chain. Consecutive utterances in a speaker's chain form a transition pair whose score combines a corpus-level transition matrix with a residual predicted from the two contextualized utterances. An auxiliary objective supervises whether each pair changes emotion but does not participate in Viterbi inference. Thus, interlocutor turns affect contextual emotion scores without being treated as transitions in another speaker's emotion trajectory. DSSM-CRF achieves 75.81% UA and 74.90% WA on IEMOCAP, and 54.72% WA and 49.31% WF1 on MELD. Matched controls demonstrate complementary gains from speaker-wise factorization and CRF modeling.
Figures & tables
Figure 1: Overview of DSSM-CRF. Frame- and utterance-level representations are fused, augmented with dialogue cues, and contextualized by the dialogue BiSSM. Speaker-wise CRF decoding uses emotion emissions and pair-conditioned transitions along each speaker’s chain.
Model
UA(%)
WA(%)
ADAN [ 25 ]
64.51
66.92
Zou et al. [ 27 ]
71.05
69.80
DST [ 8 ]
73.60
71.80
DWFormer [ 6 ]
73.90
72.30
Mel-MViTv2 [ 18 ]
–
64.03
SCQT-MaxViT [ 19 ]
–
62.49
Table 1: Performance comparison on IEMOCAP. † denotes the second-best result.
Model
WA(%)
WF1(%)
DST [ 8 ]
–
48.80 †
DWFormer [ 6 ]
–
48.50
SpeechFormer++ [ 9 ]
51.00
47.00
Vesper [ 7 ]
53.50
48.00
DFSD-EMO [ 26 ]
53.74 †
47.60
DSSM-CRF (ours)
54.72
49.31
Table 2: Performance comparison on MELD. † denotes the second-best result.
(a) IEMOCAP
Configuration
UA(%)
WA(%)
S-UA(%)
I-UA(%)
Full DSSM-CRF
75.81
74.90
59.17
79.06
− frame-level SSM
75.67
74.37
57.53
79.07
− dialogue-level SSM
75.31
73.88
59.06
78.19
− transition residual
74.09
72.79
59.74
76.62
− shift supervision
75.44
74.45
58.19
78.58
Table 3: Component ablations and matched controls. S and I denote the Shift and Inertia subsets, respectively; IEMOCAP reports subset UA and MELD reports subset WA. Bold indicates the best result in each column.
Figure 2: Validation UA and WA across different shift-loss weights on IEMOCAP.
Backbone
Setting
UA(%)
WA(%)
wav2vec 2.0 Large
Adapted classifier
53.83
48.27
+ DSSM-CRF
69.05
66.88
HuBERT-Large
Adapted classifier
60.59
57.41
+ DSSM-CRF
71.49
70.51
WavLM-Large
Adapted classifier
62.27
60.78
+ DSSM-CRF
75.81
74.90
Table 4: Backbone robustness on IEMOCAP. Bold indicates the better result for each backbone.