In this paper, we propose a Role-aware Stimulus-guided (RASG) framework for interlocutor emotion recognition, which predicts listener emotions from listener-only videos and speaker-only audios. RASG consists of Role-aware Visual Transfer (RVT) and Stimulus-guided Boundary Reasoning (SBR) modules, which address supervision mismatch due to the lack of labeled listener data and ambiguity among visually similar listener reactions whose interpretation depends on speaker context, respectively. More specifically, RVT selects speaker samples whose facial expressions support their emotion labels. It then filters listener tracks and uses reliable pseudo-labels to train a listener-centric visual expert. SBR uses a two-class language reasoner only when the visual model is uncertain. It treats speaker audio and text as context rather than direct emotion evidence to distinguish similar listener reactions. Experiments conducted on MER-Cross dataset shows that RASG achieves 76.25% on MER-Cross and improves the performance of the baseline over 17%. Our team ranks second in Track 1 (MER-Cross) of the MER Grand Challenge at ACM MM 2026.
Figures & tables
Figure 1. Role-conditioned evidence mismatch. (a) The speaker’s angry label is supported mainly by audio and text despite weak facial affect. (b) Listener anger is directly visible; speaker audio and text provide only the eliciting context. Two filmstrips compare an angry speaker sample with an angry listener sample. The speaker's facial affect remains subtle even though the speech is angry, whereas the listener directly manifests anger through facial behavior.
Figure 2. Overview of RASG . (a) RVT : Visually Aligned Speaker Supervision, Listener-domain Bootstrapping, and Listener Visual Expert Training. (b) SBR : Low-margin Uncertainty Routing, Interaction Evidence Construction, and Top-2 Pair-constrained Reasoning. The upper panel shows visually aligned speaker supervision, listener-domain bootstrapping, and listener visual expert training. The lower panel routes low-margin predictions, constructs role-separated interaction evidence, and uses a DeepSeek reasoner to select within the visual Top-2 candidate pair.
Method
Modality
WAF (%)
HuBERT-base ( Hsu et al., 2021 )
A
35.74
MacBERT-large ( Cui et al., 2020 )
T
33.68
VideoMAE-large ( Tong et al., 2022 )
V
41.08
ResNet-FER2013 ( Lian et al., 2026 ; He et al., 2016 )
V
45.86
EmoNet ( Toisoul et al., 2021 )
V
47.03
SENet-FER2013 ( Lian et al., 2026 ; Hu et al., 2018 )
V
51.55
Table 1. Official MER-Cross baseline comparison ( Lian et al., 2026 ) in WAF (%). A: speaker audio; T: speaker text; V: listener video.
Team
WAF (%) ↑
Rank
HLT-HITSZ
78.31
1
WislabGDUT (Ours)
76.25
2
CASIA
74.07
3
Table 2. Top-three teams in Track 1 (MER-Cross) of the MER Grand Challenge at ACM MM 2026. Our team is in bold.
Method
OpenFace
CLIP
RVT
SBR
WAF (%)
OpenFace + CLIP
✓
✓
✗
✗
64.28
OpenFace + fine-tuned CLIP
✓
✓
✓
✗
73.42
Full RASG
✓
✓
✓
✓
76.25
Table 3. Ablation study of RASG on MER-Cross.
Stage
WAF (%)
Increment
Cum. Gain
Official CLIP-large
58.88
–
–
+ OpenFace behavior expert
64.28
+5.40
+5.40
+ Role-aware Visual Transfer
73.42
+9.14
+14.54
+ SBR
76.25
+2.83
+17.37
Table 4. Component gains over the CLIP-large baseline.