Relational graph-driven differential denoising and diffusion attention fusion for multimodal conversation emotion recognition

Ying Liu; Yuntao Shou; Wei Ai; Tao Meng; Keqin Li

cs.CLMar 22, 2026

Relational graph-driven differential denoising and diffusion attention fusion for multimodal conversation emotion recognition

Authors: Ying Liu, Yuntao Shou, Wei Ai, Tao Meng, Keqin Li

Organizations: College of Computer and Mathematics, Central South University of Forestry and Technology, 410004, Hunan, Changsha China · Department of Computer Science, State University of New York, New Paltz, New York, 12561, USA

Abstract

In real-world scenarios, audio and video signals are often subject to environmental noise and limited acquisition conditions, resulting in extracted features containing excessive noise. Furthermore, there is an imbalance in data quality and information carrying capacity between different modalities. These two issues together lead to information distortion and weight bias during the fusion phase, impairing overall recognition performance. Most existing methods neglect the impact of noisy modalities and rely on implicit weighting to model modality importance, thereby failing to explicitly account for the predominant contribution of the textual modality in emotion understanding. To address these issues, we propose a relation-aware denoising and diffusion attention fusion model for MCER. Specifically, we first design a differential Transformer that explicitly computes the differences between two attention maps, thereby enhancing temporally consistent information while suppressing time-irrelevant noise, which leads to effective denoising in both audio and video modalities. Second, we construct modality-specific and cross-modality relation subgraphs to capture speaker-dependent emotional dependencies, enabling fine-grained modeling of intra- and inter-modal relationships. Finally, we introduce a text-guided cross-modal diffusion mechanism that leverages self-attention to model intra-modal dependencies and adaptively diffuses audiovisual information into the textual stream, ensuring more robust and semantically aligned multimodal fusion.

Figures & tables

Figure 1: Performance comparison of various models on the IEMOCAP dataset using unimodal features. T, V, and A denote the textual, visual, and acoustic modalities, respectively.

Figure 2: The overall framework is as follows: first, extract utterance-level unimodal features from audio, video, and text; then, apply a differential Transformer to audio and video features for dynamic extraction and denoising, while modeling emotional dependencies of textual features through a relational subgraph module comprising InterGAT and IntraGAT; finally, perform emotion classification by fusing multimodal features with text as the core using cross-modal diffusion attention.

Models	IEMOCAP												w-Acc	w-F1
	happy		sad		neutral		angry		excited		frustrated
	ACC	F1	ACC	F1	ACC	F1	ACC	F1	ACC	F1	ACC	F1
DialogueRNN	25.00	34.95	82.86	84.58	54.43	57.66	61.76	64.42	90.97	76.30	62.20	59.55	65.43	64.29
MMGCN	32.64	39.66	72.65	76.89	65.10	62.81	73.53	71.43	77.93	75.40	65.09	63.43	66.62	66.25
DialogueTRM	61.11	57.89	84.90	81.25	69.27	68.56	76.47	75.99	76.25	76.13	53.39	58.09	68.51	68.20
MM-DFN	44.44	44.44	77.55	80.00	71.35	66.99	75.88	70.88	74.25	76.42	58.27	61.67	67.84	67.85

Table 2: Performance on the IEMOCAP dataset. * indicates baseline models re-implemented using our extracted features. Bold values indicate the best performance.

Models	MELD														w-Acc	w-F1
	neutral		surprise		fear		sadness		joy		disgust		anger
	ACC	F1	ACC	F1	ACC	F1	ACC	F1	ACC	F1	ACC	F1	ACC	F1
DialogueRNN	82.17	76.56	46.62	47.64	0.00	0.00	21.15	24.65	49.50	51.49	0.00	0.00	48.41	46.01	60.27	57.95
MMGCN	84.32	76.96	47.33	49.63	2.00	3.64	14.90	20.39	56.97	53.76	1.47	2.82	42.61	45.23	61.34	58.41
DialogueTRM	83.20	79.41	56.94	55.27	12.00	17.39	27.88	36.48	60.45	60.30	16.18	20.18	51.01	49.79	65.10	63.80
MM-DFN	79.06	75.80	53.02	50.42	0.00	0.00	17.79	23.72	59.20	55.48	0.00	0.00	50.43	48.27	60.96	58.72

Table 3: Results on the MELD dataset.

Modality Setting	IEMOCAP		MELD
Modality Setting	w-Acc	w-F1	w-Acc	w-F1
V	40.11	31.25	23.16	31.27
A	54.75	51.17	36.15	39.01
T	69.86	69.34	65.05	65.11
V+T	69.52	68.92	65.81	65.93
A+T	74.05	72.98	65.67	65.8

Table 4: Performance under different modal settings across datasets.

Panel of Figure 3 — Figure 3: Effect of varying window sizes on model performance across two datasets.

Temporal Segmentation Setting	IEMOCAP		MELD
Temporal Segmentation Setting	w-Acc	w-F1	w-Acc	w-F1
No-overlap	72.86	72.21	66.22	66.35
50% overlap	72.92	72.28	66.41	66.55
Baseline	75.17	74.87	66.52	66.62

Table 5: Performance under different temporal segmentation settings at inference time.

Dataset	No-speaker	Noisy-speaker ( $p=0.3$ )	Ours
IEMOCAP	70.02	71.19	74.87
MELD	66.15	66.32	66.62

Table 6: Robustness of relation-subgraph construction under missing or noisy speaker information (w-F1).

Interaction	IEMOCAP		MELD
Strategy	w-Acc	w-F1	w-Acc	w-F1
w/o MSL	72.58	72.52	65.78	66.05
w MSL	75.17	74.87	66.52	66.62

Table 7: Performance of the model under different loss strategies.

Figure 4: The impact of different $\lambda_{m}$ values on the model performance. Note that a horizontal coordinate of 0 means that only fusion loss is used as the total loss.

Panel of Figure 5 — Figure 5: Performance comparison of differential strategies on the IEMOCAP and MELD datasets.

Panel of Figure 6 — Figure 6: Effect of relational subgraph modules on the overall model performance (I denotes InterGAT, A denotes IntraGAT).

Strategy		IEMOCAP		MELD
Strategy		w-Acc	w-F1	w-Acc	w-F1
SingleGraph		70.98	70.64	66.13	66.17
SubGraphs	PIS (Sum)	74.04	73.52	65.65	65.97
	PIC (Concat)	73.27	72.81	65.97	66.12
	Relncremental	74.35	73.77	65.87	66.19
	Incremental	75.17	74.87	66.52	66.62

Table 8: Performance under different interaction strategies evaluated across datasets.

Figure 7: The figure illustrates the performance comparison between our model and five baseline models on inter-speaker emotion dependency samples and intra-speaker emotion dependency samples in the IEMOCAP testing set.

Models	IEMOCAP		MELD
Models	w-Acc	w-F1	w-Acc	w-F1
MM-DFN	53.77	53.34	59.62	58.19
SDT	53.41	52.9	60.61	58.67
Ours	63.93	64.05	63.02	63.21

Table 9: Performance of the model on emotion shift samples.

Panel of Figure 8 — Figure 8: The figure shows the confusion matrices for our model evaluated on emotion shift samples from the IEMOCAP and MELD datasets, with rows representing the actual labels and columns representing the predicted labels.

Model	# Params.	IEMOCAP	MELD
Model	# Params.	Inference Time	Inference Time
MMGCN	5.63M	53.7s	75.3s
RGAT	15.28M	68.5s	146.3s
DER-GCN	78.59M	125.5s	189.7s
M3Net	7.88M	57.0s	87.6s
Ours	11.48M	61.2s	98.5s

Table 10: Model complexity and inference time comparison on IEMOCAP and MELD.

Setting	IEMOCAP		MELD
Setting	w-Acc	w-F1	w-Acc	w-F1
w/o Differential	73.84	73.19	65.93	66.05
w/o Gating	74.01	74.83	66.15	66.33
w/o Graph	69.30	69.00	65.56	65.81
w/o Diffusion	72.92	72.41	65.88	66.08
Full Model	75.17	74.87	66.52	66.62

Table 11: Results of component-wise ablation study.

$\gamma$	IEMOCAP		MELD
$\gamma$	w-Acc	w-F1	w-Acc	w-F1
0.0	73.47	72.88	65.88	66.08
0.2	74.45	74.06	66.35	66.20
0.4	74.63	74.25	66.27	66.39
0.6	74.79	74.44	66.49	66.60
0.8	74.64	74.35	66.42	66.51

Table 12: Sensitivity analysis of the diffusion fusion coefficient

\gamma

Method	IEMOCAP		MELD
Method	w-Acc	w-F1	w-Acc	w-F1
MA	73.44	73.15	64.28	64.25
EMA	74.20	73.86	65.87	65.78
Median	73.73	73.54	66.14	65.92
Ours	75.17	74.87	66.52	66.62

Table 13: Comparison with conventional denoising strategies.

Method	IEMOCAP		MELD
Method	w-Acc	w-F1	w-Acc	w-F1
Temporal diff attn	72.18	72.12	65.55	65.76
Contrastive attn	74.42	73.48	65.69	65.77
Delta-attention	74.18	73.41	65.63	65.68
Ours	75.17	74.87	66.52	66.62

Table 14: Comparison with representative difference-based attention variants under controlled settings.

Dataset	Text condition	Fixed	LSF	ADF
MELD	$\mathrm{T0}$	66.62	65.19	66.03
	$\mathrm{T0.6}$	46.28	43.51	43.96
	$\mathrm{T\emptyset}$	34.73	33.55	33.26
IEMOCAP	$\mathrm{T0}$	74.87	73.24	72.79
	$\mathrm{T0.6}$	68.61	67.39	67.57
	$\mathrm{T\emptyset}$	27.22	35.68	32.71

Table 15: w-F1 comparison of fixed-prior and learnable fusion strategies under different text conditions on MELD and IEMOCAP.

Reference shift $\Delta$	IEMOCAP		MELD
Reference shift $\Delta$	w-Acc (%)	w-F1 (%)	w-Acc (%)	w-F1 (%)
$\Delta=1$ (default)	75.17	74.87	66.52	66.62
$\Delta=2$	74.30	74.14	66.36	66.50
$\Delta=3$	74.62	74.35	66.46	66.57

Table 16: Ablation on the temporal reference shift

\Delta

in differential attention.

Figure 9: Illustration of the case dialogue window, including speaker turns, ground-truth emotion labels, and the target utterance position.

Figure 10: Heatmap of the acoustic relational-domain differential response ( $\alpha-\lambda\alpha^{ref}$ ). Rows and columns correspond to the query position $i$ and the attended position $j$ within the local window, respectively; color intensity indicates the strength of relational variation relative to the temporal reference view.

Panel of Figure 11 — Figure 11: Graph connectivity patterns on a two-speaker window from MELD. In each heatmap, row $i$ denotes the query utterance and column $j$ denotes the source utterance; darker colors indicate larger normalized weights.

Figure 12: Incremental trajectories of target-utterance predictions under varying diffusion intensities $\gamma$ (relative to $\gamma=0$ ). $\Delta P(\mathrm{GT})$ denotes the increment of the ground-truth probability; $\Delta$ probability margin denotes the increment of the probability difference between the ground-truth class and the strongest non-ground-truth class; $\Delta$ logit margin denotes the increment of the corresponding logit difference.

Panel of Figure 13 — Figure 13: Visualization of differential responses before and after gating at the target time step in the visual modality for an emotion-rich case (a) and a noisy-like case (b). In each panel, the four rows (top to bottom) show $\Delta_{\text{before}}$ , $\Delta_{\text{after}}$ , Suppress, and Retain. The dashed line marks the target key position, and the blue shaded region marks temporally neighboring key positions.

Explore similar work

To Fuse or to Drop? Dual-Path Learning for Resolving Modality Conflicts in Multimodal Emotion Recognition
May 6, 2026Yangchen Yu, Qian Chen, Jia Li +5Cross-Modal Learning Multimodal Robustness
School of Computer Science and Information Engineering, Hefei University of Technology, Hefei 230601, China · School of Computing and Information Systems, Singapore Management University, Singapore · College of Computing and Data Science, Nanyang Technological University, Singapore +1
AMB-DSGDN: Adaptive Modality-Balanced Dynamic Semantic Graph Differential Network for Multimodal Emotion Recognition
Mar 7, 2026Yunsheng Wang, Yuntao Shou, Yilong Tan +3Emotion Recognition in Conversations Multimodal Emotion Recognition
College of Computer and Mathematics, Central South University of Forestry and Technology · Department of Computer Science, State University of New York
Dynamic Fusion-Aware Graph Convolutional Neural Network for Multimodal Emotion Recognition in Conversations
Mar 22, 2026Tao Meng, Weilun Tang, Yuntao Shou +4Emotion Recognition in Conversations Multimodal Emotion Recognition
College of Computer and Mathematics, Central South University of Forestry and Technology, 410004, Hunan, Changsha, China · Department of Computer Science, State University of New York, New Paltz, New York, 12561, USA

May 6, 2026J/K move · Enter open · S save

To Fuse or to Drop? Dual-Path Learning for Resolving Modality Conflicts in Multimodal Emotion Recognition

Yangchen Yu, Qian Chen, Jia Li, Zhenzhen Hu+4

School of Computer Science and Information Engineering, Hefei University of Technology, Hefei 230601, China · School of Computing and Information Systems, Singapore Management University, Singapore · College of Computing and Data Science, Nanyang Technological University, Singapore+1

Cross-Modal Learning Multimodal Robustness Multimodal Emotion Recognition Multimodal Fusion Multimodal Understanding

Open paper AskPDF arXiv

Multimodal emotion recognition (MER) benefits from combining text, audio, and vision, yet standard fusion often fails when modalities conflict. Crucially, conflicts differ in resolvability: benign conflicts stem from missing, weak, or ambiguous cues and can be mitigated by cross-modal calibration, while severe conflicts arise from intrinsically contradictory (e.g., sarcasm) or misleading signals, for which forced fusion may amplify errors. Recognizing this, we propose Dual-Path Conflict Resolution (DCR), a unified framework that learns when to fuse and when to drop modalities. Path I (Affective Fusion Distiller, AFD) performs reverse distillation from audio/visual teachers to a textual student using temporally weighted class evidence, thereby enhancing representation-level calibration and improving fusion when alignment is beneficial. Path II (Affective Discernment Agent, ADA) formulates MER as a contextual bandit that selects among fusion and unimodal predictions based on a dual-view state and a calibration-aware reward, enabling decision-level arbitration under irreconcilable conflicts without requiring per-modality reliability labels. By taking into account the full multimodal context and coupling soft calibration with hard arbitration, DCR reconciles conflicts that can be aligned while bypassing misleading modalities when fusion is harmful. Across five benchmarks covering both dialogue-level and clip-level MER, DCR consistently outperforms competitive baselines or achieves highly competitive results. Further ablations, conflict-specific subset evaluation, and modality-selection analysis verify that AFD and ADA are complementary and jointly improve robust conflict-aware emotion recognition.

Abstract

Explore similar work

To Fuse or to Drop? Dual-Path Learning for Resolving Modality Conflicts in Multimodal Emotion Recognition

AMB-DSGDN: Adaptive Modality-Balanced Dynamic Semantic Graph Differential Network for Multimodal Emotion Recognition

Dynamic Fusion-Aware Graph Convolutional Neural Network for Multimodal Emotion Recognition in Conversations