AffectAgent: Collaborative Multi-Agent Reasoning for Retrieval-Augmented Multimodal Emotion Recognition

Zeheng Wang; Zitong Yu; Yijie Zhu; Bo Zhao; Haochen Liang; Taorui Wang; Wei Xia; Jiayu Zhang

cs.CVApr 14, 2026

AffectAgent: Collaborative Multi-Agent Reasoning for Retrieval-Augmented Multimodal Emotion Recognition

Authors: Zeheng Wang, Zitong Yu, Yijie Zhu, Bo Zhao, Haochen Liang, Taorui Wang, Wei Xia, Jiayu Zhang, +4 more

Organizations: Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), Shenzhen, Guangdong, China · Great Bay University, Dongguan, Guangdong, China · Guangdong Provincial Key Laboratory of Intelligent Information Processing & Shenzhen Key Laboratory of Media Security, Shenzhen University, Shenzhen, Guangdong, China

Abstract

LLM-based multimodal emotion recognition relies on static parametric memory and often hallucinates when interpreting nuanced affective states. In this paper, given that single-round retrieval-augmented generation is highly susceptible to modal ambiguity and therefore struggles to capture complex affective dependencies across modalities, we introduce AffectAgent, an affect-oriented multi-agent retrieval-augmented generation framework that leverages collaborative decision-making among agents for fine-grained affective understanding. Specifically, AffectAgent comprises three jointly optimized specialized agents, namely a query planner, an evidence filter, and an emotion generator, which collaboratively perform analytical reasoning to retrieve cross-modal samples, assess evidence, and generate predictions. These agents are optimized end-to-end using Multi-Agent Proximal Policy Optimization (MAPPO) with a shared affective reward to ensure consistent emotion understanding. Furthermore, we introduce Modality-Balancing Mixture of Experts (MB-MoE) and Retrieval-Augmented Adaptive Fusion (RAAF), where MB-MoE dynamically regulates the contributions of different modalities to mitigate representation mismatch caused by cross-modal heterogeneity, while RAAF enhances semantic completion under missing-modality conditions by incorporating retrieved audiovisual embeddings. Extensive experiments on MER-UniBench demonstrate that AffectAgent achieves superior performance across complex scenarios. Our code will be released at: https://github.com/Wz1h1NG/AffectAgent.

Figures & tables

Figure 1. (a) Vanilla LLMs ( Lian et al., 2025 ; Cheng et al., 2024a ) rely on static memory, struggling with nuanced emotion understanding. (b) Traditional RAG-enhanced models ( Pipoli et al., 2025 ; Wen et al., 2025 ) follow a linear retrieve-then-generate pipeline, which is inadequate for complex emotional reasoning. (c) Our AffectAgent adopts a collaborative multi-agent architecture that orchestrates query planning, retrieval, filtering, and generation for emotion recognition.

Figure 2. Overview of AffectAgent. AffectAgent performs multimodal emotion recognition through collaborative multi-agent reasoning, including affect-oriented query planning, evidence adjudication, and emotion generation. Meanwhile, RAAF and MB-MoE improve cross-modal fusion by recovering missing information and balancing modality contributions. Block diagram of AffectAgent. Multimodal inputs are processed by query-planning, evidence-adjudication, and emotion-generation agents, while RAAF recovers missing information and MB-MoE balances the contributions of available modalities.

Model	Modality			Basic				Sentiment				Fine-grained	Mean
Model	A	V	T	MER2023	MER2024	MELD	IEMOCAP	MOSI	MOSEI	SIMS	SIMS v2	OV-MERD+	Mean
Video-LLaMA ( Zhang et al., 2023a )	$\checkmark$	$\checkmark$	$\checkmark$	14.52	13.56	17.65	15.68	54.55	46.52	58.56	56.32	13.98	32.37
+ AffectAgent				22.75 $\uparrow$ 8.23	22.71 $\uparrow$ 9.15	24.07 $\uparrow$ 6.42	23.49 $\uparrow$ 7.81	58.90 $\uparrow$ 4.35	52.14 $\uparrow$ 5.62	61.70 $\uparrow$ 3.14	60.40 $\uparrow$ 4.08	22.65 $\uparrow$ 8.67	38.75 $\uparrow$ 6.38
VideoChat ( Li et al., 2025a )	$\checkmark$	$\checkmark$	$\checkmark$	16.89	15.65	19.55	16.98	56.02	48.65	59.68	57.65	15.02	34.01
+ AffectAgent				24.30 $\uparrow$ 7.41	23.87 $\uparrow$ 8.22	25.38 $\uparrow$ 5.83	23.93 $\uparrow$ 6.95	61.14 $\uparrow$ 5.12	54.96 $\uparrow$ 6.31	63.93 $\uparrow$ 4.25	61.53 $\uparrow$ 3.88	22.36 $\uparrow$ 7.34	40.15 $\uparrow$ 6.14
ChatBridge ( Zhao et al., 2023 )	$\checkmark$	$\checkmark$	$\checkmark$	18.06	16.89	20.35	25.02	56.66	49.68	58.62	56.55	15.35	35.24

Table 1. Results on MER-UniBench . For each dataset, we report its primary evaluation metric. The gray-shaded columns denote the primary ranking metrics, the best results are highlighted in bold, and the second-best results are underlined. Our AffectAgent consistently outperforms different baseline models.

Model	Modality			Basic				Sentiment				Fine-grained	Mean
Model	A	V	T	MER2023	MER2024	MELD	IEMOCAP	MOSI	MOSEI	SIMS	SIMS v2	OV-MERD+	Mean
PandaGPT ( Su et al., 2023 )	$\checkmark$	$\checkmark$	$\times$	28.15	32.40	26.50	31.20	55.30	52.10	54.20	50.50	25.40	39.53
+ AffectAgent				32.27 $\uparrow$ 4.12	35.95 $\uparrow$ 3.55	31.71 $\uparrow$ 5.21	36.03 $\uparrow$ 4.83	57.45 $\uparrow$ 2.15	54.94 $\uparrow$ 2.84	56.12 $\uparrow$ 1.92	52.91 $\uparrow$ 2.41	29.76 $\uparrow$ 4.36	43.01 $\uparrow$ 3.48
Emotion-LLaMA ( Cheng et al., 2024a )	$\checkmark$	$\checkmark$	$\times$	45.20	56.80	35.40	42.10	56.50	58.20	65.30	64.10	40.50	51.57
+ AffectAgent				49.01 $\uparrow$ 3.81	58.94 $\uparrow$ 2.14	39.52 $\uparrow$ 4.12	45.66 $\uparrow$ 3.56	58.73 $\uparrow$ 2.23	60.65 $\uparrow$ 2.45	67.16 $\uparrow$ 1.86	66.02 $\uparrow$ 1.92	44.35 $\uparrow$ 3.85	54.44 $\uparrow$ 2.87
AffectGPT ( Lian et al., 2025 )	$\checkmark$	$\checkmark$	$\times$	65.20	66.80	48.50	52.40	72.10	70.30	78.40	76.50	52.60	64.76

Table 2. Robustness evaluation under missing-modality settings. A , V , and T denote audio, video, and text, respectively. Results show that AffectAgent consistently improves performance across different backbone models under various missing-modality conditions.

Method	Basic				Sentiment				Fine-grained	Mean
Method	MER2023	MER2024	MELD	IEMOCAP	MOSI	MOSEI	SIMS	SIMS v2	OV-MERD+	Mean
No RAG	78.54	78.80	55.65	60.54	81.30	80.90	88.49	86.18	62.52	74.77
Rewrite-Retrieve-Read ( Ma et al., 2023 )	78.39	78.76	55.58	60.42	81.36	80.83	88.52	86.13	62.47	74.72
BGM ( Ke et al., 2024 )	78.68	79.03	55.88	60.77	81.62	81.18	88.73	86.41	62.81	75.01
RAG-DDR ( Li et al., 2025b )	79.12	79.38	56.33	61.27	81.98	81.54	89.13	86.82	63.21	75.42
AffectAgent (Ours)	80.69 $\uparrow$ 2.15	80.66 $\uparrow$ 1.86	59.07 $\uparrow$ 3.42	63.39 $\uparrow$ 2.85	82.73 $\uparrow$ 1.43	82.67 $\uparrow$ 1.77	89.43 $\uparrow$ 0.94	87.33 $\uparrow$ 1.15	65.13 $\uparrow$ 2.61	76.78 $\uparrow$ 2.01

Table 3. Performance comparison of different RAG methods on emotion recognition datasets. To ensure a fair comparison, all RAG methods are applied to the same MLLM using full-modality inputs (audio, video, and text). Our proposed AffectAgent consistently outperforms existing RAG baselines across all datasets.

Figure 3. Illustration of three representative case studies, showing smooth success, query deviation with a correct final prediction, and query deviation with an incorrect final prediction. Three qualitative examples trace AffectAgent from its generated query through retrieved evidence and reasoning to the final emotion prediction. The examples show a smooth success, a deviated query with a correct prediction, and a deviated query with an incorrect prediction.

SFT	Retrieval	MER2023		MER2024
SFT	Retrieval	HIT $\uparrow$	$\Delta$ HIT	HIT $\uparrow$	$\Delta$ HIT
$\times$	$\times$	20.24	—	18.32	—
$\checkmark$	$\times$	24.12	$+3.88$	22.45	$+4.13$
$\checkmark$	Naive RAG	26.58	$+6.34$	24.71	$+6.39$
$\checkmark$	AffectAgent	28.80	+8.56	27.53	+9.21

Table 4. Stepwise effectiveness of our components. We utilize the generic MLLM Video-LLaMA2 as the base model.

\Delta

HIT indicates the absolute improvement over the zero-shot baseline.

Planner	Filter	Confuse Evid.	Counter Evid.	MER2023		MER2024
Planner	Filter	Confuse Evid.	Counter Evid.	HIT $\uparrow$	$\Delta$ HIT	HIT $\uparrow$	$\Delta$ HIT
$\times$	$\checkmark$	$\checkmark$	$\checkmark$	27.15	$-1.65$	25.86	$-1.67$
$\checkmark$	$\times$	$\checkmark$	$\checkmark$	27.94	$-0.86$	26.65	$-0.88$
$\times$	$\times$	$\checkmark$	$\checkmark$	26.58	$-2.22$	24.71	$-2.82$
$\checkmark$	$\checkmark$	$\times$	$\times$	27.42	$-1.38$	26.11	$-1.42$
$\checkmark$	$\checkmark$	$\checkmark$	$\times$	28.11	$-0.69$	26.75	$-0.78$

Table 5. Ablation on Multi-Agent coordination and Evidence Structures based on Video-LLaMA2. All variants retain primary evidence.

\Delta

HIT indicates the performance drop compared to the full framework.

Figure 4. Visualization of RAAF and MB-MoE in AffectAgent on sampled MER2023 data with AffectGPT as the backbone. (a) Performance gain (pp) under different retrieval conditions. (b) Average expert routing weights under different input states. Two plots summarize the fusion modules. The first compares performance gains under different retrieval conditions, and the second compares average MB-MoE expert-routing weights for different modality input states.

Explore similar work

EmoAgent-R1: Towards Multimodal Emotion Understanding with Reinforcement Learning-based Dynamic Agent Specialization
Jul 23, 2026Lihuang Fang, Yuchen Zou, kebing Jin +1RL for Language Models Agentic RL
Guangdong University of Technology Guangzhou, China · Southern University of Science and Technology Shenzhen, China · Xi’an Jiaotong University Xi’an, China +1
AffectVerse: Emotional World Models for Multimodal Affective Computing
May 19, 2026Bo Zhao, Fanghua Ye, Yixin Ji +3Cross-Modal Learning Multimodal Large Language Models
1Great Bay University · 2Tencent · 3Tsinghua University +1
MVFA: A Multi-View Text-Guided Multimodal Fusion LLM Adapter for Sentiment Analysis and Emotion Recognition
Sep 5, 2026Pengfei Shao, Jisheng Dang, Jiawen Fang +7Multimodal Sentiment Analysis Multimodal Emotion Recognition
School of Information Science and Technology, Guangdong University of Foreign Studies, Guangdong, China · School of Information Science and Engineering, Lanzhou University, Lanzhou, China · School of Computer Science and Engineering, Sun Yat-sen University, Guangdong, China +2

Jul 23, 2026J/K move · Enter open · S save

EmoAgent-R1: Towards Multimodal Emotion Understanding with Reinforcement Learning-based Dynamic Agent Specialization

Lihuang Fang, Yuchen Zou, kebing Jin, Jinghui Qin

Guangdong University of Technology Guangzhou, China · Southern University of Science and Technology Shenzhen, China · Xi’an Jiaotong University Xi’an, China+1

RL for Language Models Agentic RL Multimodal Large Language Models Multimodal Emotion Recognition

Open paper AskPDF arXiv

Multimodal large language models (MLLMs) have achieved impressive performance in multimodal emotion recognition (MER) tasks and lifted MER to a new level that is complex emotion understanding with advanced video understanding abilities and natural language description. However, existing MLLM-based methods often use a fixed prompt to perceive the emotions, ignoring the dynamicity and complexity of the emotion source in the multimodal inputs. To address these issues, we propose a novel Reinforcement Learning-based Dynamic Agent Specialization framework (EmoAgent-R1) to optimize the emotion recognition, reasoning, and generalization abilities of an MLLM with dynamic agent specialization based on reinforcement learning. Specifically, we first adopt a cold start strategy to endow an MLLM with preliminary emotion recognition, reasoning, and agent routing ability by training with synthetic answer-conditioned chain-of-thought data and agent routing data. Then, we further train the MLLM with reinforcement learning to perceive emotions in a two-step agentic workflow with agent selection and agent specialization. To effectively train EmoAgent-R1, we propose a novel Progressive Group-Relative Policy Optimization (P-GRPO) to combine group-based relative advantages with a PMI-inspired progressive token-level modulation to transform sparse rewards into fine-grained learning signals, mitigating the coarse-grained uniform credit assignment issue in GRPO. Extensive experiments on MER benchmarks demonstrate the superiority of our EmoAgent-R1 in stronger emotion reasoning performance and improved optimization stability.

Abstract

Explore similar work

EmoAgent-R1: Towards Multimodal Emotion Understanding with Reinforcement Learning-based Dynamic Agent Specialization

AffectVerse: Emotional World Models for Multimodal Affective Computing

MVFA: A Multi-View Text-Guided Multimodal Fusion LLM Adapter for Sentiment Analysis and Emotion Recognition