Nano-EmoX: Unifying Multimodal Emotional Intelligence from Perception to Empathy

Jiahao Huang; Fengyan Lin; Xuechao Yang; Chen Feng; Kexin Zhu; Xu Yang; Zhide Chen

cs.AIMar 2, 2026

Nano-EmoX: Unifying Multimodal Emotional Intelligence from Perception to Empathy

Authors: Jiahao Huang, Fengyan Lin, Xuechao Yang, Chen Feng, Kexin Zhu, Xu Yang, Zhide Chen

Organizations: Fujian Normal University · RMIT University · Independent Researcher · Minjiang University

Abstract

The development of affective multimodal language models (MLMs) has long been constrained by a gap between low-level perception and high-level interaction, leading to fragmented affective capabilities and limited generalization. To bridge this gap, we propose a cognitively inspired three-level hierarchy that organizes affective tasks according to their cognitive depth-perception, understanding, and interaction-and provides a unified conceptual foundation for advancing affective modeling. Guided by this hierarchy, we introduce Nano-EmoX, a small-scale multitask MLM, and P2E (Perception-to-Empathy), a curriculum-based training framework. Nano-EmoX integrates a suite of omni-modal encoders, including an enhanced facial encoder and a fusion encoder, to capture key multimodal affective cues and improve cross-task transferability. The outputs are projected into a unified language space via heterogeneous adapters, empowering a lightweight language model to tackle diverse affective tasks. Concurrently, P2E progressively cultivates emotional intelligence by aligning rapid perception with chain-of-thought-driven empathy. To the best of our knowledge, Nano-EmoX is the first compact MLM (2.2B) to unify six core affective tasks across all three hierarchy levels, achieving state-of-the-art or highly competitive performance across multiple benchmarks, demonstrating excellent efficiency and generalization. The code is available at https://github.com/waHAHJIAHAO/Nano-EmoX.

Figures & tables

Figure 1: This framework organizes tasks by increasing cognitive depth: (1) Perception for direct recognition of emotional cues; (2) Understanding for inferring emotional causality and context; and (3) Emotional Interaction for establishing an emotional connection with humans. Please refer to the appendix for details.

Models	Scale	Hierarchy	MSA	MER	OV-MER	ERI	MIR	ERG
EmoLLMs [ 39 ]	7B	level 1	✓	✓	×	×	×	×
Emotion-LLaMA [ 11 ]	7.8B	level 2	×	×	✓	✓	×	×
Omni-Emotion [ 62 ]	9B	level 2	×	×	✓	✓	×	×
LGSRR [ 77 ]	7.1B	level 2	×	×	×	×	✓	×
E3RG [ 36 ]	7B	level 3	×	×	×	×	×	✓
EmoVIT [ 60 ]	8.2B	level 1&2	×	✓	×	✓	×	×

Table 1: Comparison with representative LLM-based methods under different task settings, our method unifies six core affective tasks with a smaller parameter scale.

Figure 2: The architecture of the Nano - EmoX. The visual branch extracts general visual emotional cues, the facial branch is responsible for modeling fine - grained facial details, the speech branch captures acoustic emotional cues. To balance the contribution of each modality, the fusion branch integrates key emotional cues from the audio-visual modalities and extracts complementary information. The language model integrates multimodal information and performs multitask emotion recognition.

Figure 3: The fusion encoder extracts multi-layer features from the visual and speech encoders and feeds them to three fusion experts with independent weights. Each expert extracts complementary information

E_{mf}^{i}

. Then, the gating network dynamically weighs the contribution

G_{i}

of each expert and routes the feature

E_{mf}

of the output.

Figure 4: The P2E framework consists of a three-phase instruction fine-tuning process. Phase 1 focuses on the basic emotion recognition, to ensure a smooth learning curve, phase 2 multimodal fusion and contextual understanding by incorporating the MIR task. Finally, phase 3 revisits prior knowledge and integrates a diverse set of multilevel, complex tasks governed by a predefined data mixture ratio.

Models	Scale	MER2023 [ 32 ]	MER2024 [ 33 ]	MELD [ 18 ]	IEMOCAP [ 5 ]	MOSI [ 66 ]	MOSEI [ 3 ]	SIMS [ 65 ]	SIMSV2 [ 38 ]	OV-MERD [ 31 ]	Avg.
		Hit Rate $\uparrow$ (MER)				WAF $\uparrow$ (MSA)				WAF $\uparrow$ (OV-MER)	-
SALMONN [ 56 ]	$\uparrow$ 11.7B	55.53	45.38	45.62	46.84	81.00	67.03	68.69	68.69	45.00	57.89
MiniCPM-V-2.6-8B [ 22 ]	$\uparrow$ 5.8B	46.67	45.31	40.27	36.31	74.96	57.44	74.85	75.04	50.04	55.65
Qwen-2VL-7B [ 57 ]	$\uparrow$ 5.5B	59.81	69.14	48.05	50.53	74.10	58.35	78.65	77.43	55.61	63.52
Emotion-LLaMA [ 11 ]	$\uparrow$ 5.6B	59.38	73.62	46.76	55.47	66.13	67.66	78.32	77.23	52.97	64.17
AffectGPT [ 30 ]	$\uparrow$ 6.1B	78.54	78.80	55.65	60.54	81.30	80.90	88.49	86.18	62.52	74.77

Table 2: Performance comparison on the MSA, MER and fine-grained OV-MER tasks. AffectGPT (s) marked with a

\dagger

is trained solely on the MER-caption+ dataset, as proposed in the original work [ 30 ] , while the unmarked counterparts is trained using P2E. Nano - EmoX marked with

\ddagger

uses a joint training approach. The best performance is displayed in bold , the second-best performance is underlined .

Models	Scale	Clue Overlap $\uparrow$	Label Overlap $\uparrow$
MiniCPM-V-2.6-8B [ 22 ]	$\uparrow$ 5.8B	5.13	4.74
Qwen-2VL-7B [ 57 ]	$\uparrow$ 5.5B	6.32	5.65
Emotion-LLaMA [ 11 ]	$\uparrow$ 5.6B	7.83	6.25
AffectGPT [ 30 ]	$\uparrow$ 6.1B	5.70	5.49
Omni-Emotion [ 62 ]	$\uparrow$ 6.8B	8.22	6.78
Emotion-Qwen [ 23 ]	$\uparrow$ 5.3B	8.25	8.16

Table 3: Performance comparison of models on the ERI task.

Methods	MIntRec			MIntRec 2.0
	Acc $\uparrow$	WF1 $\uparrow$	WP $\uparrow$	Acc $\uparrow$	WF1 $\uparrow$	WP $\uparrow$
GPT-4 (3 shots) [ 44 ]	63.84	63.97	68.16	42.10	43.92	53.18
MiniCPM-V-2.6-8B [ 22 ]	80.67	80.56	82.19	53.58	51.91	61.66
Qwen-2VL-7B [ 57 ]	82.92	82.79	86.75	64.19	63.31	64.39
Qwen-VL-72B [ 57 ]	86.29	86.09	86.75	66.99	66.63	67.45
LLaVA-Video-72B [ 35 ]	80.22	79.94	80.63	64.98	64.72	65.40

Table 4: Evaluation results of the model on the MIR task.

Models	Acc / Hit Rate $\uparrow$	Dist-1 $\uparrow$	Dist-2 $\uparrow$
Empatheia [ 67 ]	48.51 / -	2.69	14.76
MiniCPM-o-2.6B [ 22 ]	- / 65.8	95.20	99.60
Qwen2.5-Omni-7B [ 61 ]	- / 72.3	98.60	99.70
Ola-Omni-7B [ 40 ]	- / 75.6	98.90	99.80
Small-scale Multimodal Models
MobileVLM V2-1.7B [ 12 ]	4.07 / 21.14	75.07	96.91

Table 5: Performance comparison of models with different scales on the ERG task. MiniCPM-o-2.6B [ 22 ] , Qwen2.5-Omni-7B [ 61 ] , and Ola-Omni-7B [ 40 ] enhance their capabilities by incorporating the E3RG [ 36 ] method.

Models	Strategy	Time	MSA&MER&OV-MER	ERI	ERG
Models	Strategy	Time	Avg. $\uparrow$	Avg. $\uparrow$	Hit Rate $\uparrow$
AffectGPT (s)	Jo-T	40 h	72.26	6.71	29.33
Our Nano - EmoX	Jo-T	40 h	73.28	6.54	74.79
AffectGPT (s)	P2E	32 h	72.08	6.65	90.88
Our Nano - EmoX	P2E	32 h	74.01	6.80	91.13

Table 6: Comparison of training time and performance improvements across different training methods, evaluated on the MER-UniBench [ 30 ] , EMER [ 34 ] , and AvaMERG [ 67 ] benchmarks.

Strategy	MSA & MER & OV-MER	ERI	ERG
Strategy	Avg. $\uparrow$	Avg. $\uparrow$	Hit Rate $\uparrow$
Standard P2E	74.01	6.80	91.13
Reverse P2E	63.35 (-10.66)	6.17 (-0.63)	57.64 (-33.49)

Table 7: The comparison result between the standard P2E and the reverse P2E (from empathy to perception).

Figure 5: Visualization results on the OV-MER, MIR, ERI, and ERG tasks. Our model responds rapidly when handling perceptual tasks, whereas for interactive tasks, it first considers the interlocutor’s state before generating empathetic responses.

Variants	Modal	MSA&MER&OV-MER	ERI	ERG
Variants	Modal	Avg. $\uparrow$	Avg. $\uparrow$	Hit Rate $\uparrow$
Audio Perception	AT	69.23	5.86	76.50
Visual Perception	VT	64.13	5.78	53.36
Face Perception	FT	65.99	5.63	46.54
Visual + Face	VFT	67.26	5.94	82.61
w/o Fusion	AVFT	71.65	6.05	62.20

Table 8: Ablation results showing the influence of different branch combinations in Nano - EmoX, ‘A’, ‘V’, ‘F’, ‘M’, and ‘T’ represent Audio, Vision, Face, Fusion, and Text modalities, respectively.

Figure 6: Ablation result: (a) Training the MSA task vs. Zero shot; (b) Feature layer-wise fusion vs. cross-layer fusion; (c) Gated dynamic weighting vs. average weighting.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 1: The facial Encoder extracts multiscale facial features and fuses them via an MLP to generate a rich facial embedding

E_{f}

. Subsequently, a temporal modeling block construct the sequence to output a final facial representation, which provides the language model with critical affective visual signals

E_{f}^{c}

. Fusion experts use audio features to guide vision and extract key complementary information

E_{mf}^{i}

Task	MER	OV-MER	ERI	MIR	ERG
Identifier	[Recognition]	[Recog_OV]	[Inference]	[Intent]	[Interaction]
Datasets	CAER [ 25 ] , CREMA-D [ 7 ]	MER-Caption+ [ 30 ]	MER-Caption+ [ 30 ]	MIntRec [ 71 ]	AvaMERG [ 67 ]
	M3ED [ 75 ] , FERV39K [ 58 ]		MER-Fine [ 11 ]	MIntRec2.0 [ 70 ]
Samples	141k	36k	40.5k	7.4k	57k

Appendix

Table 1: Details of task identifiers and training datasets for diverse emotional tasks.

Models	Empathy $\uparrow$	Insight $\uparrow$	Safety $\uparrow$	Avg.
Qwen2.5-Omni-7B	3.98	4.03	4.59	4.20
Ola-Omni-7B	4.18	4.29	4.67	4.38
Small-scale Multimodal Models
MobileVLM V2-1.7B	2.25	2.84	3.73	2.94
AffectGPT (s)	4.34	4.16	4.79	4.43
Our Nano - EmoX	4.75	4.42	4.87	4.68

Appendix

Table 2: Human experts blind evaluation on the ERG task.

Speech Extract Layers	Visual Extract Layers	Expert	MSA&MER&OV-MER	ERI	ERG
Speech Extract Layers	Visual Extract Layers	Expert	Avg.	Avg.	Hit Rate
8 / 18	8 / 16	2	71.98	6.02	88.26
16 / 18	12 / 16	2	72.42	6.08	88.89
8 / 18 / 22	8 / 16 / 22	3	73.17	6.40	89.55
16 / 18 / 22	12 / 16 / 22	3	74.01	6.80	91.13
8 /16 / 18 / 22	8 / 12 / 16 / 22	4	71.09	5.70	91.12

Appendix

Table 3: Exploring the appropriate number of experts and the depth of the extraction layer, extracting from too shallow a layer will lead to a decline in performance.

Visual Tokens	MSA& MER& OV-MER	MIR	ERI	ERG
Visual Tokens	Avg. $\uparrow$	Avg. $\uparrow$	Avg. $\uparrow$	Hit Rate $\uparrow$
32	74.01	52.72	6.80	91.13
64	73.96	55.48	6.83	91.08
128	74.28	60.53	6.95	92.87

Appendix

Table 4: The result of different visual token settings.

P2E Phase3 Task Ratio (MER: OV-MER: MIR: ERI: ERG)	MER-UniBench	MIntRec	MIntRec 2.0	EMER	AvaMERG
P2E Phase3 Task Ratio (MER: OV-MER: MIR: ERI: ERG)	Avg.	WAF	WAF	Avg.	Hit Rate
0% : 20% : 20% : 25% : 35%	71.43 (-2.58)	61.29 (+3.12)	49.8 (+2.53)	6.65 (-0.15)	43.15 (-44.03)
10% : 30% : 15% : 35% : 10%	72.79 (-1.22)	62.23 (+4.06)	51.04 (+3.77)	6.64 (-0.16)	58.88 (-28.3)
18% : 20% : 20% : 25% : 18%	72.60 (-1.41)	63.41 (+5.24)	49.09 (+1.82)	6.60 (-0.20)	91.30 (+0.17)
18% : 28% : 5% : 31% : 18%	74.01	58.17	47.27	6.80	91.13
25% : 17% : 10% : 22% : 25%	72.18 (-1.83)	42.19 (-15.98)	52.09 (+4.82)	6.83 (+0.03)	87.18 (-3.95)

Appendix

Table 5: Results of the ablation study on task composition in phase 3 of P2E. This table investigates the model’s sensitivity to different proportions of training tasks.

Figure 2: More visualization results in ERI and ERG task.

Explore similar work

AffectVerse: Emotional World Models for Multimodal Affective Computing
May 19, 2026Bo Zhao, Fanghua Ye, Yixin Ji +3Cross-Modal Learning Multimodal Large Language Models
1Great Bay University · 2Tencent · 3Tsinghua University +1
Decoding Affective Nuances: Enhancing MLLMs via Hierarchical Emotion Reasoning and Contrastive Discriminative Pruning
Sep 29, 2026Cheng Ye, Weidong Chen, Zhaobo Qi +2Multimodal Large Language Models Multimodal Emotion Recognition
University of Science and Technology of China, Hefei · Harbin Institute of Technology, Weihai
MVFA: A Multi-View Text-Guided Multimodal Fusion LLM Adapter for Sentiment Analysis and Emotion Recognition
Sep 5, 2026Pengfei Shao, Jisheng Dang, Jiawen Fang +7Multimodal Sentiment Analysis Multimodal Emotion Recognition
School of Information Science and Technology, Guangdong University of Foreign Studies, Guangdong, China · School of Information Science and Engineering, Lanzhou University, Lanzhou, China · School of Computer Science and Engineering, Sun Yat-sen University, Guangdong, China +2

May 19, 2026J/K move · Enter open · S save

AffectVerse: Emotional World Models for Multimodal Affective Computing

Bo Zhao, Fanghua Ye, Yixin Ji, Sicheng Zhao+2

1Great Bay University · 2Tencent · 3Tsinghua University+1

Cross-Modal Learning Multimodal Large Language Models Multimodal Emotion Recognition Multimodal Foundation Models Affective Computing Multimodal Understanding

Open paper AskPDF arXiv

Humans infer emotions by integrating observed multimodal cues with expectations about how affective states may unfold. Existing multimodal large language models (MLLMs), however, often treat emotion recognition as static fusion over complete audiovisual-text inputs, leaving affective dynamics implicit. We propose AffectVerse, a Qwen2.5-Omni-based model equipped with an Emotion World Module (EWM), an action-free representation-level module for short-horizon latent affective prediction. \rev{EWM contains three modules: 1) Cross-Modal Temporal Imagination predicts future video/audio representations from past tokens with multi-step rollout. 2) MAMA(Modality-Aware Multi-step Attention) Belief Aggregation compresses imagined tokens into modality-aware belief tokens. 3) Belief Injection inserts these belief tokens into the LLM for affective reasoning.} AffectVerse uses future prediction as a past-conditioned self-supervised signal: it does not replace modeling observed history or require unseen signals at inference, but forces the current belief state to encode transition cues that are predictive of subsequent affective change. Across nine benchmarks, AffectVerse improves at least 2.57% over other models, while controlled ablations show additive gains from temporal imagination, cross-modal rollout, and belief aggregation. These results suggest predictive belief-state modeling is a practical alternative for affective computing.

Abstract

Explore similar work

AffectVerse: Emotional World Models for Multimodal Affective Computing

Decoding Affective Nuances: Enhancing MLLMs via Hierarchical Emotion Reasoning and Contrastive Discriminative Pruning

MVFA: A Multi-View Text-Guided Multimodal Fusion LLM Adapter for Sentiment Analysis and Emotion Recognition