Emotion Entanglement and Bayesian Inference for Multi-Dimensional Emotion Understanding

Hemanth Kotaprolu; Kishan Maharaj; Raey Zhao; Abhijit Mishra; Pushpak Bhattacharyya

cs.CLApr 1, 2026

Emotion Entanglement and Bayesian Inference for Multi-Dimensional Emotion Understanding

Authors: Hemanth Kotaprolu, Kishan Maharaj, Raey Zhao, Abhijit Mishra, Pushpak Bhattacharyya

Organizations: Indian Institute of Technology Bombay, Mumbai, India · IBM Research · University of Texas at Austin, Texas, United States

Abstract

Understanding emotions in natural language is inherently a multi-dimensional reasoning problem, where multiple affective signals interact through context, interpersonal relations, and situational cues. However, most existing emotion understanding benchmarks rely on short texts and predefined emotion labels, reducing this process to independent label prediction and ignoring the structured dependencies among emotions. To address this limitation, we introduce Emotional Scenarios (EmoScene), a theory-grounded benchmark of 4,731 contextrich scenarios annotated with an 8-dimensional emotion vector derived from Plutchik's basic emotions. Motivated by the observation that emotions rarely occur independently, we further propose an entanglement-aware Bayesian inference framework that incorporates emotion co-occurrence statistics to perform joint posterior inference over the emotion vector. This lightweight post-processing does not require any parameter updates and improves the structural consistency of predictions, and yields overall gains of 2.24% Lexical Accuracy without any additional cost. EmoScene therefore provides a challenging benchmark for studying multi-dimensional emotion understanding and the limitations of current language models.

Figures & tables

Figure 1: An example scenario from the EmoScene Dataset. The ground truth labels Joy, Surprise, and Trust must be inferred from behavioral and situational cues. The unexpected discovery ("stumbled upon", "heart skipped a beat") signals Surprise, the soft chuckle and playful outreach signal Joy, and the willingness to reconnect with Jason after a long gap signals Trust. Attending only to surface cues ("heart skipped a beat", "clammy palms") alone risks predicting Fear; the surrounding behavioral and relational context is what disambiguates the true affect.

Figure 2: Overview of our steps to construct EmoScene dataset

	Lexical Accuracy		Vector Accuracy		Hamming Loss		F1-Macro
Model	Baseline	BI	Baseline	BI	Baseline	BI	Baseline	BI
Chain-of-Thought (CoT) Prompting
Llama-3.1-8B	83.87	83.88	24.54	24.50	16.13	16.12	47.2	47.2
Llama-3.2-3B	79.76	79.76	17.40	17.40	20.24	20.24	45.1	45.1
Llama-3.1-70B	81.93	83.27	22.15	23.95	18.07	16.73	50.6	51.7
Mistral-Nemo	83.48	83.50	22.24	22.26	16.52	16.50	43.1	43.2

Table 1: Model Performance Comparison on EmoScene. We compare Baseline outputs against our Entanglement-Aware Bayesian Inference (BI) post-processing step across both Chain-of-Thought and Zero-Shot settings.

Model	$\alpha$	Baseline F1	Linear F1	Full-prior F1
Llama-3.1-8B	0.10	47.15	47.15	47.16
Llama-3.2-3B	0.00	45.13	45.13	45.13
Llama-3.1-70B	0.50	50.60	51.50	51.75
Mistral-Nemo	0.25	43.10	43.15	43.17
Qwen2.5-7B	0.75	42.81	46.15	47.89
Gemma-2-2B	0.75	43.68	46.17	47.36

Table 2: Effect of ablating the pairwise term

\theta_{ij}

at each model’s F1-optimal

\alpha

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Emotion Label counts from the EmoScene Dataset.

Statistic	Value
Total number of scenarios	4,731
Avg. scenario length (words)	82
Median scenario length (words)	82
Max scenario length (words)	129
Min scenario length (words)	49
Avg. sentences per scenario	4.24

Appendix

Table 3: Statistics of the EmoScene dataset.

Cluster	Size	Top c-TF-IDF keywords
0 – Music / concerts	201	concert, song, band
1 – Celebrations	255	warmly, celebrate, genuine
2 – School / study	241	desk staring, cluttered desk, teacher
3 – Driving / transport	170	road, bus, steering wheel
4 – Social media	182	feed, clicked, tweet
5 – Sports / injury	141	gym, teammates, injury

Appendix

Table 4: Topic clusters (k=27, elbow-optimised) of EmoScene scenarios derived from KMeans on SBERT embeddings, labelled by c-TF-IDF top keywords. The high optimal k confirms fine-grained topical diversity across the corpus.

Figure 4: UMAP projection of SBERT embeddings of all 4,731 EmoScene scenarios, coloured by KMeans cluster ( $k=27$ , Elbow optimized). Clusters separate cleanly into distinct topical regions.

Figure 5: Learned pairwise couplings $\theta_{ij}$ on EmoScene. Red cells indicate positive couplings (the pair co-occurs more often than under independence) and blue cells indicate negative couplings (mutual exclusion). The eight cells outlined in black correspond to Plutchik’s primary dyads (adjacent on his wheel; e.g., Joy+Trust = “love”). Five of the eight primary dyads carry positive couplings, including the strongest overall (Disgust+Anger = “contempt”, $+0.91$ ), and the largest negative coupling (Joy+Sadness, $-2.64$ ) coincides with Plutchik’s primary bipolar axis. The remaining three primary dyads (“submission”, “disapproval”, “remorse”) appear with negative sign, reflecting corpus-specific deviations from Plutchik’s symmetric prediction.

Panel of Figure 6 — Figure 6: Heatmap comparison of Precision and Recall for Plutchik’s 8 basic emotions across all evaluated models. Darker regions indicate lower performance, while lighter/greener regions indicate higher alignment.

Figure 7: Macro F1 as a function of $\alpha$ for each open-source model under two prior configurations: the full entanglement-aware prior (solid) and the linear-only ablation with $\theta_{ij}=0$ (dashed). Black stars mark the F1-optimal $\alpha$ used in Table 1 ; the dotted horizontal line marks the baseline ( $\alpha=0$ ).

Pattern	Llama-3.1-8B	Llama-3.2-3B	Llama-3.1-70B	Gemma-2-9B	Gemma-2-2B	Mistral-Nemo	Qwen2.5-7B
P1: Trust under-prediction (FN)	1307	1133	694	1629	1270	2020	1699
P2: Anticipation over-prediction (FP)	677	904	842	738	1424	650	1190
P3: Anticipation + Trust co-occurrence miss	557	500	428	644	596	647	734
P4: Joy + Trust co-occurrence miss	392	269	263	529	415	640	586

Appendix

Table 5: Incidence of the four dominant error patterns across six baseline LLMs on EmoScene. Each pattern is defined by a deterministic Boolean rule on the prediction array; counts are over all 4,731 scenarios. P1 – P2 are the top-2 single-emotion error modes (ranked across 8 emotions

\times

{FN, FP}); P3 – P4 are the top-2 emotion-pair co-occurrence violations (ranked across the 28 emotion pairs).

Model	Q1 (lowest)	Q2	Q3	Q4 (highest)
Qwen2.5-7B	$+1.06$	$+0.60$	$+0.31$	$+0.03$
Gemma-2-2B	$+1.49$	$+0.64$	$+0.33$	$+0.05$
Gemma-2-9B	$+0.04$	$\sim 0$	$\sim 0$	$\sim 0$
Llama-3.1-8B	$\sim 0$	$\sim 0$	$\sim 0$	$\sim 0$

Appendix

Table 6: Mean per-scenario BI gain (

\Delta

correct-emotion-cells, out of 8) within each quartile of the baseline’s logit-derived confidence. For the two models where BI is most effective (top), the gain decreases monotonically with confidence. For uniformly high-confidence baselines (bottom), the gain is small in every quartile.

Model	Self-reported ECE	Logit-derived ECE
Llama-3.1-8B	0.400	0.149
Llama-3.2-3B	0.158	0.175
Llama-3.1-70B	0.075	0.145
Mistral-Nemo	0.090	0.164
Qwen2.5-7B	0.370	0.163
Gemma-2-2B	0.118	0.232

Appendix

Table 7: Expected Calibration Error for each model under the CoT prompt.

Panel of Figure 8 — Figure 8: Scenario generation prompt used to instruct the language model.

Panel of Figure 9 — Figure 9: EmoScene Evaluation Prompt.

Panel of Figure 10 — Figure 10: Zero-Shot Evaluation Prompt.

Emotion Recognition

Emotion recognition uses language, voice, face, body movement, and physiological signals to infer emotional categories or dimensions such as valence and arousal.

See all 129 papers

E $^3$ mo-Bench: A Scalable Benchmark for Multimodal Evoked and Expressed Emotion Understanding via Bayesian Pairwise Alignment
Aug 11, 2026Lancheng Gao, Ziheng Jia, Shengyan Li +4Multimodal Large Language Models Multimodal Emotion Recognition
Shanghai Jiao Tong University
From Surface to Depth: Towards Cognitive Appraisal Reasoning in Multimodal Emotion Understanding
Oct 8, 2026Jia Li, Yichao He, Yangchen Yu +6Multimodal Emotion Recognition Emotion Recognition
Hefei University of Technology · Nanyang Technological University
EmoTrans: A Benchmark for Understanding, Reasoning, and Predicting Emotion Transitions in Multimodal LLMs
Apr 25, 2026He Hu, Tengjin Weng, Zebang Cheng +5Multimodal Large Language Models Multimodal Model Evaluation
Shenzhen University Shenzhen, China · Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) Shenzhen, China · Queen Mary University of London +3

Aug 11, 2026J/K move · Enter open · S save

E $^3$ mo-Bench: A Scalable Benchmark for Multimodal Evoked and Expressed Emotion Understanding via Bayesian Pairwise Alignment

Lancheng Gao, Ziheng Jia, Shengyan Li, Zixuan Xing+3

Shanghai Jiao Tong University

Multimodal Large Language Models Multimodal Emotion Recognition Multimodal Model Evaluation Emotion Recognition

Open paper AskPDF arXiv

Understanding both expressed and evoked emotions is critical for multimodal large language models (MLLMs) to achieve comprehensive affect-aware interactions. However, existing benchmarks typically examine expressed and evoked emotions in isolation or are constrained to coarse-grained and incomplete affective characterizations. To bridge this gap, we introduce E

^3

mo-Bench, a scalable benchmark comprising

12{,}314

question-answer pairs across

2{,}524

videos with predefined affective perspectives. It evaluates evoked and expressed emotion understanding via

3

complementary tasks: emotion perception, open-vocabulary recognition, and valence-arousal-dominance (VAD) assessment. To efficiently scale reliable continuous annotations, we propose Bayesian Pairwise Alignment, which aggregates sparse, low-burden pairwise judgments into anchor-referenced VAD estimates. Furthermore, we develop E

^3

mo-Score, a training-free agent that aggregates complementary judgments from a five-model committee to improve VAD estimation. Extensive experiments validate the effectiveness of our framework and expose a pronounced performance skew between evoked and expressed emotion paradigms. These findings, coupled with MLLMs' persistent deficits in fine-grained recognition and dimensional assessment, chart a clear course for advancing multimodal emotional intelligence.

Abstract

Emotion Recognition

Explore similar work

E3^33mo-Bench: A Scalable Benchmark for Multimodal Evoked and Expressed Emotion Understanding via Bayesian Pairwise Alignment

From Surface to Depth: Towards Cognitive Appraisal Reasoning in Multimodal Emotion Understanding

EmoTrans: A Benchmark for Understanding, Reasoning, and Predicting Emotion Transitions in Multimodal LLMs

E $^3$ mo-Bench: A Scalable Benchmark for Multimodal Evoked and Expressed Emotion Understanding via Bayesian Pairwise Alignment