The Failure Is in the Readout: Fine-Grained Emotion Recognition Benchmarks Measure Elicitation, Not Perception

Tobias Hallmen; Fabian Deuser; Robin-Nico Kampa; Norbert Oswald; Elisabeth André

cs.CVOct 6, 2026

The Failure Is in the Readout: Fine-Grained Emotion Recognition Benchmarks Measure Elicitation, Not Perception

Authors: Tobias Hallmen, Fabian Deuser, Robin-Nico Kampa, Norbert Oswald, Elisabeth André

Organizations: Chair for Human-Centered Artificial Intelligence, University of Augsburg · Institute for Distributed Intelligent Systems, University of the Bundeswehr Munich

Abstract

Fine-grained emotion recognition supports therapy tools and social robots, but it needs facial data, which raises privacy and data-protection concerns. EmoNet-Face-HQ answers that with generated portraits, expert-rated over a $40$ -category taxonomy far finer than the usual six to eight basic emotions. Under the protocol it ships with, vision-language models (VLMs) score poorly on that taxonomy, and the benchmark concludes that a dedicated fine-tuned model is necessary: Empathic-Insight-Face (EIF; Small/Large). We show that off-the-shelf VLMs match or beat that fine-tuned model when the answer is not generated but read from the logits, as one binary query per category. We keep the benchmark's images, taxonomy and ratings, and change only how the answer is read. Experts agree at $κ_w = 0.468$ on the five categories they measure most reliably. Generatively, no interval among eleven open-weight VLMs lies entirely above that anchor ( $κ_w=0.268$ - $0.486$ ). Under verification all eleven clear it, each of them significantly better at $κ_w=0.507$ - $0.586$ . Three also significantly beat EIF sitting at $κ_w = 0.551$ (Small; $0.534$ Large). The gain comes from the graded probability and not from asking a yes/no question: as a control, thresholding those same probabilities to yes/no costs 142% of the average gains and drops binarization below generative elicitation to $κ_w=0.254$ - $0.423$ . A replication on real photographs (FACES) is weaker and mixed: of the ten models that pass a validity gate, six gain, three are neutral to positive and one is negative, so the effect is not confined to synthetic data.

Figures & tables

Model	Params	Prompt	Decoding
Gemma-3-4b-it [ 16 ]	$4.30$ B	ours	greedy
Gemma-3-12b-it [ 16 ]	$12.19$ B	theirs	greedy
Gemma-4-12B-it [ 17 ]	$11.96$ B	theirs	greedy
GLM-4.6V-Flash [ 18 ]	$10.29$ B	theirs	$T=0.8$
InternVL3_5-8B [ 52 , 9 ]	$8.53$ B	theirs	greedy
MiMo-VL-7B-RL [ 57 ]	$8.31$ B	ours	$T=0.3$

Table 1: The eleven evaluated VLMs, with technical reports. All are open-weight and run locally. Parameter counts are the released safetensors totals. Prompt and decoding name the run behind the generative score we report: theirs is the benchmark’s zero-shot prompt [ 25 ] and ours our own wording of the same request ( Appendix D ), reported as the better of the two per model, the choice favorable to generative elicitation. Every model was also run under the other prompt and under the other decoding policy, and neither choice carries the result ( Appendix E ). Verification samples nothing.

Figure 2 : How the answer is read, not which model is asked, decides whether these models reach human agreement. (a) Quadratic-weighted

\kappa

with

95\%

image-bootstrap intervals under generative elicitation (open circles, dashed) and verification (filled squares). No generative interval lies entirely above the

0.468

human anchor. All eleven verification intervals do. (b) The between-model sd falls from

0.073

0.024

while the median interval narrows from

0.043

0.034

, so the spread moves inside the measurement precision rather than the measurement becoming noisier.

Figure 3 : Discard the graded logit and the gain goes with it, even though the question is unchanged. The same stored verification answers scored continuously from

P(\mathrm{yes})

(filled squares) and re-scored by thresholding them at

0.5

(open squares), and each gray line follows one model. No new inference is run and no prompt is altered, yet every model loses, and the binarized arm lands back inside the range spanned by generative elicitation (shaded, the eleven models with both elicitations).

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 5 : On real photographs the effect survives as six results, not a sweep. Change in six-way accuracy from generative to verification elicitation, with paired person-bootstrap

95\%

intervals (

171

persons). Six models gain detectably (filled squares), three have intervals spanning zero (open circles), and Gemma-3-12b-it reverses (filled triangle). MiMo-VL-7B-RL (

\dagger

, grayed) is excluded by the validity gate of Sec. 3 ; we plot it because its delta is negative, so the exclusion favors our hypothesis. Alphabetical, not sorted by score.

Explore similar work

EmoS: A High-Fidelity Multimodal Benchmark for Fine-grained Streaming Emotional Understanding
May 9, 2026Pengze Guo, Jingxi Liang, Zhiwen Xie +2Emotion Recognition Empathy
NLP2CT Lab, Department of Computer and Information Science, University of Macau · School of Computer Science, Central China Normal University · University of Macau
Why Do Vision Language Models Struggle To Recognize Human Emotions?
Apr 16, 2026Madhav Agarwal, Sotirios A. Tsaftaris, Laura Sevilla-Lara +1Vision-Language Foundation Models Emotion
The University of Edinburgh, UK
InsightVQA: High-Dimensional Emotion-Cognitive Visual Question Answering Benchmark
Jun 1, 2026Shiyu Wang, Ziyu Liu, Chaoyi Yu +6Emotion Recognition Human-Annotated Benchmark
East China Normal University, Shanghai, China

May 9, 2026, cs.CLJ/K move · Enter open · S save

EmoS: A High-Fidelity Multimodal Benchmark for Fine-grained Streaming Emotional Understanding

Pengze Guo, Jingxi Liang, Zhiwen Xie, Qifeng Wang+1

NLP2CT Lab, Department of Computer and Information Science, University of Macau · School of Computer Science, Central China Normal University · University of Macau

Emotion Recognition Empathy Multimodal Benchmarks Multilingual Benchmark Human Annotations Streaming

Open paper AskPDF arXiv

In the context of today's high-pressure, aging society, the demand for large-scale emotional models capable of providing empathetic support is more critical than ever. However, existing benchmarks fail to simultaneously achieve ecological validity, signal clarity, and reliable fine-grained labeling. We introduce EmoS, a high-fidelity bilingual benchmark designed to resolve the limitations of ecological validity and noise in existing datasets by combining strictly filtered static slices with a dynamic Streaming Monologue subset. Supported by a rigorous dual-layer human annotation pipeline, EmoS provides trusted ground truth that captures continuous emotional evolution. Empirical results show that fine-tuning MLLMs (multimodal large language models) on EmoS yields significant gains over zero-shot baselines, laying the foundation for the training and evaluation of future emotion recognition models and empathy models. The dataset and code are publicly available at https://github.com/NLP2CT/EmoS.

Abstract

Explore similar work

EmoS: A High-Fidelity Multimodal Benchmark for Fine-grained Streaming Emotional Understanding

Why Do Vision Language Models Struggle To Recognize Human Emotions?

InsightVQA: High-Dimensional Emotion-Cognitive Visual Question Answering Benchmark