Bias-Corrected Ceilings of Emotion Predictability from Human Label Variation Based on Instance-Level Fano Bounds

Keito Inoshita

cs.AIAug 16, 2026

Bias-Corrected Ceilings of Emotion Predictability from Human Label Variation Based on Instance-Level Fano Bounds

Authors: Keito Inoshita

Organizations: Faculty of Business and Commerce, Kansai University

Abstract

Emotion recognition from text keeps improving on benchmarks, yet whether an accuracy ceiling has been reached is seldom asked with discipline. Our aim is not to pin this ceiling to a single number, but to quantify how far it depends on finite annotation, estimator choice, annotation noise, and the evaluation protocol, and thereby to discipline how confidently saturation can be claimed. We propose Bias-corrected Affective Ceiling Estimation (BACE), an analysis framework that estimates a bias-corrected ceiling, separates irreducible from reducible error, and disciplines the resulting claims. An anchored Dirichlet-mixture empirical Bayes estimator, bracketed between plug-in and NSB, recovers the human-consensus distribution; an annotator split, a noise deconvolution, and a fixed claim gate then attribute error without circularity. Methodologically, unconstrained point estimates place reachability anywhere from 0.38 to 1.03, so saturation cannot be decided by any single estimator. Substantively, the only assertion passing the claim gate is that at least about 33% of a representative classifier's error on GoEmotions is irreducible, with the same pattern recurring on offensiveness and irony.

Figures & tables

Figure 1: Overview of BACE: three core layers (Estimate, Separate, Discipline) followed by a cross-task application.

Gran.	plug-in	NSB	dec. $\times$ plug	$\mathrm{Err}_{\mathrm{SOTA}}$	ceiling	gap	$R$ cons. [CI]
L2 ( $K{=}7$ )	0.258	0.391	0.197	0.388	0.150	0.238	0.386 [.368,.403]
L1 ( $K{=}4$ )	0.227	0.328	0.179	0.350	0.134	0.216	0.383 [.366,.402]

Table 1: Predictability ceilings and aleatoric/epistemic decomposition for GoEmotions (P1 exact Bayes error).

Figure 2: The reachability

R

depends on the combination of estimator, noise handling, and finite-sample correction, and the direction of its change is not constant.

Classifier	$\mathrm{Err}$	$R_{\mathrm{cons}}$ [CI]	$R_{\mathrm{EB}}$ [CI]
bhadresh-BERT	0.382	0.392 [.375,.411]	1.009 [.977,1.043]
ModernBERT-large	0.386	0.388 [.371,.405]	0.997 [.967,1.028]
SamLowe-RoBERTa	0.388	0.386 [.368,.403]	0.992 [.964,1.023]
monologg-BERT	0.419	0.357 [.342,.372]	0.918 [.891,.946]
GPT-4o-mini (fs)	0.526	0.285 [.273,.297]	0.732 [.714,.750]

Table 2: Multi-model decomposition against the same GoEmotions L2 ceiling (P1).

Figure 3: Cross-task predictability map controlling for base rate.

Task	$\mathrm{Err}_{\mathrm{SOTA}}$	$C_{\mathrm{cons}}$	$R_{\mathrm{cons}}$ [CI]	$R_{\mathrm{EB}}$	$R_{\mathrm{NSB}}$
Offensiveness (MD-Ag.)	0.332	0.137	0.413 [.404,.423]	0.640	0.618
Irony (EPIC)	0.388	0.139	0.359 [.342,.375]	0.672	0.603
Emotion (BRIGHTER)	0.275	0.116	0.422 [.416,.429]	0.482	0.543

Table 3: Cross-task SOTA decomposition against the conservative ceiling.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Inst.	Ann.	$n_{i}$ (mean/med.)	Labels
GoEmotions	57,828	82	3.41 / 3	multi-label 27+neutral
MD-Agreement	10,753	819	5 / 5	binary offensiveness
EPIC	3,000	74	4.72 / 5	binary irony
BRIGHTER eng	5,655	no ID	8.52 / 8	6 emotions, ordinal

Appendix

Table 4: Dataset statistics.

Block	Estimator	Value
Binary floor	Ishida direct ( Ishida et al., 2023 )	0.0274
Binary floor	Ushio soft-label ( Ushio et al., 2026 )	0.0274
Binary floor	BACE Beta-Binomial EB (point)	0.0369
Binary floor	BACE NSB $K{=}2$ (upper)	0.1120
L2 $H$ / ceil.	plug-in (informal)	0.713 / 0.259
L2 $H$ / ceil.	Chao–Shen ( Chao and Shen, 2003 )	1.062 / 0.205

Appendix

Table 5: Comparison with existing Bayes-error and entropy estimators.

Figure 4: Multi-model reachability at L2.

Figure 5: Ceiling on the irreducible error of GoEmotions under the plug-in, mixture empirical Bayes, and NSB estimators.

Estimator	$n$	Coverage	CI width	Bias
plug-in	3	0.00	0.028	$-0.072$
mixture EB	3	0.75	0.020	$+0.001$
NSB	3	0.00	0.023	$+0.064$
plug-in	5	0.00	0.027	$-0.044$
mixture EB	5	0.89	0.024	$+0.003$
NSB	5	0.00	0.027	$+0.040$

Appendix

Table 6: Empirical coverage of the

95

% bootstrap confidence intervals. Ground truth

C_{\mathrm{true}}=0.261

M_{\mathrm{sim}}=100

synthetic datasets,

B=1000

. Bold marks the best coverage at each

n

Figure 6: Per-label headroom.

Label	$e^{*}_{k}$	Label	$e^{*}_{k}$	Label	$e^{*}_{k}$	Label	$e^{*}_{k}$
grief	0.0033	desire	0.0175	sadness	0.0296	curiosity	0.0407
pride	0.0063	gratitude	0.0220	amusement	0.0326	realization	0.0415
relief	0.0063	surprise	0.0242	confusion	0.0336	disapproval	0.0523
nervousness	0.0087	disgust	0.0243	anger	0.0358	admiration	0.0620
remorse	0.0106	love	0.0258	joy	0.0359	annoyance	0.0622
embarrassment	0.0114	excitement	0.0264	optimism	0.0393	approval	0.0827

Appendix

Table 7: Irreducible-error floors

e^{*}_{k}

for all

28

labels (ascending in the floor).

Emotion Recognition

Emotion recognition uses language, voice, face, body movement, and physiological signals to infer emotional categories or dimensions such as valence and arousal.

See all 129 papers

Chiaroscuro for Emotions: A Contrastive Emotion Benchmark Grounded in Appraisal Theory
Sep 3, 2026Divyesh Bommana, Mohammad Saim, Tianyu JiangLLM Evaluation Classification
University of Cincinnati
VocalAffectBench: Evaluating Vocal Emotion Recognition in AI Audio Models
Aug 28, 2026Luc Debaupte, Tyler Baumgartner, Brandon Tai +3Audio-Language Model Evaluation Speech Emotion Recognition
Besimple AI, San Mateo, CA
Quality and Agreement in Multilabel Emotion Annotation: A Case Study and Evaluation Framework
Jun 19, 2026Emily Öhman, Anna KoufakouSoft-Label Learning Annotator Disagreement
Waseda University, Tokyo, Japan · Florida Gulf Coast University, Fort Myers FL, USA

Sep 3, 2026J/K move · Enter open · S save

Chiaroscuro for Emotions: A Contrastive Emotion Benchmark Grounded in Appraisal Theory

Divyesh Bommana, Mohammad Saim, Tianyu Jiang

University of Cincinnati

LLM Evaluation Classification Emotion Recognition

Open paper AskPDF arXiv

Emotion recognition benchmarks often predict one emotion per text, missing many real-world scenarios where two people arrive at opposing emotions from a single shared event. For example, a child kicks the seat in front of her in excitement while the passenger ahead grows angry. We introduce CHIARO, a 1,000 human-annotated sentence benchmark for contrastive emotion inference grounded in appraisal theory. Each scene describes one causal trigger eliciting a positive emotion in one person and a negative emotion in the other, drawn from a ten-class taxonomy. We benchmark seven frontier LLMs and four off-the-shelf emotion classifiers. The strongest LLM reaches 67.3 macro-F1, well below human agreement, while existing emotion classifiers score near chance. Beyond evaluation, CHIARO also serves as a training signal. When combined with an existing emotion corpus, the resulting downstream classifier improves on CHIARO itself and on six of ten external emotion benchmarks, which positions our dataset as a complementary signal for emotion recognition.

Abstract

Emotion Recognition

Explore similar work

Chiaroscuro for Emotions: A Contrastive Emotion Benchmark Grounded in Appraisal Theory

VocalAffectBench: Evaluating Vocal Emotion Recognition in AI Audio Models

Quality and Agreement in Multilabel Emotion Annotation: A Case Study and Evaluation Framework