Fear Is Not Always Fear: Variation in Emotion Label Equivalence Across Datasets

Emily Öhman; Anna Koufakou

cs.CLOct 9, 2026

Fear Is Not Always Fear: Variation in Emotion Label Equivalence Across Datasets

Authors: Emily Öhman, Anna Koufakou

Organizations: Waseda University, Japan · Florida Gulf Coast University, USA

Abstract

Emotion classifiers are routinely applied beyond their training data, on the assumption that same-named labels such as fear or surprise mean the same thing across resources. We test this assumption for five labels shared by GoEmotions, XED, and BRIGHTER, comparing models on identical inputs and under cross-corpus transfer. On the same texts, one fear classifier flags 71% as fearful and another 8%, yet the two largely agree on how the texts rank. Across 30 resource-label comparisons, training-label prevalence explains about half of this kind of scale mismatch, and a fitted intercept removes most of it. What recalibration cannot repair is larger: transferred models lose 55% of native discrimination in aggregate, agreement in ranking between classifiers is only weakly related to that loss, and for surprise the resources disagree about which texts express the emotion. The choice of training resource can therefore change conclusions: on two external tweet collections, BRIGHTER- and XED-trained classifiers give opposite answers about which shows more surprise, even after rank normalization, with intervals excluding zero in both directions. A shared emotion label name does not establish an interchangeable measurement target.

Figures & tables

Resource	Items	Source data	Label scheme
GoEmotions ( Demszky et al., 2020 )	54,263	Reddit comments	27 emotions + neutral, multi-label
XED (English) ( Öhman et al., 2020 )	27,202	Movie subtitles	8 Plutchik emotions + neutral
BRIGHTER (English) ( Muhammad et al., 2025 )	5,643	Social media	multi-label; intensity recoverable
SemEval-2018 E-c ( Mohammad et al., 2018 )	10,981	English tweets	11 emotions; external evaluation only

Table 1: Training resources and external evaluation data. The cross-resource analyses focus on the five labels shared by all three: anger , fear , joy , sadness , surprise .

Panel of Figure 1 — Figure 1: Compatibility across the 30 directed resource–label comparisons. Panel (a) shows held-out same-input Spearman rank agreement. Panel (b) shows residual MAE after monotonic mapping, normalized by the larger of the source- and target-head same-head cross-seed MAEs. Values above 1 indicate residual disagreement larger than this conservative seed-stability reference.

Mapping	Mean MAE	Median MAE
Raw	.193	.184
Prevalence intercept	.096	.070
L1: fitted intercept	.076	.058
L2: affine	.072	.048
L3: monotonic	.069	.046

Table 2: Held-out same-input mapping results across 30 directed resource–label comparisons. The prevalence-derived intercept uses only source and target training-label prevalences; L1–L3 are fitted on target-domain development predictions without gold emotion labels.

Direction	AUROC	AUROC gap	AP	AP gap	F1 source thr.	F1 target-dev thr.	Native F1	F1 gap
BRIGHTER $\rightarrow$ GoEmotions	.855 (.0019)	.095 (.0040)	.170 (.0092)	.425 (.0130)	.115 (.0044)	.243 (.0101)	.569 (.0105)	.325 (.0202)
BRIGHTER $\rightarrow$ XED	.786 (.0017)	.126 (.0020)	.338 (.0028)	.244 (.0050)	.319 (.0076)	.391 (.0092)	.555 (.0008)	.164 (.0098)
GoEmotions $\rightarrow$ BRIGHTER	.773 (.0118)	.115 (.0099)	.632 (.0104)	.160 (.0099)	.253 (.0020)	.582 (.0188)	.702 (.0141)	.120 (.0259)
GoEmotions $\rightarrow$ XED	.753 (.0075)	.159 (.0073)	.337 (.0035)	.245 (.0088)	.210 (.0032)	.363 (.0060)	.555 (.0008)	.192 (.0067)
XED $\rightarrow$ BRIGHTER	.753 (.0098)	.135 (.0116)	.501 (.0126)	.290 (.0126)	.023 (.0023)	.562 (.0095)	.702 (.0141)	.140 (.0108)
XED $\rightarrow$ GoEmotions	.710 (.0034)	.240 (.0032)	.111 (.0100)	.484 (.0175)	.171 (.0128)	.169 (.0162)	.569 (.0105)	.399 (.0240)

Table 3: Cross-corpus transfer averaged over five labels and three seeds. Parentheses give sample SD across seed-level five-label means. AP is average precision. Target-development and native F1 use target-development threshold selection; source-threshold F1 retains the source operating point. Gaps are native minus transferred performance.

Correction	Mean excess Brier (SD)
None (raw)	.112 (.0020)
Prevalence-derived shift	.055 (.0004)
Label-free prior-shift EM	.122 (.0077)
Source-calibrated prior-shift EM	.125 (.0036)
Intercept-only	.031 (.0002)
Platt	.029 (.0003)

Table 4: Mean excess Brier over 30 directed label-transfer units; parentheses give the SD of these means across three training seeds. The prevalence-derived shift uses training-label prevalences. The native target model receives the corresponding correction.

Label	Same- input $\rho$	Raw	Intercept	Platt	Isotonic	F1 gap
		Excess Brier
fear	.799	.404 (.0012)	.094 (.0009)	.098 (.0017)	.039 (.0136)	.033 (.0617)
joy	.445	.143 (.0012)	.080 (.0026)	.080 (.0027)	.047 (.0103)	.122 (.0528)
surprise	.329	.166 (.0017)	.097 (.0025)	.089 (.0015)	.073 (.0019)	.206 (.0234)

Table 5: XED

\rightarrow

BRIGHTER: shared-head same-input rank agreement and excess Brier relative to the equally recalibrated native BRIGHTER model. The F1 gap uses exact target-development threshold selection. Parentheses give sample SD across three seeds for excess Brier and F1 gap.

Label	BRIGHTER	GoEmotions	XED
anger	+0.011	+0.012	+0.001
fear	+0.038	+0.009	+0.006
joy	-0.036	-0.046	-0.050
sadness	+0.005	-0.000	-0.008
surprise	+0.024	-0.003	-0.015

Table 6: External collection differences in mean pooled percentile rank (SemEval-2018 E-c; 2016 minus 2017), using seed-averaged predictions. All five shared labels and all three heads are shown. Full intervals and seed ranges appear in Appendix G .

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Training condition	BRIGHTER	GoEmotions	XED
Native specialist	.713	.468	.598
Naive union (shared label heads)	.672	.418	.591
Shared encoder + dataset-specific heads	.715	.432	.595

Appendix

Table 7: Macro-F1 over each dataset’s native label inventory, averaged over three seeds; each dataset is evaluated with its own development-set thresholds.

Target	$n=25$	$n=50$	$n=100$	Full
BRIGHTER	.063/.057	.063/.059	.062/.060	.062/.060
GoEmotions	.059/.057	.039/.034	.019/.016	.011/.008
XED	.025/.021	.022/.019	.021/.019	.021/.019

Appendix

Table 8: Mean excess Brier under random few-shot recalibration. Entries are intercept/Platt.

Source	Raw	Platt	Isotonic
GoEmotions $\rightarrow$ BRIGHTER	$+0.0346$ [+0.0138, +0.0547]	$+0.0039$ [-0.0067, +0.0144]	$+0.0001$ [-0.0116, +0.0112]
XED $\rightarrow$ BRIGHTER	$+0.0453$ [+0.0225, +0.0683]	$-0.0034$ [-0.0152, +0.0078]	$-0.0037$ [-0.0155, +0.0080]

Appendix

Table 9: Difference in item-level excess squared error between raw-only-diametric and other BRIGHTER test items (raw-only minus other), averaged over the five shared labels. Brackets give 95% bootstrap confidence intervals. Positive values indicate greater transfer error on the raw-only subset.

Control	Method	GoEmotions $\rightarrow$ BRIGHTER	XED $\rightarrow$ BRIGHTER	Pooled
Label density / target presence	Raw	$-0.0022$ [-0.0136, +0.0091]	$+0.0040$ [-0.0076, +0.0156]	$+0.0009$ [-0.0098, +0.0115]
	Platt	$-0.0021$ [-0.0122, +0.0080]	$-0.0112$ [-0.0221, -0.0003]	$-0.0066$ [-0.0161, +0.0028]
	Isotonic	$-0.0051$ [-0.0161, +0.0060]	$-0.0087$ [-0.0202, +0.0028]	$-0.0069$ [-0.0169, +0.0032]
Exact released five-label pattern	Raw	$+0.0028$ [-0.0079, +0.0135]	$-0.0004$ [-0.0112, +0.0105]	$+0.0012$ [-0.0087, +0.0111]
	Platt	$+0.0049$ [-0.0051, +0.0148]	$+0.0004$ [-0.0104, +0.0112]	$+0.0026$ [-0.0067, +0.0119]
	Isotonic	$+0.0033$ [-0.0072, +0.0138]	$-0.0046$ [-0.0162, +0.0069]	$-0.0006$ [-0.0104, +0.0091]

Appendix

Table 10: Adjusted raw-only-diametric effects on excess transfer error. Entries are coefficient estimates with 95% cluster-robust confidence intervals. The first analysis controls for released-label density and focal target-label presence; the second compares only observations with the same complete released five-label pattern.

Figure 2: Raw-only-diametric effect on uncalibrated excess transfer error under progressively stricter controls. Error bars are 95% confidence intervals. The positive unadjusted association disappears after adjustment for released label structure and under exact five-label-pattern matching.

Comparison	BRIGHTER	GoEmotions	XED
BR–XED fear	+.082	+.069	-.051
BR–XED sadness	+.043	+.109	-.026
GE–XED joy	+.021	+.018	-.015

Appendix

Table 11: Descriptive rank-normalized corpus differences under each head, averaged across three seeds; item-sampling intervals are not estimated here. Positive values indicate that the first corpus has the higher mean pooled percentile rank; negative values indicate the reverse.

Label	Head	Raw	95% interval	Seed range	Rank	95% interval	Seed range
anger	BRIGHTER	+0.013	[+0.001, +0.024]	[+0.010, +0.014]	+0.011	[-0.000, +0.021]	[+0.004, +0.016]
anger	GoEmotions	+0.005	[+0.001, +0.009]	[+0.005, +0.005]	+0.012	[-0.000, +0.023]	[+0.007, +0.015]
anger	XED	+0.003	[-0.000, +0.006]	[+0.003, +0.004]	+0.001	[-0.010, +0.011]	[-0.003, +0.003]
fear	BRIGHTER	+0.043	[+0.030, +0.056]	[+0.037, +0.048]	+0.038	[+0.026, +0.050]	[+0.034, +0.041]
fear	GoEmotions	+0.005	[+0.000, +0.009]	[+0.003, +0.006]	+0.009	[-0.003, +0.020]	[+0.005, +0.010]
fear	XED	+0.005	[+0.001, +0.009]	[+0.002, +0.006]	+0.006	[-0.005, +0.018]	[-0.005, +0.009]

Appendix

Table 12: Complete external comparison under all three heads.

Resource	$\Delta$ Macro-F1	$\Delta$ Micro-F1	$\Delta$ Brier
BRIGHTER	-.00690 (.02156)	-.00772 (.01980)	+.000236 (.000549)
GoEmotions	+.00230 (.00125)	+.00365 (.00547)	-.000025 (.000043)
XED	-.00143 (.00550)	+.00135 (.00231)	+.000171 (.000177)

Appendix

Table 13: Native test-set changes after LLM-assisted revision of flagged training annotations. Changes are revised minus original; parentheses give sample SD of paired changes across three seeds. Positive F1 and negative Brier changes indicate improvement.

Emotion Recognition

Emotion recognition uses language, voice, face, body movement, and physiological signals to infer emotional categories or dimensions such as valence and arousal.

See all 129 papers

Chiaroscuro for Emotions: A Contrastive Emotion Benchmark Grounded in Appraisal Theory
Sep 3, 2026Divyesh Bommana, Mohammad Saim, Tianyu JiangLLM Evaluation Classification
University of Cincinnati
Label Authority in Speech Emotion Recognition: Intended and Perceived Ground Truth in Emozionalmente
Oct 19, 2026Marco Giordano, Stefano Giacomelli, Claudia Rinaldi +1Speech Emotion Recognition Annotator Disagreement
Department of Information Engineering, Computer Science and Mathematics (DISIM), University of L’Aquila, L’Aquila, 67100, ITA · National Inter-University Consortium for Telecommunications (CNIT), University of L’Aquila, L’Aquila, 67100, ITA
LLMs Capture Emotion Labels, Not Emotion Uncertainty: Distributional Analysis and Calibration of Human-LLM Judgment Gaps
Apr 30, 2026Keito Inoshita, Xiaokang Zhou, Akira Kawai +1LLM Evaluation Annotator Disagreement
Faculty of Business and Commerce, Kansai University, Osaka, Japan · Faculty of Business Data Science, Kansai University, Osaka, Japan · RIKEN Center for Advanced Intelligence Project, Tokyo, Japan +2

Sep 3, 2026J/K move · Enter open · S save

Chiaroscuro for Emotions: A Contrastive Emotion Benchmark Grounded in Appraisal Theory

Divyesh Bommana, Mohammad Saim, Tianyu Jiang

University of Cincinnati

LLM Evaluation Classification Emotion Recognition

Open paper AskPDF arXiv

Emotion recognition benchmarks often predict one emotion per text, missing many real-world scenarios where two people arrive at opposing emotions from a single shared event. For example, a child kicks the seat in front of her in excitement while the passenger ahead grows angry. We introduce CHIARO, a 1,000 human-annotated sentence benchmark for contrastive emotion inference grounded in appraisal theory. Each scene describes one causal trigger eliciting a positive emotion in one person and a negative emotion in the other, drawn from a ten-class taxonomy. We benchmark seven frontier LLMs and four off-the-shelf emotion classifiers. The strongest LLM reaches 67.3 macro-F1, well below human agreement, while existing emotion classifiers score near chance. Beyond evaluation, CHIARO also serves as a training signal. When combined with an existing emotion corpus, the resulting downstream classifier improves on CHIARO itself and on six of ten external emotion benchmarks, which positions our dataset as a complementary signal for emotion recognition.

Abstract

Emotion Recognition

Explore similar work

Chiaroscuro for Emotions: A Contrastive Emotion Benchmark Grounded in Appraisal Theory

Label Authority in Speech Emotion Recognition: Intended and Perceived Ground Truth in Emozionalmente

LLMs Capture Emotion Labels, Not Emotion Uncertainty: Distributional Analysis and Calibration of Human-LLM Judgment Gaps