Mixed Feelings, Single Polarity

Emily Öhman; Saif Mohammad

cs.CLOct 11, 2026

Mixed Feelings, Single Polarity

Authors: Emily Öhman, Saif Mohammad

Organizations: Waseda University, Japan · National Research Council Canada

Abstract

Emotion datasets are routinely mapped onto positive and negative valence, by assigning each emotion category one polarity. We test where this works and where it fails, using SemEval-2018 Affect in Tweets (English, Arabic, Spanish), Affective Text aligned with EmoBank, raw GoEmotions annotations, naturalistic hashtag corpora, and a controlled annotation study. Fixed projection works well when a text carries labels of one valence. It fails on a subset identifiable from the labels alone: tweets whose emotion labels point in opposite directions. These are 9.0% of polar SemEval test tweets but cause 54.8% of the polarity errors of a learned projection, are more than twice as likely to receive a neutral-or-mixed gold label, and are not rescued by modeling the full label set. Annotators produce such configurations routinely: 16.3% of multilabel GoEmotions judgments combine emotions from the dataset's own positive and negative parents. In a controlled study holding word and reader constant, context changes the coarse valence judgment in 56.8% of cases, against 19.9% for controls. We recommend projecting label configurations rather than labels, abstaining or keeping a mixed class where labels conflict, and reporting coverage alongside accuracy.

Figures & tables

Emotion	$D$ [95% CI]
Joy	.879 [.859,.898]
Love	.713 [.667,.754]
Sadness	.679 [.639,.716]
Anger	.646 [.611,.681]
Optimism	.644 [.599,.687]
Disgust	.591 [.551,.631]

Table 1: Threshold-free valence diagnosticity in SemEval-2018, averaged across English, Arabic, and Spanish test sets, with 95% bootstrap intervals. Per-language estimates appear in Appendix Table 8 .

Emotion	$n$	$D$ [95% CI]	Mean $V$	$\rho$ [95% CI]
Joy	144	.843 [.808,.875]	3.376	+.773 [.749,.795]
Sadness	139	.832 [.787,.873]	2.260	-.709 [-.737,-.679]
Disgust	19	.661 [.539,.769]	2.383	-.407 [-.454,-.357]
Fear	114	.609 [.535,.678]	2.463	-.562 [-.602,-.521]
Anger	38	.603 [.493,.706]	2.421	-.558 [-.596,-.518]
Surprise	49	.220 [.065,.370]	3.015	+.143 [.086,.197]

Table 2: Affective Text

\times

EmoBank, combined valence.

D

uses the

\geq 50

presence threshold;

\rho

uses continuous intensity. At

\geq 25

the ordering is unchanged (joy .820, sadness .735, surprise .182; surprise mean valence 2.958).

	All ( $n=2076$ )		Polar ( $n=1681$ )		Opposed ( $n=246$ )
Method	Cov.	Err.	Cov.	Err.	Cov.	Err.	Pred. 0
Fixed lookup, unsigned	92.1	20.0	94.3	3.5	48.8	55.0	0
Fixed lookup, positive	92.0	21.4	93.5	4.5	55.7	57.7	0
Averaged priors, binary	100	23.5	100	5.5	100	58.9	0
Label-set priors, binary	100	23.6	100	5.6	100	59.4	0
Selective projection	88.2	18.5	91.4	2.8	15.4	55.3	0

Table 3: Held-out SemEval-2018 emotion-to-valence projection using official V-oc labels. Cov. is prediction coverage (%); Err. is the percentage of incorrect predictions among covered items. All includes negative, neutral-or-mixed, and positive gold classes. Polar excludes gold class 0; Opposed contains conventional positive- and negative-label combinations and includes all three gold classes. Pred. 0 is the number of neutral-or-mixed predictions across the full test set. The selective projection is an illustrative abstention baseline. Polar-only evaluation is a retrospective diagnostic subset, not a deployable selection procedure.

Measure / denominator	Observed	Null
P+N, multilabel judgments	16.28%	39.53%
P+N, one-sentence multilabel	15.14%	41.02%
Any-parent, multilabel judgments	39.40%	66.90%
P+N, all emotional judgments	3.84%	9.31%
Any-parent, all emotional judgments	9.30%	15.83%
P+N, filtered item label sets	2.68%	—

Table 4: GoEmotions judgments crossing the published sentiment hierarchy. P+N: both positive- and negative-parent emotions selected. Each observed rate is compared with a null over the same denominator (Appendix A ).

Figure 1: Observed wheel placements for three illustrative lexical targets in isolation and in their two contextualized uses. Each dot is an observed Study 2 main-arm association point. The contextual columns are ordered by their observed coarse-valence balance for display only; all inferential analyses use the complete set of 50 targets.

Mechanism	Targets	Mean $\Delta_{E}$	Coarse switch
Stability control	4	.346	19.9%
Event-dependent affect	6	.427	68.8%
Mixed affect	8	.436	44.5%
Evaluative reframing	17	.451	62.4%
Sense disambiguation	13	.476	66.9%
Register	2	.488	55.2%

Table 5: Study 2 centroid displacement and coarse-label switch rate by context mechanism. Descriptive: groups differ in size and kind of target.

Handling	Emotion categories
High-diagnosticity prior	joy, love, sadness, anger, disgust
Uncertainty-aware prior	optimism, pessimism, fear, trust
No category-only polarity	anticipation, surprise

Table 6: Empirical guidance based on the minimum held-out diagnosticity

D

across SemEval languages. High diagnosticity: minimum

D\geq.50

; uncertainty-aware:

.15\leq D<.50

; no category-only polarity: minimum

D<.15

. These descriptive groupings are not independently validated projection rules, and

D

does not specify polarity direction.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Base	Co-emotion	$\Delta V$ [95% CI]	$n_{+c}$
Love	Joy	+.258 [.226,.292]	543
Sadness	Joy	+.228 [.194,.263]	86
Anticipation	Joy	+.225 [.192,.258]	128
Anger	Joy	+.202 [.172,.231]	86
Optimism	Sadness	-.205 [-.241,-.166]	67
Joy	Anger	-.199 [-.229,-.169]	86

Appendix

Table 7: Representative co-occurrence shifts in held-out SemEval-2018 data that replicated the training direction in all three languages and whose pooled 95% intervals exclude zero.

n_{+c}

: held-out items carrying both labels.

Emotion	Lang.	$n_{e}$	$D$ [95% CI]	Mean train $V$
Joy	En	450	0.883 [0.850,0.912]	0.644
Joy	Ar	292	0.896 [0.863,0.924]	0.738
Joy	Es	229	0.860 [0.819,0.896]	0.690
Love	En	167	0.658 [0.595,0.716]	0.706
Love	Ar	261	0.782 [0.732,0.830]	0.707
Love	Es	43	0.698 [0.593,0.793]	0.708

Appendix

Table 8: Per-language SemEval-2018 continuous-valence results.

n_{e}

denotes test items carrying each emotion;

D

measures threshold-free valence diagnosticity. Mean train

V

is the continuous V-reg mean and does not define categorical polarity.

Base	Co-emotion	Train dir.	Held-out $\Delta V$ [95% CI]	$n_{+c}$	Confirmed	All 3
Love	Joy	positive	+0.258 [+0.226,+0.292]	543	3/3	yes
Sadness	Joy	positive	+0.228 [+0.194,+0.263]	86	3/3	yes
Anticipation	Joy	positive	+0.225 [+0.192,+0.258]	128	3/3	yes
Trust	Joy	positive	+0.207 [+0.166,+0.249]	118	3/3	yes
Optimism	Sadness	negative	-0.205 [-0.241,-0.166]	67	3/3	yes
Optimism	Joy	positive	+0.205 [+0.182,+0.229]	751	3/3	yes

Appendix

Table 9: Directional co-occurrence discoveries carried from training to held-out evaluation (part 1 of 3). “Confirmed”: held-out languages matching the training direction over languages with sufficient support. Of 68 support-eligible ordered pairs, 62 met the same-direction training criterion; 46 reproduced that direction in all three held-out languages, and 45 of those had pooled 95% intervals excluding zero.

Base	Co-emotion	Train dir.	Held-out $\Delta V$ [95% CI]	$n_{+c}$	Confirmed	All 3
Pessimism	Sadness	negative	-0.131 [-0.161,-0.102]	240	3/3	yes
Fear	Sadness	negative	-0.128 [-0.168,-0.083]	102	3/3	yes
Trust	Optimism	positive	+0.107 [+0.060,+0.153]	114	3/3	yes
Joy	Love	positive	+0.102 [+0.085,+0.120]	543	3/3	yes
Disgust	Surprise	positive	+0.101 [+0.055,+0.147]	25	3/3	yes
Anger	Surprise	positive	+0.099 [+0.052,+0.144]	27	3/3	yes

Appendix

Table 10: Directional co-occurrence discoveries (part 2 of 3).

Base	Co-emotion	Train dir.	Held-out $\Delta V$ [95% CI]	$n_{+c}$	Confirmed	All 3
Pessimism	Anticipation	positive	+0.057 [+0.026,+0.088]	47	3/3	yes
Anger	Fear	negative	-0.038 [-0.069,-0.008]	92	3/3	yes
Sadness	Pessimism	negative	-0.034 [-0.057,-0.010]	240	3/3	yes
Optimism	Trust	negative	-0.012 [-0.043,+0.020]	114	3/3	yes
Love	Sadness	negative	-0.270 –	39	2/2	no
Fear	Optimism	positive	+0.209 –	24	2/2	no

Appendix

Table 11: Directional co-occurrence discoveries (part 3 of 3).

Emotion	$n$	Reader $D$ [95% CI]	Writer $D$ [95% CI]	Combined $D$ [95% CI]	Reader $\rho$	Writer $\rho$	Combined $\rho$
Anger	38	.614 [.502,.723]	.472 [.328,.609]	.603 [.493,.706]	-.545	-.479	-.558
Disgust	19	.642 [.537,.742]	.562 [.374,.725]	.661 [.539,.769]	-.404	-.346	-.407
Fear	114	.614 [.534,.688]	.467 [.379,.549]	.609 [.535,.678]	-.549	-.474	-.562
Joy	144	.767 [.720,.810]	.759 [.707,.810]	.843 [.808,.875]	+.731	+.684	+.773
Sadness	139	.793 [.747,.836]	.752 [.692,.808]	.832 [.787,.873]	-.677	-.632	-.709
Surprise	49	.217 [.061,.370]	.171 [.027,.320]	.220 [.065,.370]	+.123	+.142	+.143

Appendix

Table 12: Affective Text

\times

EmoBank across reader, writer, and combined valence. The ordering is preserved in all three; the combined score is slightly more diagnostic for joy and sadness, as expected from averaging two noisy ratings.

Emotion	Parent	$n_{ML}$	Purity obs.	Purity null	Opp./val. obs.	Opp./val. null
Admiration	positive	6600	0.721	0.407	0.130	0.418
Amusement	positive	3115	0.574	0.436	0.220	0.387
Approval	positive	6361	0.617	0.412	0.167	0.410
Caring	positive	2476	0.628	0.431	0.251	0.399
Desire	positive	1670	0.620	0.432	0.213	0.399
Excitement	positive	2609	0.729	0.425	0.086	0.406

Appendix

Table 13: GoEmotions category-level multilabel results. For positive and negative categories, “Opp./val.” is opposite-valence co-selection; for ambiguous categories, co-selection with either valenced parent. Nulls preserve the cardinality distribution of judgments containing each category and sample remaining labels by empirical frequency.

Rater set	Positive+negative	Any-parent
All raters	3.84%	9.30%
Drop top 1	3.93%	9.47%
Drop top 3	3.62%	7.98%
Drop top 5	3.51%	7.49%
Drop top 10	3.67%	8.13%

Appendix

Table 14: GoEmotions sensitivity to prolific raters, over all emotional rater-level judgments.

Corpus	Joy vs.	$n$	AUC [95% CI]	$D$ [95% CI]
TEC	pooled negative	17,144	.666 [.658,.675]	.333 [.317,.349]
	sadness	12,036	.648 [.638,.658]	.296 [.275,.316]
	fear	11,024	.639 [.627,.651]	.278 [.254,.303]
	anger	9,778	.736 [.723,.749]	.472 [.445,.498]
	disgust	8,987	.716 [.698,.735]	.433 [.395,.469]
TUSC	pooled neg., strict	332	.716 [.654,.776]	.433 [.308,.552]

Appendix

Table 15: Study 1D: how well residual lexical valence separates hashtag-labeled joy from negative categories after all hashtags are removed.

n

: tweets in the comparison. 95% intervals from 5,000 bootstrap resamples.

Emotion	$n$	Mean lexical $V$	Median lexical $V$
Joy	8,227	.204	.199
Sadness	3,809	.120	.125
Fear	2,797	.116	.123
Disgust	760	.077	.089
Anger	1,551	.066	.077

Appendix

Table 16: Mean lexical valence in TEC by hashtag emotion after removal of hashtag, URL, and mention material.

Inventory	$n$	AUC [95% CI]	$D$ [95% CI]	Opp.-only †
Strict	332	.716 [.654,.776]	.433 [.308,.552]	26.8%
Expanded	946	.808 [.779,.836]	.616 [.558,.672]	12.9%

Appendix

Table 17: ABCDE/TUSC joy-versus-pooled-negative robustness analysis after removal of all hashtags. † Among posts containing at least one extreme lexical cue (

|V|\geq.70

), the percentage containing extreme cues exclusively opposite to the hashtag-derived valence. The strict and expanded analyses contain 231 and 728 extreme-cued posts, respectively.

Lexical item	Positive (%)	Negative (%)	Neither (%)	Both (%)	$H_{\mathrm{norm}}$
bear	8.7	34.8	26.1	30.4	.932
escape	26.1	21.7	8.7	43.5	.907
happen	4.3	17.4	30.4	47.8	.833
stare	0.0	47.8	34.8	17.4	.739
fire	0.0	21.7	21.7	56.5	.711
surprise	34.8	0.0	8.7	56.5	.651

Appendix

Table 18: Earlier lexical annotation data used as one source for Study 2 candidate generation. Twenty-three annotators judged 41 isolated English words as Positive, Negative, Neither, or Both; four of 943 possible judgments are missing.

H_{\mathrm{norm}}

is normalized four-category response entropy. Items are ordered from highest to lowest disagreement.

Figure 20

Figure 21

Figure 2: Row-normalized transition matrix for the strongest emotion association from the isolated lexical target to a contextualized use. Rows are the isolated strongest-emotion sector and columns the contextual strongest-emotion sector.

Emotion	Min. $D$	Study 1B	Handling
Joy	0.860	strong	high-diagnosticity prior
Love	0.658	–	high-diagnosticity prior
Sadness	0.597	strong	high-diagnosticity prior
Anger	0.599	intermediate	high-diagnosticity prior
Disgust	0.519	intermediate	high-diagnosticity prior
Optimism	0.496	–	uncertainty-aware prior

Appendix

Table 19: Reference diagnosticity values and empirical handling recommendations. Study 1B gives cross-dataset evidence for six categories. Groupings follow Table 6 and are not validated prediction rules.

Sentiment Analysis

Sentiment analysis detects and measures opinions, emotions, and evaluative stance in text and multimodal data.

See all 127 papers

Quality and Agreement in Multilabel Emotion Annotation: A Case Study and Evaluation Framework
Jun 19, 2026Emily Öhman, Anna KoufakouSoft-Label Learning Annotator Disagreement
Waseda University, Tokyo, Japan · Florida Gulf Coast University, Fort Myers FL, USA
When Annotators Agree but Labels Disagree: The Projection Problem in Stance Detection
Mar 25, 2026Bowen ZhangStance Detection Annotator Disagreement
Shenzhen Technology University, Shenzhen, China
Beyond a Single Sentiment Score: Topical and Affective Dimensions of Emoji Meaning on Twitter
Oct 20, 2026Rohan James, Ioannis Christofilogiannis, Yuuv Jauhari +2Social Media Analysis Sentiment Analysis
School of Computer Science and Statistics Trinity College Dublin, Ireland

Jun 19, 2026J/K move · Enter open · S save

Quality and Agreement in Multilabel Emotion Annotation: A Case Study and Evaluation Framework

Emily Öhman, Anna Koufakou

Waseda University, Tokyo, Japan · Florida Gulf Coast University, Fort Myers FL, USA

Soft-Label Learning Annotator Disagreement Multi-Label Classification Emotion Recognition

Open paper AskPDF arXiv

Emotion annotation is inherently subjective, yet most NLP pipelines still assume "gold" labels, typically produced by majority voting, and treat annotator variation as noise. In this paper, we present a multilabel emotion annotation case study and use it to examine how annotator behavior and aggregation choices affect both agreement estimates and downstream emotion classifiers. Rather than collapsing disagreement into a single label, we represent targets as soft vote-share labels (including an intensity-weighted variant) and evaluate models using both thresholded metrics (macro-/micro-F1) and probabilistic alignment (Bernoulli cross-entropy SoftBCE), alongside data-derived disagreement diagnostics. Across annotation regimes, we show that disagreement is structured and leaves measurable traces in model behavior: hard labels may maximize F1 metrics, while soft supervision yields predictions that better reflect empirical annotator variance and uncertainty. Our results provide practical guidance for designing, aggregating, and evaluating multilabel emotion datasets when multiple interpretations are plausible.

Abstract

Sentiment Analysis

Explore similar work

Quality and Agreement in Multilabel Emotion Annotation: A Case Study and Evaluation Framework

When Annotators Agree but Labels Disagree: The Projection Problem in Stance Detection

Beyond a Single Sentiment Score: Topical and Affective Dimensions of Emoji Meaning on Twitter