Natural language understanding (NLU) is a task that enables machines to understand human language. Some tasks, such as stance detection and sentiment analysis, are closely related to individual subjective perspectives, thus termed individual-level NLU. Previously, these tasks are often simplified to text-level NLU tasks, ignoring individual factors. This not only makes inference difficult and unexplainable but often results in a large number of label errors when creating datasets. To address the above limitations, we propose a new NLU annotation guideline based on individual-level factors. Specifically, we incorporate other posts by the same individual and then annotate individual subjective perspectives after considering all individual posts. We use this guideline to expand and re-annotate the stance detection and topic-based sentiment analysis datasets. We find that error rates in the samples were as high as 31.7% and 23.3%. We further use large language models to conduct experiments on the re-annotation datasets and find that the large language models perform well on both datasets after adding individual factors. Both GPT-4o and Llama3-70B can achieve an accuracy greater than 87% on the re-annotation datasets. We also verify the effectiveness of individual factors through ablation studies. We call on future researchers to add individual factors when creating such datasets. Our re-annotation dataset can be found at https://github.com/24yearsoldstudent/Individual-NLU
Figures & tables
Figure 1: A typical example of potential label error in stance detection.
Figure 2: The process of manual re-annotation and LLM judges. In the manual re-annotation, after finding other posts related to the topic/target by individuals (network users), three annotators follow the guidelines to annotate individual-level labels. In LLM judges, the input is divided into two parts: data and prompts.
Dataset
Topics/Targets
Tweets
Semeval Stance
5
1,129
Semeval Sentiment
60
4,346
HateXplain
12
20,148
Table 1: Statistics of three datasets.
Figure 3: Some examples of correcting label errors. Using multiple posts from the same publisher can more accurately determine the user’s sentiment or stance, and can effectively explain why this label is given.
Dataset
Target
# Posts
# Errors
Error Rate
Stance
AT
133
30
29.3%
CC
114
22
19.3%
FM
114
50
43.9%
LA
130
54
41.5%
All
491
156
31.7%
Sentiment
All Topics
120
28
23.3%
Table 2: Error rate statistics across different datasets.
Model
SD
SA
HSD
Relabeled
OL
Relabeled
OL
Relabeled
OL
GPT-4o
92.46
64.77
89.17
78.33
65.79
46.05
Llama3-70B
87.58
64.56
92.50
78.33
60.53
38.16
PHI-4
79.02
61.51
92.50
76.67
53.95
35.52
Table 3: Model performance after user-context-aware relabeling. OL denotes evaluation using the original labels. SD: stance detection; SA: sentiment analysis; HSD: hate speech detection.
Model
SD
SA
HSD
MT
ST
MT
ST
MT
ST
GPT-4o
92.48
69.25
89.17
74.17
65.79
63.16
Llama3-70B
87.58
74.13
92.50
72.50
60.53
59.21
PHI-4
79.02
60.29
92.50
71.67
53.95
50.00
Table 4: Comparison between multiple-tweet and single-tweet input settings. MT denotes multiple tweets, and ST denotes a single tweet. SD: stance detection; SA: sentiment analysis; HSD: hate speech detection.
Figure 4: Performance on the Semeval stance detection dataset using different numbers of tweets as LLM input.
Figure 5: Multi-posts example. In this case, although the user’s stance is "Against" in both the new and original datasets, it is difficult or even impossible to infer the user’s stance from the text in the original dataset. After adding other tweets from the user, LLM gives an accurate prediction.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Stance Detection
Sentiment Analysis
Model
Total
Total (OL)
Total
Total (OL)
GPT-4o
94.36
68.61
88.44
77.08
LLama3-70B
90.29
68.94
92.06
76.67
PHI-4
78.87
65.96
92.06
74.88
Appendix
Table 5: The average F1 value of different models on the dataset after label correction. OL means original labels, which is the label of the original datasets without correction.
Subjective NLP datasets typically aggregate annotator judgments into a single gold label, making it difficult to diagnose whether disagreement reflects unclear criteria, collapsed distinctions, or legitimate plurality. We propose a schema-level diagnostic for auditing expert-designed annotation schemas prior to gold-label commitment, using only multi-annotator criterion judgments. The diagnostic separates two failure modes: unstable criteria with hard-to-operationalize boundaries, and systematic overlap that blurs the boundaries between mutually exclusive categories. Applied to persuasive value extraction in commercial documents, we find that disagreement is not diffuse: instability concentrates in a few criteria, while nearly half of covered sentences activate multiple categories. These signals align with where domain experts disagree, yielding an evidence-based audit for tightening guidelines, revising category structure, or reconsidering the annotation paradigm.
Nisrine Rair, Alban Goupil, Valeriu Vrabie +1
CReSTIC, Université de Reims Champagne-Ardenne, Reims, France · Chochoy Conseil, Reims, France
Large language models are increasingly used to annotate texts, but their outputs reflect some human perspectives better than others. Existing methods for correcting LLM annotation error assume a single ground truth. However, this assumption fails in subjective tasks where disagreement across demographic groups is meaningful. Here we introduce Perspective-Driven Inference, a method that treats the distribution of annotations across groups as the quantity of interest, and estimates it using a small human annotation budget. We contribute an adaptive sampling strategy that concentrates human annotation effort on groups where LLM proxies are least accurate. We evaluate on politeness and offensiveness rating tasks, showing targeted improvements for harder-to-model demographic groups relative to uniform sampling baselines, while maintaining coverage.
Human label variation has been established as a central phenomenon in NLP: the perspectives different annotators have on the same item need to be embraced. Data collection practices thus shifted towards increasing the annotator numbers and releasing disaggregated datasets, harmful language being most resourced due to its high subjectivity. While this resulted in rich information about who annotated (sociodemographics, attitudes, etc.), the what (e.g., linguistic properties of items), and their interplay has received little attention. We present the first large-scale analysis of four reference datasets for harmful language detection, bringing together annotator characteristics, linguistic properties of the items, and their interactions in a statistically informed picture. We find that interactions are crucial, revealing intersectional effects ignored in previous work, and that a strong role is played by lexical cues and annotator attitudes. Effect patterns, however, vary considerably across datasets. This urges caution about generalization and transferability.
Maximilian Maurer, Maximilian Linde, Gabriella Lapesa
1GESIS - Leibniz Institute for the Social Sciences · 2Heinrich-Heine University Düsseldorf