cs.CLFeb 18, 2025

Understanding and Tackling Label Errors in Individual-Level Nature Language Understanding

Authors: Yunpeng Xiao, Kyrie Zhixuan Zhou, Youpeng Zhao, Kai Shu

Organizations: Emory University, Atlanta, US · University of Central Florida, Orlando, US

Abstract

Natural language understanding (NLU) is a task that enables machines to understand human language. Some tasks, such as stance detection and sentiment analysis, are closely related to individual subjective perspectives, thus termed individual-level NLU. Previously, these tasks are often simplified to text-level NLU tasks, ignoring individual factors. This not only makes inference difficult and unexplainable but often results in a large number of label errors when creating datasets. To address the above limitations, we propose a new NLU annotation guideline based on individual-level factors. Specifically, we incorporate other posts by the same individual and then annotate individual subjective perspectives after considering all individual posts. We use this guideline to expand and re-annotate the stance detection and topic-based sentiment analysis datasets. We find that error rates in the samples were as high as 31.7% and 23.3%. We further use large language models to conduct experiments on the re-annotation datasets and find that the large language models perform well on both datasets after adding individual factors. Both GPT-4o and Llama3-70B can achieve an accuracy greater than 87% on the re-annotation datasets. We also verify the effectiveness of individual factors through ablation studies. We call on future researchers to add individual factors when creating such datasets. Our re-annotation dataset can be found at https://github.com/24yearsoldstudent/Individual-NLU

Figures & tables

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Beyond Black-Box Labels: Interpretable Criteria for Diagnosing Subjective NLP Tasks

    Apr 18, 2026Nisrine Rair, Alban Goupil, Valeriu Vrabie +1Annotator Disagreement

  2. Multi-Perspective LLM Annotations for Valid Analyses in Subjective Tasks

    Mar 22, 2026Navya Mehrotra, Adam Visokay, Kristina GligorićAnnotator DisagreementLLM-Assisted Annotation

  3. Who and What? Using Linguistic Features and Annotator Characteristics to Analyze Annotation Variation

    May 7, 2026Maximilian Maurer, Maximilian Linde, Gabriella LapesaAnnotator DisagreementHuman-in-the-Loop Annotation