Understanding and Tackling Label Errors in Individual-Level Nature Language Understanding
Organizations: Emory University, Atlanta, US · University of Central Florida, Orlando, US
Abstract
Natural language understanding (NLU) is a task that enables machines to understand human language. Some tasks, such as stance detection and sentiment analysis, are closely related to individual subjective perspectives, thus termed individual-level NLU. Previously, these tasks are often simplified to text-level NLU tasks, ignoring individual factors. This not only makes inference difficult and unexplainable but often results in a large number of label errors when creating datasets. To address the above limitations, we propose a new NLU annotation guideline based on individual-level factors. Specifically, we incorporate other posts by the same individual and then annotate individual subjective perspectives after considering all individual posts. We use this guideline to expand and re-annotate the stance detection and topic-based sentiment analysis datasets. We find that error rates in the samples were as high as 31.7% and 23.3%. We further use large language models to conduct experiments on the re-annotation datasets and find that the large language models perform well on both datasets after adding individual factors. Both GPT-4o and Llama3-70B can achieve an accuracy greater than 87% on the re-annotation datasets. We also verify the effectiveness of individual factors through ablation studies. We call on future researchers to add individual factors when creating such datasets. Our re-annotation dataset can be found at https://github.com/24yearsoldstudent/Individual-NLU
Figures & tables
| Dataset | Topics/Targets | Tweets |
| Semeval Stance | 5 | 1,129 |
| Semeval Sentiment | 60 | 4,346 |
| HateXplain | 12 | 20,148 |
| Dataset | Target | # Posts | # Errors | Error Rate |
| Stance | AT | 133 | 30 | 29.3% |
| CC | 114 | 22 | 19.3% | |
| FM | 114 | 50 | 43.9% | |
| LA | 130 | 54 | 41.5% | |
| All | 491 | 156 | 31.7% | |
| Sentiment | All Topics | 120 | 28 | 23.3% |
| Model | SD | SA | HSD | |||
| Relabeled | OL | Relabeled | OL | Relabeled | OL | |
| GPT-4o | 92.46 | 64.77 | 89.17 | 78.33 | 65.79 | 46.05 |
| Llama3-70B | 87.58 | 64.56 | 92.50 | 78.33 | 60.53 | 38.16 |
| PHI-4 | 79.02 | 61.51 | 92.50 | 76.67 | 53.95 | 35.52 |
| Model | SD | SA | HSD | |||
| MT | ST | MT | ST | MT | ST | |
| GPT-4o | 92.48 | 69.25 | 89.17 | 74.17 | 65.79 | 63.16 |
| Llama3-70B | 87.58 | 74.13 | 92.50 | 72.50 | 60.53 | 59.21 |
| PHI-4 | 79.02 | 60.29 | 92.50 | 71.67 | 53.95 | 50.00 |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Stance Detection | Sentiment Analysis | |||
| Model | Total | Total (OL) | Total | Total (OL) |
| GPT-4o | 94.36 | 68.61 | 88.44 | 77.08 |
| LLama3-70B | 90.29 | 68.94 | 92.06 | 76.67 |
| PHI-4 | 78.87 | 65.96 | 92.06 | 74.88 |
Stance Detection
Stance detection identifies whether people support, oppose, or remain neutral toward a target, claim, or position in text and conversation.
See all 39 papers