Human Label Variation

Human label variation (HLV), the inherent disagreement among human annotators labeling the same data, is a significant challenge in natural language processing (NLP). Current research focuses on understanding the sources of HLV (e.g., ambiguity, differing interpretations), developing methods to model and predict this variation (often employing multi-head models or techniques inspired by recommender systems), and improving model calibration by accounting for HLV rather than treating it as noise. Addressing HLV is crucial for building more robust, trustworthy, and fair NLP systems, impacting the reliability of benchmarks and the development of more accurate and interpretable models.