disAgreement Score

Disagreement scores quantify the level of inconsistency in judgments, whether from multiple human annotators or different machine learning models. Research focuses on leveraging disagreement to improve model performance, understand annotator biases (e.g., based on demographics), and enhance the generalizability of models across diverse datasets and domains. Methods range from using ensemble disagreement as a proxy for human evaluation to employing graph convolutional networks and adversarial training to minimize disagreement and improve model robustness. This work has implications for improving the reliability and fairness of machine learning systems, particularly in applications involving subjective tasks and sensitive data.