cs.CLNov 16, 2026

A Scalable Multi Task Learning Framework for Modeling Demographic Disagreements in Annotation Tasks

Authors: Soumyajit Gupta, Maria De-Arteaga, Matthew Lease

Organizations: Computer Science, School of Computing, University of Texas at Austin · Universitat Ramon Llull, ESADE · Information, School of Computing, University of Texas at Austin

Abstract

Annotators from different demographic backgrounds often disagree in systematic ways due to varying lived experiences, cultural contexts, and their relationship to the content. When such disagreement occurs, one group's perspective may sometimes warrant prioritization because they hold an epistemically advantaged standpoint or bear greater stake in the outcome. For example, when assessing toxicity of posts on social media, it may be important to prioritize a targeted group's viewpoint. We propose a Stakeholder-Aware Joint Multi-Task Learning (SAJ-MTL) framework to jointly model demographic disagreement both across- and within- annotator groups. The model uses conditional gradient routing to handle sparsity induced by group-conditional labeling, and parameterization supports customized weighting of groups in different task settings. SAJ-MTL improves group-level performance in toxicity detection through better context-aware understanding of demographic-sensitive content. Only annotator group-membership information is required, which is more practical than existing methods requiring more privacy-invasive individual-level annotator details. Our scalable architecture is also computationally efficient to support large-scale deployment.

Figures & tables

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Learning Who Disagrees: Demographic Importance Weighting for Modeling Annotator Distributions with DiADEM

    Apr 9, 2026Samay U. Shetty, Tharindu Cyril Weerasooriya, Deepak Pandita +1Annotator DisagreementLabel Distribution Learning