A Scalable Multi Task Learning Framework for Modeling Demographic Disagreements in Annotation Tasks
Authors: Soumyajit Gupta, Maria De-Arteaga, Matthew Lease
Organizations: Computer Science, School of Computing, University of Texas at Austin · Universitat Ramon Llull, ESADE · Information, School of Computing, University of Texas at Austin
Annotators from different demographic backgrounds often disagree in systematic ways due to varying lived experiences, cultural contexts, and their relationship to the content. When such disagreement occurs, one group's perspective may sometimes warrant prioritization because they hold an epistemically advantaged standpoint or bear greater stake in the outcome. For example, when assessing toxicity of posts on social media, it may be important to prioritize a targeted group's viewpoint. We propose a Stakeholder-Aware Joint Multi-Task Learning (SAJ-MTL) framework to jointly model demographic disagreement both across- and within- annotator groups. The model uses conditional gradient routing to handle sparsity induced by group-conditional labeling, and parameterization supports customized weighting of groups in different task settings. SAJ-MTL improves group-level performance in toxicity detection through better context-aware understanding of demographic-sensitive content. Only annotator group-membership information is required, which is more practical than existing methods requiring more privacy-invasive individual-level annotator details. Our scalable architecture is also computationally efficient to support large-scale deployment.
Figures & tables
Figure 1: Naive All-Pairs SAJ-MTL. Each annotator-target pair (a,t) is modeled as a separate task and dedicated head, allowing direct learning of pairwise interactions but yielding multiplicative parameter growth and data sparsity for underrepresented pairs.
Figure 2: Scalable SAJ-MTL. Annotator and target effects are factorized into independent modules and combined additively. This enables efficient modeling of annotator–target interactions while reducing complexity to linear scaling, improving both parameter efficiency and generalization in sparse settings.
Figure 3: Overview of the SAJ-MTL architecture . Input vector x is encoded using a shared encoder hϕ(x) . Annotator-group and target-group embeddings (ea,et) are projected via lightweight linear modules (Wa,Wt) and fused additively with the shared representation to form the conditional representation z(x,a,t) . Two parallel prediction heads, annotator-specific head ga and target-specific head ga , map this representation to logits that are summed to produce the annotator–target–conditioned prediction fθ(y∣x,a,t) . During training, gradients update only the shared encoder and the annotator/target modules corresponding to the active pair (a,t) . At inference, predictions may be aggregated using viewpoint prioritization weights π(a,t) enabling socially grounded or policy-driven weighting of perspectives (see Section 3.6 ).
Figure 4: TextAug Pipeline. The input text is internally predicted for toxicity based on the annotator group. An additional encoder is then used to predict conditioned on the target group as well.
Metric
Level
Joint-AWG
Joint-AG
TextAug
CondMTL
F1 ( ↑ )
Micro
0.64 ± 0.02
0.61 ± 0.01
0.52 ± 0.03
0.50 ± 0.02
Macro
0.65 ± 0.01
0.64 ± 0.02
0.55 ± 0.03
0.49 ± 0.01
Precision ( ↑ )
Micro
0.71 ± 0.01
0.67 ± 0.02
0.63 ± 0.03
0.60 ± 0.02
Macro
0.75 ± 0.02
0.73 ± 0.01
0.66 ± 0.02
0.59 ± 0.01
Recall ( ↑ )
Micro
0.59 ± 0.02
0.57 ± 0.01
0.45 ± 0.03
0.43 ± 0.02
Macro
0.59 ± 0.01
0.58 ± 0.02
0.47 ± 0.02
0.42 ± 0.02
Table 1: Prediction results on MHS for the Annotated-Target scenario. Mean performance and standard deviation are reported over 5 runs. The top results are bolded. SAJ-MTL with Joint-AWG labeling is best.
Figure 5: F1 scores over Annotator - Target groups. Both of our Joint-AWG and Joint-AG models outperform the baseline models by better capturing the interaction between annotator and target demographics.
Model
Recall
Max.
Asian
Black
Latinx
White
EO Gap ( ↓ )
Joint-AWG
0.53 ± 0.01
0.70 ± 0.01
0.55 ± 0.01
0.57 ± 0.01
0.17 ± 0.01
Joint-AG
0.53 ± 0.01
0.70 ± 0.01
0.55 ± 0.01
0.54 ± 0.01
0.17 ± 0.01
TextAug
0.42 ± 0.03
0.64 ± 0.02
0.41 ± 0.03
0.41 ± 0.03
0.23 ± 0.03
CondMTL
0.32 ± 0.02
0.61 ± 0.01
0.36 ± 0.02
0.40 ± 0.02
0.29 ± 0.02
Table 2: Per-group recall (higher is better) for each demographic group, along with the corresponding Maximum Equal Opportunity (EO) Gap ( lower is better ). Max. EO Gap is the difference between the maximum and minimum recall across demographic groups. Both Joint-AWG and Joint-AG achieve the lowest EO Gap, indicating more equitable performance across demographic groups compared to other models.
Dataset
MHS
KET
Metric
Level
Joint-AWG
Joint-AG
TextAug
CondMTL
Joint-AWG
Joint-AG
TextAug
CondMTL
F1 ( ↑ )
Micro
0.58 ± 0.02
0.56 ± 0.02
0.48 ± 0.04
0.45 ± 0.03
0.75 ± 0.03
0.72 ± 0.04
0.65 ± 0.02
0.62 ± 0.03
Macro
0.59 ± 0.01
0.57 ± 0.02
0.49 ± 0.03
0.45 ± 0.02
0.71 ± 0.02
0.68 ± 0.03
0.60 ± 0.04
0.56 ± 0.03
Precision ( ↑ )
Micro
0.68 ± 0.02
0.66 ± 0.01
0.58 ± 0.02
0.53 ± 0.01
0.77 ± 0.03
0.74 ± 0.03
0.68 ± 0.01
0.65 ± 0.04
Macro
0.73 ± 0.01
0.71 ± 0.01
0.58 ± 0.01
0.55 ± 0.01
0.71 ± 0.02
0.69 ± 0.03
0.63 ± 0.03
0.58 ± 0.05
Recall ( ↑ )
Micro
0.51 ± 0.02
0.49 ± 0.02
0.42 ± 0.04
0.41 ± 0.03
0.74 ± 0.04
0.71 ± 0.03
0.62 ± 0.03
0.59 ± 0.04
Table 3: Predictive performance on datasets for the Detected-Target scenario, averaged over five runs.
Model
Recall
Max.
Asian
Black
Latinx
White
EO Gap ( ↓ )
Joint-AWG
0.49 ± 0.02
0.64 ± 0.02
0.44 ± 0.02
0.53 ± 0.02
0.20 ± 0.02
Joint-AG
0.49 ± 0.02
0.64 ± 0.02
0.44 ± 0.02
0.50 ± 0.01
0.20 ± 0.02
TextAug
0.38 ± 0.03
0.60 ± 0.02
0.35 ± 0.03
0.40 ± 0.02
0.25 ± 0.03
CondMTL
0.30 ± 0.02
0.60 ± 0.02
0.30 ± 0.02
0.53 ± 0.02
0.30 ± 0.01
Table 4: Maximum EO Gap across demographic groups under noisy target prediction on MHS. Lower values indicate more equitable performance. Both Joint-AWG and Joint-AG maintain lower EO gaps than baseline methods despite using automatically predicted target identities.
Model type
# Params ( ↓ )
Δ ( ↓ )
Time(s) ( ↓ )
Δ ( ↓ )
Target Only (CondMTL)
763,653
-
2200
-
TextAug
67,075,795
+8684%
4800
+118%
Naive SAJ-MTL
763,653
+0%
2400
+9%
SAJ-MTL (Ours)
626,133
-18%
2850
+30%
Table 5: Memory (trainable parameters) and training time (10 epochs) comparison between models. We treat the Target-Only model as baseline for comparison, since it does not account for annotator views. While the model size of Target-Only and Naive SAJ-MTL are the same, SAJ-MTL takes longer time to train due to variation in data. The scalable SAJ variant is the best performing model in terms of trainable parameters.
Table 6: Stage-wise correspondence between identity-sensitive IRT formulations and SAJ-MTL.
Target
Annotator Group
All Annotators
Group
Asian
Black
Latinx
White
Asian
621 (1.2%)
721 (1.4%)
551 (1.1%)
5,501 (10.8%)
7,394 (14.6%)
Black
1,560 (3.1%)
2,292 (4.5%)
1,654 (3.3%)
18,545 (36.6%)
24,051 (47.4%)
Latinx
638 (1.3%)
961 (1.9%)
700 (1.4%)
6,657 (13.1%)
8,956 (17.6%)
White
711 (1.4%)
1,086 (2.1%)
706 (1.4%)
7,825 (15.4%)
10,328 (20.4%)
All Targets
3,530 (6.7%)
5,060 (10.0%)
3,611 (7.1%)
38,528 (75.9%)
50,729 (100%)
Appendix
Table 7: MHS corpus statistics showing the number of posts that match a specific annotator-target pair ( Ai,Tj ) over four demographic groups. Cells highlighted in green denote the ones considered under epistemic authority weighting.
Target
Annotator Group
All Annotators
Group
Asian
Black
Latinx
White
Unmarked
33,140 (15.3%)
56,580 (26.1%)
19,500 (9.0%)
107,480 (49.6%)
216,700 (100%)
Appendix
Table 8: KET statistics showing posts that match a specific annotator ( Ai ) over four demographic groups.
Figure 6: Target-group detection model. The final layer has ∣T∣ nodes, each containing a classification layer.
Detected Target Group
Asian
Black
Latinx
White
Annotated Target Group
Asian
5,989
151
107
1,147
Black
242
20,924
247
2,638
Latinx
121
173
7,344
1,318
White
687
985
703
7,953
Appendix
Table 9: Confusion matrix (counts) for MHS between using the target-group detection model.
Metric
Level
Joint-AWG
Joint-AG
TextAug
CondMTL
F1 ( ↑ )
Micro
0.67 ± 0.01
0.63 ± 0.02
0.56 ± 0.02
0.52 ± 0.02
Macro
0.70 ± 0.02
0.66 ± 0.01
0.57 ± 0.02
0.50 ± 0.01
Precision ( ↑ )
Micro
0.75 ± 0.02
0.70 ± 0.01
0.66 ± 0.03
0.62 ± 0.02
Macro
0.78 ± 0.01
0.74 ± 0.02
0.66 ± 0.03
0.60 ± 0.02
Recall ( ↑ )
Micro
0.61 ± 0.01
0.58 ± 0.02
0.48 ± 0.02
0.45 ± 0.01
Macro
0.63 ± 0.02
0.60 ± 0.02
0.50 ± 0.02
0.43 ± 0.02
Appendix
Table 10: Performance on MHS in the Annotated-Target Scenario using RoBERTa-base as text encoder. Mean model performance and deviations are reported over 5 runs. Best performance is highlighted in bold .
Figure 7: Synthetic experiment on scalability. Every model has a fixed frozen (non-trainable) DistilBERT representation as their minimum memory. SAJ MTL’s quadratic scaling nature with increasing groups is evident, while the Scalable variant is much more memory efficient due to having linear scaling with groups.
When humans label subjective content, they disagree, and that disagreement is not noise. It reflects genuine differences in perspective shaped by annotators' social identities and lived experiences. Yet standard practice still flattens these judgments into a single majority label, and recent LLM-based approaches fare no better: we show that prompted large language models, even with chain-of-thought reasoning, fail to recover the structure of human disagreement. We introduce DiADEM, a neural architecture that learns "how much each demographic axis matters" for predicting who will disagree and on what. DiADEM encodes annotators through per-demographic projections governed by a learned importance vector α, fuses annotator and item representations via complementary concatenation and Hadamard interactions, and is trained with a novel item-level disagreement loss that directly penalizes mispredicted annotation variance. On the DICES conversational-safety and VOICED political-offense benchmarks, DiADEM substantially outperforms both the LLM-as-a-judge and neural model baselines across standard and perspectivist metrics, achieving strong disagreement tracking (r=0.75 on DICES). The learned α weights reveal that race and age consistently emerge as the most influential demographic factors driving annotator disagreement across both datasets. Our results demonstrate that explicitly modeling who annotators are not just what they label is essential for NLP systems that aim to faithfully represent human interpretive diversity.
Samay U. Shetty, Tharindu Cyril Weerasooriya, Deepak Pandita +1
Demographic information is often used to model annotator perspectives in subjective tasks such as hate speech detection, but its benefit is inconsistent: it improves performance in some settings and behaves as noise in others. This paper asks when demographic features help. We analyze demographic gain as a function of both data split properties and modeling frameworks. For data splits, we measure annotator disagreement, namely how often annotators assign different labels to the same example, along with training size and train-test demographic coverage. We find that demographic gains concentrate in regimes with low training disagreement, high test disagreement, fine-grained ambiguity measurement, sufficient training data, and greater demographic overlap. Motivated by these regimes, we introduce a gated demographic residual model that treats demographics as a selective adjustment to text-only predictions. Experiments on MHS and POPQUORN show that this design is effective, especially on high disagreement or low confidence examples. Overall, our results suggest that demographics should not be assumed useful by default; their value depends jointly on the data regime and the modeling framework.
Humans often hold different perspectives on the same issues. In many NLP tasks, annotation disagreement can reflect valid subjective perspectives. Modeling annotator perspectives and understanding their relationship with other human factors, such as socio-demographic attributes, have received increasing attention. Prior work typically focuses on single demographic factors or limited combinations. However, in real-world settings, annotator perspectives are shaped by complex social contexts, and finer-grained socio-demographic attributes can better explain human perspectives. In this work, we propose Socio-Contrastive Learning, a method that jointly models annotator perspectives while learning socio-demographic representations. Our method provides an effective approach for the fusion of socio-demographic features and textual representations to predict annotator perspectives, outperforming standard concatenation-based methods. The learned representations further enable analysis and visualization of how demographic factors relate to variation in annotator perspectives. Our code is available at GitHub: https://github.com/Leixin-Zhang/Socio_Contrastive_Learning