A Scalable Multi Task Learning Framework for Modeling Demographic Disagreements in Annotation Tasks
Authors: Soumyajit Gupta, Maria De-Arteaga, Matthew Lease
Organizations: Computer Science, School of Computing, University of Texas at Austin · Universitat Ramon Llull, ESADE · Information, School of Computing, University of Texas at Austin
Annotators from different demographic backgrounds often disagree in systematic ways due to varying lived experiences, cultural contexts, and their relationship to the content. When such disagreement occurs, one group's perspective may sometimes warrant prioritization because they hold an epistemically advantaged standpoint or bear greater stake in the outcome. For example, when assessing toxicity of posts on social media, it may be important to prioritize a targeted group's viewpoint. We propose a Stakeholder-Aware Joint Multi-Task Learning (SAJ-MTL) framework to jointly model demographic disagreement both across- and within- annotator groups. The model uses conditional gradient routing to handle sparsity induced by group-conditional labeling, and parameterization supports customized weighting of groups in different task settings. SAJ-MTL improves group-level performance in toxicity detection through better context-aware understanding of demographic-sensitive content. Only annotator group-membership information is required, which is more practical than existing methods requiring more privacy-invasive individual-level annotator details. Our scalable architecture is also computationally efficient to support large-scale deployment.
Figures & tables
Figure 1: Naive All-Pairs SAJ-MTL. Each annotator-target pair (a,t) is modeled as a separate task and dedicated head, allowing direct learning of pairwise interactions but yielding multiplicative parameter growth and data sparsity for underrepresented pairs.
Figure 2: Scalable SAJ-MTL. Annotator and target effects are factorized into independent modules and combined additively. This enables efficient modeling of annotator–target interactions while reducing complexity to linear scaling, improving both parameter efficiency and generalization in sparse settings.
Figure 3: Overview of the SAJ-MTL architecture . Input vector x is encoded using a shared encoder hϕ(x) . Annotator-group and target-group embeddings (ea,et) are projected via lightweight linear modules (Wa,Wt) and fused additively with the shared representation to form the conditional representation z(x,a,t) . Two parallel prediction heads, annotator-specific head ga and target-specific head ga , map this representation to logits that are summed to produce the annotator–target–conditioned prediction fθ(y∣x,a,t) . During training, gradients update only the shared encoder and the annotator/target modules corresponding to the active pair (a,t) . At inference, predictions may be aggregated using viewpoint prioritization weights π(a,t) enabling socially grounded or policy-driven weighting of perspectives (see Section 3.6 ).
Figure 4: TextAug Pipeline. The input text is internally predicted for toxicity based on the annotator group. An additional encoder is then used to predict conditioned on the target group as well.
Metric
Level
Joint-AWG
Joint-AG
TextAug
CondMTL
F1 ( ↑ )
Micro
0.64 ± 0.02
0.61 ± 0.01
0.52 ± 0.03
0.50 ± 0.02
Macro
0.65 ± 0.01
0.64 ± 0.02
0.55 ± 0.03
0.49 ± 0.01
Precision ( ↑ )
Micro
0.71 ± 0.01
0.67 ± 0.02
0.63 ± 0.03
0.60 ± 0.02
Macro
0.75 ± 0.02
0.73 ± 0.01
0.66 ± 0.02
0.59 ± 0.01
Recall ( ↑ )
Micro
0.59 ± 0.02
0.57 ± 0.01
0.45 ± 0.03
0.43 ± 0.02
Macro
0.59 ± 0.01
0.58 ± 0.02
0.47 ± 0.02
0.42 ± 0.02
Table 1: Prediction results on MHS for the Annotated-Target scenario. Mean performance and standard deviation are reported over 5 runs. The top results are bolded. SAJ-MTL with Joint-AWG labeling is best.
Figure 5: F1 scores over Annotator - Target groups. Both of our Joint-AWG and Joint-AG models outperform the baseline models by better capturing the interaction between annotator and target demographics.
Model
Recall
Max.
Asian
Black
Latinx
White
EO Gap ( ↓ )
Joint-AWG
0.53 ± 0.01
0.70 ± 0.01
0.55 ± 0.01
0.57 ± 0.01
0.17 ± 0.01
Joint-AG
0.53 ± 0.01
0.70 ± 0.01
0.55 ± 0.01
0.54 ± 0.01
0.17 ± 0.01
TextAug
0.42 ± 0.03
0.64 ± 0.02
0.41 ± 0.03
0.41 ± 0.03
0.23 ± 0.03
CondMTL
0.32 ± 0.02
0.61 ± 0.01
0.36 ± 0.02
0.40 ± 0.02
0.29 ± 0.02
Table 2: Per-group recall (higher is better) for each demographic group, along with the corresponding Maximum Equal Opportunity (EO) Gap ( lower is better ). Max. EO Gap is the difference between the maximum and minimum recall across demographic groups. Both Joint-AWG and Joint-AG achieve the lowest EO Gap, indicating more equitable performance across demographic groups compared to other models.
Dataset
MHS
KET
Metric
Level
Joint-AWG
Joint-AG
TextAug
CondMTL
Joint-AWG
Joint-AG
TextAug
CondMTL
F1 ( ↑ )
Micro
0.58 ± 0.02
0.56 ± 0.02
0.48 ± 0.04
0.45 ± 0.03
0.75 ± 0.03
0.72 ± 0.04
0.65 ± 0.02
0.62 ± 0.03
Macro
0.59 ± 0.01
0.57 ± 0.02
0.49 ± 0.03
0.45 ± 0.02
0.71 ± 0.02
0.68 ± 0.03
0.60 ± 0.04
0.56 ± 0.03
Precision ( ↑ )
Micro
0.68 ± 0.02
0.66 ± 0.01
0.58 ± 0.02
0.53 ± 0.01
0.77 ± 0.03
0.74 ± 0.03
0.68 ± 0.01
0.65 ± 0.04
Macro
0.73 ± 0.01
0.71 ± 0.01
0.58 ± 0.01
0.55 ± 0.01
0.71 ± 0.02
0.69 ± 0.03
0.63 ± 0.03
0.58 ± 0.05
Recall ( ↑ )
Micro
0.51 ± 0.02
0.49 ± 0.02
0.42 ± 0.04
0.41 ± 0.03
0.74 ± 0.04
0.71 ± 0.03
0.62 ± 0.03
0.59 ± 0.04
Table 3: Predictive performance on datasets for the Detected-Target scenario, averaged over five runs.
Model
Recall
Max.
Asian
Black
Latinx
White
EO Gap ( ↓ )
Joint-AWG
0.49 ± 0.02
0.64 ± 0.02
0.44 ± 0.02
0.53 ± 0.02
0.20 ± 0.02
Joint-AG
0.49 ± 0.02
0.64 ± 0.02
0.44 ± 0.02
0.50 ± 0.01
0.20 ± 0.02
TextAug
0.38 ± 0.03
0.60 ± 0.02
0.35 ± 0.03
0.40 ± 0.02
0.25 ± 0.03
CondMTL
0.30 ± 0.02
0.60 ± 0.02
0.30 ± 0.02
0.53 ± 0.02
0.30 ± 0.01
Table 4: Maximum EO Gap across demographic groups under noisy target prediction on MHS. Lower values indicate more equitable performance. Both Joint-AWG and Joint-AG maintain lower EO gaps than baseline methods despite using automatically predicted target identities.
Model type
# Params ( ↓ )
Δ ( ↓ )
Time(s) ( ↓ )
Δ ( ↓ )
Target Only (CondMTL)
763,653
-
2200
-
TextAug
67,075,795
+8684%
4800
+118%
Naive SAJ-MTL
763,653
+0%
2400
+9%
SAJ-MTL (Ours)
626,133
-18%
2850
+30%
Table 5: Memory (trainable parameters) and training time (10 epochs) comparison between models. We treat the Target-Only model as baseline for comparison, since it does not account for annotator views. While the model size of Target-Only and Naive SAJ-MTL are the same, SAJ-MTL takes longer time to train due to variation in data. The scalable SAJ variant is the best performing model in terms of trainable parameters.
Table 6: Stage-wise correspondence between identity-sensitive IRT formulations and SAJ-MTL.
Target
Annotator Group
All Annotators
Group
Asian
Black
Latinx
White
Asian
621 (1.2%)
721 (1.4%)
551 (1.1%)
5,501 (10.8%)
7,394 (14.6%)
Black
1,560 (3.1%)
2,292 (4.5%)
1,654 (3.3%)
18,545 (36.6%)
24,051 (47.4%)
Latinx
638 (1.3%)
961 (1.9%)
700 (1.4%)
6,657 (13.1%)
8,956 (17.6%)
White
711 (1.4%)
1,086 (2.1%)
706 (1.4%)
7,825 (15.4%)
10,328 (20.4%)
All Targets
3,530 (6.7%)
5,060 (10.0%)
3,611 (7.1%)
38,528 (75.9%)
50,729 (100%)
Appendix
Table 7: MHS corpus statistics showing the number of posts that match a specific annotator-target pair ( Ai,Tj ) over four demographic groups. Cells highlighted in green denote the ones considered under epistemic authority weighting.
Target
Annotator Group
All Annotators
Group
Asian
Black
Latinx
White
Unmarked
33,140 (15.3%)
56,580 (26.1%)
19,500 (9.0%)
107,480 (49.6%)
216,700 (100%)
Appendix
Table 8: KET statistics showing posts that match a specific annotator ( Ai ) over four demographic groups.
Figure 6: Target-group detection model. The final layer has ∣T∣ nodes, each containing a classification layer.
Detected Target Group
Asian
Black
Latinx
White
Annotated Target Group
Asian
5,989
151
107
1,147
Black
242
20,924
247
2,638
Latinx
121
173
7,344
1,318
White
687
985
703
7,953
Appendix
Table 9: Confusion matrix (counts) for MHS between using the target-group detection model.
Metric
Level
Joint-AWG
Joint-AG
TextAug
CondMTL
F1 ( ↑ )
Micro
0.67 ± 0.01
0.63 ± 0.02
0.56 ± 0.02
0.52 ± 0.02
Macro
0.70 ± 0.02
0.66 ± 0.01
0.57 ± 0.02
0.50 ± 0.01
Precision ( ↑ )
Micro
0.75 ± 0.02
0.70 ± 0.01
0.66 ± 0.03
0.62 ± 0.02
Macro
0.78 ± 0.01
0.74 ± 0.02
0.66 ± 0.03
0.60 ± 0.02
Recall ( ↑ )
Micro
0.61 ± 0.01
0.58 ± 0.02
0.48 ± 0.02
0.45 ± 0.01
Macro
0.63 ± 0.02
0.60 ± 0.02
0.50 ± 0.02
0.43 ± 0.02
Appendix
Table 10: Performance on MHS in the Annotated-Target Scenario using RoBERTa-base as text encoder. Mean model performance and deviations are reported over 5 runs. Best performance is highlighted in bold .
Figure 7: Synthetic experiment on scalability. Every model has a fixed frozen (non-trainable) DistilBERT representation as their minimum memory. SAJ MTL’s quadratic scaling nature with increasing groups is evident, while the Scalable variant is much more memory efficient due to having linear scaling with groups.