Organizations: Saarland University, Saarland Informatics Campus, Germany · Utrecht University, Utrecht, The Netherlands · Queen Mary University of London, London, UK
There is growing recognition that many NLP tasks lack a single ground truth, as human judgments reflect diverse perspectives. To capture this variation, models have been developed to predict full annotation distributions rather than majority labels, while perspectivist models aim to reproduce the interpretations of individual annotators. In this work, we compare these approaches on Implicit Discourse Relation Recognition (IDRR), a highly ambiguous task where disagreement often arises from cognitive complexity rather than ideological bias. Our experiments show that existing annotator-specific models perform poorly in IDRR unless ambiguity is reduced, whereas models trained on label distributions yield more stable predictions. Further analysis indicates that frequent cognitively demanding cases drive inconsistency in human interpretation, posing challenges for perspectivist modeling in IDRR.
Figures & tables
Col
DG1.5
MDA
HSB
GHC
Train
4.5/52k
6.59/33k
0.78/4.7k
22/69k
Dev
0.6/7k
-
-
-
Test
1.2/14k
3/11k
0.16/1k
5.5/17k
#W
164
334
6
18
#A/#W
448
160
1120
4807
#A/#I
10
5
6
3.13
Table 1: Statistics of DiscoGeM 1.5 compared to Multi-Domain Agreement, MDA Leonardelli et al. (2021) ; Hate Speech-Brexit, HSB Akhtar et al. (2021) ; and GHC Kennedy et al. (2022) ; counts reported as items/annotations. #W: worker count; #A/#W: average annotation/worker ratio; #A/#I: average annotation/instance; #L: unique label count.
F1 Lv1
Acc Lv1
F1 Lv2
Acc Lv2
ST.top1
0.48
0.58
0.24
0.44
multi.top1
0.47
0.57
0.21
0.29
dist.top1
0.49
0.6
0.28 *
0.47
MT.maj
0.36
0.39
0.03
0.03
AE.maj
0.56 *
0.7 *
0.21
0.4
Table 2: Results of the single label prediction task evaluated by accuracy and macro-F1. Performance averaged per 3 runs. The highest score shown in bold. * marks significant advantage over ST, p<0.05 . Here and further on, significance is measured with a resampled paired t-test Dietterich (1998) .
Lv1
CE
JSD
MD
ED
ST.logit
1.708
0.748
0.570
0.566
multi.logit
1.520*
0.153*
0.630
0.356*
label-dist
1.341 *
0.118 *
0.547*
0.304*
AE.logit
1.402*
0.123*
0.367 *
0.216 *
Lv2
CE
JSD
MD
ED
ST.logit
3.071
0.371
1.126
0.551
Table 3: Results of the label distribution prediction task evaluated by 4 soft metrics. Results based on 3 random seeds. The lowest score shown in bold. * marks significant advantage over ST, p<0.05 .
F1 Lv1
Acc Lv1
F1 Lv2
Acc Lv2
ST.top1
0.35
0.44
0.15
0.28
multi.top1
0.35
0.44
0.14
0.20
dist.top1
0.35
0.46*
0.17*
0.29*
MT
0.33
0.34
0.08
0.10
AE.maj
0.37*
0.50*
0.11
0.21
AE
0.75 *
0.84 *
0.23 *
0.44 *
Table 4: Results of the annotator-specific label prediction task evaluated by accuracy and macro-F1, considering each annotator-specific annotation as one instance. Performance averaged per 3 runs. The highest score shown in bold. * marks significant advantage over ST, p<0.05 .
F1 Lv1
Acc Lv1
F1 Lv2
Acc Lv2
AE #E+
0.75
0.84
0.23
0.44
AE #E-
0.76
0.85
0.23
0.43
AE Low-κ
0.68
0.81
0.14
0.25
AE Med.-κ
0.74
0.83
0.22
0.46
AE High-κ
0.76
0.85
0.24
0.44
Table 5: Results of the worker-specific label prediction task in different subsets of workers. Performance averaged per 10 runs. The highest scores shown in bold.
Figure 1: Mean nPMI values between the worker and majority labels for the three clusters of workers with different levels of agreement with the majority: Cluster 0 (low agreement), Cluster 1 (medium agreement), Cluster 2 (high agreement).
Figure 2: Bias of four workers compared with the majority label based on nPMI. darker colors mean more divergence from the majority, where blue means higher tendency (positive nPMI) and red means lower tendency (negative nPMI). Abbreviations: temporal, contingency, comparison, expansion, no relation.
Figure 3: Confusion matrices of worker-specific prediction of the above workers. Gold refers to the original labels. Abbreviations: temporal, contingency, comparison, expansion, no relation.
agreement
35%
disagreement
inherent ambiguity
31%
cognitive demands:
unidentified cues
5%
sentence complexity
10%
annotation error
11%
Table 6: Distribution of disagreement categories in 300 annotations ( 100 items ×3 workers)
top1
top2
top3
X
conj.( 20% )
asynch.( 13% )
inst. ( 10% )
Y
lev-of-det ( 22% )
cond. ( 11% )
synch. ( 9% )
Z
cause ( 23% )
lev-of-det ( 11% )
goal ( 9% )
Table 7: Top 3 annotated labels by Annotators X, Y and Z in the 100 analyzed items. Abbreviations: conjunction, asychronous, instantiation, condition, sychronous.