In this thesis, I refine our understanding as to what conclusions we can reach from coreference-based evaluations by expanding existing evaluation practices and considering the extent to which evaluation results are either converging or conflicting. First, I analyze standard coreference evaluations and show that their design often leads to non-generalizable conclusions due to issues of measurement validity - including contestedness (multiple, competing definitions of coreference) and convergent validity (evaluation results that rank models differently across benchmarks). Second, I propose and implement a novel evaluation focused on testing systems' ability to infer the relative plausibility of events, a key aspect of resolving coreference. Through this extended evaluation, I find that contemporary language models demonstrate strong performance on standard benchmarks - improving over earlier baseline systems within certain domains and types of coreference - but remain sensitive to the evaluation conditions: they often fail to generalize in ways one would expect a human to be capable of when evaluation contexts are slightly modified. Taken together, these findings clarify both the strengths, such as improved accuracy over baselines on widely used evaluations, and the limitations of the current NLP paradigm, including weaknesses in measurement validity, and suggest directions for future work in developing better evaluation methods and more genuinely generalizable systems.
Figures & tables
Figure 1.1: The evolution of cuneiform written characters. Reproduced from Maspero (1916) (Creative Commons Attribution-Share Alike 4.0 International license).
Figure 1.2: Demonstrative examples of coreferences annotated in two popular datasets: OntoNotes ( Weischedel et al., 2011 ) and Winogrande ( Sakaguchi et al., 2020 ) . Square brackets mark coreferring linguistic expressions, and subscripts indicate the entities to which the expressions refer.
Figure 1.3: A visualized outline of the structure of this thesis. The body of the thesis focuses on empirical experiments. Certain experiments concern either a) canonical corpus evaluations or b) targeted evaluations of an important aspect of resolving coreference, inferring event plausibility , as outlined in Section 1.1 .
Figure 2.1: The SHRDLU program proposed by Terry Winograd which was designed to interact with a virtual world of 3D objects based on natural language instructions. On the left is a visualization of the virtual world of objects, and on the right an example interaction between a person and the program. Figure reproduced from Winograd (1972) . (M.I.T. Project MAC, Defense Technical Information Center, Public Domain)
Figure 2.2: An illustration of how model performance is most commonly evaluated using the Winogrande dataset ( Sakaguchi et al., 2020 ) as an example. For each instance in the dataset, predicted coreferences are elicited from the model being evaluated and compared against coreferences annotated by a human annotator. Accuracy is then calculated across all examples in the test set. (Certain details regarding data formatting are simplified for illustrative purposes. The spelling of “beginner” is as this example appears in the original dataset.)
Figure 2.3: A demonstrative example of an annotated clustering of linguistic expressions (“truth”) and a system-predicted clustering (“system response”). Each circle represents a referring linguistic expression, and circles in the same chain are considered coreferring as annotated (truth) or predicted (system response). Reproduced from Vilain et al. (1995) . (Creative Commons Attribution-NonCommercial-ShareAlike 3.0 International License)
Figure 2.4: A visualization of the model size needed to achieve human baseline accuracy on the Winogrande test set as estimated by Li et al. (2022) . The area of the circle on the left represents the total number parameters of the best performing open-weights language model as of 2024, LLama 3.1 405 billion ( Dubey et al., 2024 ) . Each parameter in this model is a 16-bit precision floating point decimal. The total file size of the model is ∼ 800 GB, and the model was trained using a cluster of 16,000 H100 Nvidia GPUs. By comparison, the area of the circle on the right represents the relative number of parameters to which a Gopher model is estimated to need to be scaled to achieve baseline human accuracy on the Winogrande test set (2 quadrillion parameters).
Figure 2.5: Based on an original figure from Adcock & Collier (2001) , levels and tasks of measurement. A background concept is conceptualized as a systematized concept which is measured based on the scores of indicators. In turn, these scores contribute to refining our understanding of the systematized and background concepts.
Figure 2.6: A graphical model representing a simple MM with two indicators and independent error. Inspired based on an example from Bollen (1989) .
Figure 2.7: An illustration of the Transformer model architecture proposed by Vaswani et al. (2017) . Reproduced from Jia (2019) . (Creative Commons Attribution-Share Alike 3.0 Unported license)
Figure 2.8: An example semantic hierarchy. Reproduced from Li & Abe (1998) . (Creative Commons Attribution-NonCommercial-ShareAlike 3.0 International License)
Figure 2.9: Example semantic plausibility scores and the maximizing class. Reproduced from Resnik (1997b) . (Creative Commons Attribution-NonCommercial-ShareAlike 3.0 International License)
Figure 3.1: Left : An example minimal pair from the WSC. Right : Pronouns attested in the novel Pride and Prejudice and annotated for coreference by Vala et al. (2016) .
Figure 3.2: An example instance and the corresponding variables: the pronoun x , antecedent a , and distractor candidate b .
Figure 3.3: A training set instance from the Definite Pronoun Resolution (DPR) dataset ( Rahman & Ng, 2012 ) formatted using each of the corresponding prompts. Denoted in bold is the expected model output. The GPT-3 prompt ( Brown et al., 2020b ) does not rely on gold mention span annotations. QA Prompt and Doc Prompt were presented by Le & Ritter (2023) . The multiple-choice QA (MQA) prompt was presented by Zhu et al. (2024) .
Figure 3.4: A comparison of the rule-based dcoref system ( Lee et al., 2013 ) and the Llama 3.1 8B base model prompted for PCR using various prompts. A) Systems that do not need gold mention spans. Across datasets, Llama 3.1 with the GPT-3 prompt always outperforms the dcoref baseline. B) Systems that require gold mention spans as input. In general, prompted Llama 3.1 is more accurate than dcoref on both attested and constructed instances.
Figure 3.5: Accuracies of various LMs using the QA prompt template as compared against a dcoref baseline. We find that LMs generally outperform dcoref on both attested and constructed instances.
Figure 3.6: A comparison of the accuracy of Llama 3.1 70B instruct (32-shot) against the supervised Maverick coreference resolution system. We observe that the challenge set assumption does not hold; that is, despite being generally more accurate on WSC-like datasets, the prompted LM is less accurate on datasets of pronominal expressions attested in natural corpora. From left to right, the first six datasets consist of attested examples, and the remaining five are WSC-like challenge sets.
ARRAU
Llama 3.1 70B (QA Prompt)
0.86
+ gold gender
0.87
+ gold number
0.85
+ gold enamex type
0.86
+ distance between mentions
0.84
+ gold label (oracle)
0.99
Table 3.1: Results including additional features in the model input on the ARRAU validation set.
System
OntoGUM
PreCo
WSC 273
Maverick
0.90
0.94
0.71
Llama 3.1 70B
0.84
0.88
0.82
Ensemble
0.90
0.95
0.84
Table 3.2: Accuracy of the ensemble method compared against Maverick (supervised coreference resolution) and prompted Llama 3.1 70B instruct.
Figure 3.7: An instance from the OntoNotes dataset.
Figure 3.8: An instance from the OntoGUM dataset.
Figure 3.9: An instance from the PreCo dataset.
Figure 3.10: An instance from the ARRAU dataset.
Figure 3.11: An instance from the GAP dataset.
Figure 3.12: An instance from the KnowRef-60K dataset.
Figure 3.13: An instance from the DPR dataset.
Figure 3.14: An instance from the Pronominal Winogrande (P-WG) dataset.
Dataset
Example
OntoNotes
Highway officials insist the ornamental railings on older bridges aren’t strong enough to prevent vehicles from crashing through. But other people don’t want to lose the bridges’ beautiful, sometimes historic, features.
PreCo
“Melting permafrost can also destroy trees and forests,” Weller said. “When holes in the ground form, trees fall into them and die.”
Phrase Detectives
Within 1942-1944 bicycles were also added to regimental equipment pools. … Although seeing heavy use in World War I, bicycles were largely superseded by motorized transport in more modern armies.
Table 4.1: Examples of annotated coreference (highlighted) where at least one expression could be interpreted as generic. Generic expressions tend to be annotated differently across datasets in part due to varying definitions of coreference.
Dataset
Train
Dev.
Test
Total Words (K)
OntoNotes
2,802
343
348
1,632
PreCo
36,120
500
500
12,493
Phrase Det.
695
45
45
1,321
OntoGUM
165
24
24
204
LitBank
80
10
10
211
ARRAU
444
33
75
348
Table 4.2: Number of documents for each dataset split.
Type
OntoNotes
PreCo
Phrase Detectives
Generic Mentions [dogs] can bark
Generics mentions are only annotated when they corefer with a pronoun or determinate noun phrase, or when they occur in a news headline.
All generic noun phrases and modifiers can be annotated as coreferring.
All generic noun phrases can be annotated as coreferring.
Verb Phrases it will [grow]
The head of a verb phrase can be annotated as coreferring with a determinant noun phrase.
Not annotated.
Not annotated.
Appositives [[Abe] , [the chef]]
Annotated in the dataset, but not considered coreference.
Annotated as three mentions: both noun phrases and the larger span.
Annotated in the dataset, but not considered coreference.
Copular Predicates [he] is [the teacher]
Not annotated.
In a copular structure, the referent and attribute are annotated as coreferring.
Annotated in the dataset, but not considered coreference.
Nesting [he [himself]]
When two nested mentions share a head, only the dominant mention is annotated. Proper nouns cannot contain nested mentions.
Appositives and mentions with shared heads are annotated as nested mentions.
The right-most mention in an appositive is considered referring to a distinct entity and can therefore be annotated as a nested mention. No restrictions on nesting of proper nouns.
Compound Modifiers [Taiwan] authorities
Compound modifiers are annotated if non-adjective proper nouns that are not a nationality acronym.
All compound modifiers can be annotated as coreferring.
No explicit restrictions on the annotation of compound modifiers.
Table 4.3: Noted differences in how coreference is defined and operationalized in the training datasets.
Table 4.4: Aggregated and disaggregated metrics intended to measure model performance. Scores are correlated with differences in how coreference is defined and operationalized across datasets, indicating that purported measurements of out-of-domain generalization also encompass these differences between datasets. Each column corresponds to a model trained on the respective training set: OntoNotes (ON), PreCo (PC), or Phrase Detectives (PD). For LinkAppend ( Bohnet et al., 2023 ) , we use the publicly released weights. For LingMess ( Otmazgin et al., 2022 ) , we train the model on each of the three training sets. Each row corresponds to the specified type of coreference. F1 scores are highlighted for types of coreference where the F1 score dropped significantly out-of-domain as compared to the overall difference in aggregate scores. “# Ments.” refers to the number of annotated mentions.
LingMess
Eval.
ON
PC
PD
# Ments.
PreCo (C)
64.5
77.1
50.8
PreCo (B 3 )
64.2
76.3
47.3
25,983
Nested
11.6
55.9
0.5
1,760
ON Generic
12.0
70.5
50.9
3,520
Compound
5.0
77.1
0.0
2,163
Table 4.5: Disaggregated F1 scores using the same setup as Table 4.4 , as evaluated on additional test sets. F1 scores are highlighted when the difference relative to the in-domain model is significantly larger than the same difference calculated for all mentions. C indicates CoNLL F1 score.
Model
Language Model
Prior
Ours
C2F
BERT-large
76.9
77.1
S2E
Longformer-large
80.3
80.4
WLC
RoBERTa-large
81.0
81.0
LingMess
Longformer-large
81.4
81.4
Link-Append
mT5-XXL
83.3
83.3
Table 5.1: Test set performance on OntoNotes (CoNLL-2012) as reported in prior work (“prior”) and for our reimplementation of the model (“ours”). For each model, we use the best configuration of the respective prior work.
DeBERTa-base (138M)
Method
ON
OG
GAP
Mem.
Time
C2F
77.3
63.5
77.4
6.2
71.6
S2E
78.6
63.8
78.9
1.3
20.9
WLC
79.1
64.1
78.7
1.4
25.9
LingMess
79.5
64.3
79.4
1.7
54.7
DeBERTa-large (405M)
Table 5.2: Dev. set performance using a DeBERTa encoder of the given size. ON and OG are evaluated by CoNLL F1 score, GAP by F1 score. Mem. is max memory at batch size of one (GB), and time is inference speed in ms/doc at max batch size, both calculated w.r.t. inference on ON using a single 80GB A100 GPU.
Model
LM
Size
ON
OG
GAP
Mem.
Time
C2F
DeBERTa L
452M
81.2 †
66.9
78.1
8.7
129.3
S2E
465M
80.5
65.9 †
79.9
2.7
45.3
WLC
664M
81.1 †
65.8 †
78.8
2.8
47.4
LingMess
561M
81.7
66.4
79.4
3.1
81.6
Link-Append
mT5 B
582M
66.2
45.8
63.3
5.9
5.2 e 4
mT5 L
1.23B
66.5
45.5
64.9
12.3
1.4 e 5
Table 5.3: Dev. set accuracy when controlling for all factors. ON and OG are evaluated by CoNLL F1 score, GAP by F1 score. Mem. is the max memory usage for inference at batch size one (GB), and time is inference speed (ms/doc) at max batch size, both w.r.t. ON inference using a single 80 GB A100 GPU. F1 scores without † are statistically significantly different from all other scores based on a permutation test with α=0.1 and 10k permutations following Chinchor (1995) .
Figure 5.1: OntoNotes (ON) test set CoNLL F1 score for CR models trained with language models at multiple scales. Link-Append is trained with mT5, ASP with Flan-T5, and LingMess with DeBERTa. ASP scores, as well as the two largest Link-Append model scores, were reported by the respective authors. Bohnet (2023) also noted that Link-Append has an unexpected, apparently “emergent”, scaling.
Figure 5.2: An example model output on the OntoNotes development set. “Gold” spans denote the ground-truth annotations, and “pred” spans denote predictions by the Link-Append mT5-XXL model.
Figure 6.1: Elements in the matrix are the relative plausibility score for the event “an [X] knits a [Y]” as output by a RoBERTa model fine-tuned to model plausibility. [X] and [Y] correspond to the label of the row and column, respectively. Model scores are inconsistent with respect to the two events shown on the right.
Figure 6.2: An attested event is necessarily plausible in the world, but not all plausible events are attested. By “the world” we refer to some possible world under consideration—in this sense, plausibility is an epistemic modality.
Figure 6.3: Left: The general formulation of ConceptInject ; this model takes as input an event and the full hypernym chains of each argument. Right: ConceptMax , which calculates a plausibility score for each abstraction of an event using RoBERTa, and then takes the ultimate output to be the maximum of these scores. The symbol σ represents an element-wise sigmoid function.
e
e′
animal-eat-seed
animal-eat-area
passenger-ride-bus
bus-ride-bus
fan-throw-fruit
group-throw-number
woman-seek-shelter
line-seek-issue
Table 6.1: Training examples extracted from Wikipedia. Event e is an attested event taken to be more plausible than its random perturbation e′ .
Topic
Question
Answer
cat
Does it lay eggs?
never
carrot
Can you eat it?
always
cocoon
Can it change shape?
sometimes
clock
Can I touch it?
always
Table 6.2: Example triples from the 20 Questions commonsense dataset. These are those specific examples that contain a simple question with a single s-v-o triple and no modifiers.
Pep-3k
chef-bake-cookie
✓
dog-close-door
✓
fish-throw-elephant
✗
marker-fuse-house
✗
20Q
whale-breathe-air
✓
wolf-wear-collar
✓
cat-hatch-egg
✗
Table 6.3: Representative examples taken from the validation splits of the two plausibility evaluation datasets, Pep-3k and 20Q. For simplicity, we present human judgments as plausible ( ✓ ) or implausible ( ✗ ). Details are provided in § 6.6 .
Model
Pep-3k
20Q
Avg.
n-gram
.51
.52
.52
GloVe+MLP
.55
.52
.53
RoBERTa Zero-shot
.56
.57
.56
RoBERTa
.64
.67
.66
ConceptInject
.64
.66
.65
ConceptMax
.67
.74
.70
Table 6.4: Test set results for predicting human plausibility judgements. Performance is evaluated with AUC with respect to the ground-truth, manually labeled plausibility ratings.
Figure 6.4: Outputs across conceptual abstractions for the event kid-like-marmalade from the 20Q dataset. This event is taken to be relatively plausible as the ground-truth label was “usually.”
Pep-3k
20Q
Model
CC Δ
LER
CC Δ
LER
n-gram
.06
.50
.07
.50
GloVe+MLP
.03
.61
.03
.49
RoBERTa Zero-shot
.13
.70
.12
.65
RoBERTa
.09
.52
.08
.51
ConceptInject
.08
.52
.07
.51
Table 6.5: Evaluation of inconsistency. CC Δ describes the degree to which sequences of estimates across a hypernym chain diverge from a concave sequence. LER describes how often conceptual abstractions are local extrema with respect to plausibility.
Figure 7.1: An example instance of an Abstract Meaning Representation (AMR) including annotated coreference relations. The original text for which this AMR was built is “ Maybe I can come back from this, who knows. I’ve got the most amazing university offers, but I can’t even accept them - I’ll never make the grades. ” Reproduced from Anikina et al. (2020) . (Creative Commons Attribution 4.0 International License)
Coreference resolution is typically evaluated using aggregate statistical metrics such as CoNLL-F1, which measure structural overlap between predicted and gold clusters. While widely used, these metrics offer limited diagnostic insights, penalizing errors without revealing whether a system struggles with specific semantic categories, such as people, locations, or events, and making it difficult to interpret model capabilities or derive actionable improvements. We address this gap by introducing a semantically-enhanced evaluation framework for coreference resolution. Our approach overlays Concept and Named Entity Recognition (CNER) onto coreference outputs, assigning semantic labels to nominal mentions and propagating them to entire coreference clusters. This enables the computation of typed scores aimed at evaluating mention extraction and linking capabilities stratified by semantic class. Across our experiments on OntoNotes, LitBank, and PreCo, we show that our framework uncovers systematic weaknesses that remain obscured by aggregate metrics. Furthermore, we demonstrate that these diagnostics can be used to design targeted, low-cost data augmentation strategies, achieving measurable out-of-domain improvements.
Bruno Gatti, Giuliano Martinelli, Roberto Navigli
Sapienza NLP Group, Sapienza University of Rome · 2Babelscape
Confidence estimation (CE) indicates how reliable the answers of large language models are and impacts user trust and decision-making. Existing evaluations mainly concern the alignment between confidence and correctness, but ignore the variability of language: confidence estimates should remain consistent under semantically equivalent prompts or answer variations, while changing when answer meaning differs, as this may indicate a change in correctness. Therefore, we introduce a novel evaluation framework based on three complementary properties: robustness to prompt perturbations, stability across semantically equivalent answers, and sensitivity to semantically different answers. We show that these metrics are largely independent from existing CE metrics, and that common CE methods often fail on them: while most methods achieve high robustness and stability, they struggle to distinguish semantically different answers, potentially because they do not effectively leverage generation-side information. Overall, our framework exposes overlooked limitations of current CE evaluations and provides guidance for selecting confidence estimators for real-world applications.
Yuxi Xia, Dennis Ulmer, Terra Blevins +3
Faculty of Computer Science, UniVie Doctoral School Computer Science · Faculty of Philological and Cultural Studies, University of Vienna, Austria · ILLC, University of Amsterdam, Netherlands +2
Recent advances in language models have prompted a growing body of work that questions prevailing evaluation practices. However, many such critiques have already been discussed in natural language processing (NLP), a field with a long history of methodological reflection on evaluation. We conduct a scoping review spanning four decades and develop a taxonomy of 22 recurring concerns. For each concern, we examine how recently it was discussed in the reviewed literature and whether it receives explicit attention in institutional checklists, finding that some concerns have become less visible in literature and many lack coverage in these checklists. Our work provides an overview of the major concerns in evaluation methodology that have shaped the field and identifies potential targets for renewed investigation and field-level intervention by the research community.
Ruchira Dhar, Qiwei Peng, Anders Søgaard
Department of Computer Science, University of Copenhagen