cs.CLFeb 18, 2026
SaveThe Validity of Coreference-based Evaluations of Natural Language Understanding
Organizations: School of Computer Science, McGill University, Montreal, Quebec
Abstract
In this thesis, I refine our understanding as to what conclusions we can reach from coreference-based evaluations by expanding existing evaluation practices and considering the extent to which evaluation results are either converging or conflicting. First, I analyze standard coreference evaluations and show that their design often leads to non-generalizable conclusions due to issues of measurement validity - including contestedness (multiple, competing definitions of coreference) and convergent validity (evaluation results that rank models differently across benchmarks). Second, I propose and implement a novel evaluation focused on testing systems' ability to infer the relative plausibility of events, a key aspect of resolving coreference. Through this extended evaluation, I find that contemporary language models demonstrate strong performance on standard benchmarks - improving over earlier baseline systems within certain domains and types of coreference - but remain sensitive to the evaluation conditions: they often fail to generalize in ways one would expect a human to be capable of when evaluation contexts are slightly modified. Taken together, these findings clarify both the strengths, such as improved accuracy over baselines on widely used evaluations, and the limitations of the current NLP paradigm, including weaknesses in measurement validity, and suggest directions for future work in developing better evaluation methods and more genuinely generalizable systems.
Figures & tables
Figure 1.1: The evolution of cuneiform written characters. Reproduced from Maspero (1916) (Creative Commons Attribution-Share Alike 4.0 International license).
Figure 1.2: Demonstrative examples of coreferences annotated in two popular datasets: OntoNotes ( Weischedel et al., 2011 ) and Winogrande ( Sakaguchi et al., 2020 ) . Square brackets mark coreferring linguistic expressions, and subscripts indicate the entities to which the expressions refer.
Figure 1.3: A visualized outline of the structure of this thesis. The body of the thesis focuses on empirical experiments. Certain experiments concern either a) canonical corpus evaluations or b) targeted evaluations of an important aspect of resolving coreference, inferring event plausibility , as outlined in Section 1.1 .
Figure 2.1: The SHRDLU program proposed by Terry Winograd which was designed to interact with a virtual world of 3D objects based on natural language instructions. On the left is a visualization of the virtual world of objects, and on the right an example interaction between a person and the program. Figure reproduced from Winograd (1972) . (M.I.T. Project MAC, Defense Technical Information Center, Public Domain)
Figure 2.2: An illustration of how model performance is most commonly evaluated using the Winogrande dataset ( Sakaguchi et al., 2020 ) as an example. For each instance in the dataset, predicted coreferences are elicited from the model being evaluated and compared against coreferences annotated by a human annotator. Accuracy is then calculated across all examples in the test set. (Certain details regarding data formatting are simplified for illustrative purposes. The spelling of “beginner” is as this example appears in the original dataset.)
Figure 2.3: A demonstrative example of an annotated clustering of linguistic expressions (“truth”) and a system-predicted clustering (“system response”). Each circle represents a referring linguistic expression, and circles in the same chain are considered coreferring as annotated (truth) or predicted (system response). Reproduced from Vilain et al. (1995) . (Creative Commons Attribution-NonCommercial-ShareAlike 3.0 International License)
Figure 2.4: A visualization of the model size needed to achieve human baseline accuracy on the Winogrande test set as estimated by Li et al. (2022) . The area of the circle on the left represents the total number parameters of the best performing open-weights language model as of 2024, LLama 3.1 405 billion ( Dubey et al., 2024 ) . Each parameter in this model is a 16-bit precision floating point decimal. The total file size of the model is 800 GB, and the model was trained using a cluster of 16,000 H100 Nvidia GPUs. By comparison, the area of the circle on the right represents the relative number of parameters to which a Gopher model is estimated to need to be scaled to achieve baseline human accuracy on the Winogrande test set (2 quadrillion parameters).
Figure 2.5: Based on an original figure from Adcock & Collier (2001) , levels and tasks of measurement. A background concept is conceptualized as a systematized concept which is measured based on the scores of indicators. In turn, these scores contribute to refining our understanding of the systematized and background concepts.
Figure 2.6: A graphical model representing a simple MM with two indicators and independent error. Inspired based on an example from Bollen (1989) .
Figure 2.7: An illustration of the Transformer model architecture proposed by Vaswani et al. (2017) . Reproduced from Jia (2019) . (Creative Commons Attribution-Share Alike 3.0 Unported license)
Figure 2.8: An example semantic hierarchy. Reproduced from Li & Abe (1998) . (Creative Commons Attribution-NonCommercial-ShareAlike 3.0 International License)
Figure 2.9: Example semantic plausibility scores and the maximizing class. Reproduced from Resnik (1997b) . (Creative Commons Attribution-NonCommercial-ShareAlike 3.0 International License)
Figure 3.1: Left : An example minimal pair from the WSC. Right : Pronouns attested in the novel Pride and Prejudice and annotated for coreference by Vala et al. (2016) .
Figure 3.2: An example instance and the corresponding variables: the pronoun , antecedent , and distractor candidate .
Figure 3.3: A training set instance from the Definite Pronoun Resolution (DPR) dataset ( Rahman & Ng, 2012 ) formatted using each of the corresponding prompts. Denoted in bold is the expected model output. The GPT-3 prompt ( Brown et al., 2020b ) does not rely on gold mention span annotations. QA Prompt and Doc Prompt were presented by Le & Ritter (2023) . The multiple-choice QA (MQA) prompt was presented by Zhu et al. (2024) .
Figure 3.4: A comparison of the rule-based dcoref system ( Lee et al., 2013 ) and the Llama 3.1 8B base model prompted for PCR using various prompts. A) Systems that do not need gold mention spans. Across datasets, Llama 3.1 with the GPT-3 prompt always outperforms the dcoref baseline. B) Systems that require gold mention spans as input. In general, prompted Llama 3.1 is more accurate than dcoref on both attested and constructed instances.
Figure 3.5: Accuracies of various LMs using the QA prompt template as compared against a dcoref baseline. We find that LMs generally outperform dcoref on both attested and constructed instances.
Figure 3.6: A comparison of the accuracy of Llama 3.1 70B instruct (32-shot) against the supervised Maverick coreference resolution system. We observe that the challenge set assumption does not hold; that is, despite being generally more accurate on WSC-like datasets, the prompted LM is less accurate on datasets of pronominal expressions attested in natural corpora. From left to right, the first six datasets consist of attested examples, and the remaining five are WSC-like challenge sets.
| ARRAU | |
| Llama 3.1 70B (QA Prompt) | 0.86 |
| + gold gender | 0.87 |
| + gold number | 0.85 |
| + gold enamex type | 0.86 |
| + distance between mentions | 0.84 |
| + gold label (oracle) | 0.99 |
Table 3.1: Results including additional features in the model input on the ARRAU validation set.
| System | OntoGUM | PreCo | WSC 273 |
| Maverick | 0.90 | 0.94 | 0.71 |
| Llama 3.1 70B | 0.84 | 0.88 | 0.82 |
| Ensemble | 0.90 | 0.95 | 0.84 |
Table 3.2: Accuracy of the ensemble method compared against Maverick (supervised coreference resolution) and prompted Llama 3.1 70B instruct.
Figure 3.7: An instance from the OntoNotes dataset.
Figure 3.8: An instance from the OntoGUM dataset.
Figure 3.9: An instance from the PreCo dataset.
Figure 3.10: An instance from the ARRAU dataset.
Figure 3.11: An instance from the GAP dataset.
Figure 3.12: An instance from the KnowRef-60K dataset.
Figure 3.13: An instance from the DPR dataset.
Figure 3.14: An instance from the Pronominal Winogrande (P-WG) dataset.
| Dataset | Example |
| OntoNotes | Highway officials insist the ornamental railings on older bridges aren’t strong enough to prevent vehicles from crashing through. But other people don’t want to lose the bridges’ beautiful, sometimes historic, features. |
| PreCo | “Melting permafrost can also destroy trees and forests,” Weller said. “When holes in the ground form, trees fall into them and die.” |
| Phrase Detectives | Within 1942-1944 bicycles were also added to regimental equipment pools. … Although seeing heavy use in World War I, bicycles were largely superseded by motorized transport in more modern armies. |
Table 4.1: Examples of annotated coreference (highlighted) where at least one expression could be interpreted as generic. Generic expressions tend to be annotated differently across datasets in part due to varying definitions of coreference.
| Dataset | Train | Dev. | Test | Total Words (K) |
| OntoNotes | 2,802 | 343 | 348 | 1,632 |
| PreCo | 36,120 | 500 | 500 | 12,493 |
| Phrase Det. | 695 | 45 | 45 | 1,321 |
| OntoGUM | 165 | 24 | 24 | 204 |
| LitBank | 80 | 10 | 10 | 211 |
| ARRAU | 444 | 33 | 75 | 348 |
Table 4.2: Number of documents for each dataset split.
| Type | OntoNotes | PreCo | Phrase Detectives |
| Generic Mentions [dogs] can bark | Generics mentions are only annotated when they corefer with a pronoun or determinate noun phrase, or when they occur in a news headline. | All generic noun phrases and modifiers can be annotated as coreferring. | All generic noun phrases can be annotated as coreferring. |
| Verb Phrases it will [grow] | The head of a verb phrase can be annotated as coreferring with a determinant noun phrase. | Not annotated. | Not annotated. |
| Appositives [[Abe] , [the chef]] | Annotated in the dataset, but not considered coreference. | Annotated as three mentions: both noun phrases and the larger span. | Annotated in the dataset, but not considered coreference. |
| Copular Predicates [he] is [the teacher] | Not annotated. | In a copular structure, the referent and attribute are annotated as coreferring. | Annotated in the dataset, but not considered coreference. |
| Nesting [he [himself]] | When two nested mentions share a head, only the dominant mention is annotated. Proper nouns cannot contain nested mentions. | Appositives and mentions with shared heads are annotated as nested mentions. | The right-most mention in an appositive is considered referring to a distinct entity and can therefore be annotated as a nested mention. No restrictions on nesting of proper nouns. |
| Compound Modifiers [Taiwan] authorities | Compound modifiers are annotated if non-adjective proper nouns that are not a nationality acronym. | All compound modifiers can be annotated as coreferring. | No explicit restrictions on the annotation of compound modifiers. |
Table 4.3: Noted differences in how coreference is defined and operationalized in the training datasets.
Table 4.4: Aggregated and disaggregated metrics intended to measure model performance. Scores are correlated with differences in how coreference is defined and operationalized across datasets, indicating that purported measurements of out-of-domain generalization also encompass these differences between datasets. Each column corresponds to a model trained on the respective training set: OntoNotes (ON), PreCo (PC), or Phrase Detectives (PD). For LinkAppend ( Bohnet et al., 2023 ) , we use the publicly released weights. For LingMess ( Otmazgin et al., 2022 ) , we train the model on each of the three training sets. Each row corresponds to the specified type of coreference. F1 scores are highlighted for types of coreference where the F1 score dropped significantly out-of-domain as compared to the overall difference in aggregate scores. “# Ments.” refers to the number of annotated mentions.
| LingMess | ||||
| Eval. | ON | PC | PD | # Ments. |
| PreCo (C) | 64.5 | 77.1 | 50.8 | |
| PreCo (B 3 ) | 64.2 | 76.3 | 47.3 | 25,983 |
| Nested | 11.6 | 55.9 | 0.5 | 1,760 |
| ON Generic | 12.0 | 70.5 | 50.9 | 3,520 |
| Compound | 5.0 | 77.1 | 0.0 | 2,163 |
Table 4.5: Disaggregated F1 scores using the same setup as Table 4.4 , as evaluated on additional test sets. F1 scores are highlighted when the difference relative to the in-domain model is significantly larger than the same difference calculated for all mentions. C indicates CoNLL F1 score.
| Model | Language Model | Prior | Ours |
| C2F | BERT-large | 76.9 | 77.1 |
| S2E | Longformer-large | 80.3 | 80.4 |
| WLC | RoBERTa-large | 81.0 | 81.0 |
| LingMess | Longformer-large | 81.4 | 81.4 |
| Link-Append | mT5-XXL | 83.3 | 83.3 |
Table 5.1: Test set performance on OntoNotes (CoNLL-2012) as reported in prior work (“prior”) and for our reimplementation of the model (“ours”). For each model, we use the best configuration of the respective prior work.
| DeBERTa-base (138M) | |||||
| Method | ON | OG | GAP | Mem. | Time |
| C2F | 77.3 | 63.5 | 77.4 | 6.2 | 71.6 |
| S2E | 78.6 | 63.8 | 78.9 | 1.3 | 20.9 |
| WLC | 79.1 | 64.1 | 78.7 | 1.4 | 25.9 |
| LingMess | 79.5 | 64.3 | 79.4 | 1.7 | 54.7 |
| DeBERTa-large (405M) | |||||
Table 5.2: Dev. set performance using a DeBERTa encoder of the given size. ON and OG are evaluated by CoNLL F1 score, GAP by F1 score. Mem. is max memory at batch size of one (GB), and time is inference speed in ms/doc at max batch size, both calculated w.r.t. inference on ON using a single 80GB A100 GPU.
| Model | LM | Size | ON | OG | GAP | Mem. | Time |
| C2F | DeBERTa | 452M | 81.2 † | 66.9 | 78.1 | 8.7 | 129.3 |
| S2E | 465M | 80.5 | 65.9 † | 79.9 | 2.7 | 45.3 | |
| WLC | 664M | 81.1 † | 65.8 † | 78.8 | 2.8 | 47.4 | |
| LingMess | 561M | 81.7 | 66.4 | 79.4 | 3.1 | 81.6 | |
| Link-Append | mT5 | 582M | 66.2 | 45.8 | 63.3 | 5.9 | 5.2 4 |
| mT5 | 1.23B | 66.5 | 45.5 | 64.9 | 12.3 | 1.4 5 |
Table 5.3: Dev. set accuracy when controlling for all factors. ON and OG are evaluated by CoNLL F1 score, GAP by F1 score. Mem. is the max memory usage for inference at batch size one (GB), and time is inference speed (ms/doc) at max batch size, both w.r.t. ON inference using a single 80 GB A100 GPU. F1 scores without are statistically significantly different from all other scores based on a permutation test with and 10k permutations following Chinchor (1995) .
Figure 5.1: OntoNotes (ON) test set CoNLL F1 score for CR models trained with language models at multiple scales. Link-Append is trained with mT5, ASP with Flan-T5, and LingMess with DeBERTa. ASP scores, as well as the two largest Link-Append model scores, were reported by the respective authors. Bohnet (2023) also noted that Link-Append has an unexpected, apparently “emergent”, scaling.
Figure 5.2: An example model output on the OntoNotes development set. “Gold” spans denote the ground-truth annotations, and “pred” spans denote predictions by the Link-Append mT5-XXL model.
Figure 6.1: Elements in the matrix are the relative plausibility score for the event “an [X] knits a [Y]” as output by a RoBERTa model fine-tuned to model plausibility. [X] and [Y] correspond to the label of the row and column, respectively. Model scores are inconsistent with respect to the two events shown on the right.
Figure 6.2: An attested event is necessarily plausible in the world, but not all plausible events are attested. By “the world” we refer to some possible world under consideration—in this sense, plausibility is an epistemic modality.
Figure 6.3: Left: The general formulation of ConceptInject ; this model takes as input an event and the full hypernym chains of each argument. Right: ConceptMax , which calculates a plausibility score for each abstraction of an event using RoBERTa, and then takes the ultimate output to be the maximum of these scores. The symbol represents an element-wise sigmoid function.
| animal-eat-seed | animal-eat-area |
| passenger-ride-bus | bus-ride-bus |
| fan-throw-fruit | group-throw-number |
| woman-seek-shelter | line-seek-issue |
Table 6.1: Training examples extracted from Wikipedia. Event is an attested event taken to be more plausible than its random perturbation .
| Topic | Question | Answer |
| cat | Does it lay eggs? | never |
| carrot | Can you eat it? | always |
| cocoon | Can it change shape? | sometimes |
| clock | Can I touch it? | always |
Table 6.2: Example triples from the 20 Questions commonsense dataset. These are those specific examples that contain a simple question with a single s-v-o triple and no modifiers.
| Pep-3k | chef-bake-cookie | ✓ |
| dog-close-door | ✓ | |
| fish-throw-elephant | ✗ | |
| marker-fuse-house | ✗ | |
| 20Q | whale-breathe-air | ✓ |
| wolf-wear-collar | ✓ | |
| cat-hatch-egg | ✗ |
Table 6.3: Representative examples taken from the validation splits of the two plausibility evaluation datasets, Pep-3k and 20Q. For simplicity, we present human judgments as plausible ( ✓ ) or implausible ( ✗ ). Details are provided in § 6.6 .
| Model | Pep-3k | 20Q | Avg. |
| n-gram | .51 | .52 | .52 |
| GloVe+MLP | .55 | .52 | .53 |
| RoBERTa Zero-shot | .56 | .57 | .56 |
| RoBERTa | .64 | .67 | .66 |
| ConceptInject | .64 | .66 | .65 |
| ConceptMax | .67 | .74 | .70 |
Table 6.4: Test set results for predicting human plausibility judgements. Performance is evaluated with AUC with respect to the ground-truth, manually labeled plausibility ratings.
Figure 6.4: Outputs across conceptual abstractions for the event kid-like-marmalade from the 20Q dataset. This event is taken to be relatively plausible as the ground-truth label was “usually.”
| Pep-3k | 20Q | |||
| Model | CC | LER | CC | LER |
| n-gram | .06 | .50 | .07 | .50 |
| GloVe+MLP | .03 | .61 | .03 | .49 |
| RoBERTa Zero-shot | .13 | .70 | .12 | .65 |
| RoBERTa | .09 | .52 | .08 | .51 |
| ConceptInject | .08 | .52 | .07 | .51 |
Table 6.5: Evaluation of inconsistency. CC describes the degree to which sequences of estimates across a hypernym chain diverge from a concave sequence. LER describes how often conceptual abstractions are local extrema with respect to plausibility.
Figure 7.1: An example instance of an Abstract Meaning Representation (AMR) including annotated coreference relations. The original text for which this AMR was built is “ Maybe I can come back from this, who knows. I’ve got the most amazing university offers, but I can’t even accept them - I’ll never make the grades. ” Reproduced from Anikina et al. (2020) . (Creative Commons Attribution 4.0 International License)