Quality over Quantity: Rethinking Data and Model Assumptions for Bengali Coreference Resolution
Authors: Zenith Biswas, Andrew Thomas Dyer
Organizations: Data Science and Artificial Intelligence, Saarland University, Saarbrücken, Germany · Language Science and Technology, Saarland University, Saarbrücken, Germany
Multilingual transformer models are often assumed to generalise across languages, but their effectiveness for coreference resolution in low-resource settings remains unverified. We investigate this gap in Bengali, a language with over 250 million speakers but limited annotated coreference resources. We evaluate transformer-based pre-trained encoders under multiple settings, including zero-shot, fine-tuning, data augmentation, and cross-lingual transfer. Our results challenge several common assumptions about model generalisation and data efficiency in low-resource NLP. We demonstrate that a small amount of high-quality annotated data significantly outperforms larger mixed-quality datasets (p < 0.01). We find that data augmentation, particularly back-translation, does not improve performance and can degrade results in coreference resolution. Cross-lingual transfer is also inconsistent: transfer from a distant language like English reduces performance, and transfer from the related language Hindi provides no consistent benefit across models. Several widely used multilingual models exhibit collapsed embedding spaces, rendering them unsuitable for similarity-based clustering tasks. These findings highlight the importance of annotation quality over data quantity and provide empirical guidance for developing coreference resolution systems in low-resource languages.
Figures & tables
Dataset
Docs
Tokens
Mentions
Quality
BenCoref (Train)
41
15,234
1,847
Gold
BenCoref (Dev)
10
4,012
458
Gold
BenCoref (Test)
71
29,364
3,183
Gold
TransMuCoRes-BN
100
177,134
25,402
Silver
TransMuCoRes-HI
80
167,692
20,225
Silver
OntoNotes (subset)
437
324,890
38,911
Gold
Table 1: Dataset statistics. BenCoref is gold-standard (human-annotated); TransMuCoRes datasets are silver-standard (automatically projected).
Configuration
Training Data
Docs
Zero-Shot
None
0
Mixed Baseline
BenCoref + TransMuCoRes-BN
141
Mixed + BT
Mixed + back-translated
282
Mixed + Para
Mixed + paraphrased
282
Hindi Transfer
TransMuCoRes-HI
80
English Transfer
OntoNotes subset
437
Table 2: Training configurations. BT = back-translation, Para = MLM paraphrasing. All configurations use the same BenCoref dev and test sets.
Model
MUC
B 3
CEAF e
CoNLL
MuRIL-Large
96.56
66.79
36.74
66.70
mBERT
96.69
67.11
23.19
62.33
RemBERT
96.20
65.67
26.51
62.80
BanglaBERT-Base
96.34
65.67
27.23
63.08
BERT-Base-Unc
95.70
66.22
27.26
63.06
Table 3: Per-metric F1 (%) under the Mixed configuration. High MUC alongside low CEAF e across all models indicates systematic over-linking, which the averaged CoNLL F1 obscures.
Model
Zero-Shot
Mixed
Mixed+BT
Mixed+Para
Hindi
English
Gold
Gold+BT
Gold+Para
MuRIL-Large
62.44
66.70
65.06
65.12
63.96
61.35
66.73
63.87
64.72
mBERT
63.61
62.33
63.11
63.86
63.01
61.76
64.15
64.18
64.34
RemBERT
63.20
62.80
63.09
63.52
62.83
61.32
63.88
63.23
63.81
BanglaBERT-Base
62.38
63.08
63.19
63.67
63.11
61.64
63.39
63.15
63.51
BERT-Base-Unc
63.82
63.06
63.11
63.27
62.71
62.26
63.64
63.28
63.06
Average
63.09
63.59
63.51
63.89
63.12
61.67
64.36
63.54
63.89
Table 4: CoNLL F1 (%) across all models and training configurations. Best result in bold. BT = back-translation, Para = MLM paraphrasing. All configurations evaluated on the same BenCoref test set (71 documents).
Figure 1: Embedding discriminability across all ten candidate models. Green bars show mean within-cluster (coreferent) similarity; red bars show mean between-cluster (non-coreferent) similarity. The gap ( Δ ) indicates discriminability. Models in the shaded region have gaps below 0.01 and are unsuitable for threshold-based coreference clustering.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Comparison
Δ F1
95% CI
p (corr.)
Gold vs. Mixed
+2.63pp
[+0.94, +4.54]
0.0070**
BT on Gold vs. Gold
− 0.81pp
[ − 1.41, − 0.25]
0.0156*
Para on Gold vs. Gold
− 0.46pp
[ − 0.97, +0.04]
0.2076
BT on Mixed vs. Mixed
+1.05pp
[+0.09, +2.06]
0.1470
Para on Mixed vs. Mixed
+0.83pp
[+0.05, +1.65]
0.1512
Hindi vs. Mixed
+0.99pp
[ − 0.24, +2.28]
0.2328
Appendix
Table 5: Statistical significance of key experimental comparisons. All differences are computed relative to the baseline listed in each row (e.g., “Hindi vs. Mixed” compares Hindi Transfer to the Mixed Baseline). Paired bootstrap with 10,000 iterations, Holm–Bonferroni corrected. * p<0.05 , ** p<0.01 .
Coreference resolution is a core NLP task, having a broad range of downstream applications, e.g.~machine translation, question answering, document summarization, etc. While the task is well-studied in English, comparatively less attention is dedicated to coreference resolution in other languages, especially low-resource ones. To mitigate this gap, we propose a novel coreference resolution pipeline that harnesses machine translation (MT) from English to a target low-resource language, to generate or expand training data. To automatically validate the quality of the translated samples, we back-translate the samples and assess the similarity with the original English samples via cosine similarity in the latent space of a BERT model. The resulting similarity scores are integrated into the loss function to weight training samples according to their MT cycle consistency. Extensive experiments on four low-resource languages show that our pipeline brings significant performance gains in coreference resolution. Moreover, our pipeline enables accurate coreference resolution in languages where no previous corpora were available.
Adriana-Valentina Costache, Eduard Poesina, Silviu-Florin Gheorghe +2
Department of Computer Science, University of Bucharest, Romania
We introduce CorPipe 26, our winning submission to the CRAC 2026 Shared Task on Multilingual Coreference Resolution. The fifth edition of this shared task focuses mainly on the comparison of generative LLMs and specialized systems; additionally, 5 more datasets and 2 new languages are introduced. CorPipe 26 is an improved version of CorPipe 25, with a new variant predicting empty nodes together with mentions and coreference links in a single model. Our system outperforms all other submissions in the LLM track by 2.8 percent points and all submissions in the unconstrained track by 9.5 percent points. Furthermore, we perform a series of ablation experiments with different model sizes, empty node prediction methods, and cross-lingual zero-shot evaluation. The source code and the trained models are publicly available at https://github.com/ufal/crac2026-corpipe.
Milan Straka
Charles University, Faculty of Mathematics and Physics Institute of Formal and Applied Linguistics Malostranské nám. 25, Prague, Czech Republic
Transformer-based models achieve state-of-the-art dependency parsing for high-resource languages, yet their advantage over simpler architectures in low-resource settings remains poorly understood. We evaluate four parsers---the Biaffine LSTM, Stack-Pointer Network, AfroXLMR-large, and RemBERT---across twelve typologically diverse languages, with a focus on low-resource African languages. We find that the Biaffine LSTM consistently outperforms transformer models in low-resource regimes, with transformers recovering their advantage as training data increases. The crossover falls within a resource range typical of treebanks for under-resourced languages. Morphological complexity (measured via MATTR) emerges as a significant secondary predictor of transformers' relative disadvantage after controlling for corpus size. These results indicate that the Biaffine LSTM may be better suited for syntactic tool development in low-resource regimes until sufficient annotated data is available to leverage the representational capacity of pre-trained transformers.