BLiMP-RO: A Benchmark of Linguistic Minimal Pairs for Romanian
Organizations: University of Bucharest · Universität Wien
Abstract
Evaluating grammatical knowledge in language models requires benchmarks that cover diverse linguistic phenomena and are validated against human judgments. We introduce BLiMP-RO, a Romanian minimal-pair benchmark covering 50 grammatical phenomena across ten domains. It comprises 493 linguist-constructed sentence pairs, evaluated by native speakers in five experiments, and 5,942 additional semi-automatically generated, human-validated pairs. We further investigate grammatical generalization under limited linguistic input by training GPT-2 and GPT-BERT models from scratch on Romanian BabyBabelLM data and a newly collected corpus of 10 million words from child-targeted video captions. Increasing caption training data yields approximately 9 percentage points of accuracy improvement per tenfold increase in corpus size. At matched data sizes, BabyBabelLM generally outperforms video captions, while GPT-BERT consistently outperforms GPT-2. Our findings highlight how training data quantity, composition, and model architecture shape grammatical generalization in Romanian.
Figures & tables
| Morphosyntactic agreement 77 subject-verb demonstrative adjective participial agreement adjectival article cel anaphor agreement (emphatic reflexive însu s , i ) doubled object and clitic ordinal numeral Differential object marking 30 pe in object relatives pe and clitic across referential noun types object marking across animacy classes Clitics 46 clitic doubling resumptive clitics in relatives and pe cine questions clitic order and the Person Case Constraint clitics with modal verbs clitics in negative imperatives | Binding theory 30 case condition (gerund against finite verb after a reflexive) C-command condition (antecedent of an emphatic reflexive) locality of reflexives Argument structure 29 transitivity passivizability inherent reflexive verbs Word order and movement 131 relativization (prepositions in relatives) demonstrative placement topicalization (doubled preverbal objects) definite article placement indefinite article placement preverbal depictives auxiliary and verb order adpositional phrases (nominal dependents) extraction from complex noun phrases extraction from adjuncts extraction from coordinations extraction from embedded questions extraction from clausal subjects | Mood 50 subjunctive-selecting predicates indicative-selecting predicates subjunctive and infinitive conditional future Negation 29 negative concord negative additive nici negative imperatives Nominal morphology 61 gender and number plural forms nominalization genitive and dative cardinal numerals with de vocative Null subjects 10 null against overt subject |
| BLiMP-RO | Multi | ||||
| Init. | Agr. | Ext. | |||
| Humans | 96 | [95.1, 96.9] | 93.1 | ||
| GPT-2 | |||||
| Cap. 1M | 65 | [60.8, 68.6] | 63.7 | 62 | 59 |
| BBL 1M | 65 | [62.5, 68.1] | 64.8 | 68 | 64 |
| Cap. 0–4 | 71 | [67.5, 74.5] | 69.7 | 70 | 66 |
| GPT-2 | GPT-BERT | Pretrained | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Humans | Pairs | Cap. 1M | BBL | Cap. 10M | Cap. 1M | BBL | Cap. 10M | RoGPT2 | RoBERT | |
| Agreement | 95.4 | 1011 | 54.7 | 55.4 | 65.2 | 58.7 | 55.7 | 67.5 | 95.5 | 97.6 |
| DOM | 90.1 | 389 | 55.2 | 65.8 | 67.1 | 68.0 | 70.6 | 73.3 | 94.1 | 95.9 |
| Clitics | 96.5 | 727 | 68.9 | 76.8 | 77.6 | 77.2 | 83.1 | 79.2 | 92.9 | 90.6 |
| Binding | 91.5 | 369 | 53.7 | 59.0 | 58.5 | 49.7 | 55.6 | 59.6 | 88.6 | 86.2 |
| Arguments | 97.6 | 383 | 56.1 | 62.5 | 68.0 | 62.9 | 67.5 | 70.0 | 92.2 | 98.4 |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Sum | Token | Word | SLOR | |
| GPT-2, left to right | ||||
| captions 1M | 53.0 | 64.9 | 55.1 | 64.6 |
| BBL 1M | 55.6 | 65.8 | 55.8 | 65.2 |
| captions 10M | 63.2 | 73.4 | 61.8 | 72.5 |
| GPT-BERT, left to right | ||||
| captions 1M | 57.9 | 69.7 | 57.3 | 70.1 |
| Caption words | ||||||
| 0.1M | 0.8M | 2M | 5M | Slope | BBL | |
| GPT-2 | ||||||
| order 1 | 55.6 | 62.2 | 67.1 | 70.3 | 8.8 | 5.4 |
| order 2 | 55.0 | 63.0 | 66.1 | 70.6 | 9.0 | 4.6 |
| order 3 | 56.6 | 63.5 | 66.8 | 70.0 | 8.0 | 4.1 |
| GPT-BERT | ||||||
| Training text | GPT-2 | GPT-BERT |
|---|---|---|
| Children’s books | 63.8 | 66.4 |
| Subtitles | 64.0 | 65.9 |
| Captions | 59.7 | 62.4 |
| CHILDES | 53.1 | 50.5 |
| GPT-2 | GPT-BERT | Pretrained | ||||||||
| Humans | Pairs | Cap. 1M | BBL | Cap. 10M | Cap. 1M | BBL | Cap. 10M | RoGPT2 | RoBERT | |
| Agreement | ||||||||||
| subject-verb | 95.7 | 126 | 50.8 | 51.1 | 65.9 | 60.2 | 53.8 | 72.9 | 96 | 98.4 |
| demonstrative | 96.2 | 129 | 60.8 | 64.8 | 74.9 | 66.8 | 67.6 | 74.9 | 98.4 | 100.0 |
| adjective | 94.8 | 128 | 47.0 | 52.3 | 63.0 | 54.7 | 51.9 | 67.0 | 96.1 | 99.2 |
| participial agreement | 98.8 | 130 | 51.5 | 50.6 | 54.6 | 54.0 | 50.6 | 58.8 | 92.3 | 99.2 |
LLM Evaluation
Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.
See all 2,509 papers