Evaluating grammatical knowledge in language models requires benchmarks that cover diverse linguistic phenomena and are validated against human judgments. We introduce BLiMP-RO, a Romanian minimal-pair benchmark covering 50 grammatical phenomena across ten domains. It comprises 493 linguist-constructed sentence pairs, evaluated by native speakers in five experiments, and 5,942 additional semi-automatically generated, human-validated pairs. We further investigate grammatical generalization under limited linguistic input by training GPT-2 and GPT-BERT models from scratch on Romanian BabyBabelLM data and a newly collected corpus of 10 million words from child-targeted video captions. Increasing caption training data yields approximately 9 percentage points of accuracy improvement per tenfold increase in corpus size. At matched data sizes, BabyBabelLM generally outperforms video captions, while GPT-BERT consistently outperforms GPT-2. Our findings highlight how training data quantity, composition, and model architecture shape grammatical generalization in Romanian.
Figures & tables
Morphosyntactic agreement 77 subject-verb demonstrative adjective participial agreement adjectival article cel anaphor agreement (emphatic reflexive însu s , i ) doubled object and clitic ordinal numeral Differential object marking 30 pe in object relatives pe and clitic across referential noun types object marking across animacy classes Clitics 46 clitic doubling resumptive clitics in relatives and pe cine questions clitic order and the Person Case Constraint clitics with modal verbs clitics in negative imperatives
Binding theory 30 case condition (gerund against finite verb after a reflexive) C-command condition (antecedent of an emphatic reflexive) locality of reflexives Argument structure 29 transitivity passivizability inherent reflexive verbs Word order and movement 131 relativization (prepositions in relatives) demonstrative placement topicalization (doubled preverbal objects) definite article placement indefinite article placement preverbal depictives auxiliary and verb order adpositional phrases (nominal dependents) extraction from complex noun phrases extraction from adjuncts extraction from coordinations extraction from embedded questions extraction from clausal subjects
Mood 50 subjunctive-selecting predicates indicative-selecting predicates subjunctive and infinitive conditional future Negation 29 negative concord negative additive nici negative imperatives Nominal morphology 61 gender and number plural forms nominalization genitive and dative cardinal numerals with de vocative Null subjects 10 null against overt subject
Table 1: Domains and phenomena/subdomains of BLiMP-RO. The number after a domain counts its distinct pairs.
BLiMP-RO
Multi
Init.
Agr.
Ext.
Humans
96
[95.1, 96.9]
93.1
GPT-2
Cap. 1M
65
[60.8, 68.6]
63.7
62
59
BBL 1M
65
[62.5, 68.1]
64.8
68
64
Cap. 0–4
71
[67.5, 74.5]
69.7
70
66
Table 2: SLOR Accuracy (%) for different models and pre-training scenarios. The column Init. reports accuracy on the 508 items of BLiMP-RO, with 95% bootstrap intervals over items and runs. Agr. : agreement with people, how often a model chose the same sentence as a participant, over the 26,358 answers; for people, how often two participants chose the same sentence on an item, over 671,868 pairs. Ext. : accuracy on the extended set of 5,942 further pairs that participants accepted in a validation study. Multi : accuracy on MultiBLiMP. Trained models are averaged over five runs. Cap. is captions and BBL is BabyBabelLM.
Figure 1: Accuracy (%) on each phenomenon, grouped by the domains of Table 3 . Circles are people, squares the mean of the two pretrained Romanian models and triangles the mean of GPT-2 and GPT-BERT trained on captions 1M, BBL 1M and captions 10M (Table 2 ). People are on the 508 items, models on all pairs, with the 5,942 pairs that humans accepted. A gray line joins the three marks of a phenomenon. Within a domain, rows go from the lowest human accuracy at the top to the highest at the bottom. Names are shortened from Table 1 ; anaphor agreement and the C-command condition test emphatic reflexives. The axis starts at 30. Table 7 gives the values of each model.
GPT-2
GPT-BERT
Pretrained
Humans
Pairs
Cap. 1M
BBL
Cap. 10M
Cap. 1M
BBL
Cap. 10M
RoGPT2
RoBERT
Agreement
95.4
1011
54.7
55.4
65.2
58.7
55.7
67.5
95.5
97.6
DOM
90.1
389
55.2
65.8
67.1
68.0
70.6
73.3
94.1
95.9
Clitics
96.5
727
68.9
76.8
77.6
77.2
83.1
79.2
92.9
90.6
Binding
91.5
369
53.7
59.0
58.5
49.7
55.6
59.6
88.6
86.2
Arguments
97.6
383
56.1
62.5
68.0
62.9
67.5
70.0
92.2
98.4
Table 3: Accuracy (%) by domain, with the domain names of Table 1 shortened. Humans are on the 508 items (Table 1 counts them by domain); models are on all pairs, the 508 items and the 5,942 pairs that humans accepted (Pairs counts them). Column labels as in Table 2 ; BBL is BabyBabelLM 1M.
Figure 2: Accuracy under SLOR of caption models against training words, on (a) BLiMP-RO, all 6,435 pairs: the 493 items and the 5,942 pairs that humans accepted, and (b) MultiBLiMP. The 15 nested caption sets have 0.1, 0.2, 0.3, 0.4, 0.6, 0.8, 1.2, 1.6, 2, 2.5, 3.2, 4, 5, 6.4 and 8.2 million training words, without validation text; the last is the whole caption pool (captions 10M in Table 2 ). All models use the same validation set of 160,000 words, half captions and half BabyBabelLM, only to choose the checkpoint. Small points are runs, large points their means, lines ordinary least-squares fits of the means, and labels their gain in points per tenfold increase.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Sum
Token
Word
SLOR
GPT-2, left to right
captions 1M
53.0
64.9
55.1
64.6
BBL 1M
55.6
65.8
55.8
65.2
captions 10M
63.2
73.4
61.8
72.5
GPT-BERT, left to right
captions 1M
57.9
69.7
57.3
70.1
Appendix
Table 4: Accuracy (%) on the 508 items by how a model reads a sentence and how the reading is scored. Left to right predicts each token from the tokens before it; bidirectional masks each token in turn and predicts it from the rest of the sentence (pseudo-log-likelihood); fused adds the two readings’ logits before the softmax, and only GPT-BERT is trained for both. Sum adds the log probabilities of the sentence’s tokens, Token and Word divide it by the number of tokens or words, and SLOR, the main score of Table 2 , subtracts the log probability of the same tokens under their frequencies alone and divides by the number of tokens (Section 5 ); for RoBERT, a masked token also hides the later tokens of its word in SLOR.
Caption words
0.1M
0.8M
2M
5M
Slope
Δ BBL
GPT-2
order 1
55.6
62.2
67.1
70.3
8.8
− 5.4
order 2
55.0
63.0
66.1
70.6
9.0
− 4.6
order 3
56.6
63.5
66.8
70.0
8.0
− 4.1
GPT-BERT
Appendix
Table 5: Accuracy (%) under SLOR on all 6,435 pairs for caption sets drawn from three random orders of the caption videos (Section 5 ), five runs each; order 1 gives Figure 2 . With the whole pool, shared by all three, GPT-2 reaches 72.1% and GPT-BERT 73.9%. Slope: points gained per tenfold increase in words, from a straight line through a row’s five accuracies on a log scale. Δ BBL: 0.8M captions minus BBL 1M.
Training text
GPT-2
GPT-BERT
Children’s books
63.8
66.4
Subtitles
64.0
65.9
Captions
59.7
62.4
CHILDES
53.1
50.5
Appendix
Table 6: Accuracy (%) under SLOR on all 6,435 pairs of models trained on 238,600 words from each source, all the children’s books in the BabyBabelLM training part, five runs each.
GPT-2
GPT-BERT
Pretrained
Humans
Pairs
Cap. 1M
BBL
Cap. 10M
Cap. 1M
BBL
Cap. 10M
RoGPT2
RoBERT
Agreement
subject-verb
95.7
126
50.8
51.1
65.9
60.2
53.8
72.9
96
98.4
demonstrative
96.2
129
60.8
64.8
74.9
66.8
67.6
74.9
98.4
100.0
adjective
94.8
128
47.0
52.3
63.0
54.7
51.9
67.0
96.1
99.2
participial agreement
98.8
130
51.5
50.6
54.6
54.0
50.6
58.8
92.3
99.2
Appendix
Table 7: Accuracy (%) by phenomenon, columns as in Table 3 . *Preference phenomena.
Extreme Light Infrastructure Nuclear Physics, Horia Hulubei National Institute for R & D in Physics and Nuclear Engineering, 077125 Bucharest-Magurele, Romania