cs.CLNov 13, 2026

BLiMP-RO: A Benchmark of Linguistic Minimal Pairs for Romanian

Authors: Radu-Stefan Popa, Valentin Feroiu, Adina Camelia Bleotu, Sergiu Nisioi

Organizations: University of Bucharest · Universität Wien

Abstract

Evaluating grammatical knowledge in language models requires benchmarks that cover diverse linguistic phenomena and are validated against human judgments. We introduce BLiMP-RO, a Romanian minimal-pair benchmark covering 50 grammatical phenomena across ten domains. It comprises 493 linguist-constructed sentence pairs, evaluated by native speakers in five experiments, and 5,942 additional semi-automatically generated, human-validated pairs. We further investigate grammatical generalization under limited linguistic input by training GPT-2 and GPT-BERT models from scratch on Romanian BabyBabelLM data and a newly collected corpus of 10 million words from child-targeted video captions. Increasing caption training data yields approximately 9 percentage points of accuracy improvement per tenfold increase in corpus size. At matched data sizes, BabyBabelLM generally outperforms video captions, while GPT-BERT consistently outperforms GPT-2. Our findings highlight how training data quantity, composition, and model architecture shape grammatical generalization in Romanian.

Figures & tables

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Right Tool, Right Job: Native-Language Evaluation, Tokenizer Sensitivity, and Methodological Findings from a French-Only BabyLM

    Sep 15, 2026Adam Zachary Wasserman, David BeaucheminLLM EvaluationCross-Lingual Representation Alignment

  2. Neurograf v1: An End-to-End Romanian-First Language Model Family

    Oct 9, 2026Ionut Adrian RotaruLanguage ModelingLLM Training

  3. Different Corpora, Different Skills? A Paired Study of Small Language Models

    Nov 5, 2026Edoardo Vaira, Georgios Chavales, Pierre NuguesLanguage Model PretrainingSmall Language Models