BLiMPPI: Benchmark of Linguistic Minimal Pairs for Polarity Items

Arzu Burcu Güven; Anna Rogers; Leonie Weissweiler; Joakim Nivre; Rob van der Goot

cs.CLOct 9, 2026

BLiMPPI: Benchmark of Linguistic Minimal Pairs for Polarity Items

Authors: Arzu Burcu Güven, Anna Rogers, Leonie Weissweiler, Joakim Nivre, Rob van der Goot

Organizations: IT University of Copenhagen · Leipzig University & ScaDS.AI Dresden/Leipzig · Uppsala University

Abstract

Negative polarity items (NPIs), such as "anymore" and "ever", are grammatical only in select environments, predominantly in the context of negation. Because their grammaticality depends on both syntax and semantics, they have a rich, multilingual literature in theoretical linguistics, yet work on NPIs in language models is almost entirely confined to English. We introduce BLiMPPI, a multilingual evaluation suite of over 12,000 NPI minimal pairs in 33 languages, built by a pipeline that discovers NPIs in Universal Dependencies (UD) treebanks, labels their licensing environments, and perturbs these into minimal pairs. Evaluating 12 multilingual and 2 monolingual models alongside n-gram baselines, we find that multilingual models struggle most with lower-resource languages and with pairs that are not n-gram solvable. We further examine the effects of NPI typology, language family, licensing variability, and NPI-licenser distance. Data: https://github.com/arzuburcuguven/blimppi-data

Figures & tables

Typology	Language	N	Src.	Ver.
NC (strict) N=8,171	Belarusian	171	D	MT
	Bulgarian	131	D	MT
	Croatian	71	D	Hum
	Czech	1,213	D	Hum
	Greek	87	R	Hum
	Hungarian	13	R	Hum

Table 1: Polarity-item typology, item count, NPI inventory source, and verification status for the 33 languages in BLiMPPI. NPI inventory source (Src.): R = linguistic resources, D = automatic discovery, R+D = both. All data from UD v2.18, except Dutch* ( Hoeksema, 2024 ) and English* (UD v2.18 + NPI bank).

Environm.	Detection
Standard negation	Bound negation on the predicate head; unbound negation attached to the clause head via advmod , advmod:neg , compound , compound:lvc , cop , aux , or aux:pass . Checked for Polarity=Neg and the negation marker’s lemma.
N-series	For languages without duplex negation, n-series items are added to the language configuration. Pronouns: must share the clause head with the NPI. Determiners: det of the subject or object of the NPI clause, preceding the NPI.
Matrix clause	For non-island complements ( ccomp , csubj , csubj:cop , xcomp ), walk to the matrix clause and check for scoping licensers ( Figure 1 ).
Lexical licensers	Without -type adpositions, near-negatives, negative predicates, comparatives and conditional antecedents, tracked by lemma and syntactic position.
Questions & superlatives	Feature annotations and punctuation.

Table 2: Licensing environments and how they are detected in UD trees.

Panel of Figure 1 — Figure 1: Licenser resolution for the NPI ever . \scriptsize1⃝ Climb from the NPI to its clause head lied ; no licenser is detected there. \scriptsize2⃝ Climb to the matrix clause, allowed because ccomp is scope-permeable. \scriptsize3⃝ On think , the detectors find n’t , an advmod with Polarity=Neg : a matrix licenser.

		Llama-3.1		Aya	Gemma-3				Qwen3			Apertus		Mean	Goldfish		Other
ISO	Language	8B	70B	32B	1B	4B	12B	27B	8B	14B	32B	8B	70B		100MB	1GB	5-gram
hye	Armenian	95.2	92.9	95.2	85.7	97.6	95.2	97.6	90.5	90.5	85.7	88.1	97.6	92.7	88.1	95.2	64.3
bel	Belarusian	84.2	94.2	90.1	90.1	91.8	95.9	98.2	83.0	89.5	91.8	95.3	96.5	91.7	89.5	97.1	75.4
bul	Bulgarian	83.2	93.9	69.5	87.8	96.9	96.2	99.2	87.8	90.8	88.5	98.5	99.2	91.0	92.4	96.9	73.3
cat	Catalan	93.2	97.3	91.4	89.1	92.7	96.4	98.6	89.1	93.2	94.1	95.9	95.9	93.9	97.3	97.7	55.9
zho	Chinese	94.1	97.1	88.2	88.2	91.2	91.2	97.1	88.2	94.1	97.1	94.1	88.2	92.4	73.5	88.2	73.5

Table 3: Accuracy (%) by language and model over 12,866 minimal pairs in 33 languages. Mean averages the multilingual models to its left; the columns to its right are reference points rather than systems under comparison and are excluded from it, as is the bolding. Best multilingual model per language in bold; — marks a language the model does not cover. The bottom row is the unweighted average across languages.

				Llama-3.1		Aya	Gemma-3				Qwen3			Apertus			Goldfish		Other
	Subset	Langs	Pairs	8B	70B	32B	1B	4B	12B	27B	8B	14B	32B	8B	70B	Mean	100MB	1GB	5-gram
All		33	12,866	88.5	94.6	89.0	87.9	93.5	96.5	97.9	85.4	88.7	90.9	94.4	96.8	92.0	86.5	94.8	63.2
Typology	NC	21	10,014	88.6	94.3	90.0	87.3	93.4	96.3	98.0	85.5	88.8	90.3	95.7	97.3	92.1	87.5	95.8	64.6
Typology	NPI	12	2,852	88.3	94.9	87.2	88.8	93.7	96.8	97.7	85.1	88.7	92.0	92.2	96.0	91.8	84.7	92.7	60.7
Family	Slavic	10	7,330	86.3	93.3	86.5	87.5	94.3	96.8	98.7	85.1	88.3	91.2	96.0	98.0	91.8	86.8	95.6	67.9
	Germanic	8	2,645	87.2	93.1	83.3	86.9	93.2	96.5	97.5	81.3	85.5	89.5	92.5	96.2	90.2	83.7	92.1	62.9

Table 4: Accuracy (%) by model over subsets of BLiMPPI. Every cell is a mean over the languages in the subset, each counted once; Mean averages the multilingual models to its left. Typology follows Table 1 . Rows are ordered by subset size. Best model per row in bold.

Figure 2: Model sensitivity to three benchmark properties, and ParalleLM learning curves. In the first three panels, each point is one model:

x

is the correlation between the property and accuracy,

y

is overall accuracy. Filled points are significant. Correlations: distance, Pearson over pairs; licensing entropy, Pearson over lemmas; CC volume, Spearman over languages.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Language	Category	Reason
Ancient Greek	Extinct	No living speaker community; no native judgements available
Ancient Hebrew	Extinct
Classical Armenian	Extinct
Classical Chinese	Extinct
Coptic	Extinct
Latin	Extinct

Appendix

Table 5: Universal Dependencies treebanks above

100\,000

tokens that were not included in BLiMPPI.

Language	Source	Language	Source
Armenian	Khanjian (2010)	Norwegian (Nynorsk)	Mæhlum et al. (2021)
Belarusian	—	Norwegian (Bokmål)	Mæhlum et al. (2021)
Bulgarian	Haspelmath (2001)	Persian	Haspelmath (2001)
Catalan	Haspelmath (2001) ; Espinal and Llop (2022) ; Giannakidou and Quer (1997) ; Vallduví (1994)	Polish	Haspelmath (2001)
Chinese	Xue and Pan (2023) ; Lin et al. (2014)	Romanian	Haspelmath (2001) ; Fălăuş (2008)
Croatian	Haspelmath (2001)	Russian	Haspelmath (2001)

Appendix

Table 6: Descriptive sources consulted when assembling the language configurations.

	Pairs		Accuracy (%)			Mean $\delta$
Model	Swap	Drop	Swap	Drop	$\Delta$	Swap	Drop
Goldfish 100MB	939	1,835	73.3	76.3	+3.1	2.69	2.38
Goldfish 1GB	939	1,835	91.8	88.4	-3.4	5.23	4.16
Gemma-3 1B	939	1,835	88.6	88.6	+0.0	6.92	5.48
Gemma-3 4B	939	1,835	94.0	91.7	-2.3	8.49	6.19
Apertus 8B	939	1,835	92.7	92.2	-0.5	8.54	6.40

Appendix

Table 7: Accuracy on swap items, where the licenser is replaced by a non-licensing word, against drop items, where it is deleted and the ungrammatical member is one token shorter. Mean

\delta

is the average log-probability margin within a pair. The last three rows average each kind of model separately and are not comparable with each other. Restricted to the 7 languages holding both item types.

Figure 3: One example minimal pair per language in BLiMPPI. Continued on the following page.

Figure 4: (Continued) One example minimal pair per language in BLiMPPI.

		Multi-NPI		Accuracy (%)
Language	Pairs	$n$	%	Single	Multi	$\Delta$
Armenian	42	8	19.1	86.2	88.5	$+2.3$
Belarusian	171	23	13.4	90.6	92.6	$+2.0$
Bulgarian	131	14	10.7	90.9	95.6	$+4.7$
Catalan	220	17	7.7	93.9	95.9	$+2.0$
Chinese	34	1	2.9	92.8	84.6	$-8.2$

Appendix

Table 8: Share of minimal pairs whose sentence contains more than one NPI, and accuracy on single- vs. multiple-NPI pairs, averaged over the 12 multilingual models.

\Delta

= Multi

-

Single (percentage points).

Language	Aud.	Pairs	Flagged	%
English	1	30	9	30.0
Hindi	1	15	4	26.7
German	4	30	24	20.0
Chinese	2	30	6	10.0
Danish	1	30	3	10.0
Romanian	1	30	2	6.7

Appendix

Table 9: Human audit. Aud. is the number of auditors, Pairs the sample size per auditor, Flagged the number of judgments marking a pair as not carrying the intended contrast, out of

\mathit{Aud.}\times\mathit{Pairs}

judgments. Dutch, Finnish, Hungarian, Lithuanian, Norwegian and Swedish are absent because their configurations were written with a native-speaker linguist and vetted during construction.

ISO	Acc.	Ties	OOV/sent	OOV rate	Asym.
hye	0.643	0.000	0.14	0.007	0.071
bel	0.754	0.000	0.25	0.016	0.006
bul	0.733	0.000	0.11	0.009	0.000
cat	0.559	0.000	0.25	0.010	0.009
zho	0.735	0.000	0.00	0.000	0.000
hrv	0.620	0.000	0.31	0.013	0.000

Appendix

Table 10: KenLM 5-gram baseline by language (ISO 639-3). Ties is the proportion of pairs scored identically; OOV/sent the mean number of out-of-vocabulary tokens in the acceptable member; OOV rate the OOV proportion over all tokens; Asym. the proportion of pairs whose two members differ in OOV count. Asymmetry near zero indicates that OOV mass is shared across pair members and therefore cannot account for the score difference. Colour is per column: accuracy is orange below chance and purple above; the diagnostics darken as they rise.

		Llama-3.1		Aya	Gemma-3				Qwen3			Apertus		Mean	Goldfish
ISO	Language	8B	70B	32B	1B	4B	12B	27B	8B	14B	32B	8B	70B		100MB	1GB
hye	Armenian	96.4	96.4	96.4	85.7	100.0	100.0	100.0	85.7	92.9	85.7	89.3	100.0	94.0	100.0	100.0
bel	Belarusian	85.6	95.2	92.0	95.2	92.8	96.8	99.2	86.4	89.6	96.0	93.6	97.6	93.3	94.4	98.4
bul	Bulgarian	86.6	96.9	77.3	90.7	97.9	99.0	99.0	88.7	95.9	92.8	100.0	100.0	93.7	99.0	99.0
cat	Catalan	93.5	97.6	89.4	87.8	91.9	96.7	98.4	87.0	94.3	95.1	95.1	97.6	93.7	99.2	100.0
zho	Chinese	92.0	100.0	92.0	96.0	96.0	96.0	100.0	92.0	96.0	96.0	96.0	92.0	95.3	84.0	96.0

Appendix

Table 11: Accuracy (%) on items the 5-gram baseline also gets right, by language and model (8,456 pairs in 33 languages). Mean averages the multilingual models to its left; Goldfish sits to its right as a monolingual reference point and is excluded from it, as is the bolding. Best multilingual model per language in bold. The 5-gram is absent by construction: the split is defined by its own verdict.

		Llama-3.1		Aya	Gemma-3				Qwen3			Apertus		Mean	Goldfish
ISO	Language	8B	70B	32B	1B	4B	12B	27B	8B	14B	32B	8B	70B		100MB	1GB
hye	Armenian	92.9	85.7	92.9	85.7	92.9	85.7	92.9	100.0	85.7	85.7	85.7	92.9	89.9	64.3	85.7
bel	Belarusian	80.4	91.3	84.8	76.1	89.1	93.5	95.7	73.9	89.1	80.4	100.0	93.5	87.3	76.1	93.5
bul	Bulgarian	73.5	85.3	47.1	79.4	94.1	88.2	100.0	85.3	76.5	76.5	94.1	97.1	83.1	73.5	91.2
cat	Catalan	92.8	96.9	93.8	90.7	93.8	95.9	99.0	91.8	91.8	92.8	96.9	93.8	94.2	94.8	94.8
zho	Chinese	100.0	88.9	77.8	66.7	77.8	77.8	88.9	77.8	88.9	100.0	88.9	77.8	84.3	44.4	66.7

Appendix

Table 12: Accuracy (%) on items the 5-gram baseline gets wrong, by language and model (4,356 pairs in 33 languages). Mean averages the multilingual models to its left; Goldfish sits to its right as a monolingual reference point and is excluded from it, as is the bolding. Best multilingual model per language in bold. The 5-gram is absent by construction: the split is defined by its own verdict.

Figure 5: Distribution of NPI–licenser distance (in tokens) per language.

			Goldfish 1GB		Llama 8B		EuroLLM 9B		Gemma 27B		OLMo 32B		Llama 70B		Mean
Language	Pairs_B	Pairs_M	b	m	b	m	b	m	b	m	b	m	b	m	b	m	$\Delta$
Icelandic	47	2,801	91.5	98.4	68.1	85.3	6.4	67.8	93.6	96.0	55.3	79.7	85.1	91.0	66.7	86.4	$-$ 19.7
Armenian	42	1,415	95.2	98.4	95.2	96.5	14.3	72.7	97.6	97.9	45.2	86.9	92.9	98.4	73.4	91.8	$-$ 18.4
Greek	87	1,096	93.1	98.8	79.3	97.9	95.4	99.3	96.6	99.5	69.0	96.9	92.0	99.3	87.5	98.6	$-$ 11.1
Slovak	294	4,145	96.3	95.2	68.0	85.5	89.8	96.3	95.6	95.7	50.3	77.2	84.4	91.8	80.7	90.3	$-$ 9.6
Urdu	60	550	95.0	96.4	90.0	96.7	53.3	87.6	96.7	97.5	83.3	96.5	98.3	97.5	86.1	95.4	$-$ 9.3

Appendix

Table 13: BLiMPPI against MultiBLiMP ( Jumelet et al., 2026 ) on the 28 languages both cover over 6 models reported in MultiBLiMP. Two of them, EuroLLM 9B ( Martins et al., 2025 ) and OLMo-2 32B ( Team OLMo et al., 2025 ) , are not part of our main evaluation. Each model gets a pair of columns: b is BLiMPPI and m is MultiBLiMP. Accuracy (%) per language; Pairs_B counts BLiMPPI minimal pairs and Pairs_M MultiBLiMP items. MultiBLiMP treats Serbo-Croatian as one language, so the BLiMPPI Serbian and Croatian rows are pooled over items here. Empty cells (—) are excluded from both means of that row.

\Delta

is the difference of the two means, purple where BLiMPPI is lower and orange where MultiBLiMP is.

Multilingual Language Model Evaluation

Multilingual Language Model Evaluation measures how reliably language models perform across languages, scripts, dialects, and cultural contexts.

See all 485 papers

Assessing the Impact of Language Disparity on Multilingual Linguistic Ability in Large Language Models
Sep 30, 2026Zhanyu Chen, Jaap JumeletMultilingual Language Model Evaluation Language Modeling
University of Groningen
DEPART: DEcomposing PARiTy across Multilingual LLMs
May 27, 2026Manan Uppadhyay, Prashant Kodali, Pranjal Chitale +3Multilingual Language Model Evaluation Cross-Lingual Reasoning in Language Models
Microsoft Research India · IIT Gandhinagar
"Not" is Not Lost in Translation: A Causal Analysis of Negation Representations in Multilingual LLMs
Oct 23, 2026Giang Son Nguyen, Wray Buntine, Dung D. Le +1Cross-Lingual Consistency in Language Models Multilingual Language Models
Nanyang Technological University · VinUniversity · Monash University

Sep 30, 2026J/K move · Enter open · S save

Assessing the Impact of Language Disparity on Multilingual Linguistic Ability in Large Language Models

Zhanyu Chen, Jaap Jumelet

University of Groningen

Multilingual Language Model Evaluation Language Modeling Low-Resource Language Processing

Open paper AskPDF arXiv

Claims about the grammatical competence of multilingual language models vary sharply with how competence is measured, yet the interaction between evaluation paradigm, post-training, and language resource availability has not been systematically examined. We evaluate base and post-trained models from six families on MultiBLiMP, a syntactic minimal-pair benchmark covering 101 languages, using four evaluation methods. We report three principal findings. First, post-training degrades grammatical competence, but the magnitude of this effect is reduced unevenly by model scale, while low-resource languages bear the highest cost. Second, post-trained models retain grammatical knowledge they cannot articulate through explicit prompting, yet this is measurable only in high-resource languages, because near-chance baselines in low-resource settings leave little knowledge to hide. Third, native-language prompting recovers otherwise hidden competence on low-resource languages, demonstrating that only high-resource languages can be probed directly from unprompted probabilities. We conclude that multilingual grammatical evaluation must adopt language-informed, multi-paradigm protocols to avoid systematically underestimating low-resource abilities.

Abstract

Multilingual Language Model Evaluation

Explore similar work

Assessing the Impact of Language Disparity on Multilingual Linguistic Ability in Large Language Models

DEPART: DEcomposing PARiTy across Multilingual LLMs

"Not" is Not Lost in Translation: A Causal Analysis of Negation Representations in Multilingual LLMs