Datasets for Verb Alternations across Languages: BLM Templates and Data Augmentation Strategies

Giuseppe Samo; Paola Merlo

cs.CLMar 16, 2026

Datasets for Verb Alternations across Languages: BLM Templates and Data Augmentation Strategies

Authors: Giuseppe Samo, Paola Merlo

Organizations: IDIAP Research Institute · University of Geneva

Abstract

Large language models (LLMs) have shown remarkable performance across various sentence-based linguistic phenomena, yet their ability to capture cross-sentence paradigmatic patterns, such as verb alternations, remains underexplored. In this work, we present curated paradigm-based datasets for four languages, designed to probe systematic cross-sentence knowledge of verb alternations (change-of-state and object-drop constructions in English, German and Italian, and Hebrew binyanim). The datasets comprise thousands of the Blackbird Language Matrices (BLMs) problems. The BLM task -- an RPM/ARC-like task devised specifically for language -- is a controlled linguistic puzzle where models must select the sentence that completes a pattern according to syntactic and semantic rules. We introduce three types of templates varying in complexity and apply linguistically-informed data augmentation strategies across synthetic and natural data. We provide simple baseline performance results across English, Italian, German, and Hebrew, that demonstrate the diagnostic usefulness of the datasets.

Figures & tables

Language	Structure	Change of state verbs	Object Drop verbs
English	Transitive	The artist broke the vase	The artist paints the vase
	Intransitive	The vase broke	The artist paints
Italian	Transitive	L’ artista ruppe il vaso	L’ artista dipingeva il vaso
	Intransitive	Il vaso si ruppe	L’ artista dipingeva
German	Transitive	Der Künstler zerbrach den Becher	Der Künstler malte den Becher
	Intransitive	Der Becher zerbrach	Der Künstler malte

Figure 1: Example of Change of state verbs and Object drop verbs alternation and morphological marking across languages. The verb is highlighted in bold while colours code semantic roles ( agents , patients ), morphological marking on the verb ( si in Italian) and morphological case marking on the argument nominative and accusative case.

Context
1	The artist broke the vase two years ago
2	The artist broke the vase
3	The vase broke two years ago
4	???
Answer Set
1	The vase broke the artist

Figure 2: Simplified example of a BLM template.

Context
1	The artist broke the vase
2	The artist that broke the vase is there
3	The vase that broke is there
4	???

Figure 3: Example of template of type B

Context
1	The artist will break the vase on stage
2	The vase will break during the performance
3	We met the artist that broke the vase on stage
4	The vase that broke on stage now values millions
5	The artist breaks the vase
6	???

Figure 4: Example of template of type C

Context
1	The queen chipped the scepter this morning
2	The artist broke the vase
3	The door opened at midnight
4	???
Answer Set
1	The inspector closed the case

Figure 5: A simplified example of MaxLex of the template of Figure 2 . The sentences follow the same syntax, but the words vary in each sentence.

Name	Language	Template	Augmentation Strategies
OD	En, It	A	Masking
COS	En, It	A	Masking
COS+	De, It, En	B	Prompting
Caus	He	C	Natural data

Table 1: Overview of the datasets.

COS context
1	Ag	Akt	Pat	p -NP
2	Ag	Akt	Pat	by -NP
3	Pat	Pass	by - Ag	p -NP
4	Pat	Pass	by - Ag	by -NP
5	Pat	Pass		p -NP
6	Pat	Pass		by -NP

Figure 6: BLM context sets (left) for the change-of-state group (COS) and the object drop (OD) class and answers sets (right). Italian COS and OD has two additional answers in which the presence/absence of the reflexive-like element si is absent (both GRAMMAR errors).

Context
The chef melted the butter on the stove
The chef melted the butter by mistake
The butter was melted by the chef on the stove
The butter was melted by the chef by mistake
The butter was melted on the stove
The butter was melted by mistake

Figure 7: An example for COS for English with minimal lexical variation (MinLEX).

Context
Gli artisti dipinsero questi paesaggi in una giornata
‘The artists painted these landscapes in one day’
Le attrici cantavano le stesse canzoni da qualche anno
‘The actresses had been singing the same songs for a few years’
Dei loghi saranno disegnati dalla ditta in meno di un’ora
‘Some logos will be drawn by the company in less than one hour’

Figure 8: An example for OD for Italian (with English glosses in grey) with maximal lexical variation ( MaxLex ).

T2I Context
SVO	Ag Verb Pa
S ag R	Ag rel Verb Pa
OR	Pa rel Ag Verb
S pa R	Pa rel Verb
	Answer Set
Grammar	Ag Verb

Figure 9: BLM templates for the T2I and I2T groups. Context sets and corresponding Answer sets are shown with colour-coded elements: agent (Ag), patient (Pa), verb (V), and relative markers (rel).

Context
Der Chef schmiltzt den Käse
‘The.NOM chef melts the.ACC cheese’
Der Chef der den Käse schmiltzt
‘The.NOM chef that.NOM the.ACC cheese melts’
Den Käse den der Chef schmiltzt
‘The.ACC cheese that.ACC the.NOM chef melts’

Figure 10: MinLex for COS+T2IC (Case) for German. English glosses in grey. NOM = Nominative case, ACC = Accusative case.

Figure 11: Binyanim under investigations and relative examples for the root KTB (related to the act of writing). Adapted from Kastner 2019, 574-575 . The colour codes the different overt characters representing morphemes marking the alternation.

Context
Paal		Paal		Paal		Nifal
Paal		Paal		Paal		Nifal
Nifal		Nifal		Hifil		Hifil
Nifal	…	Nifal	…	Hifil	…	Hifil
Hifil		Hufal		Hufal		Hufal
Hifil		Hufal		Hufal		Hufal

Figure 12: BLM template for the CausH.

Figure 13: An example of instantiation of sentences ( Natural ) extracted from UD and the derived synthetic data made by the verbs ( Synthetic ). Glosses are provided and the verb of the sentence is highlighted in gray.

Data	Lang	Lex	Train	Test
COS	En, It	Min/Max	2700	300
OD	En, It	Min/Max	2700	300
COS+ T2I	De Case	Min/Max	1600	400
	De mix
	En, It
COS+ I2T	De Case	Min/Max	1600	400

Table 2: Datasets, languages, training and testing instances for each language and lexical variation.

Figure 15: Error analysis across datasets.

Figure 16: Confusion matrices for Hebrew across datasets and models, for each verbal voice, reported as proportions.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

ENGLISH
Wrong Answer	Error Type	COS MinLex	COS MaxLex	OD MinLex	OD MaxLex
1 Pat Akt by -NP	Grammar	-	-	35	12
2 Ag Akt by -NP	Grammar	21	49	-	-
3 Pat Pass by - Ag	Sequence	1	1	0	0
4 Ag Pass by - Pat	Sequence	4	0	1	2
5 Pat Akt Ag	Sequence	2	1	3	0

Appendix

Table 3: Raw counts of errors for BLM-COS and OD in English and Italian across lexical variation conditions (test set n = 300).

Multilingual Language Model Evaluation

Multilingual Language Model Evaluation measures how reliably language models perform across languages, scripts, dialects, and cultural contexts.

See all 485 papers

Blackbird Language Matrices: A Framework to Investigate the Linguistic Competence of Language Models
Feb 24, 2026Paola Merlo, Chunyang Jiang, Giuseppe Samo +1LLM Evaluation LLM Interpretability
Idiap Research Institute · University of Geneva
Comparing Natural and Synthetic Structured Data: A Study of the Passive Verb Alternation in French and Italian
Mar 26, 2026Giuseppe Samo, Paola MerloSynthetic Data Evaluation Synthetic Data Pretraining
IDIAP Research Institute · University of Geneva
Light or Full Verb? A Minimal-Pair Dataset for Probing Phraseological Competence in Language Models
Jun 3, 2026Francesca Franzon, Nicolas Rosàs Gómez, Leo WannerLanguage Model Probing
Universitat Pompeu Fabra (UPF) · Catalan Institute for Research and Advanced Studies (ICREA)

Feb 24, 2026J/K move · Enter open · S save

Blackbird Language Matrices: A Framework to Investigate the Linguistic Competence of Language Models

Paola Merlo, Chunyang Jiang, Giuseppe Samo, Vivi Nastase

Idiap Research Institute · University of Geneva

LLM Evaluation LLM Interpretability Multilingual Language Model Evaluation

Open paper AskPDF arXiv

This article describes a novel language task, the Blackbird Language Matrices (BLM) task, inspired by intelligence tests, and illustrates the BLM datasets, their construction and benchmarking, and targeted experiments on chunking and systematicity. BLMs are multiple-choice problems, structured at multiple levels: within each sentence, across the input sequence, within each candidate answer. Because of their rich structure, these curated, but naturalistic datasets are key to answer some core questions about current large language models abilities: do LLMs detect linguistic objects and their properties? Do they detect and use systematic patterns across sentences? Are they more prone to linguistic or reasoning errors, and how do these interact? We show that BLMs, while challenging, can be solved at good levels of performance, in more than one language, with simple baseline models or, at better performance levels, with more tailored models. We show that their representations contain the grammatical objects and attributes relevant to solve a linguistic task. We also show that these solutions are reached by detecting systematic patterns across sentences. The paper supports the point of view that curated, structured datasets support multi-faceted investigations of properties of language and large language models. Because they present a curated, articulated structure, because they comprise both learning contexts and expected answers, and because they are partly built by hand, BLMs fall in the category of datasets that can support explainability investigations, and be useful to ask why large language models behave the way they do.

Abstract

Multilingual Language Model Evaluation

Explore similar work

Blackbird Language Matrices: A Framework to Investigate the Linguistic Competence of Language Models

Comparing Natural and Synthetic Structured Data: A Study of the Passive Verb Alternation in French and Italian

Light or Full Verb? A Minimal-Pair Dataset for Probing Phraseological Competence in Language Models