cs.CLMar 26, 2026

Comparing Natural and Synthetic Structured Data: A Study of the Passive Verb Alternation in French and Italian

Authors: Giuseppe Samo, Paola Merlo

Organizations: IDIAP Research Institute · University of Geneva

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

This study compares the impact of natural and synthetic data on training and evaluating large language models (LLMs), using the case of passive verb alternation in French and Italian. We use Blackbird Language Matrices (BLMs), structured datasets designed to probe linguistic knowledge of underlying patterns across sentence sets. We compare structured templates instantiated with natural sentences extracted from Universal Dependencies to structured templates of synthetic sentences. Experiments show that while models achieve ceiling performance when trained and tested on synthetic datasets, they do not reliably generalize to natural sentences. In contrast, models trained on natural data exhibit robust performance across both natural and synthetic test suites, demonstrating their superior ability to capture abstract linguistic patterns. These results corroborate the value of natural data and of structured set ups in linguistic evaluation for probing LLMs' syntactic and semantic knowledge.

Figures & tables

Figure 1: Example of a synthetic piece of data for transitive and intransitive forms of the Italian verb cantare ‘sing’ generated by DeepSeek (DeepSeek-AI, accessed Feb 13, 2026).
Template
StructureR ArgR STExample
Context Set
1 Ag Vact Th ?2QDoes the customer pay the bill ?
2 Ag Vact Th2DThe student gets the prize .
3 Ag Vact ?1QDoes the teacher explain ?
4 Ag Vact1DThe car moves .
Figure 2: BLM template structure instantiated with a synthetic example in English, generated with DeepSeek V.3 (section 3.2 ). Ag = Agent, Th = Theme, Vact = verb in active voice, Vpass = verb in passive voice, red elements mark interrogative markers. Number of arguments (1, 2) and sentence type (Q = question, D = declarative).
 NaturalSynthetic
Context Set
1Que signifie l’acronyme NASA ?Le garçon jette -t- il la pierre ?
 ‘what does the acronym NASA mean?’ (FQB _\_ TREC-fr-584)‘Does the boy throw the stone?’
2Leur album No Dice obtient un excellent accueil.L’équipe félicite le gagnant .
 ‘Their album No Dice obtained an excellent reception’ (fr-ud-train _\_ 06225)‘The team congratulates the winner’
3Combien de temps dure le voyage entre Tokyo et Niigata ?L’équipe coûte -t- elle ?
Figure 3: Natural and synthetic examples in French, glosses and ID number (natural data) for reference within the explored treebanks. We have coloured code the core elements of the BLM-template. Correct answer in bold.
 Example NaturalExample Synthetic
Context Set 
1Quando Innsbruck ospitò le Olimpiadi Invernali ?Lo scrittore finisce il romanzo ?
 ‘When did Innsbruck host the Winter Olympic Games?’ (ISDT, quest-991)‘Does the writer finish the novel?’
2I carabinieri gli hanno recapitato il decreto di revoca degli arresti domiciliari.Il giocatore segna un gol .
 ‘The Carabinieri delivered to him the order revoking his house arrest.’ (ISDT, isst _\_ tanl-280)‘The player scores a goal.’
3A che velocità viaggia la luce ?Il bambino disegna ?
Figure 4: Natural and synthetic examples in Italian, glosses and ID number (natural data) for reference within the explored treebanks. We have coloured code the core elements of the BLM-template. Correct answer in bold.
Figure 5: F1 scores across training and test suites in monolingual models. The grey dotted line indicates chance level.
Figure 6: Error probabilities and types of errors (indicated by the selected erroneous answer) in monolingual models. Act = Active; Pass = Passive; arg = arguments.
Figure 7: F1 scores across conditions, languages and models. The gray dotted line indicates chance level. IT = Italian, FR = French.
Figure 8: Error probabilities and types of errors (indicated by the selected erroneous answer) in multilingual models. Crosses repeat values of error bars in monolingual model for comparison (see Figure 6 ). Act=Active, Pass=Passive, arg=arguments.
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

#\# ArgsVoiceSTPatternWithout
2ActQV -[nsubj]-> Ag; V-[obj]-> Pat; Q [form="?"]Y [upos=VERB]
1ActDeclV -[nsubj]-> Ag; V-[obj]-> PatQ [form="?"]; Y [upos="VERB"]
2ActQV -[nsubj]-> Ag; Q [form="?"]V-[obj]-> Pat; Y [upos="VERB"]
1ActDeclV -[nsubj]-> AgV-[obj]-> Pat; Q [form="?"]; Y [upos="VERB"]
2PassQV -[nsubj:pass]-> Pat; V-[obl:agent]-> Ag; Q [form="?"]Y [upos="VERB"]
1PassDeclV -[nsubj:pass]-> Pat; V-[obl:agent]-> AgQ [form="?"]; Y [upos="VERB"]
Appendix
Table 1: Queries to retrieve natural instatiation example from UD treebanks with Grew-Match ( Guillaume, 2021 ) . #\# Args = number of arguments, ST = Sentence Type, Act = Active Voice, Pass = Passive Voice, Q = question, D = declarative.
More in this topic

Synthetic Data Evaluation

Synthetic data evaluation measures whether generated examples preserve the properties needed for a stated use, including fidelity, privacy, and utility.

See all 235 papers

Explore similar work

CardsList

  1. Datasets for Verb Alternations across Languages: BLM Templates and Data Augmentation Strategies

    Mar 16, 2026Giuseppe Samo, Paola MerloMultilingual Language Model Evaluation

    IDIAP Research Institute · University of Geneva

  2. Blackbird Language Matrices: A Framework to Investigate the Linguistic Competence of Language Models

    Feb 24, 2026Paola Merlo, Chunyang Jiang, Giuseppe Samo +1LLM EvaluationLLM Interpretability

    Idiap Research Institute · University of Geneva

  3. Heterogeneity in Formal Linguistic Competence of Language Models: Is Data the Real Bottleneck?

    Apr 20, 2026H S V N S Kowndinya Renduchintala, Sumit BhatiaLanguage Model PretrainingLanguage Modeling

    Media and Data Science Research Adobe Inc., India