Recent work has examined language models from a linguistic perspective to better understand how they acquire language. Most existing benchmarks focus on judging grammatical acceptability, whereas the ability to interpret meanings conveyed by grammatical forms has received much less attention. We introduce the Linguistic Minimal-Pair Benchmark for Evaluating Constructional Understanding in Language Models (CxMP), a benchmark grounded in Construction Grammar that treats form-meaning pairings, or constructions, as fundamental linguistic units. CxMP evaluates whether models can interpret the semantic relations implied by constructions, using a controlled minimal-pair design across nine construction types, including the let-alone, caused motion, and ditransitive constructions. Our results show that while syntactic competence emerges early, constructional understanding develops more gradually and remains limited even in large language models (LLMs). CxMP thus reveals persistent gaps in how language models integrate form and meaning, providing a framework for studying constructional understanding and learning trajectories in language models.
Figures & tables
Figure 1: Overview of the let-alone construction. Humans can grasp the relation between the two entities from sentences containing this construction.
Construction
Constructional Example
Diagnostic Sentence
Let-alone
He can’t help N1, let alone N2.
Helping N2/*N1 is more difficult than helping N1/*N2.
Helping N1/*N2 is easier than helping N2/*N1.
Causative-with
N1 covered N2 with a coat.
The one who provided the coat is N1/*N2.
The one who received the coat is N2/*N1.
Way-manner
N1 pushed his way into the room.
He did so while moving/*staying.
He did so with/*without effort.
Table 1: Constructions that we focus on. * indicates implausible versions.
Figure 2: Scores of each language model across constructions and variants. Solid and dashed lines at the top of the figure represent the scores of GPT-5 for variants A and B, respectively.
Figure 3: Scores of Pythia models by model size.
Figure 4: Scores of open-sci-ref models by data size.
Figure 5: Scores of Llama3.1 base/instruction models.
Model
Model sz.
Data sz.
Obj.
Score
RoBERTa-base
125M
160G
MLM
0.573
GPT-2-medium
345M
40G
CLM
0.537
RoBERTa-large
355M
160G
MLM
0.674
Pythia-410m
410M
825G
CLM
0.544
Table 2: Scores of CLM & MLM models. Obj. indicates objective and Sz. indicates size.
Figure 6: Scores of OLMo-2 on CxMP and BLiMP across training token counts (log scale). Solid/Dashed lines represent the 13B/7B model.
Figure 7: Biases of OLMo-2 on CxMP across training token counts (log scale).
Figure 8: Scores for pseudowords ( wug ) and real words.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Construction
#Items
Maj. (%)
Agr. (%)
Let_alone
112
97.32
66.07
Causative_with
112
97.32
91.96
Way_manner
56
98.21
80.36
CC
112
100.00
91.96
Conative
56
67.86
44.64
Ditransitive
112
100.00
89.29
Appendix
Table 3: Human validation results. Majority (Maj.) denotes the proportion of items where at least two of three annotators selected the correct continuation. Agreement (Agr.) denotes the proportion of items where all three annotators gave the same answer.
Grasping the semantics of rare constructions (form-meaning pairings) has been shown to be a challenging problem that has currently only been solved by the largest LLMs. It remains an open question if open-source models have robust constructional understanding, and if so, what learning dynamics underlie the acquisition of this knowledge. Focusing on a set of rare Paired-Focus constructions in English (e.g. "let alone", "much less"), we construct a novel dataset to test their meanings using both scalar adjectival semantics and general world knowledge. Testing a wide range of models differing in parameter count, architecture, and pretraining dataset size, we find that several modestly sized models are sensitive to both the forms and the meanings of Paired-Focus constructions, though models trained on human-scale data fail at all meaning evaluations. Turning to training dynamics for a set of open-checkpoint models, we find that Paired-Focus understanding emerges later in training than Paired-Focus syntactic knowledge, and that learning of Paired-Focus semantics is correlated with gains in some domains of world knowledge. Overall, our empirical results support the conclusion that modestly sized open-source models can grasp the rare Paired-Focus constructions, and demonstrate a connection between knowledge of Paired-Focus constructions and other meaning domains.
Constructed languages (conlangs) are intentionally created human languages with a rich tradition of linguistic creativity. Despite their potential for studying language learning in large language models (LLMs), existing conlangs remain largely underexplored in LLM research. We present ConlangBench, the first large-scale benchmark for evaluating and training LLMs on 21 existing conlangs. We collect over 21M conlang-English parallel sentence pairs (including 430K pairs across the 20 non-Esperanto conlangs) and 321K vocabulary entries. In bidirectional translation experiments, we find that models perform better on a posteriori conlangs, whose vocabularies are derived from natural languages, reflecting the design characteristics of conlangs. Training on ConlangBench also shows that models can learn all eight conlangs for which sufficient parallel corpora are available, while their learning curves vary depending on how the conlangs were created. Our findings suggest that conlangs provide a unique testbed for investigating how LLMs acquire low-resource languages.
In frame semantics, sentence comprehension is assumed to proceed by relating lexical meaning to background knowledge called semantic frames, thereby enabling readers to implicitly enrich the text with unstated information. Recent large language models (LLMs) have achieved strong performance across a wide range of downstream tasks. However, it remains unclear whether they can reproduce the kinds of implicit enrichment that humans naturally make during comprehension. To address this question, we introduce FrameBench, a benchmark grounded in frame semantics. FrameBench consists of multiple-choice questions that test whether models distinguish the frames evoked by the same verb across contexts. We construct the benchmark for English and Japanese using FrameNet-style resources and a generation-and-verification pipeline with native-speaker judgments. Our experiments on a diverse set of models reveal challenges for small models, while several large models surpass the human reference scores. We release the constructed FrameBench dataset and the code for dataset construction and evaluation at https://github.com/SasanoLab/FrameBench.