Interpretability research has highlighted the importance of evaluating Pretrained Language Models (PLMs) and in particular contextual embeddings against explicit linguistic theories to determine what linguistic information they encode. This study focuses on the Italian NPN (noun-preposition-noun) constructional family, challenging some of the theoretical and methodological assumptions underlying previous experimental designs and extending this type of research to a lesser-investigated language. Contextual vector representations are extracted from BERT and used as input to layer-wise probing classifiers, systematically evaluating information encoded across the model's internal layers. The results shed light on the extent to which constructional form and meaning are reflected in contextual embeddings, contributing empirical evidence to the dialogue between constructionist theory and neural language modelling
Figures & tables
Form
Meaning
Types
Tokens
1
[[x]Nia[x]Ni]MODj
Succession, iteration, distributivity
21
399
2
[[x]Nia[x]Ni]MODj
Juxtaposition, contact
27
1108
3
[[x]Nisgsu[x]Nisg]MODj
Succession, iteration, distributivity
71
178
4
[[x]Niplsu[x]Nipl]MODj
Greater plurality, accumulation
252
403
5
[[x]Niper[x]Ni]MODj
Succession, iteration, distributivity
385
3324
6
[[x]Niper[x]Ni]SUB.CLj
Inescapable presupposition
21
21
Table 1: The 8 Italian npn Cxns postulated by Masini (2024a) . This work deals with Cxns 1–4.
Figure 1: Accuracy of [UNK] (red lines, square dots) and PREP (orange lines, triangular dots) on Construction identification for the SIMPLE configuration. As in the following plots, the accuracy of the five probing classifiers resulting from the five random splits is averaged. Dashed grey line represents FastText baseline. Continuous grey lines refer to control classifiers. Figure ( 1(a) ) includes decremental training configurations, line shading becomes progressively lighter as the number of training instances decreases (480 → 240 → 120 → 60). No substantial performance differences emerge across configurations. Figure C.1 in Appendix provides a qualitative visualisation of the embedding space across layers.
Figure 2: Accuracy of [UNK] (red lines, square dots) and PREP (orange lines, triangular dots) on Construction identification for the OTHER and PSEUDO configurations. Dashed grey line represents FastText baseline.
Figure 3: Accuracy of [UNK] (red lines, square dots) and PREP (orange lines, triangular dots) on both experiments on English data from Scivetti and Schneider (2025) . Dashed grey line represents FastText baseline. Dotted grey line represents GloVe baseline.
Figure 4: Accuracy of [UNK] (red lines, square dots) and PREP (orange lines, triangular dots) on Construction disambiguation task. Dashed grey line represents FastText baseline. Dotted grey line represents morphological FastText baseline. Continuous grey lines refer to control classifiers. Model ( 4(a) ) includes decremental training configurations, line shading becomes progressively lighter as the number of training instances decreases (480 → 240 → 120 → 60). No substantial performance differences emerge across configurations. Figures C.2 and C.3 , in Appendix, provide a qualitative visualisation of the embedding space across layers.
Figure 5: Accuracy of [UNK] (red lines, square dots) and PREP (orange lines, triangular dots) on Construction disambiguation. Dashed grey line represents FastText baseline. Dotted grey line represents morphological FastText baseline.
Label
Prep
Train
Test
CXN
a
120
30
CXN
su
120
30
CXN (total)
–
240
60
DISTR
a
120
30
DISTR
su
120
30
DISTR (total)
–
240
60
Table B.1 : SIMPLE training test split configuration
Label
Prep
Train
Test
CXN
a
70
30
CXN
su
70
30
CXN (total)
–
140
60
DISTR
a
25
30
DISTR
su
115
30
DISTR (total)
–
140
60
Table B.2 : PSEUDO training test split configuration
Label
Prep
Train
Test
CXN
a
55
30
CXN
su
55
30
CXN (total)
–
110
60
DISTR
a
105
30
DISTR
su
5
30
DISTR (total)
–
110
60
Table B.3 : OTHER training test split configuration
Table B.5 : per and dopo train–test split configurations.
Figure C.1 : PCA-based three-dimensional projection of UNK BERT embeddings across layers for npn constructions (red) and distractors (red). An animated version of the visualisation is available at https://gretagorzoni00.github.io/NPN_contextual_embeddings/ .
Figure C.2 : Three-dimensional PCA projection of UNK-based BERT contextual embeddings across layers for npn semi-schematic constructions. Data points are colour-coded by semantic value: juxtaposition/contact (red), greater_plurality/accumulation (red), and succession/iteration/distributivity (green). The visualisation is based on a balanced subset of 360 instances. An animated version of the visualisation is available at https://gretagorzoni00.github.io/NPN_contextual_embeddings/ .
Figure C.3 : Three-dimensional PCA projection of PREP-based BERT contextual embeddings across layers for npn semi-schematic Constructions. Data points are color-coded by semantic value: juxtaposition/contact (red), greater_plurality/accumulation (red), and succession/iteration/distributivity (green). The visualisation is based on a balanced subset of 360 instances. An animated version of the visualisation is available at https://gretagorzoni00.github.io/NPN_contextual_embeddings/ .
Figure D.1 : Confusion matrices for the SIMPLE configuration in the Cxn identification experiment. Results for the Italian BERT model ( bert-base-italian ) are shown for [UNK] embeddings (left) and PREP embeddings (right). The results from the five random splits are aggregated by summing the prediction errors across the five corresponding probing classifiers.
Figure D.2 : Confusion matrices for the PSEUDO configuration in the Cxn identification experiment. Results for the Italian BERT model ( bert-base-italian ) are shown for [UNK] embeddings (left) and PREP embeddings (right). The results from the five random splits are aggregated by summing the prediction errors across the five corresponding probing classifiers.
Figure D.3 : Confusion matrices for the OTHER configuration in the Cxn identification experiment. Results for the Italian BERT model ( bert-base-italian ) are shown for [UNK] embeddings (left) and PREP embeddings (right). The results from the five random splits are aggregated by summing the prediction errors across the five corresponding probing classifiers.
Figure D.4 : Confusion matrices for the Cxn disambiguation experiment. Results for the Italian BERT model ( bert-base-italian ) are shown for [UNK] embeddings (left) and PREP embeddings (right). The results from the five random splits are aggregated by summing the prediction errors across the five corresponding probing classifiers.
Previous work has argued that multilingual LLMs encode language-specific features most strongly in their initial and final layers, implying that intermediate layers support language-agnostic representations. This paper reconsiders this finding by performing four different experiments: three that undertake a fine-grained analysis of representations of Portuguese and Spanish, two closely related languages, and one large-scale language identification experiment that covers over 200 languages. The results of these experiments challenge the strict separation between language-specific and language-agnostic layers in multilingual LLMs, using Llama 3.2 1B as a test case. By analyzing and manipulating contextual sentence embeddings, we demonstrate that language identity remains strongly recoverable throughout the model rather than being restricted to initial and final layers. Moreover, we find that the model tends to classify ambiguous strings as closer to Spanish than to Portuguese, revealing a skew in model behavior that differs from human judgments.
Mikaela Martins, Jonathan Dunn
Department of Linguistics, University of Illinois Urbana-Champaign
Encoder-only Transformers remain effective for discriminative and representation-learning tasks, yet Polish encoders still largely rely on BERT/RoBERTa-style architectures. We introduce Polish ModernBERT, a family of four Polish encoders available at Base and Large scales, each with 512-token and 8K context variants. We adapt the ModernBERT pretraining recipe through staged selection experiments and release a long-context benchmark covering legal topic classification, ideological decision-direction prediction, factual-consistency assessment over literary plot summaries, and human-rights violation assessment. Across 30 tasks, Polish ModernBERT achieves the best overall performance among the evaluated Polish encoders, reaching 83.99 and 85.11 for the Base-8K and Large-8K models, respectively. On long-context tasks, the 8K variants improve over matched Polish RoBERTa-8K baselines from 67.47 to 77.15 and from 75.88 to 78.49 at the Base and Large scales, respectively. The Base-8K model achieves this gain with 22% fewer parameters (149M vs.\ 190M). Efficiency measurements in representative inference setups show lower peak memory usage and latency than matched Polish RoBERTa baselines in both 512-token and 8K settings. Polish ModernBERT-8K-Base additionally achieves the best result on a Polish retrieval benchmark among the evaluated encoders below 300M parameters.
Michał Perełkiewicz, Sławomir Dadas, Rafał Poświata +1
National Information Processing Institute, Warsaw, Poland
Negative polarity items (NPIs), such as "anymore" and "ever", are grammatical only in select environments, predominantly in the context of negation. Because their grammaticality depends on both syntax and semantics, they have a rich, multilingual literature in theoretical linguistics, yet work on NPIs in language models is almost entirely confined to English. We introduce BLiMPPI, a multilingual evaluation suite of over 12,000 NPI minimal pairs in 33 languages, built by a pipeline that discovers NPIs in Universal Dependencies (UD) treebanks, labels their licensing environments, and perturbs these into minimal pairs. Evaluating 12 multilingual and 2 monolingual models alongside n-gram baselines, we find that multilingual models struggle most with lower-resource languages and with pairs that are not n-gram solvable. We further examine the effects of NPI typology, language family, licensing variability, and NPI-licenser distance. Data: https://github.com/arzuburcuguven/blimppi-data
Arzu Burcu Güven, Anna Rogers, Leonie Weissweiler +2
IT University of Copenhagen · Leipzig University & ScaDS.AI Dresden/Leipzig · Uppsala University