'Layer su Layer': Identifying and Disambiguating the Italian NPN Construction in BERT's family

Greta Gorzoni; Ludovica Pannitto; Francesca Masini

cs.CLApr 4, 2026

'Layer su Layer': Identifying and Disambiguating the Italian NPN Construction in BERT's family

Authors: Greta Gorzoni, Ludovica Pannitto, Francesca Masini

Organizations: Alma Mater Studiorum - University of Bologna

Abstract

Interpretability research has highlighted the importance of evaluating Pretrained Language Models (PLMs) and in particular contextual embeddings against explicit linguistic theories to determine what linguistic information they encode. This study focuses on the Italian NPN (noun-preposition-noun) constructional family, challenging some of the theoretical and methodological assumptions underlying previous experimental designs and extending this type of research to a lesser-investigated language. Contextual vector representations are extracted from BERT and used as input to layer-wise probing classifiers, systematically evaluating information encoded across the model's internal layers. The results shed light on the extent to which constructional form and meaning are reflected in contextual embeddings, contributing empirical evidence to the dialogue between constructionist theory and neural language modelling

Figures & tables

	Form	Meaning	Types	Tokens
1	$\,[[x]_{N_{i}}\ a\ [x]_{N_{i}}]\,_{\mathrm{MOD}_{j}}$	Succession, iteration, distributivity	21	399
2	$\,[[x]_{N_{i}}\ a\ [x]_{N_{i}}]\,_{\mathrm{MOD}_{j}}$	Juxtaposition, contact	27	1108
3	$\,[[x]_{N_{i}^{sg}}\ su\ [x]_{N_{i}^{sg}}]\,_{\mathrm{MOD}_{j}}$	Succession, iteration, distributivity	71	178
4	$\,[[x]_{N_{i}^{pl}}\ su\ [x]_{N_{i}^{pl}}]\,_{\mathrm{MOD}_{j}}$	Greater plurality, accumulation	252	403
5	$\,[[x]_{N_{i}}\ per\ [x]_{N_{i}}]\,_{\mathrm{MOD}_{j}}$	Succession, iteration, distributivity	385	3324
6	$\,[[x]_{N_{i}}\ per\ [x]_{N_{i}}]\,_{\mathrm{SUB.CL}_{j}}$	Inescapable presupposition	21	21

Table 1: The 8 Italian npn Cxns postulated by Masini (2024a) . This work deals with Cxns 1–4.

Panel of Figure 1 — Figure 1: Accuracy of [UNK] (red lines, square dots) and PREP (orange lines, triangular dots) on Construction identification for the SIMPLE configuration. As in the following plots, the accuracy of the five probing classifiers resulting from the five random splits is averaged. Dashed grey line represents FastText baseline. Continuous grey lines refer to control classifiers. Figure ( 1(a) ) includes decremental training configurations, line shading becomes progressively lighter as the number of training instances decreases (480 → 240 → 120 → 60). No substantial performance differences emerge across configurations. Figure C.1 in Appendix provides a qualitative visualisation of the embedding space across layers.

Panel of Figure 2 — Figure 2: Accuracy of [UNK] (red lines, square dots) and PREP (orange lines, triangular dots) on Construction identification for the OTHER and PSEUDO configurations. Dashed grey line represents FastText baseline.

Panel of Figure 3 — Figure 3: Accuracy of [UNK] (red lines, square dots) and PREP (orange lines, triangular dots) on both experiments on English data from Scivetti and Schneider (2025) . Dashed grey line represents FastText baseline. Dotted grey line represents GloVe baseline.

Panel of Figure 4 — Figure 4: Accuracy of [UNK] (red lines, square dots) and PREP (orange lines, triangular dots) on Construction disambiguation task. Dashed grey line represents FastText baseline. Dotted grey line represents morphological FastText baseline. Continuous grey lines refer to control classifiers. Model ( 4(a) ) includes decremental training configurations, line shading becomes progressively lighter as the number of training instances decreases (480 → 240 → 120 → 60). No substantial performance differences emerge across configurations. Figures C.2 and C.3 , in Appendix, provide a qualitative visualisation of the embedding space across layers.

Figure 5: Accuracy of [UNK] (red lines, square dots) and PREP (orange lines, triangular dots) on Construction disambiguation. Dashed grey line represents FastText baseline. Dotted grey line represents morphological FastText baseline.

Label	Prep	Train	Test
CXN	a	120	30
CXN	su	120	30
CXN (total)	–	240	60
DISTR	a	120	30
DISTR	su	120	30
DISTR (total)	–	240	60

Table B.1 : SIMPLE training test split configuration

Label	Prep	Train	Test
CXN	a	70	30
CXN	su	70	30
CXN (total)	–	140	60
DISTR	a	25	30
DISTR	su	115	30
DISTR (total)	–	140	60

Table B.2 : PSEUDO training test split configuration

Label	Prep	Train	Test
CXN	a	55	30
CXN	su	55	30
CXN (total)	–	110	60
DISTR	a	105	30
DISTR	su	5	30
DISTR (total)	–	110	60

Table B.3 : OTHER training test split configuration

Label	Prep	Train	Test
Succession	a	60	15
Succession	su	60	15
Succession (total)	–	120	30
Accumulation	su	120	30
Juxtaposition	a	120	30
total	–	360	90

Table B.4 : Semantic disambiguation train–test split configuration.

Label	Prep	Train	Test
Succession	a	30	–
Succession	su	30	–
Succession	per	–	50
Succession	dopo	–	50
Succession (total)	–	60	100
Accumulation	su	60	–

Table B.5 : per and dopo train–test split configurations.

Panel of Figure 12 — Figure C.1 : PCA-based three-dimensional projection of UNK BERT embeddings across layers for npn constructions (red) and distractors (red). An animated version of the visualisation is available at https://gretagorzoni00.github.io/NPN_contextual_embeddings/ .

Panel of Figure 13 — Figure C.2 : Three-dimensional PCA projection of UNK-based BERT contextual embeddings across layers for npn semi-schematic constructions. Data points are colour-coded by semantic value: juxtaposition/contact (red), greater_plurality/accumulation (red), and succession/iteration/distributivity (green). The visualisation is based on a balanced subset of 360 instances. An animated version of the visualisation is available at https://gretagorzoni00.github.io/NPN_contextual_embeddings/ .

Panel of Figure 14 — Figure C.3 : Three-dimensional PCA projection of PREP-based BERT contextual embeddings across layers for npn semi-schematic Constructions. Data points are color-coded by semantic value: juxtaposition/contact (red), greater_plurality/accumulation (red), and succession/iteration/distributivity (green). The visualisation is based on a balanced subset of 360 instances. An animated version of the visualisation is available at https://gretagorzoni00.github.io/NPN_contextual_embeddings/ .

Figure D.1 : Confusion matrices for the SIMPLE configuration in the Cxn identification experiment. Results for the Italian BERT model ( bert-base-italian ) are shown for [UNK] embeddings (left) and PREP embeddings (right). The results from the five random splits are aggregated by summing the prediction errors across the five corresponding probing classifiers.

Figure D.2 : Confusion matrices for the PSEUDO configuration in the Cxn identification experiment. Results for the Italian BERT model ( bert-base-italian ) are shown for [UNK] embeddings (left) and PREP embeddings (right). The results from the five random splits are aggregated by summing the prediction errors across the five corresponding probing classifiers.

Figure D.3 : Confusion matrices for the OTHER configuration in the Cxn identification experiment. Results for the Italian BERT model ( bert-base-italian ) are shown for [UNK] embeddings (left) and PREP embeddings (right). The results from the five random splits are aggregated by summing the prediction errors across the five corresponding probing classifiers.

Figure D.4 : Confusion matrices for the Cxn disambiguation experiment. Results for the Italian BERT model ( bert-base-italian ) are shown for [UNK] embeddings (left) and PREP embeddings (right). The results from the five random splits are aggregated by summing the prediction errors across the five corresponding probing classifiers.

Language Model Probing

Language model probing measures what information and capabilities models encode and how those signals relate to their behavior.

See all 275 papers

Where Are Language-Specific Features Represented in Multilingual LLMs? Everywhere
Oct 18, 2026Mikaela Martins, Jonathan DunnMultilingual Language Models Concept Localization in Language Models
Department of Linguistics, University of Illinois Urbana-Champaign
Polish ModernBERT: The Long and Short of Polish Language Understanding
Sep 1, 2026Michał Perełkiewicz, Sławomir Dadas, Rafał Poświata +1Language Model Pretraining Long-Context Language Model Evaluation
National Information Processing Institute, Warsaw, Poland
BLiMPPI: Benchmark of Linguistic Minimal Pairs for Polarity Items
Oct 9, 2026Arzu Burcu Güven, Anna Rogers, Leonie Weissweiler +2Multilingual Language Model Evaluation
IT University of Copenhagen · Leipzig University & ScaDS.AI Dresden/Leipzig · Uppsala University

Oct 18, 2026J/K move · Enter open · S save

Where Are Language-Specific Features Represented in Multilingual LLMs? Everywhere

Mikaela Martins, Jonathan Dunn

Department of Linguistics, University of Illinois Urbana-Champaign

Multilingual Language Models Concept Localization in Language Models Representation Geometry in Language Models Language Identification Linguistic Bias in Language Models

Open paper AskPDF arXiv

Previous work has argued that multilingual LLMs encode language-specific features most strongly in their initial and final layers, implying that intermediate layers support language-agnostic representations. This paper reconsiders this finding by performing four different experiments: three that undertake a fine-grained analysis of representations of Portuguese and Spanish, two closely related languages, and one large-scale language identification experiment that covers over 200 languages. The results of these experiments challenge the strict separation between language-specific and language-agnostic layers in multilingual LLMs, using Llama 3.2 1B as a test case. By analyzing and manipulating contextual sentence embeddings, we demonstrate that language identity remains strongly recoverable throughout the model rather than being restricted to initial and final layers. Moreover, we find that the model tends to classify ambiguous strings as closer to Spanish than to Portuguese, revealing a skew in model behavior that differs from human judgments.

Abstract

Language Model Probing

Explore similar work

Where Are Language-Specific Features Represented in Multilingual LLMs? Everywhere

Polish ModernBERT: The Long and Short of Polish Language Understanding

BLiMPPI: Benchmark of Linguistic Minimal Pairs for Polarity Items